DataHub Community Talks, 6 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DataHub UI Overview : DataHub Community

Description

DataHub UI Overview by Ignacio Bona: Community TownHall: Nov 6 2020

A

Let me show the screen.

B

C

Before that now she was gonna sing as a song right.

A

A

Hey good credits, um so hi everyone, I'm ignacio, um but everyone called me nacho I mean all ignacios are nachos in spain, so you can call them nacho, uh so I've been working for data hub for the last couple of years.

A

um As a ui engineer- and I will talk about the uh like briefly the ui, because there's a lot of things to talk about so I just want to give like a quick overview like how the ui structure and uh what are we doing over there, um because I think it's a little bit of a black box right now. um Okay, so um what's what is the text stack of the of the ui? Well, we have type script, which is uh it's javascript with types, so it's not very javascript right.

A

uh We have amber, which is our single page application framework um and we're also using germ workspaces to divide our add-ons or npm packages to well different packages.

A

um So I just want to give you a little bit of um overview. What is ember and justin, because maybe people are not aware so an an number app um this structure or something like that right. You have an app folder where everything all you all your code is you have you have your routes which it's all your handlers I use over there, which goes together with uh oops with the routes.ts.

A

uh That's the one on the bottom, uh the path for the routes I mean you're gonna, write a path, and then it's gonna go to the handler right and over. There is going to render a template for that route, which is in templates and in that template, you're, going to add components like a navbar, sidebar, a filter, search bar whatever right, and then your components are gonna, be in that uh folder.

A

um I put an asterisk because, um as you can see the components the hps, which is the template, it's like all the html is gonna be over there and the javascript law code is going to be in the typescript.

A

So the template is going to be together, but that only happens when we're using glimmer, which is the latest.

A

And their uh ui uh way of doing the things, so sometimes you won't see it like that. You will see the template inside temp components, uh templates, components and then the hps will be there, but that's. This is how it's supposed to be at the moment. um Okay! So now we know what uh a nap is. uh Now we use a concept called the number add-on.

A

So what is number add-on? So it's similar to an app in a way, but it's just uh it's meant to be a module, so you can share add-ons between different apps. uh It's an npm package as well. um So what you have in a number addon is an app folder, so that app folder will be actually merged uh with your app folder in your application. So whatever javascript you have there, it will appear on your application when, when everything is compiled.

A

So it has advantages like you, can create one javascript on your uh app and you can actually override one of the javascript of the phone add-on, like your. Your application should be on top of everything, so that this.

C

A

The sort of truth, but I mean it's not a good practice unless you want to do something uh strange, um but then you have your add-on folder so that atom folder is kind of private. um So it's not um well it's kind of visible, it depends. um So normally what you do I mean in your item. Folder, you will put all your components. All your real code will be there and what you put on the app folder is just a re-export. You import your item, folder and you export it there.

A

So you have visibility in your app and use that, because um for templates um like like for components and in the template, you don't explicitly import the things in in ember you just uh it is it. Is there in your? You use the name of the javascript file. um So it's like this. It's a little bit. We call it ember magic right which what is happening is it's gonna show um it's gonna show over there because you have a component and you have a javascript, so ember will see like.

A

Oh you have this component over there there's a javascript here. So this is a component, and I know that is there. So this is how amber if it works, I'm I'm going a little fast, but I don't have much time. So this is an example right. You have your add-on, which it has um one component. It has a service and it has a util.

A

Then, in your app folder of the add-on, you have a component one and javascript which exports the component one in in typescript and- and you also have service and series one- is the javascript that exports the service one in typescript.

A

So when you merge that with your app, which has component 2 and service 2, the result is something like this, where you have like these two components and these two services. um So these com, two components are available to use in any template in the app um because you see how the merge went uh and then you'll see in the add-on. There's a youtube so that does not be to be does not need to be re-exported, because it's you can import that directly from your typescript.

A

The program is more like um when you try to use a component inside a template, which is most of the time. That's why you need to do the re-export. um So that's a liquid overview. If you have more, if you want to know more, I suggest you that you can go to amber, uh but just this is like a little bit quick overview, however, works. Okay, so how we structure um data hub.

A

So first we have package slash data portal, so that's our main ember app, that's where the app is going to be so we'll go back. That is like the result here uh and then we have like multiple uh other packages or group groups um or namespaces depends on how you want to call it. So we have data hub, that's where, basically, all the code is going to be there, uh then we have natural ui, which is um like the ui framework that we use for data hub.

A

It's our own custom, ui framework. It's a long story!

A

um Then we have like ph tools, which is some like tooling, that we have for um craftsmanships and maintaining quality of the code and also like transforming the pdl's schemas into the typescript definitions, um which is uh essential actually and then, if you want to override um something we're doing not linkedin is we create our uh own folder over there, which is at linkedin and your?

A

If you want to write it will be like at your company, and then we put this kind of like the same structure of folders inside there, and then you can create all your own components over there and then you can import it um and then I will get into that a little bit deeply. So um going back so data portal right, that's the main application.

A

So here uh the goal of this is to be almost empty, because we can actually move all these logic of the app to components and add-ons. So that's our goal is to keep this like as simply as possible and and why we do this well, the thing is um so so if you want to overwrite something over here, actually you probably need to fork it because there's no easy way to modularize right, so you need to fork it and then it's a pain to mercy.

A

So we keep it as thin as we want to keep it as soon as possible. So there's nothing there. So you can. If you want to override it, it's basically a package.json with your package that you're going to have um that's. uh We want to make it more dynamically.

A

That's a future thing that thought that we are thinking about it's just uh for now. We mitigate it by making it very thin and lightweight. So there's nothing much to be to have conflicts on um yeah. And if you look at the code, you will see that um there's some components still out there. So what we're doing right now is we're trying to move all these outside it's taking its time and but we will get there.

A

Then, when we go to the nacho ui, that's a ui framework. uh It's like any other ui framework, but it's custom craft and filled with love inside linkedin, so it has like right now. It has pills, uh tables, button, drop down and avatar, which is like all these user components. Like you know, like rounded picture with the name and some some animations um we have, like the you know, the loading state which shows the thing loading. So this is like the ui framework.

A

We we there is still some natural components instead of data portal who want to move them out just so. You were we're working on that. um Then, what what do we have? How do we structure the app this is. I think this is more like interesting, um so the inside that I have we divided the app in these five blocks.

A

um We have uh youtube, which is um basically all those utility functions and maybe simple components that you can reuse. That is like not very domain specific, but it doesn't fit in nacho because it's like it's a little bit domain specific, but not like quite generic to be like ui framework, so you can put it in youtube. Oops then metadata types.

A

So over there um we put all the model type generation so like all everything, your api returns should be typed over there um right now we are generating this this type generation, but in the whole time we didn't have that, so we still have some some definitions of it we're trying to migrate. So if you see definitions over there, just ignore it, it will be migrated to this automatically creation process modeling.

A

um So that being said, metadata types is also quite empty because everything is generated.

A

Data models, so data model is uh where we create a destruction, a little bit of a structure between what the api returns to and what the ui can render. And what we do is I mean that's where the mapping exists right, so we create, for example, data sets. We create a class which is a data set that is like the ui data set and that reference, the the backend ui, uh the backend.

A

Sorry, the backend data set as well, so you can do some mapping and some massaging over there if you need some transformation of data, so that's data models, there's no components over there, because it's more like massaging data and handling data we have shared shared is basically all the components for data hub like the search bar, the navbar, the search results, the entity page components like the generic components should be there, everything that is basically data hub and then we have entities.

A

So here is where we, you store all these custom things that you want, for example, data that you want a specific graph for data set, so you put in entities uh in the past. We divided entity into different components. So, for example, you have data set component and then you have um user component and then we we stored those components inside those packages, but the problem is we had an performance issue, and so we combined them into this add-on. You totally can create your own add-on for data sets.

A

If you want to, um but we combine them into entities and inside you would see some folders like data set, which is the data set part, and then you will see users and that's the user part um second check. Okay. um So just to give you an example: uh what's inside metadata types, um so many data types, you will see something like this, like a data set it just whatever the api returns, so the dataset has a description, remove, is removed and then some aspects like separated likes.

A

um All of that, I think it's pretty simple. I mean that's what the gma actually returns.

A

If I'm correct right march.

A

So this is the the types um so basically from the ui. You want to know what the apis are returning, so this that's why this is generated because there's already a place where this is defined and what we want to do now is have some type safety. So that's why we generate these types.

A

Okay and then what is in data model? So I've talked about that abstraction. So it looks something like this, so we have this dataset entity, which is the the ui data's identity and then extend some base empty where we have some basic functionalities on that is grouping this basis entity. So and then you will see this generic, which is dataset and that's the type of the api.

A

The entity of the api is going to return and we will have an entity field inside base entity that is of this type so uh entity here. If you see one way, for example here, but I cannot hide like do you see the mouse right away.

A

Yes, yes, you see the mouse okay, so at least I'm not highlighting you see what I'm holding so you see entity over here, so the entity over here is the it's actually of type dataset. So if you see entity.description actually maps to end data set uh description right, so this is like one to one.

A

So that's why we can like one way and like an alias, but you can do like some computed uh function and you can transform the data and like, for example, if you need to combine description here with removed or some other field, you could totally can do it right. So you have more freedom and you're not like that, coupled with the data that is coming from the api and then we have other static functions, which is, for example, display name.

A

So that is how we refer this entity from a url like if you see, for example, slash entity sources for sorry, um they have a slash and then you expect like an entity name, you will see data sets. So it's like uh data hub slash data sets because it's here slash and then the id or the urn, um so that that's how we map those things, because we have this display name over here um and then we have render props, which is um it's a it's a way that we tell it's.

A

It's a basically a configuration, it's a json uh that gives us a way to render this um tell us how to render this uh entity right so, and I will talk about a little bit more about these render props. That is very important. So if you want to create your own entity, I you probably need to I mean you will need to uh specify the render props, because that's how it's gonna, that is gonna tell how data hub is. Gonna, render the browsing, the search and energy page right. um So, okay, that is a summary.

A

What is a render prop and then so it has configuration for search, it has configuration for browse and it has a configuration for entity page.

A

um So this is like I put a screenshot of like the three uh main parts of data hub right, so you have a search and then in the search uh for an entity you probably want to specify which fields you want to show like data origin here, and it will read that so, for example, in search you would say: oh data origin is a field I want to show, and also I want to show that in a filter like like here.

A

um I think I.

C

A

I had a little bit more detail about that um for browsing, so browsing is um not much to specify right now, but configuration is more like you what you want to have it or not, and that depends on the um apis if it's enabled or not enabled sorry sorry, ah what's going on, I think the touchpad when I scroll up and down it goes crazy, sorry, um so the browsing, and that depends on the apis.

A

If you have elastic search and it can return like a hierarchy for that entity, then you can enable browse, and then you have entity page the entity page. uh I will talk about more because it's more complicated than than everything else.

B

A

For search um yeah, it's like you specify what properties to show and then what filters and then additional components that you may want to render like this like component, but you can actually add more things if you want to um just to allow some flexibility and customize experience.

A

The thing uh that thing that I like from properties to show is that when you specify a property you can define, if you want to show like, for example, data origin, you can define if you want a filter for it and also, if you want an auto complete for it. So if you like, select a data or origin, it kind of maps like it knows that you're selecting like that field and so they're filtering actually that field that is showing over there.

A

So we can do more things um later, so everything also, if you're auto complete it. It knows that you're trying to auto complete that field as well. um This just basically define the properties and if you want to show it as filters and or if you want to autocomplete and that yeah, that's basically search uh browse, as I said not much to do here. Just you have to enable it or not, and that depends if you have like a specific hierarchy.

A

For example, data set you can put like if there is deployed in fabric or not and then what type of data set and then the name of the data set. It's like hdfs, you have folders, you can do that, but, for example, for users you may not have a hierarchy actually. Well. If your organization has a hierarchy like departments, you can do that. You can use that as your browsing experience right, but it depends on the entity type.

A

So my entity type might be a little bit difficult to classify um so and then you have the entity page. So in the entity page, you have uh start from top breadcrumbs, so actually breadcrumbs. uh It's not something that you specify an entity page, because uh you need an hierarchy to show breadcrumbs. So if you enable browse, you will have breadcrumbs. If you don't enable browse, you won't have breakouts for that entity.

A

Then you have the entity name and then you can add some tags that you want to show. You can add. Also in this entity there's I didn't show it, but you have properties as well. You have some custom components like health that we're showing over here and these like social actions that we have you cannot discuss. I mean you can add more um customize things. Then you may want to put in the header for that and then and the beefy part of this thing is the tab right. So we have a.

A

You can add your own tabs, uh but it's a configuration thing like I want a schema type. I want started status type tab. I want ownership tab um and then inside this tab you specify what components you want to render like. Oh, I want.

A

I have my custom component here, so I want to render this custom component inside schema and then, when you go to schema, it's going to try to render that custom opponent, and then you have your own free freedom to do whatever you want to, but as well, we give you some pre-built components like tables that you can use.

A

uh So if you generate the data for like for a table to consume, you can just it's kind of like wiring those up and then it will show a table with the data which is uh actually a very common use case. And so you can create your own entity page by just providing configuration and not adding any new component or for for your entity.

B

Okay, so natural quick time check uh a couple of more minutes.

A

Couple more minutes: okay, that's what I was checking yeah, so just an example. uh uh Well, yeah, uh sorry, it looks fine. So this is like in data modes right. We were in this layer of packages, so we now live in data models and we're going to the next one, which is share what's in shared, so it was, for example, this social action, because it's something very shareable, I'm not.

B

A

A bunch of more components over there, but it's not something that you would need to add. uh You won't need to add anything in here and as well you. This is the only thing that you may need to like uh in data hub entities. We have one for like data sets, and this is data set schema because it's very specific to data set. um So this is like a custom component for for data set okay, so how to customize that hub?

A

That's a very good question, but um I want to cut it right here. Maybe we can add an next uh next session like to be very specific, how to do it now that we know a little bit more about how is this structure and also we're running out of time?

A

So, if you're looking forward for a session like this just ping, us- and we will do it and just in snap- was something that, if that you need to do is, as I said, you need to create your custom components inside that entity's package or you just I mean you can create your own.

A

You can create your models and pass the configuration, which is a render props, and you should be done right. Of course, it's not as simple as that, but an overview is something like this.

A

um What are next for us is like we want to move all these files from data portal to data hub. So if you see a lot of migration, a lot of files changes in right now, um it's because of this. If you're trying to modify something data portal, just be careful because we're probably going to move that out. um We want to do this automatic model generation from like the open source models.

A

It's not completely flushed, so we are we're getting there um and then um we have the the meat here didn't talk about the meat here, but then, which is the like the apis that the ui talks to um we need to upgrade them as to keep them in sync. That's a very important thing right now. That's, for example, the the lineage graph. Once we do that, uh then you can.

A

You could you're going to see your the new lineage graph that we have built in the ui and the other thing is: we want to move uh to open source first, meaning that this data hub folder that right you see in open source. We are doing working on it internally and then we basically move these changes outside. So we want to change that process and actually push directly to open source and then use that as a package inside linkedin.

A

And that's basically it thank you and if you have questions yeah, you can ask questions now.

B

If we have time nah thanks nature, that's definitely interesting uh any questions. No, anyone.

A

I went a little bit fast and maybe I've been explaining some details, uh like some things very detailed, uh but um yeah yeah. We.

C

Can we can share? We will share all the slides later on on github, so people can go through them again and, of course, this recording will also be shared as well.

B