DataHub Tech Deep Dives, 20 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DataHub + OpenAPI Improvements

Description

Ryan Holstien (Acryl Data) shares details about how we're making it easier for developers to interface with DataHub via OpenAPI during the April 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

So, let's uh go on over to mr ryan, we'll be talking about open api updates today, our rest endpoints are primarily using the wrestling framework which creates uh you know, endpoints that are more in line with what linkedin's standards are internally and um have not really seen a super wide adoption throughout the rest of the open source community.

A

So with the goal with open api is to make it a lot easier to integrate with restful endpoints through other languages frameworks, and um you know, programming in the way that you want to, rather than just our limited set of sdks and what we can support right. So with the wrestling models, it goes from average schemas to our you know, kafka format and our our sdks that we support, but with an open api spec.

A

The idea is that you can take our open api spec and our models and those json schemas and apply that to any language that you want to support and have a client that you can utilize in you know across whatever languages you want.

A

So with this initial set of work, where uh we have implemented the entities post get and delete. So this mimics the ingestion and getting and deletion apis that we currently have, and that uses a slightly easier model to understand and use, then like a raw metadata change proposal, all right. uh If I can share screen I'll, go into the swagger ui.

A

So to get to this api and kind of explore these endpoints, we have exposed a swagger ui, so you can get to that by just going over into the data hub, ui and clicking this open api link, and that takes you into the swagger ui and I'm going to switch over to this tab because I already have some requests filled out.

A

So basically, what you get here is um all of the schemas that we have internally. These are processed from pdl and translated over into json schemas, and um these are- and we also have generated java models that we use in our java sdk and in inside data hub itself.

A

So, for example, for the post endpoint, um you can go in and see.

A

The schema of this request, so it has the entity type, the urn of the entity and the aspect you want to send in and, alternatively, to the earn you can send in the key aspect, similar to how a metadata change proposal has.

A

Sometimes it can be a little bit easier to have that object format to actually send in so this aspect supports any of the aspects we have inside here, so you can go in and explore these and you get fully documented descriptions.

A

Example values and you know it gives you all of the information you would need to construct a request just in this ui directly. So one thing, you'll notice is this underscore underscore type so for anything that exists in a union uh type, we have to specify what the class is. This is because not all classes have uh you know completely unique properties.

A

So with json, it's just you know raw json, you, you don't have your your class types and what have you so you have to know what is actually being sent across so that we can translate it but yeah. So there's lots of these to explore. I won't go through everything, but you can see that this will enable you to create a request to actually ingest or and and know what you're getting back.

A

uh So I'm just going to run through a quick example. So this is a ingestion example for this urn and I'm sending in a schema metadata aspect.

A

These are all part of a postman collection that I'm going to include with documentation whenever this pr gets in and so we'll just go ahead and run that, and so we get a 201 accepted response and we get the earn that got updated back. So if we go ahead and take this urn.

A

And then put it into the get.

A

And we get back our schema.

A

A

We get this response with the key and the schema metadata.

A

And this all lines up with what got sent in and then we can also do a delete of that same urn, and so this supports both soft and hard deletes, uh we'll go ahead and do a hard delete and completely remove it.

A

So that gives you the earn back that got deleted and now, if we go ahead and run that same get, we just get the key, which should they hard deleted should be completely gone.

A

So this pr is open, uh it's currently under review and it's going to get put out and not this release, but the the next one.

A

Currently, the top level api is under evolution, so we're we're trying to get feedback on this right. So this is our initial setup and we we would like to get.

A

You know, comments back and know how people want to use this and really fine tune this and get it working. So please play with it. uh Let us know what you think and uh get back to us and and let us know how we can improve it- all right. Everybody thanks! So much ryan.