DataHub / Tech Deep Dives

Hyejin Yoon (Acryl Data) gives an overview of DataHub's various APIs and new use-case-oriented guides.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Shirshanka Das (Acryl Data) shares recent speed and functionality improvements to ingesting metadata from Snowflake during the July 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

John Joyce & Tamás Nemeth go in-depth about how you can use DataHub + Airflow + Great Expectations to scalably address data reliability.


Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Ryan Holstien (Acryl Data) shares details about how we're making it easier for developers to interface with DataHub via OpenAPI during the April 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Surya Lanka & Pedro Silva (Acryl Data) share the latest advancements to managing deletes within DataHub during the April Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Kartik Darapuneni (Included Health) shares his experience building embedding Looker, Querybook, and Jupyter into DataHub.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

John Joyce (Acryl Data) shares the new Actions Framework for developing & deploying real-time outbound integrations with DataHub during the April 2022 Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

David Leifker (Zendesk) gives a demo of the new Protobuf Ingestion Source during the March Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Edward Vaisman (Wavelo) gives a demo of how you can define Dataset-to-Dataset lineage via YAML during the February 2022 Community Town Hall.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

Tamás Németh (Acryl Data) talks about how DataHub can now automatically extract Table, View, and S3 lineage using Redshift system tables.

This functionality is available as of v0.8.18

John Joyce (Acryl Data) gives a deep-dive into the DataHub Metadata Service Authentication during the November 2021 Community Town Hall

Referenced Links:

https://datahubproject.io/docs/how/auth/jaas https://datahubproject.io/docs/how/auth/sso/configure-oidc-react/
https://github.com/linkedin/datahub/blob/681ed91a0006a2d20535c0d5c30f0a68afcfab9f/docs/introducing-metadata-service-authentication.md

Shirshanka Das and Maggie Hays from Acryl Data review recent improvements to the DataHub Looker connector.

Surya Lanka and Shirshanka Das (Acryl Data) give a demo of stateful ingestion works in DataHub, ensuring that you ingest only net-new metadata to minimize redundancy and optimize ingestion performance.

Dexter Lee (Acryl Data) describes how DataHub is being instrumented for supporting performance monitoring use-cases.

Note: This was a session that was scheduled to be presented live at the townhall, but we couldn't accommodate it due to time concerns. Dexter was kind enough to record it later to share with the community.

John Joyce (Acryl Data) gives an update on recent improvements to fine-grained access control during the DataHub Community Town Hall on August 27, 2021.

John Joyce (Acryl Data) provides an update on user and group management during the DataHub Community Town Hall on August 27, 2021.

Recent developments focus on:
- New user ingestion sources! Okta & Azure AD Batch
- Just-in-Time User & Group Provisioning with OIDC - when users log in, we will automatically provision an account if they do not already have one
- Within the UI, Groups are now searchable and Group Members appear on the Groups Page

Harshal Sheth (Acryl Data) describes how Dataset Popularity is implemented in DataHub and gives a demo.

John Joyce and Gabe Lyons from Acryl Data present simplifications in deploying DataHub at the Community Townhall on June 25.
- Standalone mode can run on much less memory and requires fewer containers
- Neo4j is now optional; DataHub can run on Elastic as Graph backend.

John Joyce (Acryl Data) presents a tech deep dive on the new support for no code metadata modeling in DataHub that is going to be released as part of release 0.8.0