DataHub Metadata Day 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automated Data Ownership with DataHub

Description

Corentin Duhamel, Data Engineer from Showroomprivé, shares the importance of assigning Data Ownership and how the team has automated ownership-related processes using DataHub during Metadata Day 2022.

Learn more about DataHub: https://datahubproject.io
Join us on Slack: http://slack.datahubproject.io
Follow us on Twitter: https://twitter.com/datahubproject

A

Hello, my name is coronta, I'm a data engineer at showroom prevail and I'm going to talk about beta ownership and what we achieved to facilitate data governance.

A

Before that a short introduction of sean privy.

A

So we are a french company founded 15 years ago. We are an e-commerce platform dedicated to fashion beauty and home.

A

We collect unsold products from brands and we rework the marketing to make products look more attractive and we organize sales events.

A

We are based in paris, but we also present in italy, spain and other european countries.

A

Our data team is composed of 20 people and we have had data hub in production since last november.

A

Okay, let's get to the main topic of this talk.

A

Let's go back one year ago I arrived at showroom prive as an intern, and we had two thousands data sets on redshift, undocumented 100 dashboards and about 50 dags on airflow.

A

As a data engineer, one of my missions is to alert the right people when a pipeline goes down, but without documentation it was a struggle to get the emails of the managers.

A

So as soon as a task failed, I used to spam everyone.

A

Since then, we have come a long way thanks to data hub and setting ownership rules has been really useful to us. For several reasons.

A

It allows us to empower people. We expect owners to deliver data on time while guaranteeing the desired level of quality.

A

Also, they are asked to document the entity.

A

Setting ownership rules has allowed us to be more responsive in the event of an incident. Indeed, with the impact analysis feature, we can quickly recover the emails of all the person.

A

All the people affected in the event of a breakdown.

A

In the future, we would like to automate this process with an email alert and prevent certain tasks from starting. If there has been a problem upstream and last but not least, now we know who to reach if we need information.

A

To begin with, we try to extract owners from the source as much as possible. This has been possible on airflow and click where we just made some changes to make the format match with the url patterns.

A

For our machine learning, data sets, we usually add a prefix with a project code.

A

For example, all tables related to the churn project start with churn dash.

A

This has allowed us to add owners by data set urn pattern.

A

Here is the transformer we added in our recipe file, so john doe and I will be added as owners on all tables of the project. A and finally, we use the interface to complete what remains.

A

For owner's choice, we always put the person who provides the data, we add their team as support to always have someone when they are out of office, and sometimes we a third entity, a stakeholder or a main user.

A

As I said before, the owners are responsible for the documentation of their entities. That's why we have a policy rule which allows only owners to edit documentation.

A

After deploying data hub and ingesting our main data sources, we started a documentation work. In a few months. We have had to make a great effort of documentation.

A

We have had to solicit all the team and direct our efforts to start with the most important data sets.

A

We created a tag that extracts our entities and their owners. Thanks to lineage, we added some rules to favor influential entities.

A

A matching algorithm associates an entity to document with each data hub user and finally, we send an email to ask users to complete the documentation.

A

Another tricky subject is rights management we are very concerned about. Data sets permissions because we want to ensure users have access to the information they need to complete their jobs, and we also want to keep our data safe.

A

Thanks to no code feature, we added a permissions tab in order to view the list of users with access to the data set.

A

It allows managers to verify compliance with the principle of list. Privilege and users can see what rights they have.

A

That's all I wanted to share. I wanted to thank my team, um so datahub core team and the slack community. You have my contact details. If you have any questions.