DataHub is made up of a generic backend and a Ember-based UI. Original DataHub blog post talks about the design extensively and mentions some of the features of DataHub. Our open sourcing blog post also provides a comparison of some features between LinkedIn production DataHub vs open source DataHub. Below is a list of the latest features that are available in DataHub, as well as ones that will soon become available.
- Search: full-text & advanced search, search ranking
- Browse: browsing through a configurable hierarchy
- Schema: table & document schema in tabular and JSON format
- Coarse grain lineage: support for lineage at the dataset level, tabular & graphical visualization of downstreams/upstreams
- Ownership: surfacing owners of a dataset, viewing datasets you own
- Dataset life-cycle management: deprecate/undeprecate, surface removed datasets and tag it with "removed"
- Institutional knowledge: support for adding free form doc to any dataset
- Fine grain lineage: support for lineage at the field level [coming soon]
- Social actions: likes, follows, bookmarks [coming soon]
- Compliance management: field level tag based compliance editing [coming soon]
- Top users: frequent users of a dataset [coming soon]
- Search: full-text & advanced search, search ranking
- Browse: browsing through a configurable hierarchy [coming soon]
- Profile editing: LinkedIn style professional profile editing such as summary, skills
- Search: full-text & advanced search, search ranking
- Browse: browsing through a configurable hierarchy
- Schema history: view and diff historic versions of schemas
- GraphQL: visualization of GraphQL schemas
- Search: full-text & advanced search, search ranking
- Browse: browsing through a configurable hierarchy
- Basic information:
- Execution history: Executions and their status. Link to external service for viewing full info.
- search: full-text & advanced search, search ranking
- Browse: browsing through a configurable hierarchy
- Basic information: ownershp, dimensions, formula, input & output datasets, dashboards
- Institutional knowledge: support for adding free form doc to any metric
- search: full-text & advanced search, search ranking
- Basic information: ownership, location. Link to exzternal service for viewing the dashboard.
- Institutional knowledge: support for adding free form doc to any dashboards
You can integrate any data platform to DataHub easily. As long as you have a way of Extracting metadata from the platform and Transform that into our standard MCE format, you're free to Load/ingest metadata to DataHub from any available platform.
We have provided example ETL ingestion scripts for:
- Hive
- Kafka
- RDBMS (MySQL, Oracle, Postgres, MS SQL etc)
- Data warehouse (Snowflake, BigQuery etc)
- LDAP