In the metadata-ingestion, DataHub provides various kinds of metadata sources onboarding, including Hive, Kafka, LDAP, mySQL, and generic RDBMS as ETL scripts to feed the metadata to the GMS.
The extract process will be specific tight to the data source, hence, the data accessor should be able to reflect the correctness of the metadata from underlying data platforms.
In the transform stage, the extracted metadata should be encapsulated in a valid MetadataChangeEvent under the defined aspects and snapshots.
The load part will leverage the Kafka producer to enable the pub-sub event-based ingestion. Meanwhile, the schema validation will be involved to check metadata quality.