Support ingestion process

[zhoujinsong]

Data lake ingestion is a crucial process in Lakehouse management. Currently, there are many approaches to data lake ingestion, but none of them effectively address the following issues:

• Users typically have thousands or even tens of thousands of tables, and how to allocate resources appropriately for data lake ingestion is a significant challenge.
• In the scenario of synchronizing database tables to the data lake, the table structure in the data lake needs to automatically change along with the table structure in the database.

Amoro should attempt to integrate the data lake ingestion process to enhance the cost-effectiveness of users using the Lakehouse.

[huyuanfeng2018]

apache/iceberg#8701
Tabular will merge Kafka Connect into iceberg, maybe this is a solution

[lianneli]

too many tools to confuse, and looking forward to the new feature!

[yimlin]

In our practices, there are hundreds database tables that we need to sync to data lake.
All change datas of those tables are sent to kafka (per topic per table), and we sink those datas to data lake by using a flink job.

it works fine except it can't handle the change of table structure.

any good suggestion?