Propose optimizing module structure

[majin1102]

Let's have a look at modules of Amoro right now:

Current module structure has a very long evolution path which I don't wanna talk much. Here I mainly want to adress some problems with this module structure.

1. We could see ams-api is the most bottom module, however it is a sub module of ams. And the whole ams module is splitted into two parts by core. Format module has to depend on ams-api by depending on core. It became meaningless to organize an ams module if ams-api is meant to be a common dependency for all modules.
2. When I am developing [Feature]: Refactor AMS using function and process api #2596, I need to pull up configurations and some other classes from ams-server to ams-api. But the code in ams-server depends on core module, so the best solution is to make ams-api depend on core module, because core is the common dependency based on the positioning. But...... the fact is that core is already depending on ams-api which could cause cycle dependency.
3. Let's come back to the structure picture, the relationship of ams, format and core is not clear, even confusing. A confusing structure could lead to more unpredictable issues in the future.

So I propose to optimize module structure to clearify core, ams and format relationships. First we need to determine what module should be shared by ams and format: ams-api, core or both.
By postioning, core is designed to be shared for computing engines and service. In the past we put thrift api and generated codes in ams-api, that's why currently we have to make core depend on ams-api. But generated codes don't have to be located in ams-api, and we don't have to put ams-api under core. Instead we should put the whole ams on top of core.

The new structure shall be like:

This will make each module more cohesion and solve problems above. Meanwhile I remove metric-reporter as an independent module because it is a sub module of ams-server and the content is not that complex to be pulled out.

[link3280]

@majin1102 Good to see the proposal for improvement on the module dependencies! LGTM in general.

My 2 cents: could we extract the metric module out of ams-server in this refactoring? Cause shortly we may need the metric reporter in the optimizers as well.

[majin1102]

Sure.

How about keeping metrics independent under ams module.
The name could be metrics referring to flink

[zhoujinsong]

+1 for the new module strucuture.

[klion26]

Thanks for driving this, Optimizing the module can make we to maintain the project easier. I have some minor question here:

• does this mean ams-server and ams-optimizer will dependent on ams-api?
• Do we have any use case that ams-optimizer will dependent on table-format?

[majin1102]

does this mean ams-server and ams-optimizer will dependent on ams-api?

Yep, that's the ams-api designed for.

Do we have any use case that ams-optimizer will dependent on table-format?

I don't get what 'table format' here means, could you explain more?

[klion26]

thanks for the reply. ams-optimizer dependent on table format means the ams module will dependent on format module in the above figure.

[majin1102]

I don't think ams-optimizer should depend on format module.

Amoro is not a format project and codes under format module are engines optimization or adaption(flink、spark、trino), these engines code should be independent with ams.

In another word, format and ams-optimzer do share some dependencies under core module. That's why you could see some format related classes in core moudle.

[baiyangtx]

I don't quite understand why you would want to move ams thrift-api and it's generated codes to core. This part looks like the code which is depended by the optimizers and AMS.

Currently core is basically designed for mixed-iceberg format. I remember that the Amoro project was going to design the table-format to be plug-in, and it seemed like it violated that design goal if the core was to be put at the bottom.

In the most extreme case, if the user doesn't use mixed-format, he can compile the project without compiling the core module and mixed module.

[baiyangtx]

The most important questions about the core module is whether this module is just designed for mixed-format?

I think the code in the core module related to mixed-format should be moved under the mixed module.

[majin1102]

| Currently core is basically designed for mixed-iceberg format.

I don't think this is a common sense or initiative motivation. Core do have mixed-format related codes so that ams and format modules could share them. this doen't mean core is basically designed for mixed-iceberg format.
Instead ams-api as a api module under ams, it should be designed for ams itself, or just name it as api, which is the first designed(and at that time, we had debated if we really need two shared common modules including core and api) and has changed when we rename it under ams.

I remember that the Amoro project was going to design the table-format to be plug-in, and it seemed like it violated that design goal if the core was to be put at the bottom.
Table format is not a concept as plug-in. it is an optimized streaming format on top of Iceberg. And I think you probably could't find any docs saying table-format is a plugin like we never say Iceberg is a plugin.

[majin1102]

The most important questions about the core module is whether this module is just designed for mixed-format?

I think the code in the core module related to mixed-format should be moved under the mixed module.

Core module is not meant to include only mixed-format, if so why we need two modules? only format is just fine.
Core is meant to be shared as a common module.