Add user and permission system to Amoro

[zhoujinsong]

Currently, Amoro is mainly maintained and used by platform administrators of big data. The Amoro Dashboard provides rich management tools for data lake tables.

Amoro provides an Admin account to control user access to the Amoro Dashboard, which is simple and can meet the most basic security requirements.

However, when we want to open the Amoro Dashboard to users outside the platform administrators, such as a data developer or analyst who may also be interested in the snapshot history of data lake tables or current optimizing status. Sharing the Admin account directly with them undoubtedly brings huge security risks.

This may become a huge limitation for opening Amoro to more users, so I suggest that we may need to add a user and permission system to Amoro so that users who care about and use data lake tables can use the Amoro Dashboard.

[zhoujinsong]

About the user system

We need to support integration with existing user systems, such as Single Sign-On or OAuth. In addition, we may also need to implement a default user system based on the AMS database so that it can be easily set up and run.

About the permission system

We need to organize the functions provided by Amoro, design their permission types, and design a permission framework to control their execution. This seems to be a huge project.

Role system is a common way to simplify permission configuration. We bind a set of permissions to a role, and then only need to assign roles to users to complete permission assignment.

[baiyangtx]

Identity verification is the precondition for the permission system. I would like to have a detailed discussion about the methods and challenges of user identity verification.

Regarding user identity verification, there are currently several areas in the Amoro system that may be related to user identity:

1. Identity verification information for logining into the Amoro AMS service via Web Browser
2. Identity verification information for accessing the standard command control console through third-party clients (like MySQL-Client etc..)
3. Identity verification information for accessing the storage system through SQL execution via Terminal (possibly through a standard command line in the future)
4. Identity verification information for accessing the storage system and metastore service when the Optimizer performs Data Rewrite.
5. Identity verification information for accessing Mixed-format in the Amoro engine, or iceberg and paimon format via UnifiedCatalog in the feature.

Of the five types of authentication above:

• 1. and 2. are authentication issues related to AMS itself
• 3. concerns how AMS proxies user access to third-party systems
• 4 is an authentication issue for the Optimizer
• 5 is an authentication issue for computational tasks

I will discuss each of the four authentication issues separately.

Authentication of Logining into AMS

For web login and login through command-line tools, these two authentication methods are different, but AMS must recognize them as the same user.

Of course, if AMS provides authentication information management based on username/password, then consistency between the two authentication methods can be achieved.
However, if the web authentication is integrated with a single sign-on system, then how to maintain authentication information when accessing through the command line becomes a problem.

One feasible solution is to set a separate CLI password for each user who logins in through SSO within AMS.

Accessing external systems after logging in.

After logining into AMS, accessing an external storage system when executing SQL through Terminal or Command line requires AMS to proxy the authenticated identity of the current user. There are multiple ways for AMS to proxy the authenticated identity, such as:

• AMS can record (or obtain through plugins from third-party systems) the authentication information required for users to access the storage system (such as AK/SK, token, KeyTab/principal). This approach requires AMS to obtain sensitive information between the user and the third-party system, which is a higher risk method.

• Identity proxy can be used, such as the Kerberos Proxy protocol. This approach has requirements for the authentication protocol used by the third-party storage system, and requires AMS to participate in the same authentication system and have proxy permissions.

Authentication of Optimizer

The identity authentication of Optimizer includes several aspects.

1. What identity is used to submit Optimizer tasks to the computing cluster.
2. What identity is used by Optimizer to access the storage system.

Before 0.4, AMS tended to make one Optimizer Group handle tables from multiple Catalogs. Therefore, the identity information for submitting Optimizer tasks was configured in the Optimizer Container, and the authentication information for the Catalog held by AMS was obtained through the thrift API in Optimizer.

In the future, AMS tends to have each Catalog have its own Optimizer Group. In this design, AMS can directly use the identity information configured in the Catalog when submitting Optimizer tasks. During Optimizer execution, there is no need to re-authenticate because the identity used to access the storage system is the same as the one used to submit the task.

Even further, AMS will no longer hold the Keytab file on the Catalog and will only hold the Principal. Optimizer tasks can be submitted to the corresponding user through Kerberos Proxy, eliminating the need for AMS to hold the Keytab file on the Catalog.

Authentication in engine side.

At the engine side, user authentication should be determined by the person who submits the task. A common practice is to first use Kinit locally and then submit Spark tasks. The Amoro connector should not obtain any authentication information from AMS, but should use the identity information in the context to access the storage cluster.