Support catalog in MaxCompute Source

ruanwenjun · Aug 13, 2023 · 95762d0 · 95762d0
1 parent 843d203
commit 95762d0
Show file tree

Hide file tree

Showing 15 changed files with 646 additions and 328 deletions.
diff --git a/docs/en/connector-v2/source/Maxcompute.md b/docs/en/connector-v2/source/Maxcompute.md
@@ -26,6 +26,7 @@ Used to read data from Maxcompute.
 | partition_spec | string | no       | -             |
 | split_row      | int    | no       | 10000         |
 | common-options | string | no       |               |
+| schema         | config | no       |               |
 
 ### accessId [string]
 
@@ -59,6 +60,12 @@ Used to read data from Maxcompute.
 
 Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details.
 
+### schema [config]
+
+#### fields [Config]
+
+The schema information of upstream data.
+
 ## Examples
 
 ```hocon
@@ -71,6 +78,13 @@ source {
     table_name="<your table name>"
     #partition_spec="<your partition spec>"
     #split_row = 10000
+    schema {
+        fields {
+            name = string
+            age = int
+            gender = string 
+        }
+    }
   }
 }
 ```
@@ -80,4 +94,5 @@ source {
 ### next version
 
 - [Feature] Add Maxcompute Source Connector([3640](https://github.com/apache/seatunnel/pull/3640))
+- [Feature] Support Schema in MaxCompute Source([3640](https://github.com/apache/seatunnel/pull/5283))
 
diff --git a/docs/en/connector-v2/source/MyHours.md b/docs/en/connector-v2/source/MyHours.md
@@ -35,13 +35,13 @@ Used to read data from My Hours.
 In order to use the My Hours connector, the following dependencies are required.
 They can be downloaded via install-plugin.sh or from the Maven central repository.
 
-| Datasource | Supported Versions | Dependency                                                                                     |
-|------------|--------------------|------------------------------------------------------------------------------------------------|
-| My Hours   | universal          | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2)    | 
+| Datasource | Supported Versions |                                         Dependency                                          |
+|------------|--------------------|---------------------------------------------------------------------------------------------|
+| My Hours   | universal          | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2) |
 
 ## Source Options
 
-|            Name             |  Type   | Required | Default | Description                                                                                                                          |
+|            Name             |  Type   | Required | Default |                                                             Description                                                              |
 |-----------------------------|---------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------|
 | url                         | String  | Yes      | -       | Http request url.                                                                                                                    |
 | email                       | String  | Yes      | -       | My hours login email address.                                                                                                        |

diff --git a/...ctors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleArrayTypeInfo.java b/...ctors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleArrayTypeInfo.java
diff --git a/...nectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleMapTypeInfo.java b/...nectors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleMapTypeInfo.java
diff --git a/...tors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleStructTypeInfo.java b/...tors-v2/connector-maxcompute/src/main/java/com/aliyun/odps/type/SimpleStructTypeInfo.java