-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(DLI): import dli_template_flink resource of DLI service and add …
…unit test and document.
- Loading branch information
1 parent
7857afa
commit 850731b
Showing
7 changed files
with
857 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
--- | ||
subcategory: "Data Lake Insight (DLI)" | ||
--- | ||
|
||
# flexibleengine_dli_datasource_connection | ||
|
||
Manages a DLI datasource **enhanced** connection resource within FlexibleEngine. | ||
|
||
## Example Usage | ||
|
||
```hcl | ||
variable "name" {} | ||
variable "vpc_id" {} | ||
variable "subnet_id" {} | ||
resource "flexibleengine_dli_datasource_connection" "test" { | ||
name = var.name | ||
vpc_id = var.vpc_id | ||
subnet_id = var.subnet_id | ||
} | ||
``` | ||
|
||
## Argument Reference | ||
|
||
The following arguments are supported: | ||
|
||
* `region` - (Optional, String, ForceNew) Specifies the region in which to create the resource. | ||
If omitted, the provider-level region will be used. Changing this parameter will create a new resource. | ||
|
||
* `name` - (Required, String, ForceNew) Specifies the name of the datasource connection. | ||
The valid length is limited from can contain 1 to 64, only letters, digits and underscore (_) are allowed. | ||
The name must start with a lowercase letter and end with a lowercase letter or digit. | ||
Changing this parameter will create a new resource. | ||
|
||
* `vpc_id` - (Required, String, ForceNew) The VPC ID of the service to be connected. | ||
Changing this parameter will create a new resource. | ||
|
||
* `subnet_id` - (Required, String, ForceNew) The subnet ID of the service to be connected. | ||
Changing this parameter will create a new resource. | ||
|
||
* `route_table_id` - (Optional, String, ForceNew) The route table ID associated with the subnet of the service to be | ||
connected. Changing this parameter will create a new resource. | ||
|
||
* `queues` - (Optional, List) List of queue names that are available for datasource connections. | ||
|
||
* `hosts` - (Optional, List) The user-defined host information. A maximum of 20,000 records are supported. | ||
The [hosts](#dli_hosts) object structure is documented below. | ||
|
||
* `tags` - (Optional, Map, ForceNew) The key/value pairs to associate with the datasource connection. | ||
Changing this parameter will create a new resource. | ||
|
||
<a name="dli_hosts"></a> | ||
The `hosts` block supports: | ||
|
||
* `name` - (Required, String) The user-defined host name. | ||
|
||
* `ip` - (Required, String) IPv4 address of the host. | ||
|
||
## Attribute Reference | ||
|
||
In addition to all arguments above, the following attributes are exported: | ||
|
||
* `id` - The resource ID. | ||
|
||
* `status` - The connection status. The options are as follows: | ||
+ **ACTIVE**: The datasource connection is activated. | ||
+ **DELETED**: The datasource connection is deleted. | ||
|
||
## Import | ||
|
||
The DLI datasource connection can be imported using the `id`, e.g. | ||
|
||
```shell | ||
terraform import flexibleengine_dli_datasource_connection.test 0ce123456a00f2591fabc00385ff1234 | ||
``` | ||
|
||
Note that the imported state may not be identical to your resource definition, due to some attributes missing from the | ||
API response, security or some other reason. The missing attributes include: `tags`. | ||
It is generally recommended running `terraform plan` after importing a resource. | ||
You can then decide if changes should be applied to the resource, or the resource definition should be updated to | ||
align with the resource. Also, you can ignore changes as below. | ||
|
||
```hcl | ||
resource "flexibleengine_dli_datasource_connection" "test" { | ||
... | ||
lifecycle { | ||
ignore_changes = [ | ||
tags, | ||
] | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
--- | ||
subcategory: "Data Lake Insight (DLI)" | ||
--- | ||
|
||
# flexibleengine_dli_sql_job | ||
|
||
Manages DLI SQL job resource within FlexibleEngine | ||
|
||
## Example Usage | ||
|
||
### Create a Sql job | ||
|
||
```hcl | ||
variable "database_name" {} | ||
variable "queue_name" {} | ||
variable "sql" {} | ||
resource "flexibleengine_dli_sql_job" "test" { | ||
sql = var.sql | ||
database_name = var.database_name | ||
queue_name = var.queue_name | ||
} | ||
``` | ||
|
||
## Argument Reference | ||
|
||
The following arguments are supported: | ||
|
||
* `region` - (Optional, String, ForceNew) Specifies the region in which to create the DLI table resource. If omitted, | ||
the provider-level region will be used. Changing this parameter will create a new resource. | ||
|
||
* `sql` - (Required, String, ForceNew) Specifies SQL statement that you want to execute. | ||
Changing this parameter will create a new resource. | ||
|
||
* `database_name` - (Optional, String, ForceNew) Specifies the database where the SQL is executed. This argument does | ||
not need to be configured during database creation. Changing this parameter will create a new resource. | ||
|
||
* `queue_name` - (Optional, String, ForceNew) Specifies queue which this job to be submitted belongs. | ||
Changing this parameter will create a new resource. | ||
|
||
* `tags` - (Optional, Map, ForceNew) Specifies label of a Job. Changing this parameter will create a new resource. | ||
|
||
* `conf` - (Optional, List, ForceNew) Specifies the configuration parameters for the SQL job. Changing this parameter | ||
will create a new resource. The [conf](#dli_conf) object structure is documented below. | ||
|
||
<a name="dli_conf"></a> | ||
The `conf` block supports: | ||
|
||
* `spark_sql_max_records_per_file` - (Optional, Int, ForceNew) Maximum number of records to be written | ||
into a single file. If the value is zero or negative, there is no limit. Default value is `0`. | ||
Changing this parameter will create a new resource. | ||
|
||
* `spark_sql_auto_broadcast_join_threshold` - (Optional, Int, ForceNew) Maximum size of the table that | ||
displays all working nodes when a connection is executed. You can set this parameter to -1 to disable the display. | ||
Default value is `209715200`. Changing this parameter will create a new resource. | ||
Currently, only the configuration unit metastore table that runs the ANALYZE TABLE COMPUTE statistics no-scan | ||
command and the file-based data source table that directly calculates statistics based on data files are supported. | ||
Changing this parameter will create a new resource. | ||
|
||
* `spark_sql_shuffle_partitions` - (Optional, Int, ForceNew) Default number of partitions used to filter | ||
data for join or aggregation. Default value is `4096`. Changing this parameter will create a new resource. | ||
|
||
* `spark_sql_dynamic_partition_overwrite_enabled` - (Optional, Bool, ForceNew) In dynamic mode, Spark does not delete | ||
the previous partitions and only overwrites the partitions without data during execution. Default value is `false`. | ||
Changing this parameter will create a new resource. | ||
|
||
* `spark_sql_files_max_partition_bytes` - (Optional, Int, ForceNew) Maximum number of bytes to be packed into a | ||
single partition when a file is read. Default value is `134217728`. | ||
Changing this parameter will create a new resource. | ||
|
||
* `spark_sql_bad_records_path` - (Optional, String, ForceNew) Path of bad records. Changing this parameter will create | ||
a new resource. | ||
|
||
* `dli_sql_sqlasync_enabled` - (Optional, Bool, ForceNew) Specifies whether DDL and DCL statements are executed | ||
asynchronously. The value true indicates that asynchronous execution is enabled. Default value is `false`. | ||
Changing this parameter will create a new resource. | ||
|
||
* `dli_sql_job_timeout` - (Optional, Int, ForceNew) Sets the job running timeout interval. If the timeout interval | ||
expires, the job is canceled. Unit: `ms`. Changing this parameter will create a new resource. | ||
|
||
## Attribute Reference | ||
|
||
In addition to all arguments above, the following attributes are exported: | ||
|
||
* `id` - Indicates a resource ID in UUID format. | ||
|
||
* `owner` - User who submits a job. | ||
|
||
* `job_type` - The type of job, includes **DDL**, **DCL**, **IMPORT**, **EXPORT**, **QUERY** and **INSERT**. | ||
|
||
* `status` - Status of a job, includes **RUNNING**, **SCALING**, **LAUNCHING**, **FINISHED**, **FAILED**, | ||
and **CANCELED**. | ||
|
||
* `start_time` - Time when a job is started, in RFC-3339 format. e.g. `2019-10-12T07:20:50.52Z` | ||
|
||
* `duration` - Job running duration (unit: millisecond). | ||
|
||
* `schema` - When the statement type is DDL, the column name and type of DDL are displayed. | ||
|
||
* `rows` - When the statement type is DDL, results of the DDL are displayed. | ||
|
||
## Timeouts | ||
|
||
This resource provides the following timeouts configuration options: | ||
|
||
* `create` - Default is 20 minutes. | ||
* `delete` - Default is 45 minutes. | ||
|
||
## Import | ||
|
||
DLI SQL job can be imported by `id`, e.g. | ||
|
||
```shell | ||
terraform import flexibleengine_dli_sql_job.example 7f803d70-c533-469f-8431-e378f3e97123 | ||
``` | ||
|
||
Note that the imported state may not be identical to your resource definition, due to some attributes missing from the | ||
API response, security or some other reason. The missing attributes include: `conf`, `rows` and `schema`. | ||
It is generally recommended running `terraform plan` after importing a resource. You can then decide if changes should | ||
be applied to the resource, or the resource definition should be updated to align with the resource. Also, you can | ||
ignore changes as below. | ||
|
||
```hcl | ||
resource "flexibleengine_dli_sql_job" "test" { | ||
... | ||
lifecycle { | ||
ignore_changes = [ | ||
conf, rows, schema | ||
] | ||
} | ||
} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
--- | ||
subcategory: "Data Lake Insight (DLI)" | ||
--- | ||
|
||
# flexibleengine_dli_template_flink | ||
|
||
Manages a DLI Flink template resource within FlexibleEngine. | ||
|
||
## Example Usage | ||
|
||
```hcl | ||
variable "sql" {} | ||
resource "flexibleengine_dli_template_flink" "test" { | ||
name = "demo" | ||
type = "flink_sql_job" | ||
sql = var.sql | ||
description = "This is a demo" | ||
tags = { | ||
foo = "bar" | ||
key = "value" | ||
} | ||
} | ||
``` | ||
|
||
## Argument Reference | ||
|
||
The following arguments are supported: | ||
|
||
* `region` - (Optional, String, ForceNew) Specifies the region in which to create the resource. | ||
If omitted, the provider-level region will be used. Changing this parameter will create a new resource. | ||
|
||
* `name` - (Required, String) The name of the flink template. | ||
|
||
* `sql` - (Optional, String) The statement of the flink template. | ||
|
||
* `description` - (Optional, String) The description of the flink template. | ||
|
||
* `type` - (Optional, String, ForceNew) The type of the flink template. | ||
Valid values are **flink_sql_job** and **flink_opensource_sql_job**. | ||
Defaults to **flink_sql_job**. | ||
|
||
Changing this parameter will create a new resource. | ||
|
||
* `tags` - (Optional, Map, ForceNew) The key/value pairs to associate with the flink template. | ||
|
||
Changing this parameter will create a new resource. | ||
|
||
## Attribute Reference | ||
|
||
In addition to all arguments above, the following attributes are exported: | ||
|
||
* `id` - The resource ID. | ||
|
||
## Import | ||
|
||
The flink template can be imported using the `id`, e.g. | ||
|
||
```shell | ||
terraform import flexibleengine_dli_template_flink.test 1231 | ||
``` | ||
|
||
Note that the imported state may not be identical to your resource definition, due to some attributes missing from the | ||
API response, security or some other reason. The missing attributes include: | ||
`tags`. | ||
It is generally recommended running `terraform plan` after importing a resource. | ||
You can then decide if changes should be applied to the resource, or the resource definition should be updated to align | ||
with the resource. Also, you can ignore changes as below. | ||
|
||
```bash | ||
resource "flexibleengine_dli_template_flink" "test" { | ||
... | ||
|
||
lifecycle { | ||
ignore_changes = [ | ||
tags | ||
] | ||
} | ||
} | ||
``` |
Oops, something went wrong.