Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(DLI): import DLI template flink resource and add unit test and document. #1035

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions docs/resources/dli_datasource_connection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
subcategory: "Data Lake Insight (DLI)"
---

# flexibleengine_dli_datasource_connection

Manages a DLI datasource **enhanced** connection resource within FlexibleEngine.

## Example Usage

```hcl
variable "name" {}
variable "vpc_id" {}
variable "subnet_id" {}

resource "flexibleengine_dli_datasource_connection" "test" {
name = var.name
vpc_id = var.vpc_id
subnet_id = var.subnet_id
}
```

## Argument Reference

The following arguments are supported:

* `region` - (Optional, String, ForceNew) Specifies the region in which to create the resource.
If omitted, the provider-level region will be used. Changing this parameter will create a new resource.

* `name` - (Required, String, ForceNew) Specifies the name of the datasource connection.
The valid length is limited from can contain 1 to 64, only letters, digits and underscore (_) are allowed.
The name must start with a lowercase letter and end with a lowercase letter or digit.
Changing this parameter will create a new resource.

* `vpc_id` - (Required, String, ForceNew) The VPC ID of the service to be connected.
Changing this parameter will create a new resource.

* `subnet_id` - (Required, String, ForceNew) The subnet ID of the service to be connected.
Changing this parameter will create a new resource.

* `route_table_id` - (Optional, String, ForceNew) The route table ID associated with the subnet of the service to be
connected. Changing this parameter will create a new resource.

* `queues` - (Optional, List) List of queue names that are available for datasource connections.

* `hosts` - (Optional, List) The user-defined host information. A maximum of 20,000 records are supported.
The [hosts](#dli_hosts) object structure is documented below.

* `tags` - (Optional, Map, ForceNew) The key/value pairs to associate with the datasource connection.
Changing this parameter will create a new resource.

<a name="dli_hosts"></a>
The `hosts` block supports:

* `name` - (Required, String) The user-defined host name.

* `ip` - (Required, String) IPv4 address of the host.

## Attribute Reference

In addition to all arguments above, the following attributes are exported:

* `id` - The resource ID.

* `status` - The connection status. The options are as follows:
+ **ACTIVE**: The datasource connection is activated.
+ **DELETED**: The datasource connection is deleted.

## Import

The DLI datasource connection can be imported using the `id`, e.g.

```shell
terraform import flexibleengine_dli_datasource_connection.test 0ce123456a00f2591fabc00385ff1234
```

Note that the imported state may not be identical to your resource definition, due to some attributes missing from the
API response, security or some other reason. The missing attributes include: `tags`.
It is generally recommended running `terraform plan` after importing a resource.
You can then decide if changes should be applied to the resource, or the resource definition should be updated to
align with the resource. Also, you can ignore changes as below.

```hcl
resource "flexibleengine_dli_datasource_connection" "test" {
...

lifecycle {
ignore_changes = [
tags,
]
}
}
133 changes: 133 additions & 0 deletions docs/resources/dli_sql_job.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
---
subcategory: "Data Lake Insight (DLI)"
---

# flexibleengine_dli_sql_job

Manages DLI SQL job resource within FlexibleEngine

## Example Usage

### Create a Sql job

```hcl
variable "database_name" {}
variable "queue_name" {}
variable "sql" {}

resource "flexibleengine_dli_sql_job" "test" {
sql = var.sql
database_name = var.database_name
queue_name = var.queue_name
}
```

## Argument Reference

The following arguments are supported:

* `region` - (Optional, String, ForceNew) Specifies the region in which to create the DLI table resource. If omitted,
the provider-level region will be used. Changing this parameter will create a new resource.

* `sql` - (Required, String, ForceNew) Specifies SQL statement that you want to execute.
Changing this parameter will create a new resource.

* `database_name` - (Optional, String, ForceNew) Specifies the database where the SQL is executed. This argument does
not need to be configured during database creation. Changing this parameter will create a new resource.

* `queue_name` - (Optional, String, ForceNew) Specifies queue which this job to be submitted belongs.
Changing this parameter will create a new resource.

* `tags` - (Optional, Map, ForceNew) Specifies label of a Job. Changing this parameter will create a new resource.

* `conf` - (Optional, List, ForceNew) Specifies the configuration parameters for the SQL job. Changing this parameter
will create a new resource. The [conf](#dli_conf) object structure is documented below.

<a name="dli_conf"></a>
The `conf` block supports:

* `spark_sql_max_records_per_file` - (Optional, Int, ForceNew) Maximum number of records to be written
into a single file. If the value is zero or negative, there is no limit. Default value is `0`.
Changing this parameter will create a new resource.

* `spark_sql_auto_broadcast_join_threshold` - (Optional, Int, ForceNew) Maximum size of the table that
displays all working nodes when a connection is executed. You can set this parameter to -1 to disable the display.
Default value is `209715200`. Changing this parameter will create a new resource.
Currently, only the configuration unit metastore table that runs the ANALYZE TABLE COMPUTE statistics no-scan
command and the file-based data source table that directly calculates statistics based on data files are supported.
Changing this parameter will create a new resource.

* `spark_sql_shuffle_partitions` - (Optional, Int, ForceNew) Default number of partitions used to filter
data for join or aggregation. Default value is `4096`. Changing this parameter will create a new resource.

* `spark_sql_dynamic_partition_overwrite_enabled` - (Optional, Bool, ForceNew) In dynamic mode, Spark does not delete
the previous partitions and only overwrites the partitions without data during execution. Default value is `false`.
Changing this parameter will create a new resource.

* `spark_sql_files_max_partition_bytes` - (Optional, Int, ForceNew) Maximum number of bytes to be packed into a
single partition when a file is read. Default value is `134217728`.
Changing this parameter will create a new resource.

* `spark_sql_bad_records_path` - (Optional, String, ForceNew) Path of bad records. Changing this parameter will create
a new resource.

* `dli_sql_sqlasync_enabled` - (Optional, Bool, ForceNew) Specifies whether DDL and DCL statements are executed
asynchronously. The value true indicates that asynchronous execution is enabled. Default value is `false`.
Changing this parameter will create a new resource.

* `dli_sql_job_timeout` - (Optional, Int, ForceNew) Sets the job running timeout interval. If the timeout interval
expires, the job is canceled. Unit: `ms`. Changing this parameter will create a new resource.

## Attribute Reference

In addition to all arguments above, the following attributes are exported:

* `id` - Indicates a resource ID in UUID format.

* `owner` - User who submits a job.

* `job_type` - The type of job, includes **DDL**, **DCL**, **IMPORT**, **EXPORT**, **QUERY** and **INSERT**.

* `status` - Status of a job, includes **RUNNING**, **SCALING**, **LAUNCHING**, **FINISHED**, **FAILED**,
and **CANCELED**.

* `start_time` - Time when a job is started, in RFC-3339 format. e.g. `2019-10-12T07:20:50.52Z`

* `duration` - Job running duration (unit: millisecond).

* `schema` - When the statement type is DDL, the column name and type of DDL are displayed.

* `rows` - When the statement type is DDL, results of the DDL are displayed.

## Timeouts

This resource provides the following timeouts configuration options:

* `create` - Default is 20 minutes.
* `delete` - Default is 45 minutes.

## Import

DLI SQL job can be imported by `id`, e.g.

```shell
terraform import flexibleengine_dli_sql_job.example 7f803d70-c533-469f-8431-e378f3e97123
```

Note that the imported state may not be identical to your resource definition, due to some attributes missing from the
API response, security or some other reason. The missing attributes include: `conf`, `rows` and `schema`.
It is generally recommended running `terraform plan` after importing a resource. You can then decide if changes should
be applied to the resource, or the resource definition should be updated to align with the resource. Also, you can
ignore changes as below.

```hcl
resource "flexibleengine_dli_sql_job" "test" {
...

lifecycle {
ignore_changes = [
conf, rows, schema
]
}
}
```
81 changes: 81 additions & 0 deletions docs/resources/dli_template_flink.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
subcategory: "Data Lake Insight (DLI)"
---

# flexibleengine_dli_template_flink

Manages a DLI Flink template resource within FlexibleEngine.

## Example Usage

```hcl
variable "sql" {}

resource "flexibleengine_dli_template_flink" "test" {
name = "demo"
type = "flink_sql_job"
sql = var.sql
description = "This is a demo"

tags = {
foo = "bar"
key = "value"
}
}
```

## Argument Reference

The following arguments are supported:

* `region` - (Optional, String, ForceNew) Specifies the region in which to create the resource.
If omitted, the provider-level region will be used. Changing this parameter will create a new resource.

* `name` - (Required, String) The name of the flink template.

* `sql` - (Optional, String) The statement of the flink template.

* `description` - (Optional, String) The description of the flink template.

* `type` - (Optional, String, ForceNew) The type of the flink template.
Valid values are **flink_sql_job** and **flink_opensource_sql_job**.
Defaults to **flink_sql_job**.

Changing this parameter will create a new resource.

* `tags` - (Optional, Map, ForceNew) The key/value pairs to associate with the flink template.

Changing this parameter will create a new resource.

## Attribute Reference

In addition to all arguments above, the following attributes are exported:

* `id` - The resource ID.

## Import

The flink template can be imported using the `id`, e.g.

```shell
terraform import flexibleengine_dli_template_flink.test 1231
```

Note that the imported state may not be identical to your resource definition, due to some attributes missing from the
API response, security or some other reason. The missing attributes include:
`tags`.
It is generally recommended running `terraform plan` after importing a resource.
You can then decide if changes should be applied to the resource, or the resource definition should be updated to align
with the resource. Also, you can ignore changes as below.

```bash
resource "flexibleengine_dli_template_flink" "test" {
...

lifecycle {
ignore_changes = [
tags
]
}
}
```
Loading
Loading