bq validation

cloudsufi · Jun 6, 2023 · 4ed945b · 4ed945b
1 parent 86d084d
commit 4ed945b
Show file tree

Hide file tree

Showing 22 changed files with 528 additions and 161 deletions.
diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeValidation.feature b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeValidation.feature
@@ -1,4 +1,19 @@
-Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scenarios
+# Copyright © 2023 Cask Data, Inc.
+##
+## Licensed under the Apache License, Version 2.0 (the "License"); you may not
+## use this file except in compliance with the License. You may obtain a copy of
+## the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+## License for the specific language governing permissions and limitations under
+# the License..
+
+@CloudMySql
+Feature: CloudMySql sink- Verify CloudsqlMysql sink plugin design time validation scenarios
 
   Scenario: To verify CloudMySql sink plugin validation error message with invalid database
     Given Open Datafusion Project to configure pipeline
@@ -10,15 +25,15 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "RefName"
     Then Enter input plugin property: "database" with value: "TestDatabase"
     Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
-#    Then Click on the Get Schema button
-#    Then Verify the Output Schema matches the Expected Schema: "outputSchema"
-#    Then Validate "CloudSQL MySQL" plugin properties
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "outputSchema"
+    Then Validate "CloudSQL MySQL" plugin properties
     Then Close the Plugin Properties page
     Then Navigate to the properties page of plugin: "CloudSQL MySQL2"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
@@ -30,7 +45,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
     Then Enter input plugin property: "database" with value: "invalidDatabaseName"
     Then Enter input plugin property: "tableName" with value: "mytable"
     Then Click on the Validate button
-#    Then Verify that the Plugin is displaying an error message: "errorMessageInvalidSinkDatabase" on the header
+    Then Verify that the Plugin is displaying an error message: "errorMessageInvalidSinkDatabase" on the header
 
   Scenario: To verify CloudMySql sink plugin validation error message with invalid tablename
     Given Open Datafusion Project to configure pipeline
@@ -42,26 +57,26 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "RefName"
     Then Enter input plugin property: "database" with value: "TestDatabase"
     Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
-#    Then Click on the Get Schema button
-#    Then Verify the Output Schema matches the Expected Schema: "outputSchema"
-#    Then Validate "CloudSQL MySQL" plugin properties
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "outputSchema"
+    Then Validate "CloudSQL MySQL" plugin properties
     Then Close the Plugin Properties page
     Then Navigate to the properties page of plugin: "CloudSQL MySQL2"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "invalidRefName"
     Then Enter input plugin property: "database" with value: "TestDatabase"
     Then Enter input plugin property: "tableName" with value: "Invalidtable"
     Then Click on the Validate button
-#    Then Verify that the Plugin Property: "table" is displaying an in-line error message: "errorMessageInvalidTableName"
+    Then Verify that the Plugin Property: "table" is displaying an in-line error message: "errorMessageInvalidTableName"
 
   Scenario: To verify CloudMySql sink plugin validation error message with invalid reference Name
     Given Open Datafusion Project to configure pipeline
@@ -70,7 +85,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "invalidRef"
@@ -87,7 +102,7 @@ Feature: CloudMySql sink- Verify ,Mysql sink plugin design time validation scena
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
     Then Enter input plugin property: "referenceName" with value: "RefName"

diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeWithMacro.feature b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTimeWithMacro.feature
@@ -1,3 +1,18 @@
+# Copyright © 2023 Cask Data, Inc.
+##
+## Licensed under the Apache License, Version 2.0 (the "License"); you may not
+## use this file except in compliance with the License. You may obtain a copy of
+## the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+## License for the specific language governing permissions and limitations under
+# the License..
+
+@CloudMySql
 Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenarios
 
   Scenario: To verify CloudMySql sink plugin validation with macro enabled fields for connection section
@@ -6,15 +21,15 @@ Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenar
     When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Click on the Macro button of Property: "jdbcPluginName" and set the value to: "cloudsql-mysql"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Select radio button plugin property: "instanceType" with value: "public"
     Then Click on the Macro button of Property: "user" and set the value to: "username"
     Then Click on the Macro button of Property: "password" and set the value to: "password"
     Then Click on the Macro button of Property: "connectionArguments" and set the value to: "connectionArgumentsList"
     Then Enter input plugin property: "referenceName" with value: "sourceRef"
     Then Replace input plugin property: "database" with value: "TestDatabase"
     Then Click on the Validate button
-#    Then Validate "CloudSQL MySQL" plugin properties
+    Then Validate "CloudSQL MySQL" plugin properties
     Then Close the Plugin Properties page
 
   Scenario: To verify cloudsql sink plugin validation with macro enabled fields for basic section
@@ -23,13 +38,13 @@ Feature: CloudMySql sink- Verify CloudMySql sink plugin design time macro scenar
     When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Select radio button plugin property: "instanceType" with value: "public"
     Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "sourceRef"
     Then Replace input plugin property: "database" with value: "TestDatabase"
     Then Click on the Macro button of Property: "tableName" and set the value to: "mytable"
     Then Click on the Validate button
-#    Then Validate "CloudSQL MySQL" plugin properties
+    Then Validate "CloudSQL MySQL" plugin properties
     Then Close the Plugin Properties page
diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature
@@ -15,7 +15,7 @@
 @CloudMySql
 Feature: CloudMySql Sink - Run time scenarios
 
-  @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE
+  @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
   Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully
     Given Open Datafusion Project to configure pipeline
     When Expand Plugin group in the LHS plugins list: "Source"
@@ -24,7 +24,7 @@ Feature: CloudMySql Sink - Run time scenarios
     And Enter input plugin property: "referenceName" with value: "Reference"
     And Replace input plugin property: "project" with value: "projectId"
     And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
-    And Enter input plugin property: "dataset" with value: "dataset"
+    And Replace input plugin property: "dataset" with value: "dataset"
     And Enter input plugin property: "table" with value: "bqSourceTable"
     Then Click on the Get Schema button
     Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
@@ -36,7 +36,7 @@ Feature: CloudMySql Sink - Run time scenarios
     Then Navigate to the properties page of plugin: "CloudSQL MySQL"
     Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
     Then Select radio button plugin property: "instanceType" with value: "public"
-    Then Enter input plugin property: "connectionName" with value: "ConnectionName"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
     Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
     Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
     Then Enter input plugin property: "referenceName" with value: "RefName"
@@ -57,7 +57,98 @@ Feature: CloudMySql Sink - Run time scenarios
     Then Open and capture logs
     Then Verify the pipeline status is "Succeeded"
     Then Close the pipeline logs
+    Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
 
+  @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
+  Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully when connection arguments are set
+    Given Open Datafusion Project to configure pipeline
+    When Expand Plugin group in the LHS plugins list: "Source"
+    When Select plugin: "BigQuery" from the plugins list as: "Source"
+    And Navigate to the properties page of plugin: "BigQuery"
+    And Enter input plugin property: "referenceName" with value: "Reference"
+    And Replace input plugin property: "project" with value: "projectId"
+    And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
+    And Replace input plugin property: "dataset" with value: "dataset"
+    And Enter input plugin property: "table" with value: "bqSourceTable"
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
+    Then Validate "BigQuery" plugin properties
+    And Close the Plugin Properties page
+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "CloudSQL MySQL" to establish connection
+    Then Navigate to the properties page of plugin: "CloudSQL MySQL"
+    Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
+    Then Select radio button plugin property: "instanceType" with value: "public"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
+    Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
+    Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
+    Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
+    Then Enter input plugin property: "referenceName" with value: "RefName"
+    Then Enter input plugin property: "database" with value: "DatabaseName"
+    Then Replace input plugin property: "tableName" with value: "targetTable"
+    Then Validate "CloudSQL MySQL" plugin properties
+    Then Close the Plugin Properties page
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Wait till pipeline preview is in running state
+    Then Open and capture pipeline preview logs
+    Then Verify the preview run status of pipeline in the logs is "succeeded"
+    Then Close the pipeline logs
+    Then Close the preview
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Close the pipeline logs
+    Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
+
+
+  @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
+  Scenario: To verify data is getting transferred from BigQuery source to CloudSMySQL sink with Advanced property Connection timeout
+    Given Open Datafusion Project to configure pipeline
+    When Expand Plugin group in the LHS plugins list: "Source"
+    When Select plugin: "BigQuery" from the plugins list as: "Source"
+    And Navigate to the properties page of plugin: "BigQuery"
+    And Enter input plugin property: "referenceName" with value: "Reference"
+    And Replace input plugin property: "project" with value: "projectId"
+    And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
+    And Replace input plugin property: "dataset" with value: "dataset"
+    And Enter input plugin property: "table" with value: "bqSourceTable"
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
+    Then Validate "BigQuery" plugin properties
+    And Close the Plugin Properties page
+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "CloudSQL MySQL" to establish connection
+    Then Navigate to the properties page of plugin: "CloudSQL MySQL"
+    Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
+    Then Select radio button plugin property: "instanceType" with value: "public"
+    Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
+    Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
+    Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
+    Then Enter input plugin property: "referenceName" with value: "RefName"
+    Then Enter input plugin property: "database" with value: "DatabaseName"
+    Then Replace input plugin property: "tableName" with value: "targetTable"
+    Then Replace input plugin property: "connectionTimeout" with value: "connectionTimeout"
+    Then Validate "CloudSQL MySQL" plugin properties
+    Then Close the Plugin Properties page
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Wait till pipeline preview is in running state
+    Then Open and capture pipeline preview logs
+    Then Verify the preview run status of pipeline in the logs is "succeeded"
+    Then Close the pipeline logs
+    Then Close the preview
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Close the pipeline logs
+    Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table