diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature index 0356040fc..d63105bd8 100644 --- a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature +++ b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature @@ -57,7 +57,7 @@ Feature: CloudMySql Sink - Run time scenarios Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526 Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully when connection arguments are set @@ -102,7 +102,8 @@ Feature: CloudMySql Sink - Run time scenarios Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table + @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526 Scenario: To verify data is getting transferred from BigQuery source to CloudSMySQL sink with Advanced property Connection timeout @@ -147,6 +148,7 @@ Feature: CloudMySql Sink - Run time scenarios Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table + diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature index 2800e855f..be729b396 100644 --- a/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature +++ b/cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature @@ -63,7 +63,7 @@ Feature: CloudMySql Sink - Run time scenarios (macro) Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526 Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink using macro arguments in basic section @@ -108,7 +108,7 @@ Feature: CloudMySql Sink - Run time scenarios (macro) Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table @BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE Scenario: Verify pipeline failure message in logs when user provides invalid Table Name of plugin with Macros @@ -232,4 +232,4 @@ Feature: CloudMySql Sink - Run time scenarios (macro) Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table + Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTime.feature b/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTime.feature index b426efea3..5700f2900 100644 --- a/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTime.feature +++ b/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTime.feature @@ -56,8 +56,7 @@ Feature: CloudMySql - Verify data transfer from CloudMySql source to BigQuery si Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target BigQuery table is equal to the values from source table - + Then Validate the values of records transferred to target Big Query table is equal to the values from source table @CLOUDMYSQL_SOURCE_DATATYPES_TEST @BQ_SINK_TEST Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully with all datatypes Given Open Datafusion Project to configure pipeline @@ -99,7 +98,8 @@ Feature: CloudMySql - Verify data transfer from CloudMySql source to BigQuery si Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target BigQuery table is equal to the values from source table + Then Validate the values of records transferred to target Big Query table is equal to the values from source table + @CLOUDMYSQL_SOURCE_DATATYPES_TEST @CLOUDMYSQL_SINK_TEST @PLUGIN-20670 Scenario: To verify data is getting transferred from CloudMySql source to CloudMySql sink successfully diff --git a/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTimeMacro.feature b/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTimeMacro.feature index 5deb55deb..69d29c030 100644 --- a/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTimeMacro.feature +++ b/cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTimeMacro.feature @@ -212,7 +212,7 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target BigQuery table is equal to the values from source table + Then Validate the values of records transferred to target Big Query table is equal to the values from source table @CLOUDMYSQL_SOURCE_TEST @BQ_SINK_TEST Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully using macro arguments in basic section @@ -256,7 +256,8 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target BigQuery table is equal to the values from source table + Then Validate the values of records transferred to target Big Query table is equal to the values from source table + @CLOUDMYSQL_SOURCE_TEST @BQ_SINK_TEST Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully using macro arguments in advance section @@ -304,4 +305,5 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument Then Open and capture logs Then Verify the pipeline status is "Succeeded" Then Close the pipeline logs - Then Validate the values of records transferred to target BigQuery table is equal to the values from source table + Then Validate the values of records transferred to target Big Query table is equal to the values from source table + diff --git a/cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/BQValidation.java b/cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/BQValidation.java index c7e9ebf16..97217708c 100644 --- a/cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/BQValidation.java +++ b/cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/BQValidation.java @@ -35,6 +35,7 @@ public class BQValidation { * @return true if the values in source and target side are equal */ + public static boolean validateBQAndDBRecordValues(String sourceTable, String targetTable) throws SQLException, ClassNotFoundException, IOException, InterruptedException, ParseException { getBigQueryTableData(sourceTable, bigQueryRows); @@ -198,12 +199,12 @@ public static boolean compareResultSetWithJsonData(ResultSet rsSource, List