SCDF 2.11 not working on Custom Datasource with kubernetes version 1.26X

[sampathsri]

Description:
Currently i am using scdf 2.8.0 and it was working fine in Openshift. Recently the openshift was upgraded in the org with the kubernetes version 1.26X. Due to this the Schedules page in SCDF stopped working. It was suggested to use the latest SCDF version to overcome the issue. But the latest SCDF 2.11.0 is not loading any of the pages. (Screenshot attached). I was able to build the latest version using oracle dependency, but its not fetching the data from datasource.

Steps to reproduce:
Kubernertes -1.26X version. Build SCDF2.11.0 with oracle datasource.

Screenshots:

[onobc]

From #5416 (comment)

We tried to upgrade the scdf version to solve the above issue, but its not working. I build the latest scdf version 2.11.0 with oracle dependency. I am using oracle db and the application is not able to load data from the custom datasource. Rather it tries to get from default schema. See the error below.

2023-10-05 14:49:50.496 DEBUG 1 --- [nio-8080-exec-9] org.hibernate.SQL : select appregistr0_.id as id1_0_, appregistr0_.object_version as object_version2_0_, appregistr0_.boot_version as boot_version3_0_, appregistr0_.default_version as default_version4_0_, appregistr0_.metadata_uri as metadata_uri5_0_, appregistr0_.name as name6_0_, appregistr0_.type as type7_0_, appregistr0_.uri as uri8_0_, appregistr0_.version as version9_0_ from app_registration appregistr0_ where appregistr0_.default_version=1 order by appregistr0_.name asc fetch first ? rows only

2023-10-05 14:49:50.527 DEBUG 1 --- [nio-8080-exec-9] o.h.engine.jdbc.spi.SqlExceptionHelper : could not extract ResultSet [n/a]
java.sql.SQLSyntaxErrorException: ORA-00904: "APPREGISTR0_"."BOOT_VERSION": invalid identifier

[onobc]

The ORA-00904 error code indicates either the column name is invalid or the column does not exist. I believe this must be the latter.

@sampathsri do you mind sharing the names of the tables in your SCDF schema that uses the custom datasource?

[sampathsri]

Task Tables -TASK_DEFINITIONS,TASK_DEPLOYMENT,TASK_EXECUTION,TASK_EXECUTION_METADATA,TASK_EXECUTION_PARAMS,TASK_LOCK,TASK_TASK_BATCH
Batch Tables -
BATCH_JOB_DETAILS,BATCH_JOB_EXECUTION,BATCH_JOB_EXECUTION_CONTEXT,BATCH_JOB_EXECUTION_PARAMS,BATCH_JOB_INSTANCE,BATCH_STEP_EXECUTION,BATCH_STEP_EXECUTION_CONTEXT

[onobc]

Thanks @sampathsri

How do you manage your schema? There are quite a few DB schema changes for 2.11.0.

Flyway:

• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/java/org/springframework/cloud/dataflow/server/db/migration/oracle/V6__Boot3_Boot_Version.java
• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/java/org/springframework/cloud/dataflow/server/db/migration/oracle/V7__Boot3_Add_Task3_Batch5_Schema.java
• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/java/org/springframework/cloud/dataflow/server/db/migration/AbstractAggregateViewMigration.java

Manual:

• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/resources/schemas/oracle/V6-dataflow.sql
• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/resources/schemas/oracle/V7-dataflow.sql
• https://github.com/spring-cloud/spring-cloud-dataflow/blob/main/spring-cloud-dataflow-server-core/src/main/resources/schemas/oracle/V8-dataflow.sql

This section discusses Flyway/manual https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#configuration-local-rdbms-schema

[sampathsri]

Thanks for the suggestion. Currently the tables are managed manually and flyway is disabled. I might have to revisit all the tables with the latest schema and update it. I will try and update the thread.

[onobc]

Also, if you are moving from Dataflow 2.8 there will likely be more of those scripts you need to run. The ones I listed above are for 2.10 -> 2.11.

Looks like the only other one you will need is V5__Add_Task_Execution_Params_Indexes. It has the @Since tag of 2.10. The ones before it have a @Since tag of 2.8.

[sampathsri]

After the DB update, 2.11 started working fine without any issues. Thanks for the help.