UXPROD-3903: Upgrade MDES to version 3.0.0 (folio-org#270)

* Update POM and some src files to accommodate new schema

* Change submodule

* Update submodule configuration a bit more

* Specify source for maven/IDE intellisense

* Add http to server urls in open-api file to be able to generate sources

* Remove problematic test lines temporarily

* Update submodule

* Updated submodule folio-export-common

* Update submodule

* Update submodule

* Update submodule

* update submodule!

* Update submodule

* update submodule again

* update submodule

* update submodule

* Updated submodule folio-export-common

* Update submodule

* Updated submodule folio-export-common

* Add logic to convert from old scheduling format to new one - #3 (#6)

* changes from master

* update folio-export-common

* closes #3

* moved isLegacyJob to new util class: LegacyBursarMigrationUtil

* moved recreateLegacyJobs to util class: LegacyBursarMigrationUtil

* Update submodule

* Add sonar configuration

* Change new code recency

* Update submodule

* Update submodule

* Revert unnecessary formatting changes

* Revert more unnecessary formatting changes

* Parentheses

* Remove more unneccessary changes

* Convert old tests to use new BursarExportJob class

* Convert test to use BursarExportJob class

* Update sample data to new bursar format

* Refactor and add tests to LegacyBursarMigrationUtil - closes #9

* Update sonar.yml

* Fix some code smells

* Add lombok config

* Update module descriptor

* Revert versions

* Update version in pom.xml

* Remove jacoco

* Add test for BursarExportLegacyJobService

* Rewrite code

* Add more coverage to BursarExportLegacyJobServiceTest

* Fix test

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Fix correct UserData types

* Updated submodule folio-export-common

* Update module descriptor MDES version to 1.0

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* Add condition to new filter age

* Fix test error due to age filter

* Fix default condition to GREATER_THAN_EQUAL

* Updated submodule folio-export-common

* Merge folio-org/master pinto folio-org/ualibweb-staging

* Add back log4j2

* Fix unit tests issues

* Fix unit tests issues

* Change ports back to 8081

* Refactor entityToDto to avoid duplications

* Refactor code to avoid duplicate

* Failed assertion

* Remove uneeded imports

* Refactor job entities to avoid duplications

* Fix sonar issue

* Clean up and revert unnecessary changes

* Clean up code

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* [MODEXPS-238, MODEXPS-245] Rename LegacyJob schemas and refactor LegacyBursarMigrationUtil to BursarMigrationService  (folio-org#281)

* Rename legacy job to job with legacy bursar parameters

* Remove localhost in jobs.yaml

* Updated submodule folio-export-common

* Updated submodule folio-export-common

* MODEXPS-238: Rename LegacyJob to JobWithLegacyBursarParameters

* Updated submodule folio-export-common

* MODEXPS-245: Refactor LegacyBursarMigrationUtil to BursarMigrationService

* Update pom.xml

* Updated submodule folio-export-common

* Uncomment tests that should now work

* Resolve formatting issues

* Resolve more formatting issues

* Comment out failed assertions

* Remove duplication in pom.xml

* Set different timeout for testScheduleJobWithMultipleTriggers test

* Set different timeout for testSchedule

* Change timeout for tests

* Increase timeout for assertions in QuarzExportJobSchedulerTest

* Increase timeout on assertion for QuartzExportJobSchedulerTest

* Increase poll delay

* Increase timeout

* Decrease poll delay

* Updated submodule folio-export-common

* Fix deprecated property in BaseJob

* Increase timeout for testScheduleJobWithMultipleTriggers

* Increase even more timeout for testScheduleJobWithMultipleTriggers

* Empty-Commit to rerun SonarCloud Code Analysis

* Increase timeout for testSchedule

* Reduce timeout for testSchedule and testScheduleJobWithMultipleTriggers

* Increase timeout for testSchedule and testScheduleJobWithMultipleTriggers

* Increase timeout for testDeleteOldJobDeleteScheduler

* Increase timeout for testScheduleJobWithMultipleTriggers

* Add plugin 'find-and-replace-maven-plugin'

* Fix type mapping for LegacyBursarFeeFinesTypeMappings

* Empty-Commit to rerun Jenkins build

* Empty-Commit to rerun Jenkins build

* Empty-Commit to rerun Jenkins build

* Update submodule

* Simplify migration logic

* Fix typo

* Convert configs, too

* Rename BurSar to Bursar

* Clarify header/footer token name

* Only run bursar migration for certain versions

* Update interface to v2.0

* formatting

* Improve mocking in tenant service test

* test debugging

* revert debugging changes

* Use possessive regex

* sonar pls

* Move bursar update checking into BursarMigrationService

* Update submodule

---------

Co-authored-by: Noah Overcash <[email protected]>
Co-authored-by: Faruq Yusuff <[email protected]>
Co-authored-by: Noah Overcash <[email protected]>
Co-authored-by: Viachaslau Khandramai <[email protected]>
Co-authored-by: danetsaoo <[email protected]>
Co-authored-by: Pax Nguyen <[email protected]>

descriptors/ModuleDescriptor-template.json

@@ -18,6 +18,10 @@
       "id": "users",
       "version": "15.3 16.0"
     },
+    {
+      "id": "inventory",
+      "version": "13.0"
+    },
     {
       "id": "circulation-logs",
       "version": "1.2"
@@ -38,7 +42,7 @@
   "provides": [
     {
       "id": "data-export-spring",
-      "version": "1.0",
+      "version": "2.0",
       "handlers": [
         {
           "methods": [

lombok.config

@@ -0,0 +1 @@
+lombok.addLombokGeneratedAnnotation = true

pom.xml

@@ -24,6 +24,8 @@
 
   <properties>
     <java.version>17</java.version>
+    <maven.compiler.source>17</maven.compiler.source>
+
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
     <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 
@@ -271,7 +273,6 @@
       <artifactId>rest-assured</artifactId>
       <scope>test</scope>
     </dependency>
-
   </dependencies>
 
   <build>
@@ -333,7 +334,7 @@
             </goals>
             <configuration>
               <sources>
-                <source>${project.build.directory}/generated-sources</source>
+                <source>${project.build.directory}/generated-sources/api/src/main/java</source>
               </sources>
             </configuration>
           </execution>
@@ -406,6 +407,28 @@
         </executions>
       </plugin>
 
+      <plugin>
+        <groupId>io.github.floverfelt</groupId>
+        <artifactId>find-and-replace-maven-plugin</artifactId>
+        <version>1.1.0</version>
+        <executions>
+          <execution>
+            <id>LegacyBursarFeeFinesTypeMappings</id>
+            <phase>generate-sources</phase>
+            <goals>
+               <goal>find-and-replace</goal>
+            </goals>
+            <configuration>
+              <replacementType>file-contents</replacementType>
+              <baseDir>target/generated-sources/src/main/java/org/folio/des/domain/dto/</baseDir>
+              <fileMask>LegacyBursarFeeFinesTypeMappings.java</fileMask>
+              <findRegex>&lt;String, List&gt;</findRegex>
+              <replaceValue>&lt;String, List&lt;LegacyBursarFeeFinesTypeMapping&gt;&gt;</replaceValue>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-resources-plugin</artifactId>
@@ -490,6 +513,7 @@
           <localCheckout>true</localCheckout>
         </configuration>
       </plugin>
+
     </plugins>
   </build>
 

src/main/java/org/folio/de/entity/BaseJob.java

@@ -0,0 +1,92 @@
+package org.folio.de.entity;
+
+import io.hypersistence.utils.hibernate.type.json.JsonBinaryType;
+import jakarta.persistence.Column;
+import jakarta.persistence.EnumType;
+import jakarta.persistence.Enumerated;
+import jakarta.persistence.GeneratedValue;
+import jakarta.persistence.Id;
+import jakarta.persistence.MappedSuperclass;
+import java.util.Date;
+import java.util.List;
+import java.util.UUID;
+import lombok.Data;
+import org.folio.des.domain.dto.EntityType;
+import org.folio.des.domain.dto.ExportType;
+import org.folio.des.domain.dto.IdentifierType;
+import org.folio.des.domain.dto.JobStatus;
+import org.folio.des.domain.dto.Progress;
+import org.hibernate.annotations.GenericGenerator;
+import org.hibernate.annotations.Type;
+import org.springframework.batch.core.BatchStatus;
+import org.springframework.batch.core.ExitStatus;
+
+@MappedSuperclass
+@Data
+public abstract class BaseJob {
+
+  @Id
+  @GeneratedValue(generator = "UUID")
+  @GenericGenerator(name = "UUID", type = org.folio.des.repository.generator.CustomUUIDGenerator.class)
+  @Column(updatable = false, nullable = false)
+  private UUID id;
+
+  private String name;
+
+  private String description;
+
+  private String source;
+
+  private Boolean isSystemSource;
+
+  @Enumerated(EnumType.STRING)
+  private ExportType type;
+
+  @Enumerated(EnumType.STRING)
+  private JobStatus status;
+
+  @Type(JsonBinaryType.class)
+  @Column(columnDefinition = "jsonb")
+  private List<String> files = null;
+
+  @Type(JsonBinaryType.class)
+  @Column(columnDefinition = "jsonb")
+  private List<String> fileNames = null;
+
+  private Date startTime;
+
+  private Date endTime;
+
+  private Date createdDate;
+
+  private UUID createdByUserId;
+
+  private String createdByUsername;
+
+  private Date updatedDate;
+
+  private UUID updatedByUserId;
+
+  private String updatedByUsername;
+
+  private String outputFormat;
+
+  private String errorDetails;
+
+  @Enumerated(EnumType.STRING)
+  private BatchStatus batchStatus;
+
+  @Type(JsonBinaryType.class)
+  @Column(columnDefinition = "jsonb")
+  private ExitStatus exitStatus;
+
+  @Enumerated(EnumType.STRING)
+  private IdentifierType identifierType;
+
+  @Enumerated(EnumType.STRING)
+  private EntityType entityType;
+
+  @Type(JsonBinaryType.class)
+  @Column(columnDefinition = "jsonb")
+  private Progress progress;
+}

src/main/java/org/folio/de/entity/Job.java

@@ -3,98 +3,15 @@
 import io.hypersistence.utils.hibernate.type.json.JsonBinaryType;
 import jakarta.persistence.Column;
 import jakarta.persistence.Entity;
-import jakarta.persistence.EnumType;
-import jakarta.persistence.Enumerated;
-import jakarta.persistence.GeneratedValue;
-import jakarta.persistence.Id;
-
-import java.util.Date;
-import java.util.List;
-import java.util.UUID;
-
 import lombok.Data;
-import org.folio.des.domain.dto.ExportType;
 import org.folio.des.domain.dto.ExportTypeSpecificParameters;
-import org.folio.des.domain.dto.JobStatus;
-import org.folio.des.domain.dto.IdentifierType;
-import org.folio.des.domain.dto.EntityType;
-import org.folio.des.domain.dto.Progress;
-import org.hibernate.annotations.GenericGenerator;
 import org.hibernate.annotations.Type;
-import org.springframework.batch.core.BatchStatus;
-import org.springframework.batch.core.ExitStatus;
 
 @Entity
 @Data
-public class Job {
-
-  @Id
-  @GeneratedValue(generator = "UUID")
-  @GenericGenerator(name = "UUID", strategy = "org.folio.des.repository.generator.CustomUUIDGenerator")
-  @Column(updatable = false, nullable = false)
-  private UUID id;
-
-  private String name;
-
-  private String description;
-
-  private String source;
-
-  private Boolean isSystemSource;
-
-  @Enumerated(EnumType.STRING)
-  private ExportType type;
+public class Job extends BaseJob {
 
   @Type(JsonBinaryType.class)
   @Column(columnDefinition = "jsonb")
   private ExportTypeSpecificParameters exportTypeSpecificParameters;
-
-  @Enumerated(EnumType.STRING)
-  private JobStatus status;
-
-  @Type(JsonBinaryType.class)
-  @Column(columnDefinition = "jsonb")
-  private List<String> files = null;
-
-  @Type(JsonBinaryType.class)
-  @Column(columnDefinition = "jsonb")
-  private List<String> fileNames = null;
-
-  private Date startTime;
-
-  private Date endTime;
-
-  private Date createdDate;
-
-  private UUID createdByUserId;
-
-  private String createdByUsername;
-
-  private Date updatedDate;
-
-  private UUID updatedByUserId;
-
-  private String updatedByUsername;
-
-  private String outputFormat;
-
-  private String errorDetails;
-
-  @Enumerated(EnumType.STRING)
-  private BatchStatus batchStatus;
-
-  @Type(JsonBinaryType.class)
-  @Column(columnDefinition = "jsonb")
-  private ExitStatus exitStatus;
-
-  @Enumerated(EnumType.STRING)
-  private IdentifierType identifierType;
-
-  @Enumerated(EnumType.STRING)
-  private EntityType entityType;
-
-  @Type(JsonBinaryType.class)
-  @Column(columnDefinition = "jsonb")
-  private Progress progress;
-
 }

src/main/java/org/folio/de/entity/bursarlegacy/JobWithLegacyBursarParameters.java

@@ -0,0 +1,20 @@
+package org.folio.de.entity.bursarlegacy;
+
+import io.hypersistence.utils.hibernate.type.json.JsonBinaryType;
+import jakarta.persistence.Column;
+import jakarta.persistence.Entity;
+import jakarta.persistence.Table;
+import lombok.Data;
+import org.folio.de.entity.BaseJob;
+import org.folio.des.domain.dto.ExportTypeSpecificParametersWithLegacyBursar;
+import org.hibernate.annotations.Type;
+
+@Entity
+@Table(name = "job")
+@Data
+public class JobWithLegacyBursarParameters extends BaseJob {
+
+  @Type(JsonBinaryType.class)
+  @Column(columnDefinition = "jsonb")
+  private ExportTypeSpecificParametersWithLegacyBursar exportTypeSpecificParameters;
+}

src/main/java/org/folio/des/builder/job/BurSarFeeFinesJobCommandBuilder.java → src/main/java/org/folio/des/builder/job/BursarFeeFinesJobCommandBuilder.java

@@ -12,7 +12,7 @@
 @Service
 @Log4j2
 @RequiredArgsConstructor
-public class BurSarFeeFinesJobCommandBuilder implements JobCommandBuilder {
+public class BursarFeeFinesJobCommandBuilder implements JobCommandBuilder {
 
   private final ObjectMapper objectMapper;
 

src/main/java/org/folio/des/config/ServiceConfiguration.java

@@ -6,7 +6,7 @@
 
 import org.folio.des.builder.job.AuthorityControlJobCommandBuilder;
 import org.folio.des.builder.job.BulkEditQueryJobCommandBuilder;
-import org.folio.des.builder.job.BurSarFeeFinesJobCommandBuilder;
+import org.folio.des.builder.job.BursarFeeFinesJobCommandBuilder;
 import org.folio.des.builder.job.CirculationLogJobCommandBuilder;
 import org.folio.des.builder.job.EHoldingsJobCommandBuilder;
 import org.folio.des.builder.job.EdifactOrdersJobCommandBuilder;
@@ -33,10 +33,10 @@
 import org.folio.des.service.config.ExportConfigService;
 import org.folio.des.service.config.acquisition.EdifactOrdersExportService;
 import org.folio.des.service.config.impl.BaseExportConfigService;
-import org.folio.des.service.config.impl.BurSarFeesFinesExportConfigService;
+import org.folio.des.service.config.impl.BursarFeesFinesExportConfigService;
 import org.folio.des.service.config.impl.ExportConfigServiceResolver;
 import org.folio.des.service.config.impl.ExportTypeBasedConfigManager;
-import org.folio.des.validator.BurSarFeesFinesExportParametersValidator;
+import org.folio.des.validator.BursarFeesFinesExportParametersValidator;
 import org.folio.des.validator.ExportConfigValidatorResolver;
 import org.folio.des.validator.acquisition.EdifactOrdersExportParametersValidator;
 import org.quartz.Scheduler;
@@ -59,11 +59,11 @@ ExportConfigConverterResolver exportConfigConverterResolver(DefaultExportConfigT
   }
 
   @Bean
-  ExportConfigValidatorResolver exportConfigValidatorResolver(BurSarFeesFinesExportParametersValidator burSarFeesFinesExportParametersValidator,
+  ExportConfigValidatorResolver exportConfigValidatorResolver(BursarFeesFinesExportParametersValidator bursarFeesFinesExportParametersValidator,
                       EdifactOrdersExportParametersValidator edifactOrdersExportParametersValidator) {
     Map<String, Validator> validators = new HashMap<>();
     validators.put(ExportConfigValidatorResolver.buildKey(ExportType.BURSAR_FEES_FINES, ExportTypeSpecificParameters.class),
-      burSarFeesFinesExportParametersValidator);
+      bursarFeesFinesExportParametersValidator);
     validators.put(ExportConfigValidatorResolver.buildKey(ExportType.EDIFACT_ORDERS_EXPORT, ExportTypeSpecificParameters.class),
       edifactOrdersExportParametersValidator);
     return new ExportConfigValidatorResolver(validators);
@@ -79,10 +79,10 @@ ExportTypeBasedConfigManager exportTypeBasedConfigManager(ConfigurationClient cl
   }
 
   @Bean
-  BurSarFeesFinesExportConfigService burSarExportConfigService(ConfigurationClient client, ExportConfigValidatorResolver exportConfigValidatorResolver,
+  BursarFeesFinesExportConfigService bursarExportConfigService(ConfigurationClient client, ExportConfigValidatorResolver exportConfigValidatorResolver,
             DefaultModelConfigToExportConfigConverter defaultModelConfigToExportConfigConverter,
             ExportConfigConverterResolver  exportConfigConverterResolver, BursarExportScheduler bursarExportScheduler) {
-    return new BurSarFeesFinesExportConfigService(client, defaultModelConfigToExportConfigConverter,
+    return new BursarFeesFinesExportConfigService(client, defaultModelConfigToExportConfigConverter,
             exportConfigConverterResolver, exportConfigValidatorResolver, bursarExportScheduler);
   }
 
@@ -105,23 +105,23 @@ BaseExportConfigService baseExportConfigService(ConfigurationClient client, Expo
 
 
   @Bean
-  ExportConfigServiceResolver exportConfigServiceResolver(BurSarFeesFinesExportConfigService burSarFeesFinesExportConfigService,
+  ExportConfigServiceResolver exportConfigServiceResolver(BursarFeesFinesExportConfigService bursarFeesFinesExportConfigService,
                           EdifactOrdersExportService edifactOrdersExportService) {
     Map<ExportType, ExportConfigService> exportConfigServiceMap = new HashMap<>();
-    exportConfigServiceMap.put(ExportType.BURSAR_FEES_FINES, burSarFeesFinesExportConfigService);
+    exportConfigServiceMap.put(ExportType.BURSAR_FEES_FINES, bursarFeesFinesExportConfigService);
     exportConfigServiceMap.put(ExportType.EDIFACT_ORDERS_EXPORT, edifactOrdersExportService);
     return new ExportConfigServiceResolver(exportConfigServiceMap);
   }
 
   @Bean JobCommandBuilderResolver jobCommandBuilderResolver(BulkEditQueryJobCommandBuilder bulkEditQueryJobCommandBuilder,
-                          BurSarFeeFinesJobCommandBuilder burSarFeeFinesJobCommandBuilder,
+                          BursarFeeFinesJobCommandBuilder bursarFeeFinesJobCommandBuilder,
                           CirculationLogJobCommandBuilder circulationLogJobCommandBuilder,
                           EdifactOrdersJobCommandBuilder edifactOrdersJobCommandBuilder,
                           EHoldingsJobCommandBuilder eHoldingsJobCommandBuilder,
                           AuthorityControlJobCommandBuilder authorityControlJobCommandBuilder) {
     Map<ExportType, JobCommandBuilder> converters = new HashMap<>();
     converters.put(ExportType.BULK_EDIT_QUERY, bulkEditQueryJobCommandBuilder);
-    converters.put(ExportType.BURSAR_FEES_FINES, burSarFeeFinesJobCommandBuilder);
+    converters.put(ExportType.BURSAR_FEES_FINES, bursarFeeFinesJobCommandBuilder);
     converters.put(ExportType.CIRCULATION_LOG, circulationLogJobCommandBuilder);
     converters.put(ExportType.EDIFACT_ORDERS_EXPORT, edifactOrdersJobCommandBuilder);
     converters.put(ExportType.E_HOLDINGS, eHoldingsJobCommandBuilder);