- Materialized option doesn't handle the structure as a table #513
- Remove percentage indicator from table progress #514
- Add folder property to table structure #515
- Add a threshold for LOBs saved outside the SIARD file (#504) For more information access our wiki.
- Security fixes (Upgrade dependencies)
- Merkle Tree filter malfunction with multi-schema databases #494
- This parser does not support specification "dbptk-core" version "2.9.8" #486
- ST_GEOMETRY gives Null Pointer Exception error #487
- Improved support for Microsoft SQL Server use integrated login on Windows #488
- creating a siard do not take Postgresql own materialized views #485
- Problem with overwriting existing folder when exporting to SIARD-DK #480
- Bump UCanAccess Driver #481
- Oracle connection fails if password has special characters #471
- SIARD to SIARD migration ignoring some import-config properties #472
- 'numeric' column type without any precision or scale is assumed to be integer #477
- Missing a dependency when handling Oracle XMLType #465
- NPE during validation if mandatory fields are not filled #466
- Fallback message when certain queries return unexpected results #467
- Add strategy to ignore and report triggers that not comply with SIARD 2.1 specification #469
- Add queryOriginal metadata field to PostgreSQL DBMS #451
- PostgreSQL module not appending where and order by clauses #462
- Normalize file path inside the table content when LOBs outside SIARD file #464
- Command line output differs from the file of validation reporter #461
- Validation fails when the validator runs in parallel #458
- Fix number column in Oracle with no length specified gives error in extraction of table definition (#454)
- Fix columnConfiguration to include NON_DEFAULT values for merkle tree property
- Fix #455 Added support to choose threshold lob size
- Inventory filter module (#444) For more information access our wiki.
- Fix #450 Siard metadata edition
- Add new input type parameters
- Add more information to module parameters
- Improve DBPTK Enterprise integration when dealing with LOB
- Allow set variables in YAML import-config (#439) For more information access our wiki.
- Database archive content validation using Merkle Tree. (#433) For more information access our wiki.
- Timestamp output normalization (#440)
- Bump Jackson version from 2.10.2 to 2.10.3
- Oracle maven dependency is now retrieved from the maven central (eliminates the need to have an Oracle account and configure the maven security to download the dependency)
- Added a new module: import-config.This module is used to control which schemas, tables and columns are read from the import module to the export module. It also allows to add options for custom views, views materialization, table or view filtering and database related information such as users, roles, privileges, routines or table oriented like triggers and much more. More information can be found at our wiki.
- Added the capability to filter content from a table (#435)
- Added the capability to sort content from a table (#436)
- Added a Java property to change the prefetch size for Oracle LOB (#437)
- Added a strategy to write LOB data in a parallel fashion (#438)
- list-tables module has been replace by the import-config module. More information can be found at our wiki.
Bug fixes:
- Fix #422 - Error with package from MS Access (timestamp)
- Fix #423 - NPE when validating a SIARD file foreign keys
- Fix #426 - Description shows "null" when table comment is empty
- Fix #427 - SIARD validation - error when starting two validations at same time
- Fix #428 - Missing encoding on metadata.xml
Improvements:
- Add default port number to Oracle connections
Bug fixes:
- Fix #422 - Error with package from MS Access (timestamp)
- Fix #423 - NPE when validating a SIARD file foreign keys
- Fix #426 - Description shows "null" when table comment is empty
- Fix #427 - SIARD validation - error when starting two validations at same time
- Fix #428 - Missing encoding on metadata.xml
Improvements:
- Add default port number to Oracle connections
Bug fixes
- Fix #411 log file and -ide (disable encryption) parameter
- Fix #412 Sybase - Fatal error - NPE
- Fix #414 Opt out which views should be materialized
- Fix #415 List tables missing views
- Fix #418 Problem recreating database from SIARD file (MSSQL Server decimal max precision)
Improvements
- Improve validation reporting - related to #416
- Fix #409 DBPTK migration - microseconds should not be rounded
- Fix #338 MSSQL conversion, error on small decimal values, being set to 0
- Fix #406 Validation SIARD error on requirement 5.10
- Fix #407 validation - DBPTK memory issues (heap size java)
To solve #407 it was used an external library to remove certain objects from the heap. A new property was added to define the path where to storage the off-heap file. More information can be found at our wiki.
- Fix #403 Change T_6.4-5 requirement validation
- Fix #404 Improve SQL role metadata information fetching
- Fix #405 SIARD Validation fails on additional checks - foreign keys, triggers and check constraints
- SIARD Validator #353
Validates a SIARD against its specification and also a set of additional checks.
- Fix #392 DBPTK wrongly notices that a schema exists, but it does not
- Fix #395 SIARD can't validate - possibly due to empty table
- Fix #397 LOB options: Schema referenced in column list file, was not found in the database
- Fix #400 Malformed entry in table list
- #393 Validating error too generic - "validator expects SIARD2.1"
dbptk validate --import-file <path-to-siard> [-r <path_to_report> -a <path_to_allowed_types>]
| Option | Description |
|---|---|
| --import-file value | SIARD file to be validated |
| --report value | File where the validation report will be saved |
| --allowed value | File with allowed data types for the categories UDT or distinct, one per line. |
More information can be found at: https://github.com/keeps/db-preservation-toolkit/wiki/Validation
- SIARD Validator #353
Validates a SIARD against its specification and also a set of additional checks.
dbptk validate --import-file <path-to-siard> [-r <path_to_report> -a <path_to_allowed_types>]
| Option | Description |
|---|---|
| --import-file value | SIARD file to be validated |
| --report value | File where the validation report will be saved |
| --allowed value | File with allowed data types for the categories UDT or distinct, one per line. |
More information can be found at: https://github.com/keeps/db-preservation-toolkit/wiki/Validation
- Support connecting to DBMSs via an SSH tunnel with password authentication #370
- Fix SSH connection problems with PostgreSQL #391
- Create SIARD of postgres fails, schema name error #387
- Improve database metadata fetching strategy
- Add an option to test if a custom view query is a valid one #390
To see the help menu use: dbptk help migrate. To check a specific module, dbptk help migrate <module>
--import-sshor--export-sshto enable SSH tunnel--import-ssh-host,--import-ssh-user,--import-ssh-password,--import-ssh-portor--export-ssh-host,--export-ssh-user,--export-ssh-password,--export-ssh-port
dbptk migrate --import mysql --import-host localhost --import-user root --import-password 123456 --import-database sakila --import-ssh --import-ssh-host IP Address --import-ssh-user USER --import-ssh-password PASSWORD --import-ssh-port <BY_DEFAULT_IS_22> --export siard-2 --export-file /path/to/siard
- Support connecting to DBMSs via an SSH tunnel with password authentication #370
To see the help menu use: dbptk help migrate. To check a specific module, dbptk help migrate <module>
--import-sshor--export-sshto enable SSH tunnel--import-ssh-host,--import-ssh-user,--import-ssh-password,--import-ssh-portor--export-ssh-host,--export-ssh-user,--export-ssh-password,--export-ssh-port
dbptk migrate --import mysql --import-host localhost --import-user root --import-password 123456 --import-database sakila --import-ssh --import-ssh-host IP Address --import-ssh-user USER --import-ssh-password PASSWORD --import-ssh-port <BY_DEFAULT_IS_22> --export siard-2 --export-file /path/to/siard
- Materialised views (as tables) #348
- Add enhanced support for Sybase ASA DBMS #349
- Add support for Progress OpenEdge RDBMS #350
- Denormalize some/all table using foreign keys #351
- SIARD metadata editing #352
Regarding the denormalize some/all table using foreign keys (#351) issue, the option discussed was creating custom views from a SQL query. Check custom views section.
Use dbptk edit --import-file=<path> [OPTIONS]
--set, updates a certain attribute to a new value.
--list, lists all the possible changes in a SIARD archive.
--set dbname '<value>'. Updates the dbname attribute to the new value;--set 'schema:<schemaName>' 'table:<tableName>' 'column:<columnName>' description '<description>'. Updates the description of a certain column in a certain table from a given schema;--set 'schema:<schemaName>' description '<description>'. Updates the description of a given schema.
When using a JDBC supported import module:
-icv <path>or--import-custom-views=<path>with a path to the query list file;
dbptk migrate --import mysql --import-hostname=<hostname> --import-database=<database> --import-username=<user> --import-password=<password> ---import-custom-views=<path> --export siard-2 --export-file=<archive>
This file should follow the YAML syntax as follows:
<schema_name>:
<view_name>: |
query line 1
query line 2
(...)
<description>: <example-description>
<another_view_name>: |
query line 1
query line 2
(...)
<description>: <example-description>
Use dbptk migrate <importModule> [import module options] <exportModule> [export module options] [<filterModule(s)> [filter module options]]
- Add description option to custom view
Regarding the Denormalize some/all table using foreign keys (#351) issue, the option discussed was creating custom views from a SQL query. Check custom views section.
When using a JDBC supported import module:
-icv <path>or--import-custom-views=<path>with a path to the query list file;
dbptk migrate --import mysql --import-hostname=<hostname> --import-database=<database> --import-username=<user> --import-password=<password> ---import-custom-views=<path> --export siard-2 --export-file=<archive>
This file should follow the YAML syntax as follows:
<schema_name>:
<view_name>: |
query line 1
query line 2
(...)
<description>: <example-description>
<another_view_name>: |
query line 1
query line 2
(...)
<description>: <example-description>
Use dbptk migrate <importModule> [import module options] <exportModule> [export module options] [<filterModule(s)> [filter module options]]
- Fix bug #380
- Fix bug #381
- Improve the error handling for custom views feature
- Materialised views (as tables) #348
- Add enhanced support for Sybase ASA DBMS #349
- Add support for Progress OpenEdge RDBMS #350
- Denormalize some/all table using foreign keys #351
- SIARD metadata editing (#352)
Regarding the Denormalize some/all table using foreign keys (#351) issue, the option discussed was creating custom views from a SQL query. Check custom views section.
Use dbptk edit --import-file=<path> [OPTIONS]
--set, updates a certain attribute to a new value.
--list, lists all the possible changes in a SIARD archive.
--set dbname '<value>'. Updates the dbname attribute to the new value;--set 'schema:<schemaName>' 'table:<tableName>' 'column:<columnName>' description '<description>'. Updates the description of a certain column in a certain table from a given schema;--set 'schema:<schemaName>' description '<description>'. Updates the description of a given schema.
When using a JDBC supported import module:
-icv <path>or--import-custom-views=<path>with a path to the query list file;
dbptk migrate --import mysql --import-hostname=<hostname> --import-database=<database> --import-username=<user> --import-password=<password> ---import-custom-views=<path> --export siard-2 --export-file=<archive>
This file should follow the YAML syntax as follows:
schema_name:
view_name: |
query line 1
query line 2
...
another_view_name: |
query line 1
query line 2
...
Use dbptk migrate <importModule> [import module options] <exportModule> [export module options] [<filterModule(s)> [filter module options]]