v24.4.1

Features

• Allow non-entry if Required=False, while still enforcing validation rules if value is entered by @mialy-defelice in #1390
• Add Manifest download to the CLI by @mialy-defelice in #736
• Cross Manifest Validation: Allow first manifest to submit, if no target manifest to validate against. by @mialy-defelice in #1398
• Reorganize validation message level determination as a precursor to single rule disjunction by @GiaJordan in #1366
• Match None Manifest Validation Rule FDS-1766 by @mialy-defelice in #1387
• Add functionality to set requirements based per component by @mialy-defelice in #1401
• Updated python client to ^4.1.0 by @andrewelamb in #1403

Bug fixes

• Updated workflows to make sure uwsgi gets installed when publishing an image by @linglp in #1377
• Updated workflow to install dependencies and root project together by @linglp in #1380
• Schematic v24.2.1 AWS deployment fix by @linglp in #1378
• Schematic v24.2.1 AWS deployment fix - 2 by @linglp in #1382
• Fix wrong function name reference in function get_viz_component_attributes_explorer by @linglp in #1384
• Added installation of uWSGI in docker file by @linglp in #1381
• Avoid repeatedly calling generate_data_model_graph by @linglp in #1374
• Fix check_synapse_cache_size function; allow file size to be float by @linglp in #1389
• Added help text for file-annotations-upload parameter by @linglp in #1406
• Remove schematic init from CLI by @linglp in #1400
• Added project scope parameter in test_api.py by @linglp in #1409
• JSONLD dictionary parsing error by @mialy-defelice in #1407
• Convert nans in manifest to empty strings just prior to JSON Schema Validation by @mialy-defelice in #1402

Technical debt

• Mypy visualization module by @andrewelamb in #1373

• Mypy utils files by @andrewelamb in #1375

• Pylint schema utils by @andrewelamb in #1371

• Add pre commit hook for black by @GiaJordan in #1379

• Remove makrupsafe from toml file by @andrewelamb in #1383

• Remove temp data model references from test_api by @mialy-defelice in #1386

• Develop mypy schema utils by @andrewelamb in #1392

• Add synapse cache as a parameter to SynapseStorage by @andrewelamb in #1394

• Added file-annotations-upload parameter when submitting a manifest by @linglp in #1395

• Develop pylint schema module by @andrewelamb in #1393

• Turn pyopenssl package optional by @linglp in #1413

• Remove uneeded parameter from submit_metadata_manifest by @andrewelamb in #1410

• Schematic v24.4.1 by @lakikowolfe in #1417

New Contributors

• @lakikowolfe made their first contribution in #1417

Full Changelog: v24.2.1...v24.4.1

v24.2.1

What's Changed

Features:

• Retain formatting of Display name for annotations and table column names by @mialy-defelice in #1330
• Data Model Labels, display_label or class_label options by @mialy-defelice in #1356
• Component Based Rule Settings by @mialy-defelice in #1368

Bug Fixes

• feature-configuration-fixes by @andrewelamb in #1357
• Fixed an issue where empty rows would not be removed from manifests, and add test for load_df util function by @GiaJordan in #1346
• Increase schematic API time out by @linglp in #1358
• Deleted old version of readthedocs yaml file by @linglp in #1365

Technical Debt

• Updated poetry version in readthedocs.yml by @linglp in #1364
• Update pandas and schematic_db dependencies by @GiaJordan in #1347
• Run rule combinations tests at every push instead of only selectively by @GiaJordan in #1363
• Linted visualization module by @andrewelamb in #1359
• Move remained logic in routes.py related to manifest generator to library by @linglp in #1351
• Update parallel_map to parallel_applymap by @mialy-defelice in #1372
• Schematic v24.2.1 by @GiaJordan in #1376
• Make api packages optional by @andrewelamb in #1362
• Linted utils module by @andrewelamb in #1361
• Ran black on all modules by @andrewelamb in #1355
• Update synapse client minimum version to 3.2.0 by @GiaJordan in #1345

Full Changelog: v24.1.1...v24.2.1

Schematic Release (v24.1.1)

What's Changed

We have added a Data Model Validator that is currently run during the convert step to check that some key assumptions about the data model are fulfilled, and that is converted properly to the graph network. We have plans to expand the validator in the future and allow it to be run directly.
Users can now supply the CSV data model itself as the data model, instead of the JSONLD.
Breaking Changes: Users will be expected to regenerate their JSON-LD using the new refactor release. Previously generated JSON-LDs will not currently work with the refactored schematic.
Other Changes:

• Schema Refactor: Unit Tests for TestDataModelGraph by @mialy-defelice in #1309
• Tests for DataModelRelationships class by @andrewelamb in #1293
• Schema Refactor: Develop unit tests DataModelJsonSchema by @mialy-defelice in #1307
• Schema Refactor: Unit Tests for DataModelJSONLDParser by @mialy-defelice in #1308
• Refactor Schemas: Unit tests for DataModelJSONLD FDS-1064 by @mialy-defelice in #1313
• Refactor Schemas: Response to review comments to original PR by @mialy-defelice in #1315
• Schema Refactor: Unit Tests for DataModelParser by @mialy-defelice in #1304
• Refactor Schemas: unit tests FDS-1061 TestDataModelNodes by @mialy-defelice in #1310
• Schema Refactor: Tests for Data Model CSV Parser by @mialy-defelice in #1305
• Schema Refactor: Create tests for DataModelEdges by @GiaJordan in #1306
• Changes to address JSONLD processing by @mialy-defelice in #1321
• Revert changes related to workflow by @linglp in #1328
• Refactor Schemas: Initial Release BugFix, incorrect name causing issues with Manifest generation -- FDS-1442 by @mialy-defelice in #1338
• Revert "Refactor Schemas: Initial Release BugFix, incorrect name causing issues with Manifest generation -- FDS-1442" by @mialy-defelice in #1343
• Re-add change to change parentOf to subClassOf by @mialy-defelice in #1344
• Refactor Schemas: JSONLD Data Model Parsing, change from using labels to display names, merge develop by @mialy-defelice in #1348
• Refactor Schemas - Latest with Fixes. by @mialy-defelice in #1350
• Schematic Schemas Refactor: DM Parser by @mialy-defelice in #1126
• Schematic release v24.1.1 by @linglp in #1354
  Bug Fixes:
• Bug fix: Added missing import statements by @linglp in #1327
• Bug Fix: Fix issue with trailing comma creating an empty node by @mialy-defelice in #1329
• Bug fix: Fixed tests in test_api.py by @linglp in #1352

Full Changelog: v23.12.1...v24.1.1

Schematic release (v23.12.1)

What's Changed

Bug fixes

• Update dockerfile poetry versions FDS-1325 by @GiaJordan in #1325
• Updated poetry version when building schematic docker image for AWS deployment by @linglp in #1326
• Fixed typing that API uses by @andrewelamb in #1332
• Added fit=true parameter when setting a dataframe by @linglp in #1318
• Resolved issues with cross-manifest validation that occured when a manifest was only a single row by @GiaJordan in #1337
• Resolved an issue where a visualization endpoint intermittently returned the wrong information by @GiaJordan in #1336
• Make access token optional to fix CLI manifest submission by @mialy-defelice in #1340
• Remove great_expectations/expectations/Manifest_test_suite.json before running validation tests by @linglp in #1342

Features

• Reduce execution time for Test Schematic workflow by @GiaJordan in #1320

Others

• Move methods of creating multiple manifests from API to manifest generator by @linglp in #1333
• Update deprecated GX v0.15.x functions by @GiaJordan in #1335
• Update manifest unit test for new manifest behavior by @GiaJordan in #1334
• Update pyproject.toml notation by @GiaJordan in #1331

Full Changelog: v23.11.1...v23.12.1

v23.11.1

What's Changed

Bug Fixes:

• [BugFix]: pypi publish workflow (v23.9.3) by @andrewelamb in #1301
• [BugFix]: Addresses HTAN-250 & HTAN-258 issues: Add functions to return a display name or requirement if not available. by @mialy-defelice in #1303
• [BugFix]: Address an issue where the asset view was not being set correctly for operations involving manifest validation by @GiaJordan in #1312

New Features:

• Raise a helpful error when a datatype cannot be found in a schema by @GiaJordan in #1302
• Move API access_token parameter to request header by @GiaJordan in #1288
• Add access_token to /model/validate header by @GiaJordan in #1311
• Update store/synapse.updateDB to take sg and se by @mialy-defelice in #1291
• Add default value for access_token to another validation method FDS-1248 by @GiaJordan in #1319

Technical Debt:

• Redid poetry install in publish workflow by @andrewelamb in #1299
• Update minimum Poetry version required to 1.3.0 FDS-1218 by @GiaJordan in #1316
• Add --durations flag to pytest by @GiaJordan in #1223
• Add Component as a required attribute to the example data model by @mialy-defelice in #1259
• Project scope manifest validation tests by @GiaJordan in #1314
• Update schematic dependencies FDS-1312 by @GiaJordan in #1322
• Schematic v23.11.1 by @GiaJordan in #1317

Full Changelog: v23.9.3...v23.11.1

v23.9.3

What's Changed

• BugFix: is_class_in_schema no longer errors out when a class is not in schema by @GiaJordan in #1287
• BugFix: Fix error when use_annotations=True for record based metadata where there are no existing annotations by @GiaJordan in #1285
• Bug Fix: Adding annotations to files with -mrt file_only parameter by @mialy-defelice in #1290
• Develop markers for api tests according to credentials required FDS-1026 by @GiaJordan in #1289

Full Changelog: v23.9.2...v23.9.3

v23.9.1

What's Changed

• [bug fix] Renamed function related to setting background color of required columns by @linglp in #1276
• [bug fix] Added includesType parameter when using walk function by @linglp in #1281
• [bug fix] Do not pull annotations when use_annotations set to False and there is no existing manifests by @linglp in #1278
• Add unit test for add_root_to_component function by @linglp in #1282
• [bug fix] Skipped submission when running API tests remotely by @linglp in #1280
• Refactored logic when pulling annotations by @linglp in #1279
• [bug fix] Fix issues when generating an existing manifest as an excel sheet by @linglp in #1284
• Schematic release 23.9.1 by @andrewelamb in #1283

Full Changelog: v23.8.1...v23.9.1

v23.8.1

What's Changed

• [bug fix] Updated docker file when building docker images for schematic AWS deployment by @linglp in #1269
• update channel id to fair-data-team channel by @GiaJordan in #1272
• [bug fix] Updated manifest download endpoint to avoid querying file view by @linglp in #1270
• [bug fix] Fixed typo in manifest generator by @linglp in #1275
• Add API endpoint that returns the current version of schematic by @GiaJordan in #1277
• [bug fix] Updated .synapseCache, functions to calculate cache, and cleared manifests before each download by @linglp in #1268
• replace issue templates by @allaway in #1274
• Schematic v23.8.1 FDS-807 by @GiaJordan in #1273

Full Changelog: v23.7.1...v23.8.1

v23.7.1

Release Notes

New Features & Improvements:

• Allow users to set strictness of google sheet/Excel regex validation via the Schematic API. Previously the default schematic configuration of strict_validation=true did not allow a user to proceed if an incorrect value was entered. By exposing the strict_validation option in the manifest generation api, users are now able to select True or False. strict_validation=false will allow users to enter an incorrect value, but be served a warning. In either case, incorrect entries will not pass manifest validation.
• Breaking change for 23.7.1 - Simplified Schematic configuration - this will require config changes as of 23.7.1
  • Please adjust your config file following the documentation example and Readme:
  • https://github.com/Sage-Bionetworks/schematic/blob/develop/config_example.yml
  • Readme (see Configure config.yml File section)
• Optimized the performance in table upsert backwards compatibility by Adding functionality to the synapseStorage object to make calls to the synapse REST API by way of the functionality exposed in the synapsePythonClient.

Bug Fixes:

• Addressed regex search validation limitations in Excel and updated documentation on use.
• Ensure that entity annotations are always compliant with Synapse regardless of column name format to avoid causing an error.
• Addressed bugs to ensure that submission and annotation are occurring as expected

Technical Debt:

• Code doesn't escape the 2nd law of thermodynamics. We put energy into refactoring handling of validation rules and interactions with Synapse (so that adding features and avoiding bugs is easier later); catching errors and exceptions more robustly and specifically (so that users and clients know what's causing a problem and can handle, report, or fix it more effectively); improving coverage of automated testing (so that we reduce the likelihood of letting bugs in released versions of schematic).

For more details on specific changes, please refer to the changelog below.

Full Changelog: v23.6.3...v23.7.1

What's Changed

• Replaced synapse API calls with synapse Python client call by @linglp in #1235
• Feature fds 273 coniguration by @andrewelamb in #1219
• Expose strict_validation option for manifest/generate endpoint by @mialy-defelice in #1253
• Optimize performance in table upsert backwards compatibility scenarios by @GiaJordan in #1229
• Fix df_utils/load_df so it more accurately captures integers to prevent Regex Errors. by @mialy-defelice in #1240
• Use FAIR Data service desk for issues by @afwillia in #1257
• Update Black to 2023 version: 23.7.0 by @GiaJordan in #1262
• Update mypy to latest version by @GiaJordan in #1263
• Updated cli help text for parameter json_schema by @linglp in #1264
• Modified Nginx parameters to fix submission issue by @linglp in #1255
• modify tag by @linglp in #1251
• BugFix: Fixed an issue where files on synapse were not being annotated correctly by @GiaJordan in #1254
• [bug fix] Fixed errors when calling get_empty_manifest by @linglp in #1260
• [bug fix] Updated parameters being used when generating existing manifests in test_api.py by @linglp in #1261
• Update schematic_db minimum version to v0.0.29 by @GiaJordan in #1266
• Fix Google Sheet column info mismatch FDS-675 by @mialy-defelice in #1265
• Only add validation rules to google sheets by @mialy-defelice in #1252
• Make all entity annotations comply with synapse FDS-481 FDS-726 by @GiaJordan in #1267
• Schematic 23.7.1 - FDS-728 by @linglp in #1271

New Contributors

• @afwillia made their first contribution in #1257

Full Changelog: v23.6.3...v23.7.1

Minor release v23.6.3

What's Changed

• [Bug fix]: Fix regex in docker build workflow by @linglp in #1246
• Minor Release v23.6.3 by @linglp in #1247
• Revert "[Bug fix]: Fix regex in docker build workflow" by @linglp in #1248
• [Bug fix]: Fix regex in docker build workflow by @linglp in #1249
• Minor Release v23.6.3 by @linglp in #1250

Full Changelog: v23.6.2...v23.6.3