v19.2.0-alpha.1

Release v19.2.0-alpha.1

From feature/es-phase-2 deafd8b

v19.2.0-alpha.0

Release v19.2.0-alpha.0

#3858
The release branch was checked out from feature/es-phase-2 34a9205

v19.1.0

[v19.1.0] 2024-10-07

Migration Notes

This release contains changes listed here as well as changes listed in v19.0.0,
despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for
each Cumulus version between your current version and v19.1.0 as normal.

Added

• CUMULUS-3020

  • Updated sfEventSqsToDbRecords to allow override of the default value
    (var.rds_connection_timing_configuration.acquireTimeoutMillis / 1000) + 60)
    via a key 'sfEventSqsToDbRecords' on var.lambda_timeouts on the main cumulus module/archive module

  Please note - updating this configuration is for adavanced users only. Value changes will modify the visibility
  timeout on sfEventSqsToDbRecordsDeadLetterQueue and sfEventSqsToDbRecordsInputQueue and may lead to system
  instability.

• CUMULUS-3756

  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.

• CUMULUS-3773

  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test

Changed

• CUMULUS-3928
  • updated publish scripting to use [email protected] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0
• CUMULUS-3906
  • Bumps example ORCA deployment to version v10.0.1.

Fixed

• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects

v18.5.1

Release v18.5.1

Please note This is a backport release - changes in v18.5.1 may not yet be released in future versions, as this
is a backport/patch release on the v18.5.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Added

• CUMULUS-3773
  • Added sftpFastDownload configuration to SyncGranule task.
  • Updated @cumulus/sftp-client and @cumulus/ingest/SftpProviderClient to support both regular and fastDownload.
  • Added sftp support to FakeProvider
  • Added sftp integration test
• CUMULUS-3756
  • Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
  • This is to allow files matching specified regex to be excluded when updating the Related URLs list
  • Defaults to the current behavior of excluding no files.
• CUMULUS-3919
  • Added terraform variables disableSSL and rejectUnauthorized to tf-modules/cumulus-rds-tf module.

Changed

• CUMULUS-3928
  • updated publish scripting to use [email protected] for user email
  • updated publish scripting to use esm over common import of latest-version
  • updated bigint testing to remove intermitted failure source.
  • updated postgres dependency version
• CUMULUS-3838
  • Updated python dependencies to latest:
    • cumulus-process-py 1.4.0
    • cumulus-message-adapter-python 2.3.0

Fixed

• CUMULUS-3902
  • Update error handling to use AWS SDK V3 error classes instead of properties on js objects

v18.5.0

[v18.5.0] 2024-10-03

Migration Notes

CUMULUS-3536 Upgrading from Aurora Serverless V1 to V2

• The updates in CUMULUS-3536 require an upgrade of the postgres database.
  Please follow [Upgrading from Aurora Serverless V1 to V2]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/serverless-v2-upgrade)

Added

• CUMULUS-3536
  • Added rejectUnauthorized = false to db-provision-user-database as the Lambda
    does not have the Serverless v2 SSL certifications installed.

Changed

• CUMULUS-3725
  • Updated the default parameter group for cumulus-rds-tf to set force_ssl
    to 0. This setting for the Aurora Serverless v2 database allows non-SSL
    connections to the database, and is intended to be a temporary solution
    until Cumulus has been updated to import the RDS rds-ca-rsa2048-g1 CA bundles in Lambda environments.
    See CUMULUS-3724.

Fixed

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
• CUMULUS-3824
  • Added the missing double quote in ecs_cluster autoscaling cf template
• CUMULUS-3846
  • improve reliability of unit tests
    • tests for granules api get requests separated out to new file
    • cleanup of granule database resources to ensure no overlap
    • ensure uniqueness of execution names from getWorkflowNameIntersectFromGranuleIds
    • increase timeout in aws-client tests
• Snyk
  • Upgraded moment from 2.29.4 to 2.30.1
  • Upgraded pg from ~8.10 to ~8.12

v18.3.5

[v18.3.5] 2024-09-19

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
  • Fix async Operation migration to include Migration Count Report

v19.0.0

Deprecated

This release has been deprecated in favor of the 18.5->19.1 release series.

[v19.0.0] 2024-08-28

Dashboard Requirement

• Cumulus Dashboard v12.2.0

Breaking Changes

• This release includes Replace ElasicSearch Phase 1 updates, we no longer save collection/granule/execution records to
  ElasticSearch, the collections/granules/executions API endpoints are updated to perform operations on the postgres database.

Migration Notes

CUMULUS-3792 Add database indexes. Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3792 require a manual update to the postgres database in the production environment.
  Please follow [Update Table Indexes for CUMULUS-3792]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/update_table_indexes_CUMULUS_3792)

Replace ElasticSearch Phase 1

• CUMULUS-3238
  • Removed elasticsearch dependency from collections endpoint
• CUMULUS-3239
  • Updated executions list api endpoint and added ExecutionSearch class to query postgres
• CUMULUS-3240
  • Removed Elasticsearch dependency from executions endpoints
• CUMULUS-3639
  • Updated /collections/active endpoint to query postgres
• CUMULUS-3640
  • Removed elasticsearch dependency from granules endpoint
• CUMULUS-3641
  • Updated collections api endpoint to query postgres instead of elasticsearch except if includeStats is in the query parameters
• CUMULUS-3642
  • Adjusted queries to improve performance:
    • Used count(*) over count(id) to count rows
    • Estimated row count for large tables (granules and executions) by default for basic query
  • Updated stats summary to default to the last day
  • Updated ExecutionSearch to not include asyncOperationId by default
• CUMULUS-3688
  • Updated stats api endpoint to query postgres instead of elasticsearch
• CUMULUS-3689
  • Updated stats/aggregate api endpoint to query postgres instead of elasticsearch
  • Created a new StatsSearch class for querying postgres with the stats endpoint
• CUMULUS-3692
  • Added @cumulus/db/src/search BaseSearch and GranuleSearch classes to
    support basic queries for granules
  • Updated granules List endpoint to query postgres for basic queries
• CUMULUS-3693
  • Added functionality to @cumulus/db/src/search to support range queries
• CUMULUS-3694
  • Added functionality to @cumulus/db/src/search to support term queries
  • Updated BaseSearch and GranuleSearch classes to support term queries for granules
  • Updated granules List endpoint to search postgres
• CUMULUS-3695
  • Updated granule list api endpoint and BaseSearch class to handle sort fields
• CUMULUS-3696
  • Added functionality to @cumulus/db/src/search to support terms, not and exists queries
• CUMULUS-3699
  • Updated collections api endpoint to be able to support includeStats query string parameter
• CUMULUS-3792
  • Added database indexes to improve search performance

v18.3.4

[v18.3.4] 2024-08-27

Please note changes in v18.3.4 may not yet be released in future versions, as this
is a backport/patch release on the v18.3.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Changed

• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when fetching all rules such as in sqsMessageConsumer.

v18.4.0

[v18.4.0] 2024-08-23

Migration Notes

Important
This release requires manual intervention before applying the Terraform changes. The two tickets below require manual database updates including adding indices, modifying table constraints, and updating types. Please read and complete the steps outlined in the following two tickets, CUMULUS-3320 and CUMULUS-3449 before deploying v18.4.0.

CUMULUS-3320 Update executions table

The work for CUMULUS-3320 required index updates as well as a modification of a
table constraint. To install the update containing these changes you should:

• Pre-generate the indexes on the execution table. This can be done via manual
  procedure prior to upgrading without downtime, or done more quickly before or
  during upgrade with downtime.
• Update the executions_parent_cumulus_id_foreign constraint. This will
  require downtime as updating the constraint requires a table write lock, and
  the update may take some time.

Deployments with low volume databases and low activity and/or test/development
environments should be able to install these updates via the normal automatic
Cumulus deployment process.

Please carefully review the migration process documentation. Failure to
make these updates properly will likely result in deployment failure and/or
degraded execution table operations.

CUMULUS-3449 Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3449 requires manual update to postgres database in
  production environment. Please follow Update Cumulus_id Type and
  Indexes

Breaking Changes

Added

• CUMULUS-3320
  • Added endpoint /executions/bulkDeleteExecutionsByCollection to allow
    bulk deletion of executions from elasticsearch by collectionId
  • Added Bulk Execution Delete migration type to async operations types
• CUMULUS-3608
  • Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions
    of the variables here include notes on usage and what users should
    consider if configuring something other than the default values.
• CUMULUS-3449
  • Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id,
    files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id
    and pdrs.execution_cumulus_id
  • Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
  • Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index
    to granules table

Changed

• CUMULUS-3320
  • Updated executions table (please see Migration section and Upgrade
    Instructions for more information) to:
    • Add index on collection_cumulus_id
    • Add index on parent_cumulus_id
    • Update executions_parent_cumulus_id_foreign constraint to add ON DELETE SET NULL. This change will cause deletions in the execution table to
      allow deletion of parent executions, when this occurs the child will have
      it's parent reference set to NULL as part of the deletion operations.
• CUMULUS-3449
  • Updated @cumulus/db package and configure knex hook postProcessResponse to convert the return string
    from columns ending with "cumulus_id" to number.
• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when
    fetching all rules such as in sqsMessageConsumer.

Fixed

• CUMULUS-3817
  • updated applicable @AWS-SDK dependencies to 3.621.0 to remove inherited vulnerability from fast-xml-parser
• CUMULUS-3320
  • Execution database deletions by cumulus_id should have greatly improved
    performance as a table scan will no longer be required for each record
    deletion to validate parent-child relationships
• CUMULUS-3818
  • Fixes default value (updated to tag 52) for async-operation-image in tf-modules/cumulus.
• CUMULUS-3840
  • Fixed @cumulus/api/bin/serve to correctly use EsClient.

v18.3.3

Fixed

• CUMULUS-3824
  • Changed the ECS docker storage driver to overlay2, since devicemapper is removed in Docker Engine v25.0.
  • Removed ecs_docker_storage_driver property from cumulus module.
• CUMULUS-3836
  • Terraform configuration for cleanExecutions now correctly configures ES_HOST and lambda security group