CUMULUS-3444 : Jk/backports/cumulus 3444 v16.1.x (#3569)

* Update rds module to allow for configurable parameter group update * Add upgrade docs * Doc updates * Documentation update to reference provided instructions * Update cluster deployment to handle new parameter group seamlessly * Update docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-13.md Co-authored-by: Nate Pauzenga <[email protected]> * Jk/bugfix/cumulus 3564 (#3565) * Set webpack configuration to *not* chunk globally * Update CHANGELOG * Whitelist to pass NPM audit, should be reverted unless #3564 is merged * remove duplicate entry re: pr feedback * add missing comma to audit-ci.json --------- Co-authored-by: ecarton <[email protected]> * Update docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-13.md Co-authored-by: Nate Pauzenga <[email protected]> * Add example project config in response to PR feedback * Fix omitted pick CHANGELOG update * Fix bad CHANGELOG merge * Fixup * CHANGELOG updates * Fix documentation formatting --------- Co-authored-by: Nate Pauzenga <[email protected]> Co-authored-by: ecarton <[email protected]>
nasa · Feb 1, 2024 · 77dbeef · 77dbeef
1 parent 40a1e7f
commit 77dbeef
Show file tree

Hide file tree

Showing 40 changed files with 150 additions and 38 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,32 +16,18 @@ stores as soon as possible after upgrading to this release.
 following 18.2.0/the first forward release supporting Postgres v13, as versions
 between 16.1.x and 18.2.x+ are unsupported on Aurora Postgres v13.
 
-#### Engine Upgrade when using `cumulus-rds-tf`
+#### Database Upgrade
 
-Users utilizing the `cumulus-rds-tf` module will have upgraded/had their
-database clusters forcibly upgraded at the next maintenance window after February 29, 2024.
-
-To upgrade your engine version, we recommend a manual (outside of
-terraform) upgrade. This will result in the cluster being upgraded with a
-manually set parameter group not managed by terraform.
-
-There are several options that AWS provides for upgrading your cluster engine,
-such as using the AWS console or CLI. For more information, visit their
-[documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.modifying.html#aurora-serverless.modifying.upgrade).
-
-Once you have manually upgraded your database engine and the cluster is now on
-version 13.12+, to continue using the `cumulus-rds-tf` module *once upgraded*,
-update following module configuration values if set, or allow their defaults to
-be utilized:
+Users utilizing the `cumulus-rds-tf` module should reference [cumulus-rds-tf
+upgrade
+instructions](https://nasa.github.io/cumulus/docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-13).
 
-```terraform
-parameter_group_family = "aurora-postgresql13"
-engine_version = 13.12
-```
-
-**Please Note**: When you apply this update, the original PostgreSQL v11
-parameter group will be removed, and recreated using PG13 defaults/configured
-terraform values and it will also update the database cluster to use the new configuration.
+## Changed
+- **CUMULUS-3564**
+  - Update webpack configuration to explicitly disable chunking
+- **CUMULUS-3444**
+  - Update `cumulus-rds-tf` module to take additional parameters in support of
+    migration from Aurora PostgreSQl v11 to v13.   See Migration Notes for more details.
 
 ## [v16.1.3] 2024-1-15
 

diff --git a/audit-ci.json b/audit-ci.json
@@ -12,6 +12,8 @@
     "knex",
     "jsonwebtoken",
     "cacheable-request",
-    "http-cache-semantics"
+    "http-cache-semantics",
+    "lodash.pick",
+    "crypto-js"
   ]
 }
diff --git a/docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-13.md b/docs/upgrade-notes/upgrade-rds-cluster-tf-postgres-13.md
@@ -0,0 +1,43 @@
+---
+id: upgrade-rds-cluster-tf-postgres-13
+title: Upgrade Database Cluster to PostgreSQL v13
+hide_title: false
+---
+
+Cumulus Core as of version > 18.1.0** now supports and is tested against Aurora Postgres v13.   All users should update their datastores to this version as part of an upgrade process upon upgrading to release version 18.2.0.
+
+**This update was also backported to the 16.1.x release chain starting with v16.1.3.
+
+We recommend stopping all ingest rules if database downtime is required (e.g. you do not have a blue-green database solution or are using serverless V1) for the update  as any unavailability of the database may result in unexpected database write failures (resulting in records in the Dead Letter Archive), workflow failures or other unexpected failures.
+
+**It is also recommended that users test/evaluate the upgrade prior to performing it on their primary datastore in order to determine expected downtime and/or other related issues for their particular configuration.**
+
+## Users utilizing the `cumulus-rds-tf` module
+
+It is recommended that users manually backup and/or consider cloning their datastore in order to recover the datastore if an upgrade goes awry.
+
+Upgrading the Aurora Serverless v1 cluster will be completed via AWS console in this document and require manual steps to complete the upgrade:
+
+- Ensure a supported version (> 18.1.0 *or* 16.1.4 or a later patch version) is deployed.
+- Deploy the newest version of the `cumulus-rds-tf` module, ensuring `enable_upgrade` is set to false.   This will *only* deploy a `v13` version of your current parameter group configuration, named `<prefix>-cluster-parameter-group-v13`.
+- Shut down all ingest and other usage of the database cluster by 3rd party applications if appropriate.
+- Once this is done, utilize the AWS RDS console to `modify` the database cluster, and update the following settings:
+  - Set `Engine Version`  to the currently available Serverless v1 Postgres v13 engine (PostgreSQL 13.12 as of this instruction set’s authoring)
+  - Ensure the min/max capacity settings match expected values and have not changed
+  - DB cluster parameter group - utilize the newly created parameter group from step #2 for the update.
+- Once you have completed the modifications, click `Continue` and verify the `Summary of modifications` has the engine version and modified parameter group.
+- **Important:** Update the `Schedule modifications` to apply the change immediately.
+
+    Once this is done, apply the updates. The database upgrade will begin, and the database will shutdown/restart repeatedly.    You can monitor progress in the database cluster’s `Logs & events` tab.
+
+    Upon completion you should expect to see output similar to:
+
+    ```text
+    Database cluster engine major version has been upgraded.
+    Updated to use DBClusterParameterGroup : <prefix>-cluster-parameter-group-v13. The DB cluster will scale to apply database parameters.
+    Scaling DB cluster from 4 capacity units to 4 capacity units for this reason: Apply database parameters.
+    ```
+
+- On update completion, validate database cluster appears to have restarted with the expected configuration, non-cumulus databases, etc.
+- Update the `enable_upgrade` `rds-cluster-tf` module variable to `true`, and run `terraform init` and `terraform apply` to ensure the postgres v11 compatible parameter group is cleaned up. This should be the only change so double-check the changeset or run `terraform plan` to be sure.
+- Resume use of the database cluster.
diff --git a/example/lambdas/ftpPopulateTestLambda/webpack.config.js b/example/lambdas/ftpPopulateTestLambda/webpack.config.js
@@ -17,6 +17,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/example/lambdas/lzardsClientTest/webpack.config.js b/example/lambdas/lzardsClientTest/webpack.config.js
@@ -12,6 +12,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist', 'webpack')

diff --git a/example/rds-cluster-tf/main.tf b/example/rds-cluster-tf/main.tf
@@ -13,16 +13,19 @@ provider "aws" {
 }
 
 module "rds_cluster" {
-  source              = "../../tf-modules/cumulus-rds-tf"
-  prefix              = var.prefix
-  db_admin_username   = var.db_admin_username
-  db_admin_password   = var.db_admin_password
-  region              = var.region
-  vpc_id              = var.vpc_id != null ? var.vpc_id : data.aws_vpc.application_vpc[0].id
-  subnets             = var.subnets != null ? var.subnets : data.aws_subnet_ids.subnet_ids[0].ids
-  engine_version      = var.engine_version
-  deletion_protection = true
-  cluster_identifier  = var.cluster_identifier
-  tags                = var.tags
-  snapshot_identifier = var.snapshot_identifier
+  source                     = "../../tf-modules/cumulus-rds-tf"
+  prefix                     = var.prefix
+  db_admin_username          = var.db_admin_username
+  db_admin_password          = var.db_admin_password
+  region                     = var.region
+  vpc_id                     = var.vpc_id != null ? var.vpc_id : data.aws_vpc.application_vpc[0].id
+  subnets                    = var.subnets != null ? var.subnets : data.aws_subnets.subnet_ids[0].ids
+  engine_version             = var.engine_version
+  deletion_protection        = true
+  cluster_identifier         = var.cluster_identifier
+  tags                       = var.tags
+  snapshot_identifier        = var.snapshot_identifier
+  enable_upgrade             = var.enable_upgrade
+  parameter_group_family     = var.parameter_group_family
+  parameter_group_family_v13 = var.parameter_group_family_v13
 }
diff --git a/example/rds-cluster-tf/variables.tf b/example/rds-cluster-tf/variables.tf
@@ -78,3 +78,20 @@ variable "subnets_tag_name" {
   default = "Private application *"
 }
 
+variable "enable_upgrade" {
+  description = "Flag to enable use of updated parameter group"
+  type = bool
+  default = false
+}
+
+variable "parameter_group_family" {
+  description = "Database family to use for creating database parameter group"
+  type = string
+  default = "aurora-postgresql11"
+}
+
+variable "parameter_group_family_v13" {
+  description = "Database family to use for creating database parameter group under postgres 13 upgrade conditions"
+  type = string
+  default = "aurora-postgresql13"
+}
diff --git a/lambdas/db-migration/webpack.config.js b/lambdas/db-migration/webpack.config.js
@@ -33,6 +33,7 @@ module.exports = {
   mode: 'development',
   entry: './dist/lambda/index.js',
   output: {
+    chunkFormat: false,
     library: {
       type: 'commonjs2'
     },

diff --git a/lambdas/db-provision-user-database/webpack.config.js b/lambdas/db-provision-user-database/webpack.config.js
@@ -25,6 +25,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/lambda/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist', 'webpack')

diff --git a/lambdas/sqs-message-remover/webpack.config.js b/lambdas/sqs-message-remover/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/packages/api/webpack.config.js b/packages/api/webpack.config.js
@@ -71,6 +71,7 @@ module.exports = {
     }),
   ],
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: '[name]/index.js',
     devtoolModuleFilenameTemplate: (info) => {

diff --git a/packages/integration-tests/webpack.config.js b/packages/integration-tests/webpack.config.js
@@ -21,6 +21,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/packages/s3-credentials-endpoint/webpack.config.js b/packages/s3-credentials-endpoint/webpack.config.js
@@ -23,6 +23,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     devtoolModuleFilenameTemplate: (info) => {

diff --git a/packages/tea-map-cache/webpack.config.js b/packages/tea-map-cache/webpack.config.js
@@ -4,6 +4,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist'),
     filename: 'index.js'

diff --git a/tasks/add-missing-file-checksums/webpack.config.js b/tasks/add-missing-file-checksums/webpack.config.js
@@ -4,6 +4,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist', 'webpack'),
     filename: 'index.js'

diff --git a/tasks/discover-granules/webpack.config.js b/tasks/discover-granules/webpack.config.js
@@ -17,6 +17,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/discover-pdrs/webpack.config.js b/tasks/discover-pdrs/webpack.config.js
@@ -17,6 +17,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/files-to-granules/webpack.config.js b/tasks/files-to-granules/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/hello-world/webpack.config.js b/tasks/hello-world/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/hyrax-metadata-updates/webpack.config.js b/tasks/hyrax-metadata-updates/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/lzards-backup/webpack.config.js b/tasks/lzards-backup/webpack.config.js
@@ -4,6 +4,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist', 'webpack'),
     filename: 'index.js'

diff --git a/tasks/move-granules/webpack.config.js b/tasks/move-granules/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/orca-copy-to-archive-adapter/webpack.config.js b/tasks/orca-copy-to-archive-adapter/webpack.config.js
@@ -4,6 +4,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist', 'webpack'),
     filename: 'index.js'

diff --git a/tasks/orca-recovery-adapter/webpack.config.js b/tasks/orca-recovery-adapter/webpack.config.js
@@ -4,6 +4,7 @@ module.exports = {
   mode: 'production',
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist', 'webpack'),
     filename: 'index.js'

diff --git a/tasks/parse-pdr/webpack.config.js b/tasks/parse-pdr/webpack.config.js
@@ -17,6 +17,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/pdr-status-check/webpack.config.js b/tasks/pdr-status-check/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/post-to-cmr/webpack.config.js b/tasks/post-to-cmr/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/queue-granules/webpack.config.js b/tasks/queue-granules/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/queue-pdrs/webpack.config.js b/tasks/queue-pdrs/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/queue-workflow/webpack.config.js b/tasks/queue-workflow/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/send-pan/webpack.config.js b/tasks/send-pan/webpack.config.js
@@ -26,6 +26,7 @@ module.exports = {
   ],
   entry: './dist/src/index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     path: path.resolve(__dirname, 'dist', 'webpack'),
     filename: 'index.js'

diff --git a/tasks/sf-sqs-report/webpack.config.js b/tasks/sf-sqs-report/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/sync-granule/webpack.config.js b/tasks/sync-granule/webpack.config.js
@@ -17,6 +17,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/test-processing/webpack.config.js b/tasks/test-processing/webpack.config.js
@@ -21,6 +21,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/update-cmr-access-constraints/webpack.config.js b/tasks/update-cmr-access-constraints/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),

diff --git a/tasks/update-granules-cmr-metadata-file-links/webpack.config.js b/tasks/update-granules-cmr-metadata-file-links/webpack.config.js
@@ -6,6 +6,7 @@ module.exports = {
   mode: process.env.PRODUCTION ? 'production' : 'development',
   entry: './index.js',
   output: {
+    chunkFormat: false,
     libraryTarget: 'commonjs2',
     filename: 'index.js',
     path: path.resolve(__dirname, 'dist'),