Skip to content

v18.3.1
Compare
Choose a tag to compare
@npauzenga npauzenga released this 09 Jul 19:43
· 118 commits to master since this release
v18.3.1
0393f90
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Migration Notes

CUMULUS-3433 Update to node.js v20

The following applies only to users with a custom value configured for
async_operation_image:

  • As part of the node v20 update process, a new version (52) of the Core
    async-operation container was published - cumuluss/async
    operation     The
    default value for async_operation_image has been updated in the cumulus
    module, however if you are using an internal image repository such as ECR,
    please make sure to update your deployment configuration with the newly
    provided image.

    Users making use of a custom image configuration should note the base image
    for Core async operations must support node v20.x.

CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded

Instructions for migrating old DLA (Dead Letter Archive) messages to new format:

  • YYYY-MM-DD subfolders to organize by date
  • new top level fields for simplified search and analysis
  • captured error message

To invoke the Lambda and start the DLA migration, you can use the AWS Console or CLI:

aws lambda invoke --function-name $PREFIX-migrationHelperAsyncOperation \
  --payload $(echo '{"operationType": "DLA Migration"}' | base64) $OUTFILE
  • PREFIX is your Cumulus deployment prefix.
  • OUTFILE (optional) is the filepath where the Lambda output will be saved.

The Lambda will trigger an Async Operation and return an id such as:

{"id":"41c9fbbf-a031-4dd8-91cc-8ec2d8b5e31a","description":"Migrate Dead Letter Archive Messages",
"operationType":"DLA Migration","status":"RUNNING",
"taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}

which you can then query the Async Operations API
Endpoint for the
output or status of your request. If you want to directly observe the progress
of the migration as it runs, you can view the CloudWatch logs for your async
operations (e.g. PREFIX-AsyncOperationEcsLogs).

CUMULUS-3779 async_operations Docker image version upgrade

The async-operation Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least async-operations:52.

CUMULUS-3776 cumulus-ecs-task Docker image version upgrade

The cumulus-ecs-task Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least cumulus-ecs-task:2.1.0.

Breaking Changes

  • CUMULUS-3618

    • Modified @cumulus/es-client/search.BaseSearch:
      • Removed static class method es in favor of new class for managing
        elasticsearch clients EsClient which allows for credential
        refresh/reset. Updated api/es-client code to
        utilize new pattern. Users making use of @cumulus/es-client should
        update their code to make use of the new EsClient create/initialize pattern.
      • Added helper method getEsClient to encapsulate logic to create/initialize
        a new EsClient.

  • CUMULUS-2889

    • Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs
      client from the @cumulus/aws-client package.

  • CUMULUS-2890

    • Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client
      from the @cumulus/aws-client package.

  • CUMULUS-3323

    • Updated @cumulus/db to by default set the ssl option for knex, and
      reject non-SSL connections via use of the rejectUnauthorized configuration
      flag. This causes all Cumulus database connections to require SSL (CA or
      self-signed) and reject connectivity if the database does not provide SSL.
      Users using serverless v1/cumulus-rds-tf should not be impacted by this
      change as certs are provided by default. Users using databases that do not
      provide SSL should update their database secret with the optional value
      disableSSL set to true
    • Updated cumulus-rds-tf to set rds.force_ssl to 1, forcing SSL enabled
      connections in the db_parameters configuration. Users of this module
      defining their own db_parameters should make this configuration change to allow only SSL
      connections to the RDS datastore.

  • CUMULUS-2897

    • Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client
      from the @cumulus/aws-client package.

  • CUMULUS-3779

    • Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to
      support aws-sdk v3 changes.

Added

  • CUMULUS-3614
    • tf-modules/monitoring module now deploys Glue table for querying dead-letter-archive messages.
  • CUMULUS-3616
    • Added user guide on querying dead-letter-archive messages using AWS Athena.
  • CUMULUS-3433
    • Added importGot helper method to import got as an ESM module in
      CommmonJS typescript/webpack clients.
  • CUMULUS-3606
    • Updated with additional documentation covering tunneling configuration
      using a PKCS11 provider

Changed

  • CUMULUS-3735
    • Remove unused getGranuleIdsForPayload from @cumulus/api/lib
  • CUMULUS-3746
    • cicd unit test error log changed to environment unique name
  • CUMULUS-3717
    • Update @cumulus/ingest/HttpProviderClient to use direct injection test mocks, and remove rewire from unit tests
  • CUMULUS-3720
    • add cicd unit test error logging to s3 for testing improvements
  • CUMULUS-3433
    • Updated all node.js lambda dependencies to node 20.x/20.12.2
    • Modified @cumulus/ingest unit test HTTPs server to accept localhost POST
      requests, and removed nock dependency from tests involving fs.Readstream
      and got due to a likely incompatibility with changes in node v18, got,
      fs.Readstream and nock when used in combination in units
      (sindresorhus/got#2341)
    • Updated got dependency in @cumulus/ingest to use @cumulus/common
      dynamic import helper / got > v10 in CommonJS.
    • Updated all Core lambdas to use cumulus-message-adapter-js v2.2.0
  • CUMULUS-3629
    • dla guarantees de-nested SQS message bodies, preferring outermost metadata as found.
    • dla uses execution Name as filename and ensures no ':' or '/' characters in name
  • CUMULUS-3570
    • Updated Kinesis docs to support latest AWS UI and recommend server-side encryption.
  • CUMULUS-3519
    • Updates SQS and SNS code to AWS SDK V3 Syntax
  • CUMULUS-3609
    • Adds dla-migration lambda to async-operations to be used for updating existing DLA records
    • Moved hoistCumulusMessageDetails function from write-db-dlq-records-to-s3 lambda to @cumulus/message/DeadLetterMessage
  • CUMULUS-3613
    • Updated writeDbRecordsDLQtoS3 lambda to write messages to YYYY-MM-DD subfolder of S3 dead letter archive.
  • CUMULUS-3518
    • Update existing usage of @cumulus/aws-client lambda service to use AWS SDK v3 send syntax
    • Update Discover Granules lambda default memory to 1024 MB
  • CUMULUS-3600
    • Update docs to clarify CloudFront HTTPS DIT requirements.
  • CUMULUS-2892
    • Updates aws-client's EC2 client to use AWS SDK v3.
  • CUMULUS-2896
    • Updated Secrets Manager code to AWS SDK v3.
  • CUMULUS-2901
    • Updated STS code to AWS SDK v3.
  • CUMULUS-2898
    • Update Step Functions code to AWS SDK v3
  • CUMULUS-2902
    • Removes aws-sdk from es-client package by replacing credential fetching with
      the @aws-sdk/credential-providers AWS SDK v3 package.
    • Removes aws-sdk from all cumulus packages and replaces usages with AWS SDK v3 clients.
  • CUMULUS-3456
    • Added stateMachineArn, executionArn, collectionId, providerId, granules, status, time, and error fields to Dead Letter Archive message
    • Added cumulusError field to records in sfEventSqsToDbRecordsDeadLetterQueue
  • CUMULUS-3323
    • Added disableSSL as a valid database secret key - setting this in your database credentials will
      disable SSL for all Core database connection attempts.
    • Added rejectUnauthorized as a valid database secret key - setting
      this to false in your database credentials will allow self-signed certs/certs with an unrecognized authority.
    • Updated the default parameter group for cumulus-rds-tf to set force_ssl
      to 1. This setting for the Aurora Serverless v1 database disallows non-SSL
      connections to the database, and is intended to help enforce security
      compliance rules. This update can be opted-out by supplying a non-default
      db_parameters set in the terraform configuration.
  • CUMULUS-3425
    • Update @cumulus/lzards-backup task to either respect the lzards_provider
      terraform configuration value or utilize lzardsProvider as part of the task
      workflow configuration
    • Minor refactor of @cumulus/lzards-api-client to:
      • Use proper ECMAScript import for @cumulus/launchpad-auth
      • Update incorrect docstring
  • CUMULUS-3497
    • Updated example/cumulus-tf/orca.tf to use v9.0.4
  • CUMULUS-3610
    • Updated aws-client's ES client to use AWS SDK v3.
  • CUMULUS-3617
    • Added lambdas to migrate DLA messages to YYYY-MM-DD subfolder
    • Updated @cumulus/aws-client/S3/recursivelyDeleteS3Bucket to handle bucket with more than 1000 objects.
  • CUMULUS-2891
    • Updated ECS code to aws sdk v3

Fixed

  • CUMULUS-3715
    • Update ProvisionUserDatabase lambda to correctly pass in knex/node debug
      flags to knex custom code
  • CUMULUS-3721
    • Update lambda:GetFunctionConfiguration policy statement to fix error related to resource naming
  • CUMULUS-3701
    • Updated @cumulus/api to no longer improperly pass PATCH/PUT null values to Eventbridge rules
  • CUMULUS-3618
    • Fixed @cumulus/es-client credentialing issue in instance where
      lambda/Fargate task runtime would exceed the timeout for the es-client. Added retry/credential
      refresh behavior to @cumulus/es-client/indexer.genericRecordUpdate to ensure record indexing
      does not fail in those instances.
    • Updated index-from-database lambda to utilize updated es-client to prevent
      credentialing timeout in long-running ECS jobs.
  • CUMULUS-3323
    • Minor edits to errant integration test titles (dyanmo->postgres)
  • AWS-SDK v3 Exclusion (v18.3.0 fix)*
    • Excludes aws-sdk v3 from packages to reduce overall package size. With the requirement of Node v20
      packaging the aws-sdk v3 with our code is no longer necessary and prevented some packages from being
      published to npm.
Assets 8
Loading