Serverless Data Lake Framework 2.7.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• fix datalakeLibrary case in workshop files by @cnfait in #403
• update version to 2.7.0-alpha.0 by @cnfait in #404
• redesign sdlf-team and sdlf-dataset by @cnfait in #405
• add glue database for raw bucket by @gwenika in #406
• add glue database for analytics bucket by @cnfait in #407
• add glue crawler for raw bucket by @cnfait in #408
• add glue crawler for analytics bucket by @cnfait in #409
• sdlf-dataset can be deployed without a team by @cnfait in #410
• workshop examples: add stage- prefix to stages by @cnfait in #411
• remove replicate lambda by @cnfait in #412
• [sdlf-foundations][cdk] remove legacy resources by @cnfait in #413
• [sdlf-dataset][cdk] add peh, manifests tables by @cnfait in #414
• cdk: sdlf-dataset/sdlf-team design changes by @cnfait in #415
• [sdlf-team][sdlf-dataset][cdk] fix imports by @cnfait in #416
• [cdk] Enable key rotation for all KMS keys by @cnfait in #417
• [sdlf-dataset] deployment fixes by @cnfait in #418
• [sdlf-foundations][cdk] remove CentralBucket reference by @cnfait in #419
• [sdlf-foundations][cdk] CfnDataLakeSettings by @cnfait in #420
• [sdlf-foundations][cdk] data_bucket method to handle raw/stage/analytics buckets creation by @cnfait in #421
• rework sdlf-foundations (cdk) deployment with seedfarmer by @cnfait in #422
• rework sdlf-dataset (cdk) deployment with seedfarmer by @cnfait in #423
• [sdlf-dataset][cdk] data_catalog method to handle raw/stage/analytics catalogs creation by @cnfait in #424
• [sdlf-foundations][cdk] avoid using a hardcoded stack name by @cnfait in #425
• [sdlf-dataset][cdk] avoid using a hardcoded stack name by @cnfait in #426
• [sdlf-dataset][cdk] update dynamo table names by @cnfait in #427
• [cdk] remove pEnvironment parameter by @cnfait in #428
• [sdlf-dataset][cdk] remove pPipelineDetails parameter by @cnfait in #429
• [sdlf-foundations][cdk] replace octagon with sdlf in dynamo table name by @cnfait in #430
• [cdk] ssm parameters naming change by @cnfait in #431
• [sdlf-dataset][cdk] add sdlf:dataset lf-tag association for raw and analytics databases by @cnfait in #432
• [sdlf-dataset][cdk] add EMR Security Configuration SSM parameter by @cnfait in #433
• [sdlf-dataset][cdk] ruff linting by @cnfait in #434
• [sdlf-foundations][cdk] dynamo table: let cloudformation do the naming by @cnfait in #435
• [sdlf-foundations][cdk] ruff linting by @cnfait in #436
• [sdlf-team][cdk] ruff linting by @cnfait in #437
• [sdlf-pipeline][sdlf-stage-*][cdk] remove sdlf-team dependency by @cnfait in #438
• Ruff linting for all CDK constructs by @cnfait in #439
• [sdlf-foundations][cdk] ruff formatting by @cnfait in #440
• [sdlf-foundations][cdk] metadata for seedfarmer by @cnfait in #441
• [sdlf-dataset][cdk] metadata for seedfarmer by @cnfait in #442
• add seedfarmer.gitmodules to gitignore by @cnfait in #443
• [sdlf-foundations][cdk] ability to provide parameter values without --parameters by @cnfait in #444
• [sdlf-dataset][cdk] ability to provide parameter values without CfnParameter by @cnfait in #445
• [sdlf-pipeline][cdk] remove pEnv parameter by @cnfait in #446
• [sdlf-dataset][cdk] org and domain parameters by @cnfait in #447
• [cdk] rework sdlf-pipeline deployment with seedfarmer by @cnfait in #448
• [sdlf-datalakeLibrary] remove legacy octagon support by @cnfait in #449
• [sdlf-pipeline][sdlf-stage-lambda][cdk] rewrite by @cnfait in #450
• [sdlf-foundations/team/dataset/stage-lambda] cloudformation version without legacy resources by @tomaszwrzonski in #451
• [sdlf-stage-lambda] remove pEnvironment by @cnfait in #452
• [sdlf-foundations] use new ssm parameter format by @cnfait in #453
• [sdlf-foundations] remove object metadata dynamo table by @cnfait in #454
• [sdlf-team] remove unused parameters by @cnfait in #455
• [sdlf-dataset] use new ssm parameter format by @cnfait in #456
• [sdlf-stage-glue] cfn without legacy by @cnfait in #457
• [sdlf-team][cdk] remove unused parameters by @cnfait in #458
• [sdlf-stage-lambda/glue] new ssm parameter format, remove subscription filters by @cnfait in #459
• [sdlf-stage-lambda/glue] remove unused ELK parameters by @cnfait in #460
• [sdlf-foundations] support for multiple instances by @cnfait in #461
• [sdlf-foundations] support for 'anycompany' org name by @cnfait in #462
• [sdlf-foundations] add Deployment Instance output, rename pOrg/pDomain SSM by @cnfait in #463
• [sdlf-dataset] support for multiple instances by @cnfait in #464
• [sdlf-dataset] remove unused parameter by @cnfait in #465
• [sdlf-pipeline] cfn without legacy by @cnfait in #466
• [workshop build] sdlf-utils manifest files by @cnfait in #467
• [sdlf-stage-lambda/glue] update input and output ssm parameters by @cnfait in #468
• [sdlf-datalakeLibrary] support for multiple instances by @cnfait in #469
• [sdlf-stage-lambda] pass instance parameters to datalakeLibrary by @cnfait in https://github.com/awslabs/aws-serverless-data-lake-framewor...

Serverless Data Lake Framework 2.7.0-alpha.1

Work is ongoing on version 2.7.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.7.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-dataset][cdk] data_catalog method to handle raw/stage/analytics catalogs creation by @cnfait in #424
• [sdlf-foundations][cdk] avoid using a hardcoded stack name by @cnfait in #425
• [sdlf-dataset][cdk] avoid using a hardcoded stack name by @cnfait in #426
• [sdlf-dataset][cdk] update dynamo table names by @cnfait in #427
• [cdk] remove pEnvironment parameter by @cnfait in #428
• [sdlf-dataset][cdk] remove pPipelineDetails parameter by @cnfait in #429
• [sdlf-foundations][cdk] replace octagon with sdlf in dynamo table name by @cnfait in #430
• [cdk] ssm parameters naming change by @cnfait in #431
• [sdlf-dataset][cdk] add sdlf:dataset lf-tag association for raw and analytics databases by @cnfait in #432
• [sdlf-dataset][cdk] add EMR Security Configuration SSM parameter by @cnfait in #433
• [sdlf-dataset][cdk] ruff linting by @cnfait in #434
• [sdlf-foundations][cdk] dynamo table: let cloudformation do the naming by @cnfait in #435
• [sdlf-foundations][cdk] ruff linting by @cnfait in #436
• [sdlf-team][cdk] ruff linting by @cnfait in #437
• [sdlf-pipeline][sdlf-stage-*][cdk] remove sdlf-team dependency by @cnfait in #438
• Ruff linting for all CDK constructs by @cnfait in #439
• [sdlf-foundations][cdk] ruff formatting by @cnfait in #440
• [sdlf-foundations][cdk] metadata for seedfarmer by @cnfait in #441
• [sdlf-dataset][cdk] metadata for seedfarmer by @cnfait in #442
• add seedfarmer.gitmodules to gitignore by @cnfait in #443
• [sdlf-foundations][cdk] ability to provide parameter values without --parameters by @cnfait in #444
• [sdlf-dataset][cdk] ability to provide parameter values without CfnParameter by @cnfait in #445

Full Changelog: 2.7.0-alpha.0...2.7.0-alpha.1

Serverless Data Lake Framework 2.7.0-alpha.0

Work is ongoing on version 2.7.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.7.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• fix datalakeLibrary case in workshop files by @cnfait in #403
• update version to 2.7.0-alpha.0 by @cnfait in #404
• redesign sdlf-team and sdlf-dataset by @cnfait in #405
• add glue database for raw bucket by @gwenika in #406
• add glue database for analytics bucket by @cnfait in #407
• add glue crawler for raw bucket by @cnfait in #408
• add glue crawler for analytics bucket by @cnfait in #409
• sdlf-dataset can be deployed without a team by @cnfait in #410
• workshop examples: add stage- prefix to stages by @cnfait in #411
• remove replicate lambda by @cnfait in #412
• [sdlf-foundations][cdk] remove legacy resources by @cnfait in #413
• [sdlf-dataset][cdk] add peh, manifests tables by @cnfait in #414
• cdk: sdlf-dataset/sdlf-team design changes by @cnfait in #415
• [sdlf-team][sdlf-dataset][cdk] fix imports by @cnfait in #416
• [cdk] Enable key rotation for all KMS keys by @cnfait in #417
• [sdlf-dataset] deployment fixes by @cnfait in #418
• [sdlf-foundations][cdk] remove CentralBucket reference by @cnfait in #419
• [sdlf-foundations][cdk] CfnDataLakeSettings by @cnfait in #420
• [sdlf-foundations][cdk] data_bucket method to handle raw/stage/analytics buckets creation by @cnfait in #421
• rework sdlf-foundations (cdk) deployment with seedfarmer by @cnfait in #422
• rework sdlf-dataset (cdk) deployment with seedfarmer by @cnfait in #423

Full Changelog: 2.6.0...2.7.0-alpha.0

Serverless Data Lake Framework 2.6.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• Mark Octagon Configuration, Events, Artifacts, Metrics as legacy by @cnfait in #400
• mark Octagon-DataSchemas as legacy by @cnfait in #401
• update version to 2.6.0 by @cnfait in #402

Full Changelog: 2.5.0...2.6.0

Serverless Data Lake Framework 2.5.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• new stage: sdlf-stage-lambda by @cnfait in #338
• force use of cfn-lint 0.87.7 since release 1.3.0 break cloudformation-cli by @cnfait in #341
• new stage: sdlf-stage-ecsfargate by @cnfait in #342
• new stage: emrserverless by @cnfait in #344
• new stage: sdlf-stage-glue by @cnfait in #345
• [sdlf-stage-*] generic get_transform_details in routing lambda by @cnfait in #346
• pin cfn-lint to <1 instead of 0.87.7 by @cnfait in #349
• [sdlf-pipeline] support for pipelines with no source event by @cnfait in #350
• [sdlf-stage-*] put metadata in transform field, handle decimal metadata by @cnfait in #351
• [vpc] support shared vpc subnets by @cnfait in #348
• Preview: CDK Python by @cnfait in #353
• build stage-lambda and stage-glue by default by @cnfait in #355
• [sdlf-team][cdk] lakeformation data_location_access permissions by @cnfait in #356
• [sdlf-dataset][cdk] lake formation resources by @cnfait in #357
• avoid code duplication in domain-cicd by @cnfait in #359
• [sdlf-team][cdk] lakeformation tag and tag permissions by @cnfait in #360
• [sdlf-stage-emrserverless/ecsfargate] CDK Python by @cnfait in #361
• [sdlf-foundations/team][cdk] kms removal policy by @cnfait in #362
• [sdlf-*][cdk] set removal policy on stateful resources by @cnfait in #363
• [sdlf-*][cdk] extends Construct instead of Stack, add requirements.txt by @cnfait in #364
• SeedFarmer Support by @cnfait in #365
• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374
• poetry/pyproject.toml for sdlf cdk modules by @cnfait in #378
• [sdlf-monitoring] logs account policy for subscription filters by @cnfait in #379
• [sdlf-monitoring] log group for data firehose lambda by @cnfait in #380
• upgrade to seedfarmer 5.0.0 (from 4.0.4) by @cnfait in #381
• [sdlf-monitoring] firehose lambda: use logging, do not merge log events by @cnfait in #382
• [sdlf-stage-*] peh_id is under transform now by @cnfait in #383
• [sdlf-datalakeLibrary] seedfarmer support by @cnfait in #384
• skip poetry install if cfn module by @cnfait in #385
• workshop seedfarmer manifest examples by @cnfait in #386
• update version to 2.5.0-beta.1 by @cnfait in #387
• seedfarmer - make sure setuptools is available by @cnfait in #388
• delete cfn stack properly with seedfarmer destroy by @cnfait in #389
• avoid hardcoding SDLF deployment role name in LakeFormation admin list by @cnfait in #390
• use globbing for cfn-lint, set ignore_templates by @cnfait in #391
• update dependencies by @cnfait in #392
• update version to 2.5.0-beta.2 by @cnfait in #393
• avoid substitution issue on macosx by @cnfait in #394
• avoid buildspec yaml syntax issues by @cnfait in #395
• [sdlf-cicd] lambda:TagResource for event-source-mapping by @cnfait in #396
• [sdlf-cicd] lambda:TagResource for event-source-mapping in team role by @cnfait in #397
• (sdlf-stage-*) don't set default dataset by @cnfait in #398
• update version to 2.5.0 by @cnfait in #399

Full Changelog: 2.4.0...2.5.0

Serverless Data Lake Framework 2.5.0-beta.2

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This beta release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• seedfarmer - make sure setuptools is available by @cnfait in #388
• delete cfn stack properly with seedfarmer destroy by @cnfait in #389
• avoid hardcoding SDLF deployment role name in LakeFormation admin list by @cnfait in #390
• use globbing for cfn-lint, set ignore_templates by @cnfait in #391
• update dependencies by @cnfait in #392
• update version to 2.5.0-beta.2 by @cnfait in #393
• avoid substitution issue on macosx by @cnfait in #394
• avoid buildspec yaml syntax issues by @cnfait in #395
• [sdlf-cicd] lambda:TagResource for event-source-mapping by @cnfait in #396
• [sdlf-cicd] lambda:TagResource for event-source-mapping in team role by @cnfait in #397

Full Changelog: 2.5.0-beta.1...2.5.0-beta.2

Serverless Data Lake Framework 2.5.0-beta.1

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This beta release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-monitoring] logs account policy for subscription filters by @cnfait in #379
• [sdlf-monitoring] log group for data firehose lambda by @cnfait in #380
• upgrade to seedfarmer 5.0.0 (from 4.0.4) by @cnfait in #381
• [sdlf-monitoring] firehose lambda: use logging, do not merge log events by @cnfait in #382
• [sdlf-stage-*] peh_id is under transform now by @cnfait in #383
• [sdlf-datalakeLibrary] seedfarmer support by @cnfait in #384
• skip poetry install if cfn module by @cnfait in #385
• workshop seedfarmer manifest examples by @cnfait in #386
• update version to 2.5.0-beta.1 by @cnfait in #387

Full Changelog: 2.5.0-beta.0...2.5.0-beta.1

Serverless Data Lake Framework 2.5.0-beta.0

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• poetry/pyproject.toml for sdlf cdk modules by @cnfait in #378

Full Changelog: 2.5.0-alpha.2...2.5.0-beta.0

Serverless Data Lake Framework 2.5.0-alpha.2

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374

Full Changelog: 2.5.0-alpha.1...2.5.0-alpha.2

Serverless Data Lake Framework 2.5.0-alpha.1

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• SeedFarmer Support by @cnfait in #365

Full Changelog: 2.5.0-alpha.0...2.5.0-alpha.1