Serverless Data Lake Framework 2.7.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• fix datalakeLibrary case in workshop files by @cnfait in #403
• update version to 2.7.0-alpha.0 by @cnfait in #404
• redesign sdlf-team and sdlf-dataset by @cnfait in #405
• add glue database for raw bucket by @gwenika in #406
• add glue database for analytics bucket by @cnfait in #407
• add glue crawler for raw bucket by @cnfait in #408
• add glue crawler for analytics bucket by @cnfait in #409
• sdlf-dataset can be deployed without a team by @cnfait in #410
• workshop examples: add stage- prefix to stages by @cnfait in #411
• remove replicate lambda by @cnfait in #412
• [sdlf-foundations][cdk] remove legacy resources by @cnfait in #413
• [sdlf-dataset][cdk] add peh, manifests tables by @cnfait in #414
• cdk: sdlf-dataset/sdlf-team design changes by @cnfait in #415
• [sdlf-team][sdlf-dataset][cdk] fix imports by @cnfait in #416
• [cdk] Enable key rotation for all KMS keys by @cnfait in #417
• [sdlf-dataset] deployment fixes by @cnfait in #418
• [sdlf-foundations][cdk] remove CentralBucket reference by @cnfait in #419
• [sdlf-foundations][cdk] CfnDataLakeSettings by @cnfait in #420
• [sdlf-foundations][cdk] data_bucket method to handle raw/stage/analytics buckets creation by @cnfait in #421
• rework sdlf-foundations (cdk) deployment with seedfarmer by @cnfait in #422
• rework sdlf-dataset (cdk) deployment with seedfarmer by @cnfait in #423
• [sdlf-dataset][cdk] data_catalog method to handle raw/stage/analytics catalogs creation by @cnfait in #424
• [sdlf-foundations][cdk] avoid using a hardcoded stack name by @cnfait in #425
• [sdlf-dataset][cdk] avoid using a hardcoded stack name by @cnfait in #426
• [sdlf-dataset][cdk] update dynamo table names by @cnfait in #427
• [cdk] remove pEnvironment parameter by @cnfait in #428
• [sdlf-dataset][cdk] remove pPipelineDetails parameter by @cnfait in #429
• [sdlf-foundations][cdk] replace octagon with sdlf in dynamo table name by @cnfait in #430
• [cdk] ssm parameters naming change by @cnfait in #431
• [sdlf-dataset][cdk] add sdlf:dataset lf-tag association for raw and analytics databases by @cnfait in #432
• [sdlf-dataset][cdk] add EMR Security Configuration SSM parameter by @cnfait in #433
• [sdlf-dataset][cdk] ruff linting by @cnfait in #434
• [sdlf-foundations][cdk] dynamo table: let cloudformation do the naming by @cnfait in #435
• [sdlf-foundations][cdk] ruff linting by @cnfait in #436
• [sdlf-team][cdk] ruff linting by @cnfait in #437
• [sdlf-pipeline][sdlf-stage-*][cdk] remove sdlf-team dependency by @cnfait in #438
• Ruff linting for all CDK constructs by @cnfait in #439
• [sdlf-foundations][cdk] ruff formatting by @cnfait in #440
• [sdlf-foundations][cdk] metadata for seedfarmer by @cnfait in #441
• [sdlf-dataset][cdk] metadata for seedfarmer by @cnfait in #442
• add seedfarmer.gitmodules to gitignore by @cnfait in #443
• [sdlf-foundations][cdk] ability to provide parameter values without --parameters by @cnfait in #444
• [sdlf-dataset][cdk] ability to provide parameter values without CfnParameter by @cnfait in #445
• [sdlf-pipeline][cdk] remove pEnv parameter by @cnfait in #446
• [sdlf-dataset][cdk] org and domain parameters by @cnfait in #447
• [cdk] rework sdlf-pipeline deployment with seedfarmer by @cnfait in #448
• [sdlf-datalakeLibrary] remove legacy octagon support by @cnfait in #449
• [sdlf-pipeline][sdlf-stage-lambda][cdk] rewrite by @cnfait in #450
• [sdlf-foundations/team/dataset/stage-lambda] cloudformation version without legacy resources by @tomaszwrzonski in #451
• [sdlf-stage-lambda] remove pEnvironment by @cnfait in #452
• [sdlf-foundations] use new ssm parameter format by @cnfait in #453
• [sdlf-foundations] remove object metadata dynamo table by @cnfait in #454
• [sdlf-team] remove unused parameters by @cnfait in #455
• [sdlf-dataset] use new ssm parameter format by @cnfait in #456
• [sdlf-stage-glue] cfn without legacy by @cnfait in #457
• [sdlf-team][cdk] remove unused parameters by @cnfait in #458
• [sdlf-stage-lambda/glue] new ssm parameter format, remove subscription filters by @cnfait in #459
• [sdlf-stage-lambda/glue] remove unused ELK parameters by @cnfait in #460
• [sdlf-foundations] support for multiple instances by @cnfait in #461
• [sdlf-foundations] support for 'anycompany' org name by @cnfait in #462
• [sdlf-foundations] add Deployment Instance output, rename pOrg/pDomain SSM by @cnfait in #463
• [sdlf-dataset] support for multiple instances by @cnfait in #464
• [sdlf-dataset] remove unused parameter by @cnfait in #465
• [sdlf-pipeline] cfn without legacy by @cnfait in #466
• [workshop build] sdlf-utils manifest files by @cnfait in #467
• [sdlf-stage-lambda/glue] update input and output ssm parameters by @cnfait in #468
• [sdlf-datalakeLibrary] support for multiple instances by @cnfait in #469
• [sdlf-stage-lambda] pass instance parameters to datalakeLibrary by @cnfait in #470
• [sdlf-stage-lambda] lambda permission on bucket key by @cnfait in #471
• [sdlf-stage-glue] pass instance parameters to datalakeLibrary by @cnfait in #472
• [sdlf-utils] remove legacy team from legislators example by @cnfait in #473
• [sdlf-utils] reuse artifacts_bucket in legislators deploy script by @cnfait in #474
• [sdlf-team] update input and output ssm parameters by @cnfait in #475
• [sdlf-utils] use aws cfn deploy instead of create/update-stack by @cnfait in #476
• [sdlf-utils] remove single bucket support in legislators script by @cnfait in #477
• [sdlf-utils] remove unused STAGE_BUCKET from legislators script by @cnfait in #478
• ruff formatting/linting by @cnfait in #479
• [sdlf-stage-glue] Glue Number of Workers configurable through cfn parameter by @cnfait in #480
• [sdlf-stage-glue] Glue Worker Type configurable through cfn parameter by @cnfait in #481
• [sdlf-stage-glue] Glue Crawler configurable through cfn parameter by @cnfait in #482
• [sdlf-stage-lambda/glue] remove pOrg and pDomain requirements, not used for anything by @cnfait in #483
• [sdlf-stage-glue] Glue job name configurable through cfn parameter by @cnfait in #484
• [sdlf-stage-glue] handling of dataset parameters by @cnfait in #485
• [sdlf-pipeline/stage-glue] handling of dataset parameters by @cnfait in #486
• [sdlf-stage-lambda] handling of dataset parameters by @cnfait in #487
• remove legacy ObjectMetadata table references by @cnfait in #488
• [sdlf-stage-lambda] handling of storage parameters by @cnfait in #489
• [sdlf-dataset] handling of storage parameters by @cnfait in #490
• [sdlf-stage-glue] handling of storage parameters by @cnfait in #491
• [sdlf-team] handling of storage parameters by @cnfait in #492
• [sdlf-stage-glue] max items per batch configurable through cfn parameter by @cnfait in #493
• [sdlf-stage-glue] glue arguments configurable through cfn parameter by @cnfait in #494
• [sdlf-utils] handle non-existing vpc support ssm parameter in legislators example by @cnfait in #495
• [sdlf-*] restore legacy seedfarmer deployspec by @cnfait in #496

Full Changelog: 2.6.0...2.7.0