Serverless Data Lake Framework 1.5.2

What's Changed

• enable versioning on ELK stack bucket by @cnfait in #139

Full Changelog: 1.5.1...1.5.2

Serverless Data Lake Framework 1.5.1

Bug Fix

• Create a role for Lake Formation data access by @cnfait in #138

Full Changelog: 1.5.0...1.5.1

Thanks

We thank @Druizm128 for raising the issue!

Serverless Data Lake Framework 1.5.0

Features & Enhancements

• ELK Update by @cnfait in #136
• rework sdlf-cicd rCodeBuildRole IAM role to avoid using wildcards by @cnfait in #130
• avoid wildcards in sdlf-lakeformation-admin role permissions by @cnfait in #132
• avoid wildcards in data quality lambda permissions by @cnfait in #131
• disable cfn_nag W11 on CodeCommit roles by @cnfait in #133
• update awswrangler (aws sdk for pandas) to the latest 2.x version by @ntlohi in #134

Full Changelog: 1.4.0...1.5.0

Thanks

We thank all the contributors/users for their work on this release, in particular @ntlohi.

Serverless Data Lake Framework 1.4.0

Noteworthy

• AWS Partition Support by @cnfait in #128
  • SDLF can now be deployed on GovCloud (us-gov-west-1)

Features & Enhancements

• update codebuild image from standard:4.0 to amazonlinux2-x86_64-standard:4.0 by @cnfait in #113
• validate.sh: replace flake8, isort with ruff by @cnfait in #126
• Support for specifying glue arguments in dynamodb dataset table by @cnfait in #127
• add emr tagging permissions by @cnfait in #129

Full Changelog: 1.3.1...1.4.0

Thanks

We thank all the contributors/users for their work on this release.

Serverless Data Lake Framework 1.3.1

Bug Fixes

• fix pipeline stage dynamodb entry creation by @cnfait in 675517a
• use deequ 1.1.0 instead of 1.2.2 as it breaks glue jobs by @cnfait and @piers-walter-ibm in 5065cd5
• deploy failed when you deploy a new environment (missing sns permission) by @YuliemAlavez in #118

Minor Changes

• gitlab support: readme file by @cnfait in b456258
• remove executable bit from json files by @cnfait in in #112
• fix minor typo by @cnfait in #114

Features & Enhancements

• add cfn-lint to validate.sh by @cnfait in #115

Full Changelog: 1.3.0...1.3.1

Thanks

We thank all the contributors/users for their work on this release, in particular @YuliemAlavez and @piers-walter-ibm.

Serverless Data Lake Framework 1.3.0

Noteworthy

• Third-party SCM support (mirroring to CodeCommit): GitLab🔥
• As of version 1.1.0 released on December, 7th 2022, there is now a public roadmap.

Features & Enhancements

• third-party scm support: gitlab by @cnfait in #104
• enable versioning on central/raw/stage/analytics buckets by @cnfait in #106
• add security configuration to sdlf-dataset glue crawler by @cnfait in #107
• encrypt cloudtrail logs when using externally-provided bucket by @cnfait in #108

Full Changelog: 1.2.0...1.3.0

Thanks

We thank all the contributors/users for their work on this release.

Serverless Data Lake Framework 1.2.0

Noteworthy

• As of version 1.1.0 released on December, 7th 2022, there is now a public roadmap.
• As of version 1.1.0 released on December, 7th 2022, the main branch of the repository has been renamed to main from master. This is to be in line with what other projects the team is working on are using. master is still available with the same content as main to avoid breaking existing workflows. Currently only master is supported by SDLF CICD infrastructure however.
• As of version 1.1.0 released on December, 7th 2022, Semantic Versioning is now used for SDLF releases. This is to be in line with other projects from the same team.

Bug Fixes

• correct and clean manifests and cloudfront examples by @mariandumitrascu-p in #71
• fix bitbucket team pipeline when checking repositories by @cnfait in #103 - Thanks @YuliemAlavez!

Features & Enhancements

• Python 3.9 as default for Lambda functions, Lambda layers and CodeBuild runtimes by @cnfait in #93
• Align GlueVersion to 2.0 for all Glue jobs by @cnfait in #94
• Update Deequ from 1.0.X to Deequ 1.2.2-spark2.4 by @cnfait in #95
• Update ElasticSearch domain from 6.3 to 6.8 by @cnfait in #96
• Add simple shell script and configuration files to help improve code quality by @cnfait in #97
• isort by @cnfait in #98
• black by @cnfait in #99
• flake8 by @cnfait in #100
• shellcheck by @cnfait in #101

Full Changelog: 1.1.0...1.2.0

Thanks

We thank all the contributors/users for their work on this release.

Serverless Data Lake Framework 1.1.0

Noteworthy

• This release is just a snapshot of the repository as of December, 7th 2022. There is no new feature or change if you already pulled the code from the main branch.
• There is now a public roadmap.
• The main branch of the repository has been renamed to main from master. This is to be in line with what other projects the team is working on are using. master is still available with the same content as main to avoid breaking existing workflows.
• Semantic Versioning is now used for SDLF releases. This is to be in line with other projects from the same team.

Features & Enhancements

• Added bucket policies to enforce in transit encryption for s3 buckets #14
• Update catalog lambda to handle S3 multipart upload events #19
• Update catalog lambda to support DeleteMarkerCreated events #24
• 3rd party SCM providers - Azure DevOps integration #22
• Bumping Wrangler to 2.3.0 and removing ListBucket condition
• 3rd party SCM providers - Bitbucket integration #26
• Enable python 3.8 runtime for non-default lambda layers #29
• Add alias option for target e-mail #32
• Enable Manifest Based Processing in SDLF #30
• Adding Glue Jobs Deployer utility #34
• Feature to add pre-existing whl files without having to build them #39
• Adding deploy mode for datasets #40
• Enable NodeToNodeEncryptionOptions (CFN_Nag W85) #43
• Add update stack logic for cross-account team role stack #44
• Adding Data Lake testing #45
• Enable tracing for step functions #49
• Lambda cloudwatch log encryption retention #46
• Add template protection function #48
• Update key and bucket retention policies #50
• Adding PutLifecycleConfiguration permission
• Adding in a CloudFormation template that sets up automated testing for CodeCommit Pull Requests #47
• Datalake Workload Management #52
• Point-in-time recovery (PITR) enabled for DynamoDB tables #53
• Modifying user agent
• Adding few more examples and public references #58
• Sqoop ingestion extension #57
• Reducing size policy #62
• Removing slf4j logger calls
• EMR security configuration #59
• Python runtime updated #67

Bug Fixes

• Adding missing sdlf-utils and reinstating PubRef
• Correct typo of Glue Job's name #33
• Deleting additional Images, fixing README and parameters-dev errors #42
• Fixing Topic Modelling Example
• Sqoop ingestion minor fixes #66
• Fix unsupported resource arn format on rXXBucketLakeFormationS3Registration resources #77
• Fix S3 buckets ARN - Lakeformation integration #75

Documentation

• Adjusting Contributing file to latest template
• Adjusting workshop URLs to support i18n
• Better documentation for new service connection strategy #25

Thanks

We thank all the contributors/users for their work on this release.

Full Changelog: v1.0.4.0...1.1.0