From a2cdacc948bf401dc059ca071d5c441381f525c9 Mon Sep 17 00:00:00 2001 From: Tess Ferrandez Date: Thu, 22 Aug 2024 10:18:24 +0200 Subject: [PATCH] Refactor CI/CD (#1064) * re-factor ci/cd * re-factor ci/cd * fix links * Update docs/CI-CD/dev-sec-ops/secrets-management/README.md Co-authored-by: Mark Jervelund * Update docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md Co-authored-by: Mark Jervelund --------- Co-authored-by: Mark Jervelund --- docs/.pages | 4 + docs/CI-CD/.pages | 6 ++ .../CICD.md => CI-CD/README.md} | 23 +++++- .../continuous-delivery.md} | 2 +- .../continuous-integration.md} | 2 +- docs/CI-CD/dev-sec-ops/.pages | 3 + docs/CI-CD/dev-sec-ops/README.md | 21 +++++ ...ure-devops-service-connection-security.md} | 21 ++--- .../dependency-and-container-scanning.md} | 0 .../evaluate-open-source-software.md} | 0 .../images/approvals-and-checks.png | Bin .../dev-sec-ops}/images/branch-control.png | Bin .../dev-sec-ops/penetration-testing.md} | 0 .../dev-sec-ops}/secrets-management/README.md | 26 ++++-- .../credential_scanning.md | 4 +- .../secrets-management}/recipes/README.md | 0 .../recipes/detect-secrets-ado.md | 8 +- .../recipes/detect-secrets.md | 0 .../secrets-management}/secrets_rotation.md | 0 .../static-code-analysis.md | 0 .../gitops/deploying-with-gitops.md} | 0 .../gitops/github-workflows.md} | 0 .../images/GitopsWorflowVsTraditionalPush.jpg | Bin .../Workflow-Designs-Dependent-Workflows.png | Bin ...Workflow-Designs-Independent-Workflows.png | Bin .../gitops/secret-management/README.md | 0 ...ure-devops-secret-management-per-branch.md | 0 .../secret-rotation-in-pods.md | 0 .../images/blue_green.png | Bin .../images/canary_release.png | Bin .../images/environments.png | Bin .../images/example_release_flow.png | Bin .../images}/repository-properties.png | Bin docs/CI-CD/recipes/.pages | 6 ++ .../recipes/cd-on-low-code-solutions.md} | 0 .../ci-pipeline-for-better-documentation.md} | 3 +- .../recipes/ci-with-jupyter-notebooks.md} | 4 +- .../runtime-variables/README.md | 0 .../examples/commit-example.yaml | 0 .../examples/pr-example.yaml | 0 .../runtime-variables/images/AzDoExample.PNG | Bin .../runtime-variables/images/CommitFalse.PNG | Bin .../runtime-variables/images/CommitTrue.PNG | Bin .../runtime-variables/images/DevAWorkflow.png | Bin .../runtime-variables/images/DevBWorkflow.png | Bin .../runtime-variables/images/PRExample.PNG | Bin .../runtime-variables/images/PRTrue.PNG | Bin .../images/workflow-diagram.png | Bin .../recipes}/inclusive-linting.md | 0 ...reusing-devcontainers-within-a-pipeline.md | 72 +++++++++++++++++ .../save-output-to-variable-group.md | 0 ...are-common-variables-naming-conventions.md | 0 .../terraform-structure-guidelines.md | 0 docs/ISE.md | 2 +- docs/{index.md => README.md} | 38 +-------- docs/continuous-delivery/.pages | 5 -- docs/continuous-delivery/recipes/README.md | 11 --- .../recipes/terraform/README.md | 5 -- docs/continuous-integration/.pages | 5 -- .../dev-sec-ops/README.md | 21 ----- .../dev-sec-ops/azure-devops/README.md | 8 -- .../dev-sec-ops/secret-management/README.md | 27 ------- .../devcontainers/README.md | 74 ------------------ ... => engineering-fundamentals-checklist.md} | 4 +- ...md => the-first-week-of-an-ise-project.md} | 21 +++-- linkcheck.json | 2 +- 66 files changed, 190 insertions(+), 238 deletions(-) create mode 100644 docs/.pages create mode 100644 docs/CI-CD/.pages rename docs/{continuous-integration/CICD.md => CI-CD/README.md} (58%) rename docs/{continuous-delivery/README.md => CI-CD/continuous-delivery.md} (99%) rename docs/{continuous-integration/README.md => CI-CD/continuous-integration.md} (99%) create mode 100644 docs/CI-CD/dev-sec-ops/.pages create mode 100644 docs/CI-CD/dev-sec-ops/README.md rename docs/{continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md => CI-CD/dev-sec-ops/azure-devops-service-connection-security.md} (83%) rename docs/{continuous-integration/dev-sec-ops/dependency-container-scanning/README.md => CI-CD/dev-sec-ops/dependency-and-container-scanning.md} (100%) rename docs/{continuous-integration/dev-sec-ops/evaluate-oss/README.md => CI-CD/dev-sec-ops/evaluate-open-source-software.md} (100%) rename docs/{continuous-integration/dev-sec-ops/azure-devops => CI-CD/dev-sec-ops}/images/approvals-and-checks.png (100%) rename docs/{continuous-integration/dev-sec-ops/azure-devops => CI-CD/dev-sec-ops}/images/branch-control.png (100%) rename docs/{continuous-integration/dev-sec-ops/penetration-testing/README.md => CI-CD/dev-sec-ops/penetration-testing.md} (100%) rename docs/{continuous-delivery => CI-CD/dev-sec-ops}/secrets-management/README.md (78%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/credential_scanning.md (90%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/recipes/README.md (100%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/recipes/detect-secrets-ado.md (82%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/recipes/detect-secrets.md (100%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/secrets_rotation.md (100%) rename docs/{continuous-integration/dev-sec-ops/secret-management => CI-CD/dev-sec-ops/secrets-management}/static-code-analysis.md (100%) rename docs/{continuous-delivery/gitops/deploying/README.md => CI-CD/gitops/deploying-with-gitops.md} (100%) rename docs/{continuous-delivery/recipes/github-workflows/README.md => CI-CD/gitops/github-workflows.md} (100%) rename docs/{continuous-delivery/gitops/deploying => CI-CD/gitops}/images/GitopsWorflowVsTraditionalPush.jpg (100%) rename docs/{continuous-delivery/recipes/github-workflows => CI-CD/gitops}/images/Workflow-Designs-Dependent-Workflows.png (100%) rename docs/{continuous-delivery/recipes/github-workflows => CI-CD/gitops}/images/Workflow-Designs-Independent-Workflows.png (100%) rename docs/{continuous-delivery => CI-CD}/gitops/secret-management/README.md (100%) rename docs/{continuous-delivery => CI-CD}/gitops/secret-management/azure-devops-secret-management-per-branch.md (100%) rename docs/{continuous-delivery => CI-CD}/gitops/secret-management/secret-rotation-in-pods.md (100%) rename docs/{continuous-delivery => CI-CD}/images/blue_green.png (100%) rename docs/{continuous-delivery => CI-CD}/images/canary_release.png (100%) rename docs/{continuous-delivery => CI-CD}/images/environments.png (100%) rename docs/{continuous-delivery => CI-CD}/images/example_release_flow.png (100%) rename docs/{continuous-integration/ci-in-data-science/working-with-notebooks/assets => CI-CD/images}/repository-properties.png (100%) create mode 100644 docs/CI-CD/recipes/.pages rename docs/{continuous-delivery/low-code-solutions/README.md => CI-CD/recipes/cd-on-low-code-solutions.md} (100%) rename docs/{continuous-integration/markdown-linting/README.md => CI-CD/recipes/ci-pipeline-for-better-documentation.md} (94%) rename docs/{continuous-integration/ci-in-data-science/working-with-notebooks/README.md => CI-CD/recipes/ci-with-jupyter-notebooks.md} (97%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/README.md (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/examples/commit-example.yaml (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/examples/pr-example.yaml (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/AzDoExample.PNG (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/CommitFalse.PNG (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/CommitTrue.PNG (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/DevAWorkflow.png (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/DevBWorkflow.png (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/PRExample.PNG (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/PRTrue.PNG (100%) rename docs/{continuous-delivery/devops-provider-recipes => CI-CD/recipes}/github-actions/runtime-variables/images/workflow-diagram.png (100%) rename docs/{continuous-integration => CI-CD/recipes}/inclusive-linting.md (100%) create mode 100644 docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md rename docs/{continuous-delivery => CI-CD}/recipes/terraform/save-output-to-variable-group.md (100%) rename docs/{continuous-delivery => CI-CD}/recipes/terraform/share-common-variables-naming-conventions.md (100%) rename docs/{continuous-delivery => CI-CD}/recipes/terraform/terraform-structure-guidelines.md (100%) rename docs/{index.md => README.md} (59%) delete mode 100644 docs/continuous-delivery/.pages delete mode 100644 docs/continuous-delivery/recipes/README.md delete mode 100644 docs/continuous-delivery/recipes/terraform/README.md delete mode 100644 docs/continuous-integration/.pages delete mode 100644 docs/continuous-integration/dev-sec-ops/README.md delete mode 100644 docs/continuous-integration/dev-sec-ops/azure-devops/README.md delete mode 100644 docs/continuous-integration/dev-sec-ops/secret-management/README.md delete mode 100644 docs/continuous-integration/devcontainers/README.md rename docs/{ENG-FUNDAMENTALS-CHECKLIST.md => engineering-fundamentals-checklist.md} (95%) rename docs/{SPRINT-STRUCTURE.md => the-first-week-of-an-ise-project.md} (92%) diff --git a/docs/.pages b/docs/.pages new file mode 100644 index 0000000000..4c2ec50e13 --- /dev/null +++ b/docs/.pages @@ -0,0 +1,4 @@ +nav: + - ... + - CI/CD: ci-cd + - ... diff --git a/docs/CI-CD/.pages b/docs/CI-CD/.pages new file mode 100644 index 0000000000..1da2a21b0a --- /dev/null +++ b/docs/CI-CD/.pages @@ -0,0 +1,6 @@ +nav: + - Continuous Delivery: continuous-delivery + - Continuous Integration: continuous-integration + - DevSecOps: dev-sec-ops + - GitOps: gitops + - ... diff --git a/docs/continuous-integration/CICD.md b/docs/CI-CD/README.md similarity index 58% rename from docs/continuous-integration/CICD.md rename to docs/CI-CD/README.md index de5da3f64a..164cfb651d 100644 --- a/docs/continuous-integration/CICD.md +++ b/docs/CI-CD/README.md @@ -1,10 +1,10 @@ -# Continuous Integration and Delivery +# Continuous Integration and Continuous Delivery -Continuous Integration is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed. +[**Continuous Integration (CI)**](continuous-integration.md) is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed. Paired with an automated testing approach, continuous integration also allows us to also test the integrated build such that we can verify that not only does the code base still build correctly, but also is still functionally correct. This is also a best practice for building robust and flexible software systems. -Continuous Delivery takes the Continuous Integration concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage. +[**Continuous Delivery (CD)**](continuous-delivery.md) takes the **Continuous Integration (CI)** concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage. The goal of all of this is to ensure that the main branch is always shippable, meaning that we could, if we needed to, take a build from the main branch of our code base and ship it on production. @@ -14,6 +14,23 @@ Our expectation is that CI/CD should be used in all the engineering projects tha For a much deeper understanding of all of these concepts, the books [Continuous Integration](https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380) and [Continuous Delivery](https://www.amazon.com/gp/product/0321601912) provide a comprehensive background. +## Why CI/CD + +- We want to have an automated build and deployment of our software +- We want automated configuration of all components +- We want to be able to quickly re-build the environment from scratch in case of disaster +- We want the latest version of the code to always be deployed to our dev/test environments +- We want a reliable release strategy, where the policies for release are well understood by all + +## The fundamentals + +- We run a quality pipeline (with linting, unit tests etc.) on each PR/update of the main branch +- All cloud resources (including secrets and permissions) are provisioned through infrastructure as code templates – ex. Terraform, Bicep (ARM), Pulumi etc. +- All release candidates are deployed to a non-production environment through an automated process (ex Azure DevOps or Github pipelines) +- Releases are deployed to the production environment through an automated process +- Release rollbacks are carried out through a repeatable process +- Our release pipeline runs automated tests, validating all release candidate artifact(s) end-to-end against a non-production environment + ## Tools ### Azure Pipelines diff --git a/docs/continuous-delivery/README.md b/docs/CI-CD/continuous-delivery.md similarity index 99% rename from docs/continuous-delivery/README.md rename to docs/CI-CD/continuous-delivery.md index ad795a6b4f..e34e2df3c8 100644 --- a/docs/continuous-delivery/README.md +++ b/docs/CI-CD/continuous-delivery.md @@ -139,7 +139,7 @@ Try to limit the number of versions of your application running parallel in prod Low code solutions have increased their participation in the applications and processes and because of that it is required that a proper conjunction of disciplines improve their development. -Here is a guide for [continuous deployment for Low Code Solutions](low-code-solutions/README.md). +Here is a guide for [continuous deployment for Low Code Solutions](recipes/cd-on-low-code-solutions.md). ## References diff --git a/docs/continuous-integration/README.md b/docs/CI-CD/continuous-integration.md similarity index 99% rename from docs/continuous-integration/README.md rename to docs/CI-CD/continuous-integration.md index bc43027ec8..7f0ebc5db8 100644 --- a/docs/continuous-integration/README.md +++ b/docs/CI-CD/continuous-integration.md @@ -45,7 +45,7 @@ An automated build should encompass the following principles: - Code across an engineering team must be formatted to agreed coding standards. Such standards keep code consistent, and most importantly easy for the team and customer(s) to read and refactor. Code styling consistency encourages collective ownership for project scrum teams and our partners. - There are several open source code style validation tools available to choose from ([code style checks](https://github.com/checkstyle/checkstyle), [StyleCop](https://en.wikipedia.org/wiki/StyleCop)). The [Code Review recipes section](../code-reviews/recipes/README.md) of the playbook has suggestions for linters and preferred styles for a number of languages. -- Your code and documentation should avoid the use of non-inclusive language wherever possible. Follow the [Inclusive Linting section](inclusive-linting.md) to ensure your project promotes an inclusive work environment for both the team and for customers. +- Your code and documentation should avoid the use of non-inclusive language wherever possible. Follow the [Inclusive Linting section](recipes/inclusive-linting.md) to ensure your project promotes an inclusive work environment for both the team and for customers. - We recommend incorporating security analysis tools within the build stage of your pipeline such as: code credential scanner, security risk detection, static analysis, etc. For Azure DevOps, you can add a security scan task to your pipeline by installing the [Microsoft Security Code Analysis Extension](https://secdevtools.azurewebsites.net/#pills-onboard). GitHub Actions supports a similar extension with the [RIPS security scan solution](https://github.com/marketplace/actions/rips-security-scan). - Code standards are maintained within a single configuration file. There should be a step in your build pipeline that asserts code in the latest commit conforms to the known style definition. diff --git a/docs/CI-CD/dev-sec-ops/.pages b/docs/CI-CD/dev-sec-ops/.pages new file mode 100644 index 0000000000..10ac82557c --- /dev/null +++ b/docs/CI-CD/dev-sec-ops/.pages @@ -0,0 +1,3 @@ +nav: + - Azure DevOps service connection security: azure-devops-service-connection-security + - ... diff --git a/docs/CI-CD/dev-sec-ops/README.md b/docs/CI-CD/dev-sec-ops/README.md new file mode 100644 index 0000000000..0879fd4792 --- /dev/null +++ b/docs/CI-CD/dev-sec-ops/README.md @@ -0,0 +1,21 @@ +# DevSecOps + +## The concept of DevSecOps + +DevSecOps or DevOps security is about introducing security earlier in the life cycle of application development (a.k.a shift-left), thus minimizing the impact of vulnerabilities and bringing security closer to development team. + +## Why + +By embracing shift-left mentality, DevSecOps encourages organizations to bridge the gap that often exists between development and security teams to the point where many of the security processes are automated and are effectively handled by the development team. + +## DevSecOps Practices + +This section covers different tools, frameworks and resources allowing introduction of DevSecOps best practices to your project at early stages of development. +Topics covered: + +1. [Credential Scanning](./secrets-management/credential_scanning.md) - automatically inspecting a project to ensure that no secrets are included in the project's source code. +1. [Secrets Rotation](./secrets-management/secrets_rotation.md) - automated process by which the secret, used by the application, is refreshed and replaced by a new secret. +1. [Static Code Analysis](./secrets-management/static-code-analysis.md) - analyze source code or compiled versions of code to help find security flaws. +1. [Penetration Testing](./penetration-testing.md) - a simulated attack against your application to check for exploitable vulnerabilities. +1. [Container Dependencies Scanning](./dependency-and-container-scanning.md) - search for vulnerabilities in container operating systems, language packages and application dependencies. +1. [Evaluation of Open Source Libraries](./evaluate-open-source-software.md) - make it harder to apply open source supply chain attacks by evaluating the libraries you use. diff --git a/docs/continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md b/docs/CI-CD/dev-sec-ops/azure-devops-service-connection-security.md similarity index 83% rename from docs/continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md rename to docs/CI-CD/dev-sec-ops/azure-devops-service-connection-security.md index df14b0d832..8e257b2ca5 100644 --- a/docs/continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md +++ b/docs/CI-CD/dev-sec-ops/azure-devops-service-connection-security.md @@ -13,10 +13,11 @@ Securing Service Connections can be achieved by using several methods. - [Project permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints#project-permissions---cross-project-sharing-of-service-connections) can be configured to ensure only certain Azure DevOps projects are able to use the Service Connection. After using the above methods, what is secured is **who** can use the Service Connections. -What still *isn't* secured however, is **what** can be done with the Service Connections. +What still *isn't* secured however, is **what** can be done with the Service Connections. + +Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users. +An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources. Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done. -Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users. -An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources. Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done. ```yaml pool: vmImage: ubuntu-latest @@ -36,12 +37,12 @@ steps: ## Pipeline Security caveat -YAML pipelines can be triggered without the need for a pull request, this introduces a security risk. +YAML pipelines can be triggered without the need for a pull request, this introduces a security risk. -In good practice, [Pull Requests](../../../code-reviews/pull-requests.md) and [Code Reviews](../../../code-reviews/README.md) should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues. -However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer. +In good practice, [Pull Requests](../../code-reviews/pull-requests.md) and Code Reviews should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues. +However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer. -The configuration of *when* a pipeline should be triggered is specified in the YAML Pipeline itself and therefore a pipeline can be configured to execute on changes in a temporary branch. In this temporary branch, any changes made to the pipeline itself will be executed without being reviewed. +The configuration of *when* a pipeline should be triggered is specified in the YAML Pipeline itself and therefore a pipeline can be configured to execute on changes in a temporary branch. In this temporary branch, any changes made to the pipeline itself will be executed without being reviewed. If the given pipeline has been granted [Pipeline-level permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints#pipeline-permissions) to use a specific Service Connection, any command can be executed using that Service Connection, without anyone reviewing the command. Since Service Connections can have a lot of permissions in the external service, executing any pipeline without review could potentially have big consequences. @@ -54,13 +55,13 @@ Configuration can be done in the Approvals and Checks menu on the Service Connec ### Branch Control -By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch. +By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch. By configuring Branch Control to only allow the main branch (and potentially release branches) you can ensure a YAML Pipeline can only use the Service Connection after any changes to that pipeline have been merged into the main branch, and therefore has passed any Pull Requests checks and Code Reviews. As an additional check, Branch Control can verify if Branch Protections (like required Pull Requests and Code Reviews) are actually configured on the allowed branches. -With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore. +With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore. -**Note: When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.* +> **Note:** When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.* ![BranchControl](images/branch-control.png) diff --git a/docs/continuous-integration/dev-sec-ops/dependency-container-scanning/README.md b/docs/CI-CD/dev-sec-ops/dependency-and-container-scanning.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/dependency-container-scanning/README.md rename to docs/CI-CD/dev-sec-ops/dependency-and-container-scanning.md diff --git a/docs/continuous-integration/dev-sec-ops/evaluate-oss/README.md b/docs/CI-CD/dev-sec-ops/evaluate-open-source-software.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/evaluate-oss/README.md rename to docs/CI-CD/dev-sec-ops/evaluate-open-source-software.md diff --git a/docs/continuous-integration/dev-sec-ops/azure-devops/images/approvals-and-checks.png b/docs/CI-CD/dev-sec-ops/images/approvals-and-checks.png similarity index 100% rename from docs/continuous-integration/dev-sec-ops/azure-devops/images/approvals-and-checks.png rename to docs/CI-CD/dev-sec-ops/images/approvals-and-checks.png diff --git a/docs/continuous-integration/dev-sec-ops/azure-devops/images/branch-control.png b/docs/CI-CD/dev-sec-ops/images/branch-control.png similarity index 100% rename from docs/continuous-integration/dev-sec-ops/azure-devops/images/branch-control.png rename to docs/CI-CD/dev-sec-ops/images/branch-control.png diff --git a/docs/continuous-integration/dev-sec-ops/penetration-testing/README.md b/docs/CI-CD/dev-sec-ops/penetration-testing.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/penetration-testing/README.md rename to docs/CI-CD/dev-sec-ops/penetration-testing.md diff --git a/docs/continuous-delivery/secrets-management/README.md b/docs/CI-CD/dev-sec-ops/secrets-management/README.md similarity index 78% rename from docs/continuous-delivery/secrets-management/README.md rename to docs/CI-CD/dev-sec-ops/secrets-management/README.md index 1241b46cbc..81a643499b 100644 --- a/docs/continuous-delivery/secrets-management/README.md +++ b/docs/CI-CD/dev-sec-ops/secrets-management/README.md @@ -1,12 +1,24 @@ -# Secrets Management +# Secrets management -Secrets Management refers to the way in which we protect configuration settings and other sensitive data which, if -made public, would allow unauthorized access to resources. Examples of secrets are usernames, passwords, api keys, SAS -tokens etc. +Secret management refers to the tools and practices used to manage digital authentication credentials (like API keys, tokens, passwords, and certificates). These secrets are used to protect access to sensitive data and services, making their management critical for security. We should assume any repo we work on may go public at any time and protect our secrets, even if the repo is initially private. +## Importance of Secrets Management + +In modern software development, applications often need to interact with other software components, APIs, and services. These interactions often require authentication, which is typically handled using secrets. If these secrets are not managed properly, they can be exposed, leading to potential security breaches. + +## Best Practices for Secrets Management + +1. **Centralized Secret Storage:** Store all secrets in a centralized, encrypted location. This reduces the risk of secrets being lost or exposed. +1. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets. +1. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised. +1. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities. +1. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error. + +Remember, the goal of secret management is to protect sensitive information from unauthorized access and potential security threats. + ## General Approach The general approach is to keep secrets in separate configuration files that are not checked in @@ -19,7 +31,7 @@ the Azure CLI to do the same is a useful time-saving utility. See [az webapp con It's best practice to maintain separate secrets configurations for each environment that you run. e.g. dev, test, prod, local etc -The [secrets-per-branch recipe](../gitops/secret-management/azure-devops-secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment. +The [secrets-per-branch recipe](../../gitops/secret-management/azure-devops-secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment. > Note: even if the secret was only pushed to a feature branch and never merged, it's still a part of the git history. Follow [these instructions](https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository) to remove any sensitive data and/or regenerate any keys and other sensitive information added to the repo. If a key or secret made it into the code base, rotate the key/secret so that it's no longer active @@ -53,7 +65,7 @@ These techniques make the loading of secrets transparent to the developer. For .NET SDK (version 2.0 or higher) we have `dotnet secrets`, a tool provided by the .NET SDK that allows you to manage and protect sensitive information, such as API keys, connection strings, and other secrets, during development. The secrets are stored securely on your machine and can be accessed by your .NET applications. ```shell -# Initialize dotnet secret +# Initialize dotnet secret dotnet user-secrets init # Adding secret @@ -176,4 +188,4 @@ The following steps lay out a clear pathway to creating new secrets and then uti ### Validation -Automated credential scanning can be performed on the code regardless of the programming language. Read more about it [here](../../continuous-integration/dev-sec-ops/secret-management/credential_scanning.md) +Automated [credential scanning](credential_scanning.md) can be performed on the code regardless of the programming language. diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/credential_scanning.md b/docs/CI-CD/dev-sec-ops/secrets-management/credential_scanning.md similarity index 90% rename from docs/continuous-integration/dev-sec-ops/secret-management/credential_scanning.md rename to docs/CI-CD/dev-sec-ops/secrets-management/credential_scanning.md index bdc3171417..b6b5dda85b 100644 --- a/docs/continuous-integration/dev-sec-ops/secret-management/credential_scanning.md +++ b/docs/CI-CD/dev-sec-ops/secrets-management/credential_scanning.md @@ -4,9 +4,9 @@ Credential scanning is the practice of automatically inspecting a project to ens ## Why Credential scanning -Including secrets in a project's source code is a significant risk, as it might make those secrets available to unwanted parties. Even if it seems that the source code is accessible to the same people who are privy to the secrets, this situation is likely to change as the project grows. Spreading secrets in different places makes them harder to manage, access control, and revoke efficiently. Secrets that are committed to source control are also harder to discard of, since they will persist in the source's history. +Including secrets in a project's source code is a significant risk, as it might make those secrets available to unwanted parties. Even if it seems that the source code is accessible to the same people who are privy to the secrets, this situation is likely to change as the project grows. Spreading secrets in different places makes them harder to manage, access control, and revoke efficiently. Secrets that are committed to source control are also harder to discard of, since they will persist in the source's history. Another consideration is that coupling the project's code to its infrastructure and deployment specifics is limiting and considered a bad practice. From a software design perspective, the code should be independent of the runtime configuration that will be used to run it, and that runtime configuration includes secrets. - As such, there should be a clear boundary between code and secrets: secrets should be managed outside of the source code (read more [here](../../../continuous-delivery/secrets-management/README.md)) and credential scanning should be employed to ensure that this boundary is never violated. +As such, there should be a clear boundary between code and secrets: secrets should be managed outside of the source code and credential scanning should be employed to ensure that this boundary is never violated. ## Applying Credential Scanning diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/recipes/README.md b/docs/CI-CD/dev-sec-ops/secrets-management/recipes/README.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/secret-management/recipes/README.md rename to docs/CI-CD/dev-sec-ops/secrets-management/recipes/README.md diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets-ado.md b/docs/CI-CD/dev-sec-ops/secrets-management/recipes/detect-secrets-ado.md similarity index 82% rename from docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets-ado.md rename to docs/CI-CD/dev-sec-ops/secrets-management/recipes/detect-secrets-ado.md index 0ec0b2c155..c6b85c2de4 100644 --- a/docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets-ado.md +++ b/docs/CI-CD/dev-sec-ops/secrets-management/recipes/detect-secrets-ado.md @@ -12,13 +12,13 @@ Proposed Azure DevOps Pipeline contains multiple steps described below: 1. Install detect-secrets using pip 1. Run detect-secrets tool 1. Publish results in the Pipeline Artifact - > NOTE: It's an optional step, but for future investigation .json file with results may be helpful. + > **Note:** It's an optional step, but for future investigation .json file with results may be helpful. 1. Analyzing detect-secrets results - > NOTE: This step does a simple analysis of the .json file. If any secret has been detected, then break the build with exit code 1. + > **Note:** This step does a simple analysis of the .json file. If any secret has been detected, then break the build with exit code 1. -> NOTE: The below example has 2 jobs: for Linux and Windows agents. You do not have to use both jobs - just adjust the pipeline to your needs. +> **Note:** The below example has 2 jobs: for Linux and Windows agents. You do not have to use both jobs - just adjust the pipeline to your needs. > -> NOTE: Windows example does not use the latest version of detect-secrets. It is related to the bug in the detect-secret tool (see more in [Issue#452](https://github.com/Yelp/detect-secrets/issues/452)). It is highly recommended to monitor the fix for the issue and use the latest version if possible by removing version tag `==1.0.3` in the pip install command. +> **Note:** Windows example does not use the latest version of detect-secrets. It is related to the bug in the detect-secret tool (see more in [Issue#452](https://github.com/Yelp/detect-secrets/issues/452)). It is highly recommended to monitor the fix for the issue and use the latest version if possible by removing version tag `==1.0.3` in the pip install command. ```yaml trigger: diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets.md b/docs/CI-CD/dev-sec-ops/secrets-management/recipes/detect-secrets.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets.md rename to docs/CI-CD/dev-sec-ops/secrets-management/recipes/detect-secrets.md diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/secrets_rotation.md b/docs/CI-CD/dev-sec-ops/secrets-management/secrets_rotation.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/secret-management/secrets_rotation.md rename to docs/CI-CD/dev-sec-ops/secrets-management/secrets_rotation.md diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/static-code-analysis.md b/docs/CI-CD/dev-sec-ops/secrets-management/static-code-analysis.md similarity index 100% rename from docs/continuous-integration/dev-sec-ops/secret-management/static-code-analysis.md rename to docs/CI-CD/dev-sec-ops/secrets-management/static-code-analysis.md diff --git a/docs/continuous-delivery/gitops/deploying/README.md b/docs/CI-CD/gitops/deploying-with-gitops.md similarity index 100% rename from docs/continuous-delivery/gitops/deploying/README.md rename to docs/CI-CD/gitops/deploying-with-gitops.md diff --git a/docs/continuous-delivery/recipes/github-workflows/README.md b/docs/CI-CD/gitops/github-workflows.md similarity index 100% rename from docs/continuous-delivery/recipes/github-workflows/README.md rename to docs/CI-CD/gitops/github-workflows.md diff --git a/docs/continuous-delivery/gitops/deploying/images/GitopsWorflowVsTraditionalPush.jpg b/docs/CI-CD/gitops/images/GitopsWorflowVsTraditionalPush.jpg similarity index 100% rename from docs/continuous-delivery/gitops/deploying/images/GitopsWorflowVsTraditionalPush.jpg rename to docs/CI-CD/gitops/images/GitopsWorflowVsTraditionalPush.jpg diff --git a/docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Dependent-Workflows.png b/docs/CI-CD/gitops/images/Workflow-Designs-Dependent-Workflows.png similarity index 100% rename from docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Dependent-Workflows.png rename to docs/CI-CD/gitops/images/Workflow-Designs-Dependent-Workflows.png diff --git a/docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Independent-Workflows.png b/docs/CI-CD/gitops/images/Workflow-Designs-Independent-Workflows.png similarity index 100% rename from docs/continuous-delivery/recipes/github-workflows/images/Workflow-Designs-Independent-Workflows.png rename to docs/CI-CD/gitops/images/Workflow-Designs-Independent-Workflows.png diff --git a/docs/continuous-delivery/gitops/secret-management/README.md b/docs/CI-CD/gitops/secret-management/README.md similarity index 100% rename from docs/continuous-delivery/gitops/secret-management/README.md rename to docs/CI-CD/gitops/secret-management/README.md diff --git a/docs/continuous-delivery/gitops/secret-management/azure-devops-secret-management-per-branch.md b/docs/CI-CD/gitops/secret-management/azure-devops-secret-management-per-branch.md similarity index 100% rename from docs/continuous-delivery/gitops/secret-management/azure-devops-secret-management-per-branch.md rename to docs/CI-CD/gitops/secret-management/azure-devops-secret-management-per-branch.md diff --git a/docs/continuous-delivery/gitops/secret-management/secret-rotation-in-pods.md b/docs/CI-CD/gitops/secret-management/secret-rotation-in-pods.md similarity index 100% rename from docs/continuous-delivery/gitops/secret-management/secret-rotation-in-pods.md rename to docs/CI-CD/gitops/secret-management/secret-rotation-in-pods.md diff --git a/docs/continuous-delivery/images/blue_green.png b/docs/CI-CD/images/blue_green.png similarity index 100% rename from docs/continuous-delivery/images/blue_green.png rename to docs/CI-CD/images/blue_green.png diff --git a/docs/continuous-delivery/images/canary_release.png b/docs/CI-CD/images/canary_release.png similarity index 100% rename from docs/continuous-delivery/images/canary_release.png rename to docs/CI-CD/images/canary_release.png diff --git a/docs/continuous-delivery/images/environments.png b/docs/CI-CD/images/environments.png similarity index 100% rename from docs/continuous-delivery/images/environments.png rename to docs/CI-CD/images/environments.png diff --git a/docs/continuous-delivery/images/example_release_flow.png b/docs/CI-CD/images/example_release_flow.png similarity index 100% rename from docs/continuous-delivery/images/example_release_flow.png rename to docs/CI-CD/images/example_release_flow.png diff --git a/docs/continuous-integration/ci-in-data-science/working-with-notebooks/assets/repository-properties.png b/docs/CI-CD/images/repository-properties.png similarity index 100% rename from docs/continuous-integration/ci-in-data-science/working-with-notebooks/assets/repository-properties.png rename to docs/CI-CD/images/repository-properties.png diff --git a/docs/CI-CD/recipes/.pages b/docs/CI-CD/recipes/.pages new file mode 100644 index 0000000000..af7ff8a0a7 --- /dev/null +++ b/docs/CI-CD/recipes/.pages @@ -0,0 +1,6 @@ +nav: + - CD on low code solutions: cd-on-low-code-solutions.md + - CI pipeline for better documentation: ci-pipeline-for-better-documentation.md + - CI with jupyter notebooks: ci-with-jupyter-notebooks.md + - GitHub actions: github-actions + - ... diff --git a/docs/continuous-delivery/low-code-solutions/README.md b/docs/CI-CD/recipes/cd-on-low-code-solutions.md similarity index 100% rename from docs/continuous-delivery/low-code-solutions/README.md rename to docs/CI-CD/recipes/cd-on-low-code-solutions.md diff --git a/docs/continuous-integration/markdown-linting/README.md b/docs/CI-CD/recipes/ci-pipeline-for-better-documentation.md similarity index 94% rename from docs/continuous-integration/markdown-linting/README.md rename to docs/CI-CD/recipes/ci-pipeline-for-better-documentation.md index 5f6679c401..539f08fe86 100644 --- a/docs/continuous-integration/markdown-linting/README.md +++ b/docs/CI-CD/recipes/ci-pipeline-for-better-documentation.md @@ -4,8 +4,7 @@ Most projects start with spikes, where developers and analysts produce lots of documentation. -Sometimes, these documents don't have a standard and each team member writes them accordingly with their preference. Add to that -the time a reviewer will spend confirming grammar, searching for typos or non-inclusive language. +Sometimes, these documents don't have a standard and each team member writes them accordingly with their preference. Add to that the time a reviewer will spend confirming grammar, searching for typos or non-inclusive language. This pipeline helps address that! diff --git a/docs/continuous-integration/ci-in-data-science/working-with-notebooks/README.md b/docs/CI-CD/recipes/ci-with-jupyter-notebooks.md similarity index 97% rename from docs/continuous-integration/ci-in-data-science/working-with-notebooks/README.md rename to docs/CI-CD/recipes/ci-with-jupyter-notebooks.md index c8f8d9b199..4492de28a3 100644 --- a/docs/continuous-integration/ci-in-data-science/working-with-notebooks/README.md +++ b/docs/CI-CD/recipes/ci-with-jupyter-notebooks.md @@ -1,4 +1,4 @@ -# Data Science Pipeline +# CI with Jupyter Notebooks As Azure DevOps doesn't allow code reviewers to comment directly in Jupyter Notebooks, Data Scientists(DSs) have to convert the notebooks to scripts before they commit and push these files to the repository. @@ -41,7 +41,7 @@ We can add a pipeline with the following steps to the repository to run in `ipyn 1. Go to the *Project Settings* -> *Repositories* -> *Security* -> *User Permissions* 1. Add the *Build Service* in *Users* the permission to *Contribute* - ![Contribute](assets/repository-properties.png) + ![Contribute](../images/repository-properties.png) 1. Create a new pipeline. In the newly created pipeline we add: diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/README.md b/docs/CI-CD/recipes/github-actions/runtime-variables/README.md similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/README.md rename to docs/CI-CD/recipes/github-actions/runtime-variables/README.md diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/commit-example.yaml b/docs/CI-CD/recipes/github-actions/runtime-variables/examples/commit-example.yaml similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/commit-example.yaml rename to docs/CI-CD/recipes/github-actions/runtime-variables/examples/commit-example.yaml diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/pr-example.yaml b/docs/CI-CD/recipes/github-actions/runtime-variables/examples/pr-example.yaml similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/examples/pr-example.yaml rename to docs/CI-CD/recipes/github-actions/runtime-variables/examples/pr-example.yaml diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/AzDoExample.PNG b/docs/CI-CD/recipes/github-actions/runtime-variables/images/AzDoExample.PNG similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/AzDoExample.PNG rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/AzDoExample.PNG diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitFalse.PNG b/docs/CI-CD/recipes/github-actions/runtime-variables/images/CommitFalse.PNG similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitFalse.PNG rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/CommitFalse.PNG diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitTrue.PNG b/docs/CI-CD/recipes/github-actions/runtime-variables/images/CommitTrue.PNG similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/CommitTrue.PNG rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/CommitTrue.PNG diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevAWorkflow.png b/docs/CI-CD/recipes/github-actions/runtime-variables/images/DevAWorkflow.png similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevAWorkflow.png rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/DevAWorkflow.png diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevBWorkflow.png b/docs/CI-CD/recipes/github-actions/runtime-variables/images/DevBWorkflow.png similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/DevBWorkflow.png rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/DevBWorkflow.png diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRExample.PNG b/docs/CI-CD/recipes/github-actions/runtime-variables/images/PRExample.PNG similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRExample.PNG rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/PRExample.PNG diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRTrue.PNG b/docs/CI-CD/recipes/github-actions/runtime-variables/images/PRTrue.PNG similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/PRTrue.PNG rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/PRTrue.PNG diff --git a/docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/workflow-diagram.png b/docs/CI-CD/recipes/github-actions/runtime-variables/images/workflow-diagram.png similarity index 100% rename from docs/continuous-delivery/devops-provider-recipes/github-actions/runtime-variables/images/workflow-diagram.png rename to docs/CI-CD/recipes/github-actions/runtime-variables/images/workflow-diagram.png diff --git a/docs/continuous-integration/inclusive-linting.md b/docs/CI-CD/recipes/inclusive-linting.md similarity index 100% rename from docs/continuous-integration/inclusive-linting.md rename to docs/CI-CD/recipes/inclusive-linting.md diff --git a/docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md b/docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md new file mode 100644 index 0000000000..b30c2b30ec --- /dev/null +++ b/docs/CI-CD/recipes/reusing-devcontainers-within-a-pipeline.md @@ -0,0 +1,72 @@ +# Reusing dev containers within a pipeline + +Given a repository with a local development container a.k.a. dev container that contains all the tooling required for development, would it make sense to reuse that container for running the tooling in the Continuous Integration pipelines? + +## Options for building devcontainers within pipeline + +There are three ways to build devcontainers within pipeline: + +- With [GitHub - devcontainers/ci](https://github.com/devcontainers/ci) builds the container with the `devcontainer.json`. Example here: [devcontainers/ci · Getting Started](https://github.com/devcontainers/ci/blob/main/docs/github-action.md#getting-started). +- With [GitHub - devcontainers/cli](https://github.com/devcontainers/cli), which is the same as the above, but using the underlying CLI directly without tasks. +- Building the `DockerFile` with `docker build`. This option excludes all configuration/features specified within the `devcontainer.json`. + +## Considered options + +- Run CI pipelines in native environment +- Run CI pipelines in the dev container via building image locally +- Run CI pipelines in the dev container with a container registry + +Here are below pros and cons for both approaches: + +### Run CI pipelines in native environment + +| Pros | Cons | +| -- | -- | +| Can use any pipeline tasks available | Need to keep two sets of tooling and their versions in sync | +| No container registry | Can take some time to start, based on tools/dependencies required | +| Agent will always be up to date with security patches | The dev container should always be built within each run of the CI pipeline, to verify the changes within the branch haven't broken anything | + +### Run CI pipelines in the dev container without image caching + +| Pros | Cons | +| -- | -- | +| Utilities scripts will work out of the box | Need to rebuild the container for each run, given that there may be changes within the branch being built | +| Rules used (for linting or unit tests) will be the same on the CI | Not everything in the container is needed for the CI pipeline¹ | +| No surprise for the developers, local outputs (of linting for instance) will be the same in the CI | Some pipeline tasks will not be available | +| All tooling and their versions defined in a single place | Building the image for each pipeline run is slow² | +| Tools/dependencies are already present || +| The dev container is being tested to include all new tooling in addition to not being broken || + +> ¹: container size can be reduced by exporting the layer that contains only the tooling needed for the CI pipeline +> +> ²: could be mitigated via adding image caching without using a container registry + +### Run CI pipelines in the dev container with image registry + +| Pros | Cons | +| -- | -- | +| Utilities scripts will work out of the box | Need to rebuild the container for each run, given that there may be changes within the branch being built | +| No surprise for the developers, local outputs (of linting for instance) will be the same in the CI | Not everything in the container is needed for the CI pipeline¹ | +| Rules used (for linting or unit tests) will be the same on the CI | Some pipeline tasks will not be available² | +| All tooling and their versions defined in a single place | Require access to a container registry to host the container within the pipeline³ | +| Tools/dependencies are already present || +| The dev container is being tested to include all new tooling in addition to not being broken || +| Publishing the container built from `devcontainer.json` allows you to reference it in the cacheFrom in `devcontainer.json` (see [docs](https://containers.dev/implementors/json_reference/#image-specific)). By doing this, VS Code will use the published image as a layer cache when building || + +> ¹: container size can be reduces by exporting the layer that contains only the tooling needed for the CI pipeline. This would require building the image without tasks +> +> ²: using container jobs in AzDO you can use all tasks (as far as I can tell). Reference: [Dockerizing DevOps V2 - AzDO container jobs - DEV Community](https://dev.to/eliises/dockerizing-devops-v2-azdo-container-jobs-3hbf) +> +> ³: within GH actions, the default Github Actions token can be used for accessing GHCR without setting up separate registry, see the example below. +> **Note:** This does not build the `Dockerfile` together with the `devcontainer.json` + +```yaml +    - uses: whoan/docker-build-with-cache-action@v5 +        id: cache +        with: +          username: $GITHUB_ACTOR +          password: "${{ secrets.GITHUB_TOKEN }}" +          registry: docker.pkg.github.com +          image_name: devcontainer +          dockerfile: .devcontainer/Dockerfile +``` diff --git a/docs/continuous-delivery/recipes/terraform/save-output-to-variable-group.md b/docs/CI-CD/recipes/terraform/save-output-to-variable-group.md similarity index 100% rename from docs/continuous-delivery/recipes/terraform/save-output-to-variable-group.md rename to docs/CI-CD/recipes/terraform/save-output-to-variable-group.md diff --git a/docs/continuous-delivery/recipes/terraform/share-common-variables-naming-conventions.md b/docs/CI-CD/recipes/terraform/share-common-variables-naming-conventions.md similarity index 100% rename from docs/continuous-delivery/recipes/terraform/share-common-variables-naming-conventions.md rename to docs/CI-CD/recipes/terraform/share-common-variables-naming-conventions.md diff --git a/docs/continuous-delivery/recipes/terraform/terraform-structure-guidelines.md b/docs/CI-CD/recipes/terraform/terraform-structure-guidelines.md similarity index 100% rename from docs/continuous-delivery/recipes/terraform/terraform-structure-guidelines.md rename to docs/CI-CD/recipes/terraform/terraform-structure-guidelines.md diff --git a/docs/ISE.md b/docs/ISE.md index ff0425c391..eeff8db1a1 100644 --- a/docs/ISE.md +++ b/docs/ISE.md @@ -1,4 +1,4 @@ -# Who We Are +# Who is ISE (Industry Solutions Engineering) Our team, ISE (Industry Solutions Engineering), works side-by-side with customers to help them tackle their toughest technical problems both in the cloud and on the edge. We meet customers where they are, work in the languages they use, with the open source frameworks they use, and on the operating systems they use. We work with enterprises and start-ups across many industries from financial services to manufacturing. Our work covers a broad spectrum of domains including IoT, machine learning, and high scale compute. Our "superpower" is that we work closely with both our customers’ engineering teams and Microsoft’s product engineering teams, developing real-world expertise that we can use to help our customers grow their business and help Microsoft improve our products and services. diff --git a/docs/index.md b/docs/README.md similarity index 59% rename from docs/index.md rename to docs/README.md index 6bf536aa89..82bc617797 100644 --- a/docs/index.md +++ b/docs/README.md @@ -9,19 +9,15 @@ An engineer working for a [ISE](ISE.md) project... This is our playbook. All contributions are welcome! Please feel free to submit a pull request to get involved. -## Why Have A Playbook +## Why have a Playbook * To increase overall efficiency for team members and the whole team in general. * To reduce the number of mistakes and avoid common pitfalls. * To strive to be better engineers and learn from other people's shared experience. -## "The" Checklist +If you do nothing else follow the [Engineering Fundamentals Checklist](./engineering-fundamentals-checklist.md)! -If you do nothing else follow the [Engineering Fundamentals Checklist](ENG-FUNDAMENTALS-CHECKLIST.md)! - -## Structure of a Sprint - -The [structure of a sprint](SPRINT-STRUCTURE.md) is a breakdown of the sections of the playbook according to the structure of an Agile sprint. +The [first week of an ISE project](./the-first-week-of-an-ise-project.md) is a breakdown of the sections of the playbook according to the structure of an Agile sprint. ## General Guidance @@ -36,34 +32,6 @@ The [structure of a sprint](SPRINT-STRUCTURE.md) is a breakdown of the sections * Report product issues found and provide clear and repeatable engineering feedback! * We all own our code and each one of us has an obligation to make all parts of the solution great. -## QuickLinks - -* [Engineering Fundamentals Checklist](ENG-FUNDAMENTALS-CHECKLIST.md) -* [Structure of a Sprint](SPRINT-STRUCTURE.md) - -## Engineering Fundamentals - -* [Accessibility](accessibility/README.md) -* [Agile Development](agile-development/README.md) -* [Automated Testing](automated-testing/README.md) -* [Code Reviews](code-reviews/README.md) -* [Continuous Delivery (CD)](continuous-delivery/README.md) -* [Continuous Integration (CI)](continuous-integration/README.md) -* [Design](design/readme.md) -* [Developer Experience](developer-experience/README.md) -* [Documentation](documentation/README.md) -* [Engineering Feedback](engineering-feedback/README.md) -* [Observability](observability/README.md) -* [Security](security/README.md) -* [Privacy](privacy/README.md) -* [Source Control](source-control/README.md) -* [Reliability](reliability/README.md) - -## Fundamentals for Specific Technology Areas - -* [Machine Learning Fundamentals](machine-learning/README.md) -* [User-Interface Engineering](user-interface-engineering/README.md) - ## Contributing See [CONTRIBUTING.md](https://github.com/microsoft/code-with-engineering-playbook/tree/master/CONTRIBUTING.md) for contribution guidelines. diff --git a/docs/continuous-delivery/.pages b/docs/continuous-delivery/.pages deleted file mode 100644 index 5233920c6f..0000000000 --- a/docs/continuous-delivery/.pages +++ /dev/null @@ -1,5 +0,0 @@ -nav: - - Azure DevOps: azure-devops - - DevOps provider recipes: devops-provider-recipes - - GitOps: gitops - - ... \ No newline at end of file diff --git a/docs/continuous-delivery/recipes/README.md b/docs/continuous-delivery/recipes/README.md deleted file mode 100644 index 88fa469901..0000000000 --- a/docs/continuous-delivery/recipes/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Recipes - -## Github - -- [Github workflows](./github-workflows/README.md) - -## Terraform - -- [Save output to variable group](./terraform/save-output-to-variable-group.md) -- [Share common variables naming conventions](./terraform/share-common-variables-naming-conventions.md) -- [Terraform structure guidelines](./terraform/terraform-structure-guidelines.md) \ No newline at end of file diff --git a/docs/continuous-delivery/recipes/terraform/README.md b/docs/continuous-delivery/recipes/terraform/README.md deleted file mode 100644 index d163c83702..0000000000 --- a/docs/continuous-delivery/recipes/terraform/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Terraform recipes - -- [Save output to variable group](./save-output-to-variable-group.md) -- [Share common variables naming conventions](./share-common-variables-naming-conventions.md) -- [Terraform structure guidelines](./terraform-structure-guidelines.md) \ No newline at end of file diff --git a/docs/continuous-integration/.pages b/docs/continuous-integration/.pages deleted file mode 100644 index 73e6208844..0000000000 --- a/docs/continuous-integration/.pages +++ /dev/null @@ -1,5 +0,0 @@ -nav: - - CI in data science: ci-in-data-science - - DevSecOps: dev-sec-ops - - Dev conainters: devcontainers - - markdown-linting \ No newline at end of file diff --git a/docs/continuous-integration/dev-sec-ops/README.md b/docs/continuous-integration/dev-sec-ops/README.md deleted file mode 100644 index 57fc3b0508..0000000000 --- a/docs/continuous-integration/dev-sec-ops/README.md +++ /dev/null @@ -1,21 +0,0 @@ -# DevSecOps - -## The concept of DevSecOps - -DevSecOps or DevOps security is about introducing security earlier in the life cycle of application development (a.k.a shift-left), thus minimizing the impact of vulnerabilities and bringing security closer to development team. - -## Why - -By embracing shift-left mentality, DevSecOps encourages organizations to bridge the gap that often exists between development and security teams to the point where many of the security processes are automated and are effectively handled by the development team. - -## DevSecOps Practices - -This section covers different tools, frameworks and resources allowing introduction of DevSecOps best practices to your project at early stages of development. -Topics covered: - -1. [Credential Scanning](./secret-management/credential_scanning.md) - automatically inspecting a project to ensure that no secrets are included in the project's source code. -1. [Secrets Rotation](./secret-management/secrets_rotation.md) - automated process by which the secret, used by the application, is refreshed and replaced by a new secret. -1. [Static Code Analysis](./secret-management/static-code-analysis.md) - analyze source code or compiled versions of code to help find security flaws. -1. [Penetration Testing](./penetration-testing/README.md) - a simulated attack against your application to check for exploitable vulnerabilities. -1. [Container Dependencies Scanning](./dependency-container-scanning/README.md) - search for vulnerabilities in container operating systems, language packages and application dependencies. -1. [Evaluation of Open Source Libraries](./evaluate-oss/README.md) - make it harder to apply open source supply chain attacks by evaluating the libraries you use. diff --git a/docs/continuous-integration/dev-sec-ops/azure-devops/README.md b/docs/continuous-integration/dev-sec-ops/azure-devops/README.md deleted file mode 100644 index a12e35e689..0000000000 --- a/docs/continuous-integration/dev-sec-ops/azure-devops/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Azure DevOps - -Write something about Azure DevOps here. - - -## Table of Contents - -- [Service connection security](./service-connection-security.md) diff --git a/docs/continuous-integration/dev-sec-ops/secret-management/README.md b/docs/continuous-integration/dev-sec-ops/secret-management/README.md deleted file mode 100644 index 85acf5b97c..0000000000 --- a/docs/continuous-integration/dev-sec-ops/secret-management/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# Secret management - -Secret Management refers to the tools and practices used to manage digital authentication credentials (like API keys, tokens, passwords, and certificates). These secrets are used to protect access to sensitive data and services, making their management critical for security. - -## Importance of Secret Management - -In modern software development, applications often need to interact with other software components, APIs, and services. These interactions often require authentication, which is typically handled using secrets. If these secrets are not managed properly, they can be exposed, leading to potential security breaches. - -## Best Practices for Secret Management - -1. **Centralized Secret Storage:** Store all secrets in a centralized, encrypted location. This reduces the risk of secrets being lost or exposed. - -2. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets. - -3. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised. - -4. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities. - -5. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error. - -Remember, the goal of secret management is to protect sensitive information from unauthorized access and potential security threats. - -## Pages - -- [Credential Scanning](./credential_scanning.md) -- [Secrets Rotation](./secrets_rotation.md) -- [Static code analysis](./static-code-analysis.md) diff --git a/docs/continuous-integration/devcontainers/README.md b/docs/continuous-integration/devcontainers/README.md deleted file mode 100644 index dbdea5e97a..0000000000 --- a/docs/continuous-integration/devcontainers/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Reusing dev containers within a pipeline - -Given a repository with a local development container aka [dev container](../devcontainers/README.md) that contains all the tooling required for development, would it make sense to reuse that container for running the tooling in the Continuous Integration pipelines? - -## Options for building devcontainers within pipeline - -There are three ways to build devcontainers within pipeline: - -- With [GitHub - devcontainers/ci](https://github.com/devcontainers/ci) builds the container with the `devcontainer.json`. Example here: [devcontainers/ci · Getting Started](https://github.com/devcontainers/ci/blob/main/docs/github-action.md#getting-started). -- With [GitHub - devcontainers/cli](https://github.com/devcontainers/cli), which is the same as the above, but using the underlying CLI directly without tasks. -- Building the `DockerFile` with `docker build`. This option excludes all configuration/features specified within the `devcontainer.json`. - -## Considered Options - -- Run CI pipelines in native environment -- Run CI pipelines in the dev container via building image locally -- Run CI pipelines in the dev container with a container registry - -Here are below pros and cons for both approaches: - -### Run CI pipelines in native environment - -| Pros | Cons | -|-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| -| Can use any pipeline tasks available | Need to keep two sets of tooling and their versions in sync | -| No container registry | Can take some time to start, based on tools/dependencies required | -| Agent will always be up to date with security patches | The dev container should always be built within each run of the CI pipeline, to verify the changes within the branch haven't broken anything | - -### Run CI pipelines in the dev container without image caching - -| Pros | Cons | -|----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| -| Utilities scripts will work out of the box | Need to rebuild the container for each run, given that there may be changes within the branch being built | -| Rules used (for linting or unit tests) will be the same on the CI | Not everything in the container is needed for the CI pipeline¹ | -| No surprise for the developers, local outputs (of linting for instance) will be the same in the CI | Some pipeline tasks will not be available | -| All tooling and their versions defined in a single place | Building the image for each pipeline run is slow² | -| Tools/dependencies are already present | | -| The dev container is being tested to include all new tooling in addition to not being broken | | - -> ¹: container size can be reduces by exporting the layer that contains only the tooling needed for the CI pipeline -> -> ²: could be mitigated via adding image caching without using a container registry - -### Run CI pipelines in the dev container with image registry - -| Pros | Cons | -|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| -| Utilities scripts will work out of the box | Need to rebuild the container for each run, given that there may be changes within the branch being built | -| No surprise for the developers, local outputs (of linting for instance) will be the same in the CI | Not everything in the container is needed for the CI pipeline¹ | -| Rules used (for linting or unit tests) will be the same on the CI | Some pipeline tasks will not be available ² | -| All tooling and their versions defined in a single place | Require access to a container registry to host the container within the pipeline³ | -| Tools/dependencies are already present | | -| The dev container is being tested to include all new tooling in addition to not being broken | | -| Publishing the container built from `devcontainer.json` allows you to reference it in the cacheFrom in `devcontainer.json` (see [docs](https://containers.dev/implementors/json_reference/#image-specific)). By doing this, VS Code will use the published image as a layer cache when building | | - -> ¹: container size can be reduces by exporting the layer that contains only the tooling needed for the CI pipeline. This would require building the image without tasks -> -> ²: using container jobs in AzDO you can use all tasks (as far as I can tell). Reference: [Dockerizing DevOps V2 - AzDO container jobs - DEV Community](https://dev.to/eliises/dockerizing-devops-v2-azdo-container-jobs-3hbf) -> -> ³: within GH actions, the default Github Actions token can be used for accessing GHCR without setting up separate registry, see the example below. -> **NOTE:** This does not build the `Dockerfile` together with the `devcontainer.json` - -```yaml -    - uses: whoan/docker-build-with-cache-action@v5 -        id: cache -        with: -          username: $GITHUB_ACTOR -          password: "${{ secrets.GITHUB_TOKEN }}" -          registry: docker.pkg.github.com -          image_name: devcontainer -          dockerfile: .devcontainer/Dockerfile -``` - - diff --git a/docs/ENG-FUNDAMENTALS-CHECKLIST.md b/docs/engineering-fundamentals-checklist.md similarity index 95% rename from docs/ENG-FUNDAMENTALS-CHECKLIST.md rename to docs/engineering-fundamentals-checklist.md index 6a7764a7cd..fed811f3cf 100644 --- a/docs/ENG-FUNDAMENTALS-CHECKLIST.md +++ b/docs/engineering-fundamentals-checklist.md @@ -10,7 +10,7 @@ This checklist helps to ensure that our projects meet our Engineering Fundamenta - [ ] Commit history is consistent and commit messages are informative (what, why). - [ ] Consistent branch naming conventions. - [ ] Clear documentation of repository structure. -- [ ] Secrets are not part of the commit history or made public. (see [Credential scanning](continuous-integration/dev-sec-ops/secret-management/credential_scanning.md)) +- [ ] Secrets are not part of the commit history or made public. (see [Credential scanning](./CI-CD/dev-sec-ops/secrets-management/credential_scanning.md)) - [ ] Public repositories follow the [OSS guidelines](source-control/README.md#creating-a-new-repository), see `Required files in default branch for public repositories`. More details on [source control](source-control/README.md) @@ -35,7 +35,7 @@ More details on [automated testing](automated-testing/README.md) - [ ] Project uses CD to manage deployments to a replica environment before PRs are merged. - [ ] Main branch is always shippable. -More details on [continuous integration](continuous-integration/README.md) and [continuous delivery](continuous-delivery/README.md) +More details on [continuous integration](./CI-CD/continuous-integration.md) and [continuous delivery](./CI-CD/continuous-delivery.md) ## Security diff --git a/docs/SPRINT-STRUCTURE.md b/docs/the-first-week-of-an-ise-project.md similarity index 92% rename from docs/SPRINT-STRUCTURE.md rename to docs/the-first-week-of-an-ise-project.md index 4433a1dd42..e94854366a 100644 --- a/docs/SPRINT-STRUCTURE.md +++ b/docs/the-first-week-of-an-ise-project.md @@ -1,4 +1,4 @@ -# Structure of a Sprint +# The first week of an ISE Project The purpose of this document is to: @@ -6,9 +6,8 @@ The purpose of this document is to: - Provide content in a logical structure which reflects the engineering process - Extensible hierarchy to allow teams to share deep subject-matter expertise -## The first week of an ISE Project -### Before starting the project +## Before starting the project - [ ] Discuss and start writing the Team Agreements. Update these documents with any process decisions made throughout the project - [Working Agreement](agile-development/advanced-topics/team-agreements/working-agreements.md) @@ -23,7 +22,7 @@ The purpose of this document is to: - [INVEST](https://en.wikipedia.org/wiki/INVEST_(mnemonic)) in good User Stories and Acceptance Criteria - [Non-Functional Requirements Guidance](design/design-patterns/non-functional-requirements-capture-guide.md) -### Day 1 +## Day 1 - [ ] [Plan the first sprint](agile-development/basics/ceremonies.md#sprint-planning) - Agree on a sprint goal, and how to measure the sprint progress @@ -35,34 +34,34 @@ The purpose of this document is to: - Agree on how to separate unit tests from integration, load and smoke tests - Design the first test cases - [ ] [Decide on branch naming](source-control/naming-branches.md) -- [ ] [Discuss security needs and verify that secrets are kept out of source control](continuous-delivery/gitops/secret-management/azure-devops-secret-management-per-branch.md) +- [ ] [Discuss security needs and verify that secrets are kept out of source control](./CI-CD/dev-sec-ops/secrets-management/README.md) -### Day 2 +## Day 2 - [ ] [Set up Source Control](source-control/README.md) - Agree on [best practices for commits](source-control/git-guidance/README.md#commit-best-practices) -- [ ] [Set up basic Continuous Integration with linters and automated tests](continuous-integration/README.md) +- [ ] [Set up basic Continuous Integration with linters and automated tests](./CI-CD/continuous-integration.md) - [ ] [Set up meetings for Daily Stand-ups and decide on a Process Lead](agile-development/basics/ceremonies.md#stand-up) - Discuss purpose, goals, participants and facilitation guidance - Discuss timing, and how to run an efficient stand-up - [ ] [If the project has sub-teams, set up a Scrum of Scrums](agile-development/advanced-topics/effective-organization/scrum-of-scrums.md) -### Day 3 +## Day 3 - [ ] [Agree on code style](code-reviews/README.md) and on [how to assign Pull Requests](code-reviews/pull-requests.md) - [ ] [Set up Build Validation for Pull Requests (2 reviewers, linters, automated tests)](code-reviews/README.md) and agree on [Definition of Done](agile-development/advanced-topics/team-agreements/definition-of-done.md) - [ ] [Agree on a Code Merging strategy](source-control/merge-strategies.md) and update the [CONTRIBUTING.md](resources/templates/CONTRIBUTING.md) - [ ] [Agree on logging and observability frameworks and strategies](observability/README.md) -### Day 4 +## Day 4 -- [ ] [Set up Continuous Deployment](continuous-delivery/README.md) +- [ ] [Set up Continuous Deployment](./CI-CD/continuous-delivery.md) - Determine what environments are appropriate for this solution - For each environment discuss purpose, when deployment should trigger, pre-deployment approvers, sing-off for promotion. - [ ] [Decide on a versioning strategy](source-control/component-versioning.md) - [ ] Agree on how to [Design a feature and conduct a Design Review](design/design-reviews/README.md) -### Day 5 +## Day 5 - [ ] Conduct a [Sprint Demo](agile-development/basics/ceremonies.md#sprint-demo) - [ ] Conduct a [Retrospective](agile-development/basics/ceremonies.md#retrospectives) diff --git a/linkcheck.json b/linkcheck.json index 47bc45face..91b1bfcc12 100755 --- a/linkcheck.json +++ b/linkcheck.json @@ -69,5 +69,5 @@ ], "only_errors": true, "cache_duration": "24h", - "cache_output_path": "/github/workspace/megalinter-reports/linchcheck-cache" + "cache_output_path": "/github/workspace/megalinter-reports/linkcheck-cache" } \ No newline at end of file