re-factor ci/cd

docs/.pages

@@ -0,0 +1,4 @@
+nav:
+  - ...
+  - CI/CD: ci-cd
+  - ...

docs/CI-CD/.pages

@@ -0,0 +1,6 @@
+nav:
+  - Continuous Delivery: continuous-delivery
+  - Continuous Integration: continuous-integration
+  - DevSecOps: dev-sec-ops
+  - GitOps: gitops
+  - ...

docs/continuous-integration/CICD.md → docs/CI-CD/README.md

@@ -1,10 +1,10 @@
-# Continuous Integration and Delivery
+# Continuous Integration and Continuous Delivery
 
-Continuous Integration is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed.
+[**Continuous Integration (CI)**](continuous-integration.md) is the engineering practice of frequently committing code in a shared repository, ideally several times a day, and performing an automated build on it. These changes are built with other simultaneous changes to the system, which enables early detection of integration issues between multiple developers working on a project. Build breaks due to integration failures are treated as the highest priority issue for all the developers on a team and generally work stops until they are fixed.
 
 Paired with an automated testing approach, continuous integration also allows us to also test the integrated build such that we can verify that not only does the code base still build correctly, but also is still functionally correct. This is also a best practice for building robust and flexible software systems.
 
-Continuous Delivery takes the Continuous Integration concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage.
+[**Continuous Delivery (CD)**](continuous-delivery.md) takes the **Continuous Integration (CI)** concept further to also test deployments of the integrated code base on a replica of the environment it will be ultimately deployed on. This enables us to learn early about any unforeseen operational issues that arise from our changes as quickly as possible and also learn about gaps in our test coverage.
 
 The goal of all of this is to ensure that the main branch is always shippable, meaning that we could, if we needed to, take a build from the main branch of our code base and ship it on production.
 
@@ -14,6 +14,23 @@ Our expectation is that CI/CD should be used in all the engineering projects tha
 
 For a much deeper understanding of all of these concepts, the books [Continuous Integration](https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380) and [Continuous Delivery](https://www.amazon.com/gp/product/0321601912) provide a comprehensive background.
 
+## Why CI/CD
+
+- We want to have an automated build and deployment of our software
+- We want automated configuration of all components
+- We want to be able to quickly re-build the environment from scratch in case of disaster
+- We want the latest version of the code to always be deployed to our dev/test environments
+- We want a reliable release strategy, where the policies for release are well understood by all
+
+## The fundamentals
+
+- We run a quality pipeline (with linting, unit tests etc.) on each PR/update of the main branch
+- All cloud resources (including secrets and permissions) are provisioned through infrastructure as code templates – ex. Terraform, Bicep (ARM), Pulumi etc.
+- All release candidates are deployed to a non-production environment through an automated process (ex Azure DevOps or Github pipelines)
+- Releases are deployed to the production environment through an automated process
+- Release rollbacks are carried out through a repeatable process
+- Our release pipeline runs automated tests, validating all release candidate artifact(s) end-to-end against a non-production environment
+
 ## Tools
 
 ### Azure Pipelines

docs/CI-CD/dev-sec-ops/.pages

@@ -0,0 +1,3 @@
+nav:
+  - Azure DevOps service connection security: azure-devops-service-connection-security
+  - ...

docs/continuous-integration/dev-sec-ops/azure-devops/service-connection-security.md → docs/CI-CD/dev-sec-ops/azure-devops-service-connection-security.md

@@ -13,10 +13,11 @@ Securing Service Connections can be achieved by using several methods.
 - [Project permissions](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints#project-permissions---cross-project-sharing-of-service-connections) can be configured to ensure only certain Azure DevOps projects are able to use the Service Connection.
 
 After using the above methods, what is secured is **who** can use the Service Connections.
-What still *isn't* secured however, is **what** can be done with the Service Connections.  
+What still *isn't* secured however, is **what** can be done with the Service Connections.
+
+Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users.
+An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources.  Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done.
 
-Because Service Connections have all the necessary permissions in the external services, it is crucial to secure Service Connections so they cannot be misused by accident or by malicious users.  
-An example of this is a Azure DevOps Pipeline that uses a Service Connection to an Azure Resource Group (or entire subscription) to list all resources and then delete those resources.  Without the correct security in place, it could be possible to execute this Pipeline, without any validation or reviews being done.  
 ```yaml
 pool:
   vmImage: ubuntu-latest
@@ -36,10 +37,10 @@ steps:
 
 ## Pipeline Security caveat
 
-YAML pipelines can be triggered without the need for a pull request, this introduces a security risk.  
+YAML pipelines can be triggered without the need for a pull request, this introduces a security risk.
 
-In good practice, [Pull Requests](../../../code-reviews/pull-requests.md) and [Code Reviews](../../../code-reviews/README.md) should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues.  
-However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer.  
+In good practice, [Pull Requests](../../../code-reviews/pull-requests.md) and [Code Reviews](../../../code-reviews/README.md) should be used to ensure the code that is being deployed, is being reviewed by a second person and potentially automatically being checked for vulnerabilities and other security issues.
+However, YAML Pipelines can be executed without the need for a Pull Request and Code Reviews. This allows the (malicious) user to make changes using the Service Connection which would normally require a reviewer.
 
 The configuration of *when* a pipeline should be triggered is specified in the YAML Pipeline itself and therefore a pipeline can be configured to execute on changes in a temporary branch. In this temporary branch, any changes made to the pipeline itself will be executed without being reviewed.  
 
@@ -54,13 +55,13 @@ Configuration can be done in the Approvals and Checks menu on the Service Connec
 
 ### Branch Control
 
-By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch.  
+By configuring Branch Control on a Service Connection, you can control that the Service Connection can only be used in a YAML Pipeline if the pipeline is running from a specific branch.
 
 By configuring Branch Control to only allow the main branch (and potentially release branches) you can ensure a YAML Pipeline can only use the Service Connection after any changes to that pipeline have been merged into the main branch, and therefore has passed any Pull Requests checks and Code Reviews.
 As an additional check, Branch Control can verify if Branch Protections (like required Pull Requests and Code Reviews) are actually configured on the allowed branches.
 
-With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore.  
+With Branch Control in place, in combination with Branch Protections, it is not possible anymore to run any commands against a Service Connection without having multiple persons review the commands. Therefore accidental, or malicious, mis-use of the permissions a Service Connection has is not possible anymore.
 
-**Note: When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.*
+> **Note:** When setting a wildcard for the Allowed Branches, anyone could still create a branch matching that wildcard and would be able to use the Service Connection. Using [git permissions](https://learn.microsoft.com/en-us/azure/devops/repos/git/require-branch-folders#enforce-permissions) it can be configured so only administrators are allowed to create certain branches, like release branches.*
 
 ![BranchControl](images/branch-control.png)

docs/continuous-integration/dev-sec-ops/secret-management/README.md → docs/CI-CD/dev-sec-ops/secret-management/README.md

@@ -9,19 +9,9 @@ In modern software development, applications often need to interact with other s
 ## Best Practices for Secret Management
 
 1. **Centralized Secret Storage:** Store all secrets in a centralized, encrypted location. This reduces the risk of secrets being lost or exposed.
-
-2. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets.
-
-3. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised.
-
-4. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities.
-
-5. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error.
+1. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets.
+1. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised.
+1. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities.
+1. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error.
 
 Remember, the goal of secret management is to protect sensitive information from unauthorized access and potential security threats.
-
-## Pages
-
-- [Credential Scanning](./credential_scanning.md)
-- [Secrets Rotation](./secrets_rotation.md)
-- [Static code analysis](./static-code-analysis.md)

docs/continuous-integration/dev-sec-ops/secret-management/credential_scanning.md → docs/CI-CD/dev-sec-ops/secret-management/credential_scanning.md

@@ -4,7 +4,7 @@ Credential scanning is the practice of automatically inspecting a project to ens
 
 ## Why Credential scanning
 
-Including secrets in a project's source code is a significant risk, as it might make those secrets available to unwanted parties. Even if it seems that the source code is accessible to the same people who are privy to the secrets, this situation is likely to change as the project grows. Spreading secrets in different places makes them harder to manage, access control, and revoke efficiently. Secrets that are committed to source control are also harder to discard of, since they will persist in the source's history.  
+Including secrets in a project's source code is a significant risk, as it might make those secrets available to unwanted parties. Even if it seems that the source code is accessible to the same people who are privy to the secrets, this situation is likely to change as the project grows. Spreading secrets in different places makes them harder to manage, access control, and revoke efficiently. Secrets that are committed to source control are also harder to discard of, since they will persist in the source's history.
 Another consideration is that coupling the project's code to its infrastructure and deployment specifics is limiting and considered a bad practice. From a software design perspective, the code should be independent of the runtime configuration that will be used to run it, and that runtime configuration includes secrets.
  As such, there should be a clear boundary between code and secrets: secrets should be managed outside of the source code (read more [here](../../../continuous-delivery/secrets-management/README.md)) and credential scanning should be employed to ensure that this boundary is never violated.
 

docs/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets-ado.md → docs/CI-CD/dev-sec-ops/secret-management/recipes/detect-secrets-ado.md

@@ -12,13 +12,13 @@ Proposed Azure DevOps Pipeline contains multiple steps described below:
 1. Install detect-secrets using pip
 1. Run detect-secrets tool
 1. Publish results in the Pipeline Artifact
-   > NOTE: It's an optional step, but for future investigation .json file with results may be helpful.
+   > **Note:** It's an optional step, but for future investigation .json file with results may be helpful.
 1. Analyzing detect-secrets results
-   > NOTE: This step does a simple analysis of the .json file. If any secret has been detected, then break the build with exit code 1.
+   > **Note:** This step does a simple analysis of the .json file. If any secret has been detected, then break the build with exit code 1.
 
-> NOTE: The below example has 2 jobs: for Linux and Windows agents. You do not have to use both jobs - just adjust the pipeline to your needs.
+> **Note:** The below example has 2 jobs: for Linux and Windows agents. You do not have to use both jobs - just adjust the pipeline to your needs.
 >
-> NOTE: Windows example does not use the latest version of detect-secrets. It is related to the bug in the detect-secret tool (see more in [Issue#452](https://github.com/Yelp/detect-secrets/issues/452)). It is highly recommended to monitor the fix for the issue and use the latest version if possible by removing version tag `==1.0.3` in the pip install command.
+> **Note:** Windows example does not use the latest version of detect-secrets. It is related to the bug in the detect-secret tool (see more in [Issue#452](https://github.com/Yelp/detect-secrets/issues/452)). It is highly recommended to monitor the fix for the issue and use the latest version if possible by removing version tag `==1.0.3` in the pip install command.
 
 ```yaml
 trigger:

docs/CI-CD/recipes/.pages

@@ -0,0 +1,6 @@
+nav:
+  - CD on low code solutions: cd-on-low-code-solutions.md
+  - CI pipeline for better documentation: ci-pipeline-for-better-documentation.md
+  - CI with jupyter notebooks: ci-with-jupyter-notebooks.md
+  - GitHub actions: github-actions
+  - ...

docs/continuous-integration/markdown-linting/README.md → docs/CI-CD/recipes/ci-pipeline-for-better-documentation.md

@@ -4,8 +4,7 @@
 
 Most projects start with spikes, where developers and analysts produce lots of documentation.
 
-Sometimes, these documents don't have a standard and each team member writes them accordingly with their preference. Add to that
-the time a reviewer will spend confirming grammar, searching for typos or non-inclusive language.
+Sometimes, these documents don't have a standard and each team member writes them accordingly with their preference. Add to that the time a reviewer will spend confirming grammar, searching for typos or non-inclusive language.
 
 This pipeline helps address that!
 

docs/continuous-integration/ci-in-data-science/working-with-notebooks/README.md → docs/CI-CD/recipes/ci-with-jupyter-notebooks.md

@@ -1,4 +1,4 @@
-# Data Science Pipeline
+# CI with Jupyter Notebooks
 
 As Azure DevOps doesn't allow code reviewers to comment directly in Jupyter Notebooks, Data Scientists(DSs) have
 to convert the notebooks to scripts before they commit and push these files to the repository.
@@ -41,7 +41,7 @@ We can add a pipeline with the following steps to the repository to run in `ipyn
 
 1. Go to the *Project Settings* -> *Repositories* -> *Security* -> *User Permissions*
 1. Add the *Build Service* in *Users* the permission to *Contribute*
-    ![Contribute](assets/repository-properties.png)
+    ![Contribute](../images/repository-properties.png)
 1. Create a new pipeline.
 
 In the newly created pipeline we add: