Fix layout issues and add overview pages that are missing in some sec…

…tions (#1012)

* Fix layout

* more cleanup

* add explaining text.

* updated readme.md with directory of folder.
The 3 docs could also be merged instead

* fixe a few files

* fix a few spelling issues

* ignore some acronyms

* fix paths

* fixing broken paths

* fix a few more linking issues

* fix links

* undo curly brackets

* fix two linting errors

---------

Co-authored-by: Mark Jervelund <[email protected]>
Co-authored-by: Shiran Rubin <[email protected]>

.github/workflows/mega-linter.yml

@@ -47,7 +47,7 @@ jobs:
 
       # Upload MegaLinter artifacts
       - name: Archive production artifacts
-        if: ${{ success() }} || ${{ failure() }}
+        if: always()
         uses: actions/upload-artifact@v2
         with:
           name: MegaLinter reports

.mega-linter.yml

@@ -14,7 +14,6 @@ ENABLE_LINTERS:
   - YAML_YAMLLINT
   - MDLINKCHECK_LINKCHECK
 
-
 SPELL_CSPELL_DISABLE_ERRORS: true
 MARKDOWN_MARKDOWN_LINK_CHECK_DISABLE_ERRORS: true
 SHOW_ELAPSED_TIME: true

docs/SPRINT-STRUCTURE.md

@@ -35,7 +35,7 @@ The purpose of this document is to:
   - Agree on how to separate unit tests from integration, load and smoke tests
   - Design the first test cases
 - [ ] [Decide on branch naming](source-control/naming-branches.md)
-- [ ] [Discuss security needs and verify that secrets are kept out of source control](continuous-delivery/azure-devops/secret-management-per-branch.md)
+- [ ] [Discuss security needs and verify that secrets are kept out of source control](continuous-delivery/gitops/secret-management/azure-devops-secret-management-per-branch.md)
 
 ### Day 2
 

docs/automated-testing/e2e-testing/recipes/README.md

@@ -0,0 +1,4 @@
+# Templates
+
+- [Gauge Framework](gauge-framework.md)
+- [Postman](postman-testing.md)

docs/automated-testing/performance-testing/load-testing.md

@@ -97,6 +97,7 @@ Here are a few popular load testing frameworks you may consider, and the languag
 In the case where a specific workload application is not being provided and the focus is instead on the system, here are a few popular sample workload applications you may consider.
 
 - **HttpBin** ([Python](https://github.com/postmanlabs/httpbin), [GoLang](https://github.com/mccutchen/go-httpbin)) - Supports variety of endpoint types and language implementations. Can echo data used in request.
+<!-- // cSpell:ignore NGSA, IMDB -->
 - **NGSA** ([Java](https://github.com/retaildevcrews/ngsa-java), [C#](https://github.com/retaildevcrews/ngsa-java)) - Intended for Kubernetes Platform and Monitoring Testing. Built on top of IMDB data store with many CRUD endpoints available. Does not need to have a live database connection.
 - **MockBin** (<https://github.com/Kong/mockbin>) - Allows you to generate custom endpoints to test, mock, and track HTTP requests & responses between libraries, sockets and APIs.
 

docs/automated-testing/tech-specific-samples/README.md

@@ -0,0 +1,4 @@
+# Tech specific samples
+
+- [azdo-container-dev-test-release](azdo-container-dev-test-release/README.md)
+- [blobstorage-unit-tests](blobstorage-unit-tests/README.md)

docs/automated-testing/templates/README.md

@@ -0,0 +1,4 @@
+# Templates
+
+- [case-study-template](./case-study-template.md)
+- [test-type-template](./test-type-template.md)

docs/code-reviews/faq.md

@@ -37,8 +37,7 @@ Slow code reviews might cause delays in delivering features and cause frustratio
 Checkout the [Tools](./tools.md) for help on how to perform reviews out of Visual Studio or Visual Studio Code.
 
 ## How can we enforce code review policies?
-
-[By configuring Branch Policies](tools.md#Configuring Branch Policies), you can easily enforce code reviews rules.
+By configuring [Branch Policies](./tools.md#Configuring-Branch-Policies) , you can easily enforce code reviews rules.
 
 ## We pair or mob. How should this reflect in our code reviews?
 

docs/code-reviews/pull-requests.md

@@ -22,7 +22,7 @@ The requirements of pull requests can and should be enforced by policies, which
     * Ensure the code compiles and runs without errors or warnings
     * Write and/or update tests to cover the changes and make sure all new and existing tests pass
     * Write and/or update the documentation to match the changes
-1. Once convinced the criteria above are met, create and submit a new pull request adhering to the [pull request template](pull-request-template/pull-request-template.md)
+1. Once convinced the criteria above are met, create and submit a new pull request adhering to the [pull request template](./pull-request-template.md)
 1. Follow the [code review](./process-guidance/README.md) process to merge the changes to the main codebase
 
 The following diagram illustrates this approach.
@@ -78,7 +78,7 @@ One popular specification for open-source projects and others is the [Convention
 
 The `<type>` in this message can be selected from a list of types defined by the team, but many projects use the [list of commit types from the Angular open-source project](https://github.com/angular/angular/blob/22b96b9/CONTRIBUTING.md#type). It should be clear that `scope`, `body` and `footer` elements are **optional**, but having a required `type` and short description enables the features mentioned above.
 
-See also [Pull Request Template](pull-request-template/pull-request-template.md)
+See also [Pull Request Template](./pull-request-template.md)
 
 ## Resources
 

docs/continuous-delivery/README.md

@@ -139,7 +139,7 @@ Try to limit the number of versions of your application running parallel in prod
 
 Low code solutions have increased their participation in the applications and processes and because of that it is required that a proper conjunction of disciplines improve their development.
 
-Here is a guide for [continuous deployment for Low Code Solutions](low-code-solutions/low-code-solutions.md).
+Here is a guide for [continuous deployment for Low Code Solutions](low-code-solutions/README.md).
 
 ## References
 

docs/continuous-delivery/azure-devops/secret-management-per-branch.md → docs/continuous-delivery/gitops/secret-management/azure-devops-secret-management-per-branch.md

@@ -1,12 +1,12 @@
 # Azure DevOps: Managing Settings on a Per-Branch Basis
 
-When using [Azure DevOps Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) for CI/CD, it's convenient to leverage the built-in [pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables) for [secrets management](../secrets-management/README.md), but using pipeline variables for secrets management has its disadvantages:
+When using [Azure DevOps Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) for CI/CD, it's convenient to leverage the built-in [pipeline variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables) for [secrets management](./README.md), but using pipeline variables for secrets management has its disadvantages:
 
 - *Pipeline variables are managed outside the code that references them.* This makes it easy to introduce drift between the source code and the secrets, e.g. adding a reference to a new secret in code but forgetting to add it to the pipeline variables (leads to confusing build breaks), or deleting a reference to a secret in code and forgetting to remote it from the pipeline variables (leads to confusing pipeline variables).
 
 - *Pipeline variables are global shared state.* This can lead to confusing situations and hard to debug problems when developers make concurrent changes to the pipeline variables which may override each other. Having a single global set of pipeline variables also makes it impossible for secrets to vary per environment (e.g. when using a branch-based deployment model where 'master' deploys using the production secrets, 'development' deploys using the staging secrets, and so forth).
 
-A solution to these limitations is to manage secrets in the Git repository jointly with the project's source code. As described in [secrets management](../secrets-management/README.md), don't check secrets into the repository in plain text. Instead we can add an encrypted version of our secrets to the repository and enable our CI/CD agents and developers to decrypt the secrets for local usage with some pre-shared key. This gives us the best of both worlds: a secure storage for secrets as well as side-by-side management of secrets and code.
+A solution to these limitations is to manage secrets in the Git repository jointly with the project's source code. As described in [secrets management](README.md), don't check secrets into the repository in plain text. Instead we can add an encrypted version of our secrets to the repository and enable our CI/CD agents and developers to decrypt the secrets for local usage with some pre-shared key. This gives us the best of both worlds: a secure storage for secrets as well as side-by-side management of secrets and code.
 
 ```sh
 # first, make sure that we never commit our plain text secrets and generate a strong encryption key

docs/continuous-delivery/recipes/README.md

@@ -0,0 +1,11 @@
+# Recipes
+
+## Github
+
+- [Github workflows](./github-workflows/README.md)
+
+## Terraform
+
+- [Save output to variable group](./terraform/save-output-to-variable-group.md)
+- [Share common variables naming conventions](./terraform/share-common-variables-naming-conventions.md)
+- [Terraform structure guidelines](./terraform/terraform-structure-guidelines.md)

docs/continuous-delivery/recipes/terraform/README.md

@@ -0,0 +1,5 @@
+# Terraform recipes
+
+- [Save output to variable group](./save-output-to-variable-group.md)
+- [Share common variables naming conventions](./share-common-variables-naming-conventions.md)
+- [Terraform structure guidelines](./terraform-structure-guidelines.md)

docs/continuous-delivery/secrets-management/README.md

@@ -19,7 +19,7 @@ the Azure CLI to do the same is a useful time-saving utility. See [az webapp con
 
 It's best practice to maintain separate secrets configurations for each environment that you run. e.g. dev, test, prod, local etc
 
-The [secrets-per-branch recipe](./../azure-devops/secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment.
+The [secrets-per-branch recipe](../gitops/secret-management/azure-devops-secret-management-per-branch.md) describes a simple way to manage separate secrets configurations for each environment.
 
 > Note: even if the secret was only pushed to a feature branch and never merged, it's still a part of the git history. Follow [these instructions](https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository) to remove any sensitive data and/or regenerate any keys and other sensitive information added to the repo. If a key or secret made it into the code base, rotate the key/secret so that it's no longer active
 

docs/continuous-integration/dev-sec-ops/README.md

@@ -15,6 +15,6 @@ Topics covered:
 
 1. [Credential Scanning](./secret-management/credential_scanning.md) - automatically inspecting a project to ensure that no secrets are included in the project's source code.
 1. [Secrets Rotation](./secret-management/secrets_rotation.md) - automated process by which the secret, used by the application, is refreshed and replaced by a new secret.
-1. [Static Code Analysis](./static-code-analysis/static_code_analysis.md) - analyze source code or compiled versions of code to help find security flaws.
-1. [Penetration Testing](./penetration-testing/penetration_testing.md) - a simulated attack against your application to check for exploitable vulnerabilities.
-1. [Container Dependencies Scanning](./dependency-container-scanning/dependency_container_scanning.md) - search for vulnerabilities in container operating systems, language packages and application dependencies.
+1. [Static Code Analysis](./secret-management/static-code-analysis.md) - analyze source code or compiled versions of code to help find security flaws.
+1. [Penetration Testing](./penetration-testing/README.md) - a simulated attack against your application to check for exploitable vulnerabilities.
+1. [Container Dependencies Scanning](./dependency-container-scanning/README.md) - search for vulnerabilities in container operating systems, language packages and application dependencies.

docs/continuous-integration/dev-sec-ops/azure-devops/README.md

@@ -0,0 +1,8 @@
+# Azure DevOps
+
+Write something about Azure DevOps here.
+
+
+## Table of Contents
+
+- [Service connection security](./service-connection-security.md)

docs/continuous-integration/dev-sec-ops/secret-management/README.md

@@ -0,0 +1,27 @@
+# Secret management
+
+Secret Management refers to the tools and practices used to manage digital authentication credentials (like API keys, tokens, passwords, and certificates). These secrets are used to protect access to sensitive data and services, making their management critical for security.
+
+## Importance of Secret Management
+
+In modern software development, applications often need to interact with other software components, APIs, and services. These interactions often require authentication, which is typically handled using secrets. If these secrets are not managed properly, they can be exposed, leading to potential security breaches.
+
+## Best Practices for Secret Management
+
+1. **Centralized Secret Storage:** Store all secrets in a centralized, encrypted location. This reduces the risk of secrets being lost or exposed.
+
+2. **Access Control:** Implement strict access control policies. Only authorized entities should have access to secrets.
+
+3. **Rotation of Secrets:** Regularly change secrets to reduce the risk if a secret is compromised.
+
+4. **Audit Trails:** Keep a record of when and who accessed which secret. This can help in identifying suspicious activities.
+
+5. **Automated Secret Management:** Automate the processes of secret creation, rotation, and deletion. This reduces the risk of human error.
+
+Remember, the goal of secret management is to protect sensitive information from unauthorized access and potential security threats.
+
+## Pages
+
+- [Credential Scanning](./credential_scanning.md)
+- [Secrets Rotation](./secrets_rotation.md)
+- [Static code analysis](./static-code-analysis.md)

docs/continuous-integration/dev-sec-ops/secret-management/credential_scanning.md

@@ -21,7 +21,7 @@ To implement credential scanning for a project, consider the  following:
 
 ## Credential Scanning Frameworks and Tools
 
-Recipes and Scenarios-
+Recipes and Scenarios -
 
 1. [detect-secrets](./recipes/detect-secrets.md) is an aptly named module for detecting secrets within a code base.
 1. Use [detect-secrets inside Azure DevOps Pipeline](./recipes/detect-secrets-ado.md)

docs/continuous-integration/dev-sec-ops/secret-management/recipes/README.md

@@ -0,0 +1,4 @@
+# Recipes
+
+- [Detect secrets](./detect-secrets.md)
+- [Detect secrets on Azure DevOps](./detect-secrets-ado.md)

docs/design/exception-handling/readme.md

@@ -1,6 +1,8 @@
+# Exception handling
+
 ## Exception constructs
 
-Almost all language platforms offer a construct of exception or equivalent to handle error scenarios. The underlying platform, used libraries or the authored code can "throw" exceptions to initiate an error flow. Some of the advantages of using exceptions are - 
+Almost all language platforms offer a construct of exception or equivalent to handle error scenarios. The underlying platform, used libraries or the authored code can "throw" exceptions to initiate an error flow. Some of the advantages of using exceptions are -
 
 1. Abstract different kind of errors
 2. Breaks the control flow from different code structures
@@ -17,16 +19,16 @@ Here is some guidance on exception handling in .Net
 
 ## Custom exceptions
 
-Although the platform offers numerous types of exceptions, often we need custom defined exceptions to arrive at an optimal low level design for error handling. The advantages of using custom exceptions are - 
+Although the platform offers numerous types of exceptions, often we need custom defined exceptions to arrive at an optimal low level design for error handling. The advantages of using custom exceptions are -
 
 1. Define exceptions specific to business domain of the requirement. E.g. InvalidCustomerException
-2. Wrap system/platform exceptions to define more generic system exception so that overall code base is more tech stack agnostic. E.g DatabaseWriteException which wraps MongoWriteException. 
-3. Enrich the exception with more information about the code flow of the error. 
+2. Wrap system/platform exceptions to define more generic system exception so that overall code base is more tech stack agnostic. E.g DatabaseWriteException which wraps MongoWriteException.
+3. Enrich the exception with more information about the code flow of the error.
 4. Enrich the exception with more information about the data context of the error. E.g. RecordId in property in DatabaseWriteException which carries the Id of the record failed to update.
-5. Define custom error message which is more business user friendly or support team friendly. 
+5. Define custom error message which is more business user friendly or support team friendly.
 
 ### Custom exception hierarchy
-Below diagram shows a sample hierarchy of custom exceptions. 
+Below diagram shows a sample hierarchy of custom exceptions.
 
 1. It defines a BaseException class which derives from System.Exception class and parent of all custom exceptions. BaseException also has additional properties for ActionCode and ResultCode. ActionCode represents the "flow" in which the error happened. ResultCode represents the exact error that happened. These additional properties help in defining different error handling flows in the catch blocks.  
 2. Defines a number of System exceptions which derive from SystemException class. They will address all the errors generated by the technical aspects of the code. Like connectivity, read, write, buffer overflow etc
@@ -35,7 +37,7 @@ Below diagram shows a sample hierarchy of custom exceptions.
 ![ClassDiagram1](https://github.com/SudhirChandra/code-with-engineering-playbook/assets/23739807/1234e529-67ab-4a14-8f7d-fc5c41006015)
 
 ## Error details in API response
-When an error occurs in an API, it has to rendered as response with all the necessary fields. There can be custom response schema drafted for these purposes. But one of the popular formats is the problem detail structure - 
+When an error occurs in an API, it has to rendered as response with all the necessary fields. There can be custom response schema drafted for these purposes. But one of the popular formats is the problem detail structure -
 
 [Problem details](https://datatracker.ietf.org/doc/html/rfc7807)
 

docs/developer-experience/README.md

@@ -100,7 +100,7 @@ Assign hotkeys to each of the essential tasks.
 
 The F5 contract aims for the ability to run the end-to-end solution with the following steps.
 
-1. Clone - git clone [`my-repo-url-here`]
+1. Clone - git clone \[`my-repo-url-here`\]
 2. Configure - set any configuration values that need to be unique to the individual (i.e. update a .env file)
 3. Press F5 - launch the solution with debugging attached.
 
@@ -146,11 +146,11 @@ Automate the ability to execute each essential task across all solution componen
 
 When this is not implemented, the engineers must repeat each of the essential tasks manually for each component in the solution. In this situation, the number of steps required to perform each essential task is multiplied by the number of components in the system
 
-[Configuration steps + Build steps + Start/Debug steps + Stop steps + Run test steps + Documenting all of the above] * [many solution components] = TOO MANY STEPS
+\[Configuration steps + Build steps + Start/Debug steps + Stop steps + Run test steps + Documenting all of the above\] * \[many solution components\] = TOO MANY STEPS
 
 VS.
 
-[Configuration steps + Build steps + Start/Debug steps + Stop steps + Run test steps + Documenting all of the above] * [1 solution] = MINIMUM NUMBER OF STEPS
+\[Configuration steps + Build steps + Start/Debug steps + Stop steps + Run test steps + Documenting all of the above\] * \[1 solution\] = MINIMUM NUMBER OF STEPS
 
 ### Observability
 
@@ -162,11 +162,11 @@ There are many observability strategies a developer can use alongside best engin
 
 Splitting a solution across multiple repositories can negatively impact the above measures. This can also negatively impact other areas such as Pull Requests, Automated Testing, Continuous Integration, and Continuous Delivery. Similar to the IDE instances, the negative impact is multiplied by the number of repositories.
 
-[Clone steps + Branching steps + Commit steps + CI steps + Pull Request reviews & merges ] * [many source code repositories] = TOO MANY STEPS
+\[Clone steps + Branching steps + Commit steps + CI steps + Pull Request reviews & merges \] * \[many source code repositories\] = TOO MANY STEPS
 
 VS.
 
-[Clone steps + Branching steps + Commit steps + CI steps + Pull Request reviews & merges ] * [1 source code repository] = MINIMUM NUMBER OF STEPS
+\[Clone steps + Branching steps + Commit steps + CI steps + Pull Request reviews & merges \] * \[1 source code repository\] = MINIMUM NUMBER OF STEPS
 
 #### Atomic Pull Requests
 

docs/documentation/best-practices/README.md

@@ -0,0 +1,5 @@
+# Best Practices
+
+- [Replacing Documentation with Automation](automation.md)
+- [Establishing and Managing Documentation](establish-and-manage.md)
+- [Creating Good Documentation](good-documentation.md)