Merge branch 'update-db-migrations-docs' of github.com:bitwarden/cont…

…ributing-docs into ps/migrations

# Conflicts:
#	docs/contributing/database-migrations/index.md

docs/contributing/database-migrations/edd.mdx

@@ -1,16 +1,18 @@
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
 
-# Evolutionary Database Design
+import refactoringPhases from "./stages_refactoring.jpg";
+import eddStateMachine from "./edd_state_machine.jpg";
+
+# Evolutionary database design
 
 At Bitwarden we follow
 [Evolutionary Database Design (EDD)](https://en.wikipedia.org/wiki/Evolutionary_database_design).
 EDD describes a process where the database schema is continuously updated while still ensuring
 compatibility with older releases by using database transition phases.
 
 In short the Database Schema for the Bitwarden Server **must** support the previous release of the
-server. The database migrations will be performed before the code deployment, and in the event of a
-release rollback the database schema will **not** be updated.
+server at any given time.
 
 <bitwarden>
 
@@ -24,22 +26,67 @@ For background on this decision please see the [Evolutionary Database Design RFD
 
 ## Design
 
-### Nullable
+Two types of database changes exist: destructive and non-destructive \[[1](./edd#further-reading)\].
+A destructive change is any database change that requires an accompanying code change to continue
+working as expected. A non-destructive change is the opposite: a database change that does not
+require a code change to allow the application to continue working as expected.
+
+### Non-destructive database changes
+
+An example of a non-destructive change is almost always using either nullable fields or default
+values in database tables, views, and stored procedures. This as a standard for any such changes.
+This will allow stored procedures to omit columns, which is a requirement when running both old and
+new code.
+
+### Destructive changes
+
+The current release process couples database and code changes, even a new column can even be
+considered a destructive change if the default value of the column is a non-constant value that
+needs to be computed from elsewhere.
+
+Destructive database changes are handled elegantly by breaking them up into three phases: _Start_,
+_Transition_ and _End_.
+
+<div style={{ margin: "1em" }}>
+  <img src={refactoringPhases} alt="Refactoring Phases" />
+  <div style={{ fontSize: 12, textAlign: "center" }}>
+    Refactoring Phases [<a href="./edd#further-reading">1</a>]
+  </div>
+</div>
+
+The terminology is tweaked to be more easily understandable in relation to two types of deployment
+processes: always-on environments and offline update environments. The terms: _Initial_ Phase
+(instead of _Start_), _Transition_ Phase, and _Finalization_ Phase (instead of _End_).
 
-Database tables, views and stored procedures should almost always use either nullable fields or have
-a default value. Since this will allow stored procedures to omit columns, which is a requirement
-when running both old and new code.
+#### Initial phase
 
-### EDD Process
+- Compatible with _X.1.0_ **and** _X.2.0_ application code changes
+- Represents the beginning of a database change
+- Updates the database schema to support any new functionality while also maintaining old
+  functionality
+- Supports both the previous version of code and the one being upgraded to
+- Run during upgrade
+- Must execute quickly to minimize downtime.
 
-The EDD breaks up each database migration into three phases. _Start_, _Transition_ and _End_.
+#### Transition phase
 
-![Refactoring Stages](./stages_refactoring.jpg)
-[https://www.martinfowler.com/articles/evodb.html#TransitionPhase](https://www.martinfowler.com/articles/evodb.html#TransitionPhase)
+- Compatible with _X.1.0_ **and** _X.2.0_ application code changes
+- The time between initial migration and finalization
+- Exists to provide an opportunity to rollback server to _X.1.0_ version prior to breaking changes
+- Only data population migrations may be run at this time, if they are needed
+  - Optional step, required only when migrating data would be too slow to execute during the initial
+    migration. This might be a column population, index creation, anything to prepare the database
+    for the _X.2.0_ version
+  - Must be run as a background task during the Transition phase.
+  - These MUST run in a way where the database stays responsive during the full migration
+- Schema changes are NOT to be run during this phase.
 
-This necessitates two different database migrations. The first migration adds new content and is
-backwards compatible with the existing code. The second migration removes content and is not
-backwards compatible with that same code prior to the first migration.
+#### Finalization phase
+
+- Only compatible with _X.2.0_ application code; represents the point of no return for this
+  migration
+- Removes columns, data, and fallback code required to support _X.1.0_ version
+- Should be run as a typical migration either during a subsequent upgrade
 
 ### Example
 
@@ -73,7 +120,7 @@ actions.
 :::
 
 <Tabs>
-<TabItem value="first" label="First Migration" default>
+<TabItem value="first" label="Initial Migration" default>
 
 ```sql
 -- Add Column
@@ -120,7 +167,7 @@ END
 ```
 
 </TabItem>
-<TabItem value="data" label="Data Migration">
+<TabItem value="data" label="Transition Migration">
 
 ```sql
 UPDATE [dbo].Customer SET
@@ -129,7 +176,7 @@ WHERE FirstName IS NULL
 ```
 
 </TabItem>
-<TabItem value="second" label="Second Migration">
+<TabItem value="second" label="Finalization Migration">
 
 ```sql
 -- Remove Column
@@ -173,65 +220,95 @@ END
 </TabItem>
 </Tabs>
 
-## Workflow
+## Bitwarden EDD process
+
+There are some unique constraints to how Bitwarden has to implement this process.
+
+- Bitwarden Production environments are required to be on at all times
+- Self-host instances must support the same database change process; however, they do not have the
+  same always-on application constraint
+- Minimization of manual steps in the process
+
+The process to support all of these constraints is a complex one. Below is an image of a state
+machine that will hopefully help visualize the process and what it supports. It assumes that all
+database changes follow the standards that are laid out in [Migrations](./).
+
+---
+
+![Bitwarden EDD State Machine](./edd_state_machine.jpg) \[Open Image in a new tab for better
+viewing\]
+
+---
 
-The Bitwarden specific workflow for writing migrations are described below.
+### Online environments
 
-### Developer
+Schema migrations and data migrations as just migrations. The underlying implementation issue is
+orchestrating the runtime constraints on the migration. Eventually, all migrations will end up in
+`DbScripts`. However, to orchestrate the running of _Transition_ and associated _Finalization_
+migrations, they are kept outside of `DbScripts` until the correct timing.
 
-The development flow is described in [Migrations](./).
+In environments with always-on applications, _Transition_ scripts must be run after the new code has
+been rolled out. To execute a full deploy, all new migrations in `DbScripts` are run, the new code
+is rolled out, and then all _Transition_ migrations in the `DbScripts_transition` directory are run
+as soon as all of the new code services are online. In the case of a critical failure after the new
+code is rolled out, a Rollback would be conducted (see Rollbacks below). _Finalization_ migrations
+will not be run until the start of the next deploy when they are moved into `DbScripts`.
 
-### Devops
+After this deploy, to prep for the next release, all migrations in `DbScripts_transition` are moved
+to `DbScripts` and then all migrations in `DbScripts_finalization` are moved to `DbScripts`,
+conserving their execution order for a clean install. For the current branching strategy, PRs will
+be open against `master` when `rc` is cut to prep for this release. This PR automation will also
+handle renaming the migration file and updating any reference of `[dbo_finalization]` to `[dbo]`.
 
-#### On `rc` cut
+The next deploy will pick up the newly added migrations in `DbScripts` and set the previously
+repeatable _Transition_ migrations to no longer be repeatable, execute the _Finalization_
+migrations, and then execute any new migrations associated with the code changes that are about to
+go out.
 
-Create a PR moving the future scripts.
+The the state of migrations in the different directories at any one time is is saved and versioned
+in the Migrator Utility which supports the phased migration process in both types of environments.
 
-- `DbScripts_future` to `DbScripts`, prefix the script with the current date, but retain the
-  existing date.
-- `dbo_future` to `dbo`.
-  <bitwarden>
-    <li>
-      Create a ticket in Jira with a `Due Date` of the release date to ensure future migrations are
-      merged in and ready to be executed. Set the ticket that created the future migration as a
-      blocker.
-    </li>
-  </bitwarden>
+### Offline environments
 
-#### After server release
+The process for offline environments is similar to the always-on ones. However, since they do not
+have the constraint of always being on, the _Initial_ and _Transition_ migrations will be run one
+after the other:
 
-1. Run whatever data migration scripts might be needed. (This might need to be batched and executed
-   until all the data has been migrated)
-2. After having the server run for a while execute the future migration script to clean up the
-   database.
+- Stop the Bitwarden stack as done today
+- Start the database
+- Run all new migrations in `DbScripts` (both _Finalization_ migrations from the last deploy and any
+  _Initial_ migrations from the deploy currently going out)
+- Run all _Transition_ migrations
+- Restart the Bitwarden stack.
 
 ## Rollbacks
 
 In the event the server release failed and needs to be rolled back, it should be as simple as just
 re-deploying the previous version again. The database will **stay** in the transition phase until a
-hotfix can be released, and the server can be updated.
+hotfix can be released, and the server can be updated. Once a hotfix is ready to go out, it is
+deployed the _Transition_ migrations are rerun to verify that the DB is in the state that it is
+required to be in.
 
-The goal is to resolve the issue quickly and re-deploy the fixed code to minimize the time the
-database stays in the transition phase. Should a feature need to be completely pulled, a new
-migration needs to be written to undo the database changes and the future migration will also need
-to be updated to work with the database changes. This is generally not recommended since pending
-migrations (for other releases) will need to be revisited.
+Should a feature need to be completely pulled, a new migration needs to be written to undo the
+database changes and the future migration will also need to be updated to work with the database
+changes. This is generally not recommended since pending migrations (for other releases) will need
+to be revisited.
 
 ## Testing
 
 Prior to merging a PR please ensure that the database changes run well on the currently released
 version. We currently do not have an automated test suite for this and it’s up to the developers to
 ensure their database changes run correctly against the currently released version.
 
-## Further Reading
+## Further reading
 
-- [Evolutionary Database Design](https://martinfowler.com/articles/evodb.html) (Particularly
-  [All database changes are database refactorings](https://martinfowler.com/articles/evodb.html#AllDatabaseChangesAreMigrations))
-- [The Agile Data (AD) Method](http://agiledata.org/) (Particularly
-  [Catalog of Database Refactorings](http://agiledata.org/essays/databaseRefactoringCatalog.html))
-- [Refactoring Databases: Evolutionary Database](https://databaserefactoring.com/)
-- Refactoring Databases: Evolutionary Database Design (Addison-Wesley Signature Series (Fowler))
-  ISBN-10: 0321774515
+1. [Evolutionary Database Design](https://martinfowler.com/articles/evodb.html) (Particularly
+   [All database changes are database refactorings](https://martinfowler.com/articles/evodb.html#AllDatabaseChangesAreMigrations))
+2. [The Agile Data (AD) Method](http://agiledata.org/) (Particularly
+   [Catalog of Database Refactorings](http://agiledata.org/essays/databaseRefactoringCatalog.html))
+3. [Refactoring Databases: Evolutionary Database](https://databaserefactoring.com/)
+4. Refactoring Databases: Evolutionary Database Design (Addison-Wesley Signature Series (Fowler))
+   ISBN-10: 0321774515
 
 [edd-rfd]:
   https://bitwarden.atlassian.net/wiki/spaces/PIQ/pages/177701412/Adopt+Evolutionary+database+design

docs/contributing/database-migrations/mssql.md

@@ -11,7 +11,7 @@ For instructions on how to apply database migrations, please refer to the
 
 :::
 
-## SQL Database project
+## SQL database project
 
 :::warning
 
@@ -50,13 +50,13 @@ Since we follow [Evolutionary Database Design _(EDD)_](./edd.mdx), any migration
 existing columns most likely needs to be split into at least two parts: a backwards compatible
 transition phase, and a non-backwards compatible phase.
 
-### Best Practices
+### Best practices
 
 When writing a migration script there are a couple of best practices we follow. Please check the
 [T-SQL Code Style][code-style-sql] for more details. But the most important aspect is ensuring the
 script can be re-run on the database multiple times without producing any errors or data loss.
 
-### Backwards Compatible
+### Backwards compatible
 
 Since we follow _EDD_ the first migration needs to retain backwards compatibility with existing
 production code.
@@ -71,14 +71,14 @@ ensure the existing sprocs first checks the new location before falling back to
 also need to ensure we continue updating the old data columns, since in case a rollback is necessary
 no data should be lost.
 
-### Data Migration
+### Data migration
 
 We now need to write a script that migrates any data from the old location to the new locations.
 This script should ideally be written in a way that supports batching, i.e. execute for X number of
 rows at a time. This helps avoiding locking the database. When running the scripts against the
 server please keep running it until it affects `0 rows`.
 
-### Non-backwards Compatible
+### Non-backwards compatible
 
 These changes should be written from the perspective of "all data has been migrated". And any old
 _Stored Procedures_ that were kept around for backwards compatibility should be removed. Any logic
@@ -87,15 +87,26 @@ for syncing old and new data should also be removed in this step.
 Since `Sql/dbo` represents the current state we need to introduce a "future" state which we will
 call `dbo_future`.
 
-1. Copy the relevant `.sql` files from `src/Sql/dbo` to `src/Sql/dbo_future`.
+1. Copy the relevant `.sql` files from `src/Sql/dbo` to `src/Sql/dbo_finalization`.
 2. Remove the backwards compatibility which is no longer needed.
-3. Write a new Migration and place it in `src/Migrator/DbScripts_future`, name it
-   `YYYY-0M-FutureMigration.sql`.
+3. Write a new Migration and place it in `src/Migrator/DbScripts_finalization`, name it
+   `YYYY-0M-FinalizationMigration.sql`.
    - Typically migrations are designed to be run in sequence. However since the migrations in
      DbScripts_future can be run out of order, care must be taken to ensure they remain compatible
      with the changes to DbScripts. In order to achieve this we only keep a single migration, which
      executes all backwards incompatible schema changes.
 
+### [Not Yet Implemented] Manual MSSQL migrations
+
+There may be a need for a migration to be run outside of our normal update process. These types of
+migrations should be saved for very exceptional purposes. One such reason could be an Index rebuild.
+
+1. Write a new Migration with a prefixed current date and place it in
+   `src/Migrator/DbScripts_manual`
+2. After it has been run against our Cloud environments and we are satisfied with the outcome,
+   create a PR to move it to `DbScripts`. This will enable it to be run by our Migrator processes in
+   self-host and clean installs of both cloud and self-host environments
+
 [repository]:
   https://docs.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/infrastructure-persistence-layer-design
 [dapper]: https://github.com/DapperLib/Dapper