feat: refine ISLM asset naming

udi-prime/src/main/postgres/islm/README.md

@@ -1,10 +1,21 @@
 
 # Information Schema Lifecycle Manager (ISLM)
 
-## Overview
 The Information Schema Lifecycle Manager (ISLM) is a PostgreSQL-native schema migration system designed to manage and automate database schema evolution. By encapsulating schema evolution (migration) logic within PostgreSQL stored routines, ISLM provides a robust and manageable approach to database schema evolution. It operates within a dedicated schema named `info_schema_lifecycle` and uses stateless helper functions (`islm_*`) to help DBAs orchestrate migrations based on naming conventions.
 
-## Significant Features
+The ISLM infrastructure is designed not to actually execute SQL but to provide state management and helper functions that generate SQL which can then be executed. DBAs review the generated code and execute the scripts. This design ensures that ISLM is as safe as possible, providing helpers to prepare SQL but not execute it directly.
+
+ISLM helps with state management by maintaining the lifecycle and status of migration routines, ensuring that all schema changes are tracked and managed effectively.
+
+Benefits:
+- Simplicity: ISLM is simple and straightforward to use, operating entirely within PostgreSQL without the need for external tools.
+- PostgreSQL-Native: ISLM leverages PostgreSQL's native functionality, ensuring compatibility and ease of use within PostgreSQL environments.
+- Revisionable: Since each migration is just a stored procedure, schema evoluation can be revisioned via Git.
+- Focus on Documentation: ISLM's design allows teams to focus on clear and thorough documentation of database schema evolution. 
+
+There is an ISLM "Control Plane" Deno TypeScript CLI utility called `islmctl.ts`, which is essentially a convenience wrapper around `psql` and all the ISLM stored procedures.
+
+## Significant Components
 - **Schema and Bootstrap Procedure**: 
   - The `info_schema_lifecycle` schema contains all ISLM-related objects.
   - The `islm_bootstrap_idempotent` procedure creates essential tables: `islm_infrastructure_artifact` and `islm_governance`.
@@ -17,61 +28,28 @@ The Information Schema Lifecycle Manager (ISLM) is a PostgreSQL-native schema mi
 
 - **Instruction and Script Generation**:
   - `islm_migration_instruction`: Discovers migration routines and generates instructions for their execution, based on the current state and idempotency.
-  - `islm_migration_script`: Generates a SQL script for executing migration routines, based on the instructions from `islm_migration_instruction`.
-
-## Benefits
-- **Simplicity**: ISLM is simple and straightforward to use, operating entirely within PostgreSQL without the need for external tools.
-- **PostgreSQL-Native**: ISLM leverages PostgreSQL's native functionality, ensuring compatibility and ease of use within PostgreSQL environments.
-- **Focus on Documentation**: ISLM's design allows teams to focus on clear and thorough documentation of database schema evolution.
-
-## Analysis of Alternatives
-
-### Flyway
-- **Versioned Migrations**: Each migration has a version, a description, and a checksum. Migrations can be written in SQL or Java.
-- **Checksum Validation**: Flyway validates on startup that the migrations applied to the database match the ones available locally.
-- **Repair Functionality**: Provides a repair command to correct the schema history table if needed.
-- **Callbacks**: Custom operations can be performed before/after each migration, or before/after all migrations.
-- **Configuration Options**: Extensive configuration options available through files, environment variables, and command-line arguments.
-- **Cross-Team Development**: Supports working in a team environment with branching and merging.
-
-### Liquibase
-- **XML, YAML, JSON, and SQL Formats**: Changelog files can be written in various formats.
-- **Database-Agnostic Syntax**: Offers a database-agnostic syntax for changesets, which are translated into database-specific SQL.
-- **Changeset Execution**: Changesets can include preconditions, rollback code, and can be executed in transactions.
-- **Database Refactoring**: Manages a sequence of changes to the database schema, including complex refactoring.
-- **Command Line and Maven/Gradle Integration**: Offers command-line tools and integrates with Maven and Gradle.
-
-### ISLM
-- **Simplicity**: ISLM is simple and straightforward to use. Instead of creating an external strategy for PostgreSQL, ISLM does everything inside Postgres.
-
-### Summary
-- **Flyway**: Best for projects needing versioned migrations, checksum validation, and extensive configuration options, particularly in Java-based environments.
-- **Liquibase**: Ideal for projects requiring a database-agnostic approach, multiple changelog formats, and integration with build tools like Maven and Gradle.
-- **ISLM**: Best for teams looking for a straightforward solution that operates entirely within PostgreSQL without the need for external tools.
+  - **`islm_migration_script`**: Generates a SQL script for executing migration routines, based on the instructions from `islm_migration_instruction`. `islm_migration_script` will be the script you use most often.
 
 ## Usage
 To start using ISLM, follow these steps:
 
-### 1. Load the islm-prime.psql File
+### 1. Load the islm-driver.psql File
 Use the `psql` command-line tool to load the islm-prime.psql file into your PostgreSQL database:
 
 ```sh
-psql -f islm-prime.psql
+psql -f islm-driver.psql
 ```
 
-### 2. Bootstrap ISLM
-Run the `islm_bootstrap_idempotent` procedure to create the necessary tables and objects:
+You can use `islmctl.ts evolve up` (see below) or `psql` directly.
 
-```sql
-CALL "info_schema_lifecycle"."islm_bootstrap_idempotent"();
-```
+There's a special stored procedure called `islm_bootstrap_idempotent`, which runs automatically when `islm-driver.psql` is loaded, to create the necessary tables and objects. Search for `islm_bootstrap_idempotent` in `islm-driver.psql` to learn more.
 
-### 3. Define Migration Routines
+### 2. Define Migration Routines
 Follow the naming conventions to define your migration routines. The expected naming conventions are:
 - `migrate_vYYYY_MM_DD_HH_MM_stateful_XYZ`
 - `migrate_vYYYY_MM_DD_HH_MM_idempotent_XYZ`
 
-### 4. Use Helper Functions to Manage Migrations
+### 3. Use Helper Functions to Manage Migrations
 By default, all helper functions use the `info_schema_lifecycle` schema and other default arguments. You can pass in other arguments as appropriate.
 
 #### Discover Migration Routines
@@ -88,13 +66,6 @@ To get the migration status for each candidate routine:
 SELECT * FROM "info_schema_lifecycle"."migration_routine_state"();
 ```
 
-#### Generate Migration Instructions
-To discover migration routines and generate instructions for their execution:
-
-```sql
-SELECT * FROM "info_schema_lifecycle"."islm_migration_instruction"();
-```
-
 #### Generate Migration Script
 To generate a SQL script for executing migration routines:
 
@@ -104,5 +75,46 @@ SELECT "info_schema_lifecycle"."islm_migration_script"();
 
 For detailed usage and examples, refer to the source code and comments within the islm-prime.psql file.
 
+### Using `islmctl.ts`
+The `islmctl.ts` utility can be used to manage schema migrations more conveniently than using `psql` alone. The following `islmctl.ts evolve` subcommands are available:
+
+- `up`: Use `psql` to load migration stored procedures into the database.
+- `candidates`: Use `psql` to check for migration candidates (stored procedures).
+- `state`: Use `psql` to get the migration state for each candidate routine.
+- `script`: Use `psql` to generate migration scripts based on the current state.
+- `test`: Use `psql` to execute test scripts for ISLM infrastructure.
+- `omnibus-fresh`: Freshen the given connection ID by dropping and recreating the schema. This is a dangerous subcommand and should only be used in a sandbox environemtn.
+
+To see the commands available in `islmctl.ts evolve`, use the following command:
+
+```sh
+islmctl.ts evolve --help
+```
+
+## Analysis of Alternatives
+
+### Flyway
+- **Versioned Migrations**: Each migration has a version, a description, and a checksum. Migrations can be written in SQL or Java.
+- **Checksum Validation**: Flyway validates on startup that the migrations applied to the database match the ones available locally.
+- **Repair Functionality**: Provides a repair command to correct the schema history table if needed.
+- **Callbacks**: Custom operations can be performed before/after each migration, or before/after all migrations.
+- **Configuration Options**: Extensive configuration options available through files, environment variables, and command-line arguments.
+- **Cross-Team Development**: Supports working in a team environment with branching and merging.
+
+### Liquibase
+- **XML, YAML, JSON, and SQL Formats**: Changelog files can be written in various formats.
+- **Database-Agnostic Syntax**: Offers a database-agnostic syntax for changesets, which are translated into database-specific SQL.
+- **Changeset Execution**: Changesets can include preconditions, rollback code, and can be executed in transactions.
+- **Database Refactoring**: Manages a sequence of changes to the database schema, including complex refactoring.
+- **Command Line and Maven/Gradle Integration**: Offers command-line tools and integrates with Maven and Gradle.
+
+### ISLM
+- **Simplicity**: ISLM is simple and straightforward to use. Instead of creating an external strategy for PostgreSQL, ISLM does everything inside Postgres.
+
+### Summary
+- **Flyway**: Best for projects needing versioned migrations, checksum validation, and extensive configuration options, particularly in Java-based environments.
+- **Liquibase**: Ideal for projects requiring a database-agnostic approach, multiple changelog formats, and integration with build tools like Maven and Gradle.
+- **ISLM**: Best for teams looking for a straightforward solution that operates entirely within PostgreSQL without the need for external tools.
+
 ## License
 This project is licensed under the MIT License - see the LICENSE file for details.

udi-prime/src/main/postgres/islm/islm-driver.psql

@@ -0,0 +1,4 @@
+-- setup ISLM infrastructure (tables and stored routines)
+\ir islm-infrastructure.psql
+
+-- add all files that are used by the app/service

udi-prime/src/main/postgres/islm/islm-prime-test.psql → udi-prime/src/main/postgres/islm/islm-infrastructure-test.psql

@@ -278,6 +278,10 @@ BEGIN
     CALL info_schema_lifecycle_assurance.islm_test_structs();
     CALL info_schema_lifecycle_assurance.islm_test_candidates();
     CALL info_schema_lifecycle_assurance.islm_test_migrations_unapplied();
+
+    -- TODO: add more test cases
+    -- CALL info_schema_lifecycle_assurance.islm_test_migrations_partial_applied();
+    -- CALL info_schema_lifecycle_assurance.islm_test_migrations_all_applied();
 
     -- Additional tests can be added here in future
 END;

udi-prime/src/main/postgres/islm/islmctl.ts

@@ -101,11 +101,11 @@ const CLI = new Command()
   })
   .command("evolve", new Command()
     .description("ISLM Schema Evoluation Handler")    
-    .command("up", "Use psql to execute migration scripts")
+    .command("up", "Use psql to load migration scripts into the database")
       .option("-s, --src <path:string>", "Source location for artifacts", { required: true, default: path.fromFileUrl(import.meta.resolve("./")) })
       .option("-t, --target <path:string>", "Target location for generated artifacts", { required: true, default: cleanableTarget("/evolve") })
-      .option("--destroy-fname <file-name:string>", "Filename of the generated destroy script in target", { default: "islm-prime-destroy.psql" })
-      .option("--driver-fname <file-name:string>", "Filename of the generated construct script in target", { default: "islm-prime.psql" })
+      .option("--destroy-fname <file-name:string>", "Filename of the generated destroy script in target", { default: "islm-infrastructure-destroy.psql" })
+      .option("--driver-fname <file-name:string>", "Filename of the generated construct script in target", { default: "islm-driver.psql" })
       .option("--psql <path:string>", "`psql` command", { required: true, default: "psql" })
       .option("--destroy-first", "Destroy objects before migration")
       .option("--log-results <path:string>", "Store `psql` results in this log file", { default: `./islmctl-migrate-${new Date().toISOString()}.log` })
@@ -156,11 +156,32 @@ const CLI = new Command()
           console.error(`WARNING: ${psqlErrors} ${options.psql} error(s) occurred, see log file ${options.logResults}`);
         }
       })
-    .command("test", "Use psql to execute test scripts")
+    .command("candidates", "Use psql to check for migration candidates (stored procedures)")
+      .option("--psql <path:string>", "`psql` command", { required: true, default: "psql" })
+      .option("-c, --conn-id <id:string>", "pgpass connection ID to use for psql", { required: true, default: "UDI_PRIME_DESTROYABLE_DEVL" })
+      .action(async (options) => {
+        const psqlCreds = pgpassPsqlArgs(options.connId);
+        console.log((await $.raw`${options.psql} ${psqlCreds} -c "SELECT * FROM info_schema_lifecycle.migration_routine_candidate();"`.captureCombined().lines()).join("\n"));
+      })
+    .command("state", "Use psql to get the migration state for each candidate routine")
+      .option("--psql <path:string>", "`psql` command", { required: true, default: "psql" })
+      .option("-c, --conn-id <id:string>", "pgpass connection ID to use for psql", { required: true, default: "UDI_PRIME_DESTROYABLE_DEVL" })
+      .action(async (options) => {
+        const psqlCreds = pgpassPsqlArgs(options.connId);
+        console.log((await $.raw`${options.psql} ${psqlCreds} -c "SELECT * FROM info_schema_lifecycle.migration_routine_state();"`.captureCombined().lines()).join("\n"));
+      })
+    .command("script", "Use psql to generate migration scripts based on current state")
+      .option("--psql <path:string>", "`psql` command", { required: true, default: "psql" })
+      .option("-c, --conn-id <id:string>", "pgpass connection ID to use for psql", { required: true, default: "UDI_PRIME_DESTROYABLE_DEVL" })
+      .action(async (options) => {
+        const psqlCreds = pgpassPsqlArgs(options.connId);
+        console.log((await $.raw`${options.psql} ${psqlCreds} -q -t -A -P border=0 -X -c "SELECT * FROM info_schema_lifecycle.islm_migration_script();"`.captureCombined().lines()).join("\n"));
+      })
+    .command("test", "Use psql to execute test scripts for ISLM infrastructure")
       .option("-s, --src <path:string>", "Source location for artifacts", { required: true, default: path.fromFileUrl(import.meta.resolve("./")) })
       .option("-t, --target <path:string>", "Target location for generated artifacts", { required: true, default: cleanableTarget("/evolve") })
       .option("--psql <path:string>", "`psql` command", { required: true, default: "psql" })
-      .option("--suite-fname <file-name:string>", "Filename of the generated test suite script in target", { default: "islm-prime-test.psql" })
+      .option("--suite-fname <file-name:string>", "Filename of the generated test suite script in target", { default: "islm-infrastructure-test.psql" })
       .option("--log-results <path:string>", "Store `psql` results in this log file", { default: `./islmctl-test-${new Date().toISOString()}.log` })
       .option("-c, --conn-id <id:string>", "pgpass connection ID to use for psql", { required: true, default: "UDI_PRIME_DESTROYABLE_DEVL" })
       .type("pg-client-min-messages-level", postreSqlClientMinMessagesLevelCliffyEnum)
@@ -195,7 +216,7 @@ const CLI = new Command()
           console.error(`WARNING: ${psqlErrors} ${options.psql} error(s) occurred, see log file ${options.logResults}`);
         }
       })
-    .command("omnibus-fresh", "Freshen the given connection ID")
+    .command("omnibus-fresh", "Freshen the given connection ID by dropping and recreating the schema")
       .option("-c, --conn-id <id:string>", "pgpass connection ID to use for psql", { required: true, default: "UDI_PRIME_DESTROYABLE_DEVL" })
       .action(async (options) => {
         await CLI.parse(["evolve", "up", "--destroy-first", "--conn-id", options.connId]);