Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support migrations in subdirectories #1226

Closed
wants to merge 4 commits into from
Closed

feat: support migrations in subdirectories #1226

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c65486a
feat: support migrations in subdirectories
mythmon Jul 8, 2024
fa3d39c
move `mockMigrationClient` to before its uses
mythmon Jul 9, 2024
5bb1805
Add support for year prefixes to getTimestamp
mythmon Jul 9, 2024
caa669d
turn off subdir processing by default and sort using locale sort
mythmon Aug 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions bin/node-pg-migrate.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ const parser = yargs(process.argv.slice(2))
},
[migrationFilenameFormatArg]: {
defaultDescription: '"timestamp"',
choices: ['timestamp', 'utc'],
choices: ['timestamp', 'utc', 'year/utc', 'year/timestamp'],
describe:
'Prefix type of migration filename (Only valid with the create action)',
type: 'string',
Expand Down Expand Up @@ -381,7 +381,11 @@ function readJson(json: unknown): void {
MIGRATIONS_FILENAME_FORMAT,
migrationFilenameFormatArg,
json,
(val): val is `${FilenameFormat}` => val === 'timestamp' || val === 'utc'
(val): val is `${FilenameFormat}` =>
val === 'timestamp' ||
val === 'utc' ||
val === 'year/utc' ||
val === 'year/timestamp'
);
TEMPLATE_FILE_NAME = applyIf(
TEMPLATE_FILE_NAME,
Expand Down
50 changes: 25 additions & 25 deletions docs/src/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,31 +80,31 @@ More on that below.

You can adjust defaults by passing arguments to `node-pg-migrate`:

| Argument | Aliases | Default | Description |
| --------------------------- | ------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `config-file` | `f` | `undefined` | The file with migration JSON config |
| `config-value` | | `db` | Name of config section with db options |
| `schema` | `s` | `public` | The schema(s) on which migration will be run, used to set `search_path` |
| `create-schema` | | `false` | Create the configured schema if it doesn't exist |
| `database-url-var` | `d` | `DATABASE_URL` | Name of env variable with database url string |
| `migrations-dir` | `m` | `migrations` | The directory containing your migration files |
| `migrations-schema` | | same value as `schema` | The schema storing table which migrations have been run |
| `create-migrations-schema` | | `false` | Create the configured migrations schema if it doesn't exist |
| `migrations-table` | `t` | `pgmigrations` | The table storing which migrations have been run |
| `ignore-pattern` | | `undefined` | Regex pattern for file names to ignore |
| `migration-filename-format` | | `utc` | Choose prefix of file, `utc` (`20200605075829074`) or `timestamp` (`1591343909074`) |
| `migration-file-language` | `j` | `js` | Language of the migration file to create (`js`, `ts` or `sql`) |
| `template-file-name` | | `undefined` | Utilize a custom migration template file with language inferred from its extension. The file should export the up method, accepting a MigrationBuilder instance. |
| `tsconfig` | | `undefined` | Path to tsconfig.json. Used to setup transpiling of TS migration files. (Also sets `migration-file-language` to typescript, if not overridden) |
| `envPath` | | `same level where it's invoked` | Retrieve the path to a .env file. This feature proves handy when dealing with nested projects or when referencing a global .env file. |
| `timestamp` | | `false` | Treats number argument to up/down migration as timestamp (running up migrations less or equal to timestamp or down migrations greater or equal to timestamp) |
| `check-order` | | `true` | Check order of migrations before running them, to switch it off supply `--no-check-order` |
| `single-transaction` | | `true` | Combines all pending migrations into a single transaction so that if any migration fails, all will be rolled back, to switch it off supply `--no-single-transaction` |
| `no-lock` | | `false` | Disables locking mechanism and checks |
| `fake` | | `false` | Mark migrations as run without actually performing them, (use with caution!) |
| `decamelize` | | `false` | Runs `decamelize` on table/column/etc. names |
| `verbose` | | `true` | Print all debug messages like DB queries run, to switch it off supply `--no-verbose` |
| `reject-unauthorized` | | `undefined` | Sets ssl `rejectUnauthorized` parameter. Use for e.g. self-signed certificates on the server. [see](https://node-postgres.com/announcements#2020-02-25) |
| Argument | Aliases | Default | Description |
| --------------------------- | ------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `config-file` | `f` | `undefined` | The file with migration JSON config |
| `config-value` | | `db` | Name of config section with db options |
| `schema` | `s` | `public` | The schema(s) on which migration will be run, used to set `search_path` |
| `create-schema` | | `false` | Create the configured schema if it doesn't exist |
| `database-url-var` | `d` | `DATABASE_URL` | Name of env variable with database url string |
| `migrations-dir` | `m` | `migrations` | The directory containing your migration files |
| `migrations-schema` | | same value as `schema` | The schema storing table which migrations have been run |
| `create-migrations-schema` | | `false` | Create the configured migrations schema if it doesn't exist |
| `migrations-table` | `t` | `pgmigrations` | The table storing which migrations have been run |
| `ignore-pattern` | | `undefined` | Regex pattern for file names to ignore |
| `migration-filename-format` | | `utc` | Choose prefix of file, `utc` (`20200605075829074`) or `timestamp` (`1591343909074`), or either format prefixed with `year/` (`2020/20200605075829074` or `2020/1591343909074`) |
| `migration-file-language` | `j` | `js` | Language of the migration file to create (`js`, `ts` or `sql`) |
| `template-file-name` | | `undefined` | Utilize a custom migration template file with language inferred from its extension. The file should export the up method, accepting a MigrationBuilder instance. |
| `tsconfig` | | `undefined` | Path to tsconfig.json. Used to setup transpiling of TS migration files. (Also sets `migration-file-language` to typescript, if not overridden) |
| `envPath` | | `same level where it's invoked` | Retrieve the path to a .env file. This feature proves handy when dealing with nested projects or when referencing a global .env file. |
| `timestamp` | | `false` | Treats number argument to up/down migration as timestamp (running up migrations less or equal to timestamp or down migrations greater or equal to timestamp) |
| `check-order` | | `true` | Check order of migrations before running them, to switch it off supply `--no-check-order` |
| `single-transaction` | | `true` | Combines all pending migrations into a single transaction so that if any migration fails, all will be rolled back, to switch it off supply `--no-single-transaction` |
| `no-lock` | | `false` | Disables locking mechanism and checks |
| `fake` | | `false` | Mark migrations as run without actually performing them, (use with caution!) |
| `decamelize` | | `false` | Runs `decamelize` on table/column/etc. names |
| `verbose` | | `true` | Print all debug messages like DB queries run, to switch it off supply `--no-verbose` |
| `reject-unauthorized` | | `undefined` | Sets ssl `rejectUnauthorized` parameter. Use for e.g. self-signed certificates on the server. [see](https://node-postgres.com/announcements#2020-02-25) |

For SSL connection to DB you can set `PGSSLMODE` environment variable to value
from [list](https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNECT-SSLMODE) other
Expand Down
55 changes: 43 additions & 12 deletions src/migration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

import { createReadStream, createWriteStream } from 'node:fs';
import { mkdir, readdir } from 'node:fs/promises';
import { basename, extname, join, resolve } from 'node:path';
import { extname, join, relative, resolve } from 'node:path';
import { cwd } from 'node:process';
import type { QueryResult } from 'pg';
import type { DBConnection } from './db';
Expand All @@ -32,6 +32,8 @@ export interface RunMigration {
export enum FilenameFormat {
timestamp = 'timestamp',
utc = 'utc',
year_timestamp = 'year/timestamp',
year_utc = 'year/utc',
}

export interface CreateOptionsTemplate {
Expand All @@ -44,7 +46,7 @@ export interface CreateOptionsDefault {
}

export type CreateOptions = {
filenameFormat?: FilenameFormat | `${FilenameFormat}`;
filenameFormat?: FilenameFormat;
} & (CreateOptionsTemplate | CreateOptionsDefault);

const SEPARATOR = '_';
Expand All @@ -53,10 +55,13 @@ export async function loadMigrationFiles(
dir: string,
ignorePattern?: string
): Promise<string[]> {
const dirContent = await readdir(`${dir}/`, { withFileTypes: true });
const dirContent = await readdir(`${dir}/`, {
withFileTypes: true,
recursive: true,
});
const files = dirContent
.map((file) => (file.isFile() || file.isSymbolicLink() ? file.name : null))
.filter((file): file is string => Boolean(file))
.flatMap((file) => (file.isFile() || file.isSymbolicLink() ? [file] : []))
.map((file) => relative(dir, join(file.parentPath ?? file.path, file.name)))
.sort();
const filter = new RegExp(`^(${ignorePattern})$`);
return ignorePattern === undefined
Expand Down Expand Up @@ -129,12 +134,6 @@ export class Migration implements RunMigration {
// ensure the migrations directory exists
await mkdir(directory, { recursive: true });

const now = new Date();
const time =
filenameFormat === FilenameFormat.utc
? now.toISOString().replace(/\D/g, '')
: now.valueOf();

const templateFileName =
'templateFileName' in options
? resolve(cwd(), options.templateFileName)
Expand All @@ -145,6 +144,7 @@ export class Migration implements RunMigration {
const suffix = getSuffixFromFileName(templateFileName);

// file name looks like migrations/1391877300255_migration-title.js
const time = Migration.filenameTime(filenameFormat);
const newFile = join(directory, `${time}${SEPARATOR}${name}.${suffix}`);

// copy the default migration template to the new file location
Expand Down Expand Up @@ -186,7 +186,10 @@ export class Migration implements RunMigration {
) {
this.db = db;
this.path = migrationPath;
this.name = basename(migrationPath, extname(migrationPath));
this.name = relative(options.dir, migrationPath).replace(
new RegExp(`${extname(migrationPath)}$`),
''
);
this.timestamp = getTimestamp(logger, this.name);
this.up = up;
this.down = down;
Expand All @@ -195,6 +198,34 @@ export class Migration implements RunMigration {
this.logger = logger;
}

private static filenameTime(filenameFormat: FilenameFormat): string {
const now = new Date();
switch (filenameFormat) {
case FilenameFormat.utc: {
return now.toISOString().replace(/\D/g, '');
}

case FilenameFormat.year_utc: {
return join(
now.getFullYear().toString(),
now.toISOString().replace(/\D/g, '')
mythmon marked this conversation as resolved.
Show resolved Hide resolved
);
}

case FilenameFormat.timestamp: {
return now.valueOf().toString();
}

case FilenameFormat.year_timestamp: {
return join(now.getFullYear().toString(), now.valueOf().toString());
}

default: {
throw new Error(`unrecognized filenameFormat: ${filenameFormat}`);
}
}
}

_getMarkAsRun(action: MigrationAction): string {
const schema = getMigrationTableSchema(this.options);

Expand Down
Loading