diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 76fffe5..120ba82 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -58,7 +58,7 @@ and [Poetry](https://python-poetry.org/docs/cli/#build) to create releases and t The release process is automated through GitHub Actions. Here is the process: - Create a Pull Request from `develop` to `main`. -- Merge the Pull Request. This can be a merge commit or a squash and merge. +- Merge the Pull Request. This must create a merge commit. - The merge will trigger the Release GitHub Action defined in [this workflow](.github/workflows/release.yaml). The Release GitHub Action does the following: @@ -69,8 +69,6 @@ The Release GitHub Action does the following: The action is triggered by any push to main. -Here is the relevant part of the GitHub Action: - > [!NOTE] > The release action will be triggered by any push to `main` only if the 'CI' job in the 'release.yaml' workflow succeeds. > Python Semantic Release will take care of version number update, tag creation and release creation. diff --git a/README.md b/README.md index ddf8960..1a33755 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ - [`list`](#list) - [CLI: Options](#cli-options) +[Full CLI documentation](docs/CLI_REFERENCE.md) ## Why this tool? @@ -96,6 +97,15 @@ List available versions: gsutil ls gs://vertex-pipelines-deployer ``` +### Add to requirements + +It's better to get the .tar.gz archive from gcs, and version it. + +Then add the following line to your `requirements.in` file: +```bash +file:my/path/to/vertex_deployer-$VERSION.tar.gz +``` + ## Usage ### Setup @@ -164,7 +174,7 @@ gcloud artifacts repositories add-iam-policy-binding ${GAR_PIPELINES_REPO_ID} \ --role="roles/artifactregistry.admin" ``` -You can use the deployer CLI (see example below) or import [`VertexPipelineDeployer`](deployer/deployer.py) in your code (try it yourself). +You can use the deployer CLI (see example below) or import [`VertexPipelineDeployer`](deployer/pipeline_deployer.py) in your code (try it yourself). ### Folder Structure diff --git a/docs/CLI_REFERENCE.md b/docs/CLI_REFERENCE.md new file mode 100644 index 0000000..30db910 --- /dev/null +++ b/docs/CLI_REFERENCE.md @@ -0,0 +1,119 @@ +# `vertex-deployer` + +**Usage**: + +```console +$ vertex-deployer [OPTIONS] COMMAND [ARGS]... +``` + +**Options**: + +* `--log-level, -log [TRACE|DEBUG|INFO|SUCCESS|WARNING|ERROR|CRITICAL]`: Set the logging level. [default: LoguruLevel.INFO] +* `--version, -v / --no-version`: Display the version number and exit. [default: no-version] +* `--install-completion`: Install completion for the current shell. +* `--show-completion`: Show completion for the current shell, to copy it or customize the installation. +* `--help`: Show this message and exit. + +**Commands**: + +* `check`: Check that pipelines are valid. +* `create`: Create files structure for a new pipeline. +* `deploy`: Compile, upload, run and schedule pipelines. +* `list`: List all pipelines. + +## `vertex-deployer check` + +Check that pipelines are valid. + +Checking that a pipeline is valid includes: + +* Checking that the pipeline can be imported. It must be a valid python module with a +`pipeline` function decorated with `@kfp.dsl.pipeline`. + +* Checking that the pipeline can be compiled using `kfp.compiler.Compiler`. + +* Checking that config files in `{CONFIG_ROOT_PATH}/{pipeline_name}` are corresponding to the +pipeline parameters definition, using Pydantic. + +--- + +**This command can be used to check pipelines in a Continuous Integration workflow.** + +**Usage**: + +```console +$ vertex-deployer check [OPTIONS] +``` + +**Options**: + +* `--pipeline-name []` +* `--all, -a / --no-all`: Whether to check all pipelines. [default: no-all] +* `--config-filepath, -cfp PATH`: Path to the json/py file with parameter values and input artifacts to check. If not specified, all config files in the pipeline dir will be checked. +* `--raise-error, -re / --no-raise-error, -nre`: Whether to raise an error if the pipeline is not valid. [default: no-raise-error] +* `--help`: Show this message and exit. + +## `vertex-deployer create` + +Create files structure for a new pipeline. + +**Usage**: + +```console +$ vertex-deployer create [OPTIONS] PIPELINE_NAME +``` + +**Arguments**: + +* `PIPELINE_NAME`: The name of the pipeline to create. [required] + +**Options**: + +* `--config-type, -ct [json|py]`: The type of the config to create. [default: ConfigType.json] +* `--help`: Show this message and exit. + +## `vertex-deployer deploy` + +Compile, upload, run and schedule pipelines. + +**Usage**: + +```console +$ vertex-deployer deploy [OPTIONS] PIPELINE_NAME:{} +``` + +**Arguments**: + +* `PIPELINE_NAME:{}`: The name of the pipeline to run. [required] + +**Options**: + +* `--env-file PATH`: The environment file to use. +* `--compile, -c / --no-compile, -nc`: Whether to compile the pipeline. [default: compile] +* `--upload, -u / --no-upload, -nu`: Whether to upload the pipeline to Google Artifact Registry. [default: no-upload] +* `--run, -r / --no-run, -nr`: Whether to run the pipeline. [default: no-run] +* `--schedule, -s / --no-schedule, -ns`: Whether to create a schedule for the pipeline. [default: no-schedule] +* `--cron TEXT`: Cron expression for scheduling the pipeline. To pass it to the CLI, use hyphens e.g. '0-10-*-*-*'. +* `--delete-last-schedule, -dls / --no-delete-last-schedule`: Whether to delete the previous schedule before creating a new one. [default: no-delete-last-schedule] +* `--tags TEXT`: The tags to use when uploading the pipeline. [default: latest] +* `--config-filepath, -cfp PATH`: Path to the json/py file with parameter values and input artifacts to use when running the pipeline. +* `--config-name, -cn TEXT`: Name of the json/py file with parameter values and input artifacts to use when running the pipeline. It must be in the pipeline config dir. e.g. `config_dev.json` for `./vertex/configs/{pipeline-name}/config_dev.json`. +* `--enable-caching, -ec / --no-enable-caching`: Whether to enable caching when running the pipeline. [default: no-enable-caching] +* `--experiment-name, -en TEXT`: The name of the experiment to run the pipeline in. Defaults to '{pipeline_name}-experiment'. +* `--local-package-path, -lpp PATH`: Local dir path where pipelines will be compiled. [default: vertex/pipelines/compiled_pipelines] +* `--help`: Show this message and exit. + +## `vertex-deployer list` + +List all pipelines. + +**Usage**: + +```console +$ vertex-deployer list [OPTIONS] +``` + +**Options**: + +* `--with-configs, -wc / --no-with-configs, -nc`: Whether to list config files. [default: no-with-configs] +* `--help`: Show this message and exit.