Job import export (#16)

* Add 'dbt-cloud job export' command * Add 'dbt-cloud job import' command * Add 'dbt-cloud job delete' command Co-authored-by: Simo Tumelius <[email protected]>
data-mie · Dec 25, 2021 · bc2bb12 · bc2bb12
1 parent 3cae73a
commit bc2bb12
Show file tree

Hide file tree

Showing 11 changed files with 390 additions and 59 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -53,5 +53,14 @@ jobs:
           dbt-cloud job run | tee run.json
           echo ::set-output name=run_id::$(cat run.json | jq .data.id -r)
 
-      - name: Test 'dbt cloud run get'
-        run: dbt-cloud run get --run-id ${{ steps.job_run.outputs.run_id }}
+      - name: Test 'dbt-cloud run get'
+        run: dbt-cloud run get --run-id ${{ steps.job_run.outputs.run_id }}
+
+      - name: Test 'dbt-cloud job export'
+        run: dbt-cloud job export | tee job.json
+
+      - name: Test 'dbt-cloud job import'
+        run: cat job.json | dbt-cloud job import | tee job_imported.json
+
+      - name: Test 'dbt-cloud job delete'
+        run: dbt-cloud job delete --job-id $(cat job_imported.json | jq .data.id)
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,12 @@
+# VSCode
+.vscode/
+
+# Python
 **/*.egg-info/
 venv/
-*.pyc
+*.pyc
+
+
+# pytest-cov
+.coverage
+cov_html/
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # dbt-cloud-cli
 
-`dbt-cloud-cli` is a command line interface for [dbt Cloud API v2.0](https://docs.getdbt.com/dbt-cloud/api-v2). It abstracts the REST API calls in an easy-to-use interface that can be incorporated into automated and manual (ad-hoc) workloads. Here are some example use cases for `dbt-cloud-cli`:
+`dbt-cloud-cli` is a command line interface for [dbt Cloud API](https://docs.getdbt.com/dbt-cloud/api-v2). It abstracts the REST API calls in an easy-to-use interface that can be incorporated into automated and manual (ad-hoc) workloads. Here are some example use cases for `dbt-cloud-cli`:
 
 1. Triggering dbt Cloud jobs in CI/CD: You can use [dbt-cloud job run](#dbt-cloud-job-run) in a CI/CD workflow (e.g., Github Actions) to trigger a dbt Cloud job that runs and tests the changes in a commit branch
 2. Setting up dbt Cloud jobs: You can use [dbt-cloud job create](#dbt-cloud-job-create) to create standardized jobs between dbt Cloud projects.
@@ -30,6 +30,9 @@ The following environment variables are used as argument defaults:
 * [dbt-cloud job run](#dbt-cloud-job-run)
 * [dbt-cloud job get](#dbt-cloud-job-get)
 * [dbt-cloud job create](#dbt-cloud-job-create)
+* [dbt-cloud job delete](#dbt-cloud-job-delete)
+* [dbt-cloud job export](#dbt-cloud-job-export)
+* [dbt-cloud job import](#dbt-cloud-job-import)
 * [dbt-cloud run get](#dbt-cloud-run-get)
 
 ## dbt-cloud job run
@@ -235,6 +238,192 @@ dbt-cloud job create --project-id REFACTED --environment-id 49819 --name "Create
 }
 ```
 
+## dbt-cloud job delete
+
+This command deletes a job in a dbt Cloud project. Note that this command uses an undocumented v3 API endpoint.
+
+### Usage
+
+```bash
+>> dbt-cloud job delete --job-id 48474
+{
+  "status": {
+    "code": 200,
+    "is_success": true,
+    "user_message": "Success!",
+    "developer_message": ""
+  },
+  "data": {
+    "execution": {
+      "timeout_seconds": 0
+    },
+    "generate_docs": false,
+    "run_generate_sources": false,
+    "id": 48474,
+    "account_id": REDACTED,
+    "project_id": REDACTED,
+    "environment_id": 49819,
+    "name": "Do nothing!",
+    "dbt_version": null,
+    "created_at": "2021-12-25T10:12:29.114456+00:00",
+    "updated_at": "2021-12-25T10:12:29.814383+00:00",
+    "execute_steps": [
+      "dbt run -s not_a_model"
+    ],
+    "state": 2,
+    "deferring_job_definition_id": null,
+    "lifecycle_webhooks": false,
+    "lifecycle_webhooks_url": null,
+    "triggers": {
+      "github_webhook": false,
+      "git_provider_webhook": null,
+      "custom_branch_only": true,
+      "schedule": false
+    },
+    "settings": {
+      "threads": 4,
+      "target_name": "default"
+    },
+    "schedule": {
+      "cron": "0 * * * *",
+      "date": {
+        "type": "every_day"
+      },
+      "time": {
+        "type": "every_hour",
+        "interval": 1
+      }
+    },
+    "is_deferrable": false,
+    "generate_sources": false,
+    "cron_humanized": "Every hour",
+    "next_run": null,
+    "next_run_humanized": null
+  }
+}
+```
+
+## dbt-cloud job export
+
+This command exports a dbt Cloud job as JSON to a file and can be used in conjunction with [dbt-cloud job import](#dbt-cloud-job-import) to copy jobs between dbt Cloud projects.
+
+### Usage
+
+```bash
+>> dbt-cloud job export | tee job.json
+{
+  "execution": {
+    "timeout_seconds": 0
+  },
+  "generate_docs": false,
+  "run_generate_sources": false,
+  "account_id": REDACTED,
+  "project_id": REDACTED,
+  "environment_id": 49819,
+  "name": "Do nothing!",
+  "dbt_version": null,
+  "created_at": "2021-11-18T15:19:03.185668+00:00",
+  "updated_at": "2021-12-25T09:17:12.788186+00:00",
+  "execute_steps": [
+    "dbt run -s not_a_model"
+  ],
+  "state": 1,
+  "deferring_job_definition_id": null,
+  "lifecycle_webhooks": false,
+  "lifecycle_webhooks_url": null,
+  "triggers": {
+    "github_webhook": false,
+    "git_provider_webhook": null,
+    "custom_branch_only": true,
+    "schedule": false
+  },
+  "settings": {
+    "threads": 4,
+    "target_name": "default"
+  },
+  "schedule": {
+    "cron": "0 * * * *",
+    "date": {
+      "type": "every_day"
+    },
+    "time": {
+      "type": "every_hour",
+      "interval": 1
+    }
+  },
+  "is_deferrable": false,
+  "generate_sources": false,
+  "cron_humanized": "Every hour",
+  "next_run": null,
+  "next_run_humanized": null
+}
+```
+
+## dbt-cloud job import
+
+This command imports a dbt Cloud job from exported JSON. You can use JSON manipulation tools (e.g., [jq](https://stedolan.github.io/jq/)) to modify the job definition before importing it.
+
+### Usage
+
+```bash
+>> cat job.json | jq '.environment_id = 49819 | .name = "Imported job"' | dbt-cloud job import
+{
+  "status": {
+    "code": 201,
+    "is_success": true,
+    "user_message": "Success!",
+    "developer_message": ""
+  },
+  "data": {
+    "execution": {
+      "timeout_seconds": 0
+    },
+    "generate_docs": false,
+    "run_generate_sources": false,
+    "id": 48475,
+    "account_id": REDACTED,
+    "project_id": REDACTED,
+    "environment_id": 49819,
+    "name": "Imported job",
+    "dbt_version": null,
+    "created_at": "2021-12-25T10:40:13.193129+00:00",
+    "updated_at": "2021-12-25T10:40:13.193149+00:00",
+    "execute_steps": [
+      "dbt run -s not_a_model"
+    ],
+    "state": 1,
+    "deferring_job_definition_id": null,
+    "lifecycle_webhooks": false,
+    "lifecycle_webhooks_url": null,
+    "triggers": {
+      "github_webhook": false,
+      "git_provider_webhook": null,
+      "custom_branch_only": true,
+      "schedule": false
+    },
+    "settings": {
+      "threads": 4,
+      "target_name": "default"
+    },
+    "schedule": {
+      "cron": "0 * * * *",
+      "date": {
+        "type": "every_day"
+      },
+      "time": {
+        "type": "every_hour",
+        "interval": 1
+      }
+    },
+    "is_deferrable": false,
+    "generate_sources": false,
+    "cron_humanized": "Every hour",
+    "next_run": null,
+    "next_run_humanized": null
+  }
+}
+```
+
 ## dbt-cloud run get
 This command prints a dbt Cloud run status JSON response. For more information on the API endpoint arguments and response, run `dbt-cloud run get --help` and check out the [dbt Cloud API docs](https://docs.getdbt.com/dbt-cloud/api-v2#operation/getRunById).
 

diff --git a/dbt_cloud/args.py b/dbt_cloud/args.py
@@ -1,6 +1,7 @@
 import click
 import os
 from pydantic import BaseModel, validator, Field
+from dbt_cloud.serde import json_to_dict
 
 
 class ArgsBaseModel(BaseModel):
@@ -42,12 +43,9 @@ def field_not_none(cls, value, field):
         else:
             return value
 
-    def get_payload(self, exclude_keys=["api_token", "account_id", "job_id"]) -> dict:
-        payload = self.dict()
-        payload = {
-            key: value for key, value in payload.items() if key not in exclude_keys
-        }
-        return payload
+    def get_payload(self, exclude=["api_token", "account_id", "job_id"]) -> dict:
+        payload = self.json(exclude=set(exclude))
+        return json_to_dict(payload)
 
 
 class DbtCloudArgsBaseModel(ArgsBaseModel):

diff --git a/dbt_cloud/cli.py b/dbt_cloud/cli.py
@@ -1,13 +1,17 @@
 import json
 import time
 import click
+from pathlib import Path
+from dbt_cloud.args import DbtCloudArgsBaseModel
 from dbt_cloud.job import (
     DbtCloudJob,
+    DbtCloudJobArgs,
     DbtCloudJobRunArgs,
     DbtCloudJobGetArgs,
     DbtCloudJobCreateArgs,
 )
 from dbt_cloud.run import DbtCloudRunStatus, DbtCloudRunGetArgs
+from dbt_cloud.serde import json_to_dict, dict_to_json
 from dbt_cloud.exc import DbtCloudException
 
 
@@ -26,7 +30,7 @@ def job_run():
     pass
 
 
-@job.command()
+@job.command(help="Triggers a dbt Cloud job run and returns a status JSON response.")
 @DbtCloudJobRunArgs.click_options
 @click.option(
     f"--wait/--no-wait",
@@ -35,7 +39,7 @@ def job_run():
 )
 def run(wait, **kwargs):
     args = DbtCloudJobRunArgs(**kwargs)
-    job = DbtCloudJob(**args.dict())
+    job = args.get_job()
     response, run = job.run(args=args)
     if wait:
         while True:
@@ -49,27 +53,72 @@ def run(wait, **kwargs):
                     f"Job run failed with {status.name} status. For more information, see {href}."
                 )
             time.sleep(5)
-    click.echo(json.dumps(response.json(), indent=2))
+    click.echo(dict_to_json(response.json()))
     response.raise_for_status()
 
 
-@job.command()
+@job.command(help="Returns the details of a dbt Cloud job.")
 @DbtCloudJobGetArgs.click_options
 def get(**kwargs):
     args = DbtCloudJobGetArgs(**kwargs)
     job = DbtCloudJob(**args.dict())
     response = job.get(order_by=args.order_by)
-    click.echo(json.dumps(response.json(), indent=2))
+    click.echo(dict_to_json(response.json()))
     response.raise_for_status()
 
 
-@job.command()
+@job.command(help="Creates a job in a dbt Cloud project.")
 @DbtCloudJobCreateArgs.click_options
 def create(**kwargs):
     args = DbtCloudJobCreateArgs(**kwargs)
     job = DbtCloudJob(job_id=None, **args.dict())
     response = job.create(args)
-    click.echo(json.dumps(response.json(), indent=2))
+    click.echo(dict_to_json(response.json()))
+    response.raise_for_status()
+
+
+@job.command(help="Deletes a job from a dbt Cloud project.")
+@DbtCloudJobArgs.click_options
+def delete(**kwargs):
+    args = DbtCloudJobArgs(**kwargs)
+    job = args.get_job()
+    response = job.delete()
+    click.echo(dict_to_json(response.json()))
+    response.raise_for_status()
+
+
+@job.command(help="Exports a dbt Cloud job as JSON to a file.")
+@DbtCloudJobArgs.click_options
+@click.option(
+    "-f",
+    "--file",
+    default="-",
+    type=click.File("w"),
+    help="Export file path.",
+)
+def export(file, **kwargs):
+    args = DbtCloudJobArgs(**kwargs)
+    job = args.get_job()
+    exclude = ["id"]
+    file.write(job.to_json(exclude=exclude))
+
+
+@job.command(help="Imports a dbt Cloud job from exported JSON.", name="import")
+@DbtCloudArgsBaseModel.click_options
+@click.option(
+    "-f",
+    "--file",
+    default="-",
+    type=click.File("r"),
+    help="Import file path.",
+)
+def import_job(file, **kwargs):
+    args = DbtCloudArgsBaseModel(**kwargs)
+    job_create_kwargs = json_to_dict(file.read())
+    job_create_args = DbtCloudJobCreateArgs(**job_create_kwargs)
+    job = DbtCloudJob(job_id=None, **args.dict())
+    response = job.create(job_create_args)
+    click.echo(dict_to_json(response.json()))
     response.raise_for_status()
 
 
@@ -79,5 +128,5 @@ def get(**kwargs):
     args = DbtCloudRunGetArgs(**kwargs)
     run = args.get_run()
     response, _ = run.get_status()
-    click.echo(json.dumps(response.json(), indent=2))
+    click.echo(dict_to_json(response.json()))
     response.raise_for_status()