Skip to content

Commit

Permalink
Update to 0.12.4, add in docs about validating HTTP requests/responses
Browse files Browse the repository at this point in the history
  • Loading branch information
pflooky committed Dec 6, 2024
1 parent 3bfc1d6 commit aafbbf8
Show file tree
Hide file tree
Showing 10 changed files with 159 additions and 7 deletions.
5 changes: 5 additions & 0 deletions docs/docs/guide/data-source/database/cassandra.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,3 +335,8 @@ Also check the HTML report, found at `docker/sample/report/index.html`, that get
was executed.

![Sample report](../../../../sample/report/report_screenshot.png)

### Validation

If you want to validate data from Cassandra,
[follow the validation documentation found here to help guide you](../../../validation.md).
5 changes: 5 additions & 0 deletions docs/docs/guide/data-source/database/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,3 +311,8 @@ Also check the HTML report, found at `docker/sample/report/index.html`, that get
was executed.

![Sample report](../../../../sample/report/report_screenshot.png)

### Validation

If you want to validate data from MySQL,
[follow the validation documentation found here to help guide you](../../../validation.md).
5 changes: 5 additions & 0 deletions docs/docs/guide/data-source/database/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,3 +316,8 @@ Also check the HTML report, found at `docker/sample/report/index.html`, that get
was executed.

![Sample report](../../../../sample/report/report_screenshot.png)

### Validation

If you want to validate data from Postgres,
[follow the validation documentation found here to help guide you](../../../validation.md).
1 change: 0 additions & 1 deletion docs/docs/guide/data-source/file/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ Make sure your class extends `PlanRun`.
tasks:
- name: "iceberg_account_table"
dataSourceName: "customer_accounts"
enabled: true
```

=== "UI"
Expand Down
142 changes: 142 additions & 0 deletions docs/docs/guide/data-source/http/http.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,32 @@ Make sure your class extends `PlanRun`.
}
```

=== "YAML"

In `docker/data/custom/plan/my-http.yaml`:
```yaml
name: "my_http_plan"
description: "Create account data via HTTP from OpenAPI metadata"
tasks:
- name: "http_task"
dataSourceName: "my_http"
```

In `application.conf`:
```
flags {
enableGeneratePlanAndTasks = true
}
folders {
generatedReportsFolderPath = "/opt/app/data/report"
}
```

=== "UI"

1. Click on `Advanced Configuration` towards the bottom of the screen
3. Click on `Folder` and enter `/tmp/data-caterer/report` for `Generated Reports Folder Path`

We will enable generate plan and tasks so that we can read from external sources for metadata and save the reports
under a folder we can easily access.

Expand Down Expand Up @@ -122,6 +148,30 @@ We have kept the following endpoints to test out:
.count(count.records(2))
```

=== "YAML"

In `docker/data/custom/task/http/openapi-task.yaml`:
```
name: "http_task"
steps:
- name: "my_petstore"
options:
metadataSourceType: "openapi"
schemaLocation: "/opt/app/mount/http/petstore.json"
count:
records: 2
```

=== "UI"

1. Click on `Connection` tab at the top
1. Click on `Select data source type` and select `OpenAPI/Swagger`
1. Provide a `Name` and `Schema Location` pointing to where you have stored your OpenAPI specification file
1. Click `Create`
1. Click on `Home` tab at the top
1. Click on `Generation` and tick the `Auto from metadata source` checkbox
1. Click on `Select metadata source` and select the OpenAPI metadata source you just created

The above defines that the schema will come from an OpenAPI document found on the pathway defined. It will then generate
2 requests per request method and endpoint combination.

Expand Down Expand Up @@ -197,6 +247,28 @@ knowledge to link all the `id` values together.
execute(myPlan, conf, httpTask)
```

=== "YAML"

In `docker/data/custom/plan/my-http.yaml`:
```yaml
name: "my_http_plan"
description: "Create account data via HTTP from OpenAPI metadata"
tasks:
- name: "http_task"
dataSourceName: "my_http"

sinkOptions:
foreignKeys:
- - "my_http.POST/pets.bodyContent.id"
- - "my_http.DELETE/pets/{id}.pathParamid"
- - "my_http.GET/pets/{id}.pathParamid"
- []
```

=== "UI"

![Generate relationship in UI](../diagrams/ui/data-caterer-ui-generate-relationship.png)

Let's test it out by running it again

```shell
Expand Down Expand Up @@ -243,6 +315,38 @@ Given the `id` column is a nested column as noted in the foreign key, we can alt
.count(count.records(2))
```

=== "YAML"

In `docker/data/custom/task/http/openapi-task.yaml`:
```
name: "http_task"
steps:
- name: "my_petstore"
options:
metadataSourceType: "openapi"
schemaLocation: "/opt/app/mount/http/petstore.json"
count:
records: 2
schema:
fields:
- name: "bodyContent"
fields:
- name: "id"
options:
regex: "ID[0-9]{8}"
```

=== "UI"

1. Click on `Generation` and tick the `Manual` checkbox
1. Click on `+ Field`
1. Add name as `bodyContent`
1. Click on `Select data type` and select `struct`
1. Click on `+ Field`
1. Add name as `id`
1. Click on `Select data type` and select `string`
1. Click `Regex` and enter `ID[0-9]{8}`

We first get the column `bodyContent`, then get the nested schema and get the column `id` and add metadata stating that
`id` should follow the patter `ID[0-9]{8}`.

Expand Down Expand Up @@ -298,4 +402,42 @@ want to alter this value, you can do so via the below configuration. The lowest
...
```

=== "YAML"

In `docker/data/custom/task/http/openapi-task.yaml`:
```
name: "http_task"
steps:
- name: "my_petstore"
options:
metadataSourceType: "openapi"
schemaLocation: "/opt/app/mount/http/petstore.json"
rowsPerSecond: "1"
...
```

### Validation

Once you have generated HTTP requests, you may also want to validate the responses to ensure your service is responding
as expected.

The following fields are made available to you to validate against:

| Field | Inner Field | Data Type | Example |
|----------|-------------|---------------------|-------------------------------|
| request | method | String | GET |
| request | url | String | http://localhost:8080/my/path |
| request | headers | Map[String, String] | Content-Length -> 200 |
| request | body | String | my-body |
| request | startTime | Long | 1733408207499 |
| response | contentType | String | application/json |
| response | headers | Map[String, String] | Content-Length -> 200 |
| response | body | String | my-body |
| response | statusCode | Int | 200 |
| response | statusText | String | OK |
| response | timeTakenMs | Long | 120 |

If you want to validate data from an HTTP source,
[follow the validation documentation found here to help guide you](../../../validation.md).

Check out the full example under `AdvancedHttpPlanRun` in the example repo.
1 change: 0 additions & 1 deletion docs/docs/guide/data-source/metadata/data-contract-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ Make sure your class extends `PlanRun`.
tasks:
- name: "csv_account_file"
dataSourceName: "customer_accounts"
enabled: true
```

In `application.conf`:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ Make sure your class extends `PlanRun`.
tasks:
- name: "csv_account_file"
dataSourceName: "customer_accounts"
enabled: true
```

In `application.conf`:
Expand Down
2 changes: 0 additions & 2 deletions docs/docs/guide/scenario/data-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,6 @@ Make sure your class extends `PlanRun`.
tasks:
- name: "csv_account_file"
dataSourceName: "customer_accounts"
enabled: true
```

=== "UI"
Expand Down Expand Up @@ -926,7 +925,6 @@ below.
tasks:
- name: "csv_account_file"
dataSourceName: "customer_accounts"
enabled: true

sinkOptions:
foreignKeys:
Expand Down
2 changes: 1 addition & 1 deletion docs/get-started/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ image: "https://data.catering/diagrams/logo/data_catering_logo.svg"

1. Docker
```shell
docker run -d -i -p 9898:9898 -e DEPLOY_MODE=standalone --name datacaterer datacatering/data-caterer:0.12.3
docker run -d -i -p 9898:9898 -e DEPLOY_MODE=standalone --name datacaterer datacatering/data-caterer:0.12.4
```
2. [Open localhost:9898](http://localhost:9898)

Expand Down
2 changes: 1 addition & 1 deletion helm/data-caterer/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ image:
repository: "datacatering/data-caterer"
pullPolicy: "IfNotPresent"
# Overrides the image tag whose default is the chart appVersion.
tag: "0.12.3"
tag: "0.12.4"

imagePullSecrets: []
nameOverride: ""
Expand Down

0 comments on commit aafbbf8

Please sign in to comment.