forked from pflooky/data-caterer-docs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add in new section for report, add in alert configuration and externa…
…l validation source pages, clean up package names to use updated path of io.github.datacatering, ensure all links point to repos in data-catering
- Loading branch information
Showing
78 changed files
with
10,906 additions
and
1,388 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,7 +5,7 @@ | |
| Ensure you have `docker` installed and running. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| cd data-caterer-example && ./run.sh | ||
| #check results under docker/sample/report/index.html folder | ||
| ``` | ||
|
|
@@ -23,7 +23,7 @@ Sample report can also be seen [**here**](../sample/report/html/index.html) | |
| 1. Join the [Slack Data Catering Slack group here](https://join.slack.com/t/data-catering/shared_invite/zt-2664ylbpi-w3n7lWAO~PHeOG9Ujpm~~w) | ||
| 2. Get an API_KEY by using slash command `/token` in the Slack group (will only be visible to you) | ||
| 3. | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| cd data-caterer-example && export DATA_CATERING_API_KEY=<insert api key> | ||
| ./run.sh | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,7 +8,7 @@ Two main ways to deploy and run Data Caterer: | |
| ## Docker | ||
|
|
||
| To package up your class along with the Data Caterer base image, you can follow | ||
| the [Dockerfile that is created for you here](https://github.com/pflooky/data-caterer-example/blob/main/Dockerfile). | ||
| the [Dockerfile that is created for you here](https://github.com/data-catering/data-caterer-example/blob/main/Dockerfile). | ||
|
|
||
| Then you can run the following: | ||
|
|
||
|
|
@@ -19,13 +19,13 @@ docker build -t <my_image_name>:<my_image_tag> . | |
|
|
||
| ## Helm | ||
|
|
||
| [Link to sample helm on GitHub here](https://github.com/pflooky/data-caterer-example/tree/main/helm/data-caterer) | ||
| [Link to sample helm on GitHub here](https://github.com/data-catering/data-caterer-example/tree/main/helm/data-caterer) | ||
|
|
||
| Update | ||
| the [configuration](https://github.com/pflooky/data-caterer-example/blob/main/helm/data-caterer/templates/configuration.yaml) | ||
| the [configuration](https://github.com/data-catering/data-caterer-example/blob/main/helm/data-caterer/templates/configuration.yaml) | ||
| to your own data connections and configuration or own image created from above. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| helm install data-caterer ./data-caterer-example/helm/data-caterer | ||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,7 +20,7 @@ for the tables you configure. | |
| First, we will clone the data-caterer-example repo which will already have the base project setup required. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| ``` | ||
|
|
||
| If you already have a Cassandra instance running, you can skip to [this step](#plan-setup). | ||
|
|
@@ -59,15 +59,15 @@ metadata information about tables and columns from the below tables. | |
|
|
||
| Create a new Java or Scala class. | ||
|
|
||
| - Java: `src/main/java/com/github/pflooky/plan/MyAdvancedCassandraJavaPlan.java` | ||
| - Scala: `src/main/scala/com/github/pflooky/plan/MyAdvancedCassandraPlan.scala` | ||
| - Java: `src/main/java/io/github/datacatering/plan/MyAdvancedCassandraJavaPlan.java` | ||
| - Scala: `src/main/scala/io/github/datacatering/plan/MyAdvancedCassandraPlan.scala` | ||
|
|
||
| Make sure your class extends `PlanRun`. | ||
|
|
||
| === "Java" | ||
|
|
||
| ```java | ||
| import com.github.pflooky.datacaterer.java.api.PlanRun; | ||
| import io.github.datacatering.datacaterer.java.api.PlanRun; | ||
|
|
||
| public class MyAdvancedCassandraJavaPlan extends PlanRun { | ||
| } | ||
|
|
@@ -76,7 +76,7 @@ Make sure your class extends `PlanRun`. | |
| === "Scala" | ||
|
|
||
| ```scala | ||
| import com.github.pflooky.datacaterer.api.PlanRun | ||
| import io.github.datacatering.datacaterer.api.PlanRun | ||
|
|
||
| class MyAdvancedCassandraPlan extends PlanRun { | ||
| } | ||
|
|
@@ -124,7 +124,7 @@ defined under`docker/data/cql/customer.cql`. This table should already be setup | |
| [step](#cassandra-setup). We can check if the table is setup already via the following command: | ||
|
|
||
| ```shell | ||
| docker exec host.docker.internal cqlsh -e 'describe account.accounts; describe account.account_status_history;' | ||
| docker exec docker-cassandraserver-1 cqlsh -e 'describe account.accounts; describe account.account_status_history;' | ||
| ``` | ||
|
|
||
| Here we should see some output that looks like the below. This tells us what schema we need to follow when generating | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,7 +25,7 @@ Creating a data generator based on an [OpenAPI/Swagger](https://spec.openapis.or | |
| First, we will clone the data-caterer-example repo which will already have the base project setup required. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| ``` | ||
|
|
||
| ### HTTP Setup | ||
|
|
@@ -44,15 +44,15 @@ docker ps | |
|
|
||
| Create a new Java or Scala class. | ||
|
|
||
| - Java: `src/main/java/com/github/pflooky/plan/MyAdvancedHttpJavaPlanRun.java` | ||
| - Scala: `src/main/scala/com/github/pflooky/plan/MyAdvancedHttpPlanRun.scala` | ||
| - Java: `src/main/java/io/github/datacatering/plan/MyAdvancedHttpJavaPlanRun.java` | ||
| - Scala: `src/main/scala/io/github/datacatering/plan/MyAdvancedHttpPlanRun.scala` | ||
|
|
||
| Make sure your class extends `PlanRun`. | ||
|
|
||
| === "Java" | ||
|
|
||
| ```java | ||
| import com.github.pflooky.datacaterer.java.api.PlanRun; | ||
| import io.github.datacatering.datacaterer.java.api.PlanRun; | ||
| ... | ||
|
|
||
| public class MyAdvancedHttpJavaPlanRun extends PlanRun { | ||
|
|
@@ -66,7 +66,7 @@ Make sure your class extends `PlanRun`. | |
| === "Scala" | ||
|
|
||
| ```scala | ||
| import com.github.pflooky.datacaterer.api.PlanRun | ||
| import io.github.datacatering.datacaterer.api.PlanRun | ||
| ... | ||
|
|
||
| class MyAdvancedHttpPlanRun extends PlanRun { | ||
|
|
@@ -266,7 +266,7 @@ want to alter this value, you can do so via the below configuration. The lowest | |
| === "Java" | ||
|
|
||
| ```java | ||
| import com.github.pflooky.datacaterer.api.model.Constants; | ||
| import io.github.datacatering.datacaterer.api.model.Constants; | ||
|
|
||
| ... | ||
| var httpTask = http("my_http", Map.of(Constants.ROWS_PER_SECOND(), "1")) | ||
|
|
@@ -276,7 +276,7 @@ want to alter this value, you can do so via the below configuration. The lowest | |
| === "Scala" | ||
|
|
||
| ```scala | ||
| import com.github.pflooky.datacaterer.api.model.Constants.ROWS_PER_SECOND | ||
| import io.github.datacatering.datacaterer.api.model.Constants.ROWS_PER_SECOND | ||
|
|
||
| ... | ||
| val httpTask = http("my_http", options = Map(ROWS_PER_SECOND -> "1")) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,7 +20,7 @@ for the topics you configure. | |
| First, we will clone the data-caterer-example repo which will already have the base project setup required. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| ``` | ||
|
|
||
| If you already have a Kafka instance running, you can skip to [this step](#plan-setup). | ||
|
|
@@ -39,15 +39,15 @@ docker-compose up -d kafka | |
|
|
||
| Create a new Java or Scala class. | ||
|
|
||
| - Java: `src/main/java/com/github/pflooky/plan/MyAdvancedKafkaJavaPlan.java` | ||
| - Scala: `src/main/scala/com/github/pflooky/plan/MyAdvancedKafkaPlan.scala` | ||
| - Java: `src/main/java/io/github/datacatering/plan/MyAdvancedKafkaJavaPlan.java` | ||
| - Scala: `src/main/scala/io/github/datacatering/plan/MyAdvancedKafkaPlan.scala` | ||
|
|
||
| Make sure your class extends `PlanRun`. | ||
|
|
||
| === "Java" | ||
|
|
||
| ```java | ||
| import com.github.pflooky.datacaterer.java.api.PlanRun; | ||
| import io.github.datacatering.datacaterer.java.api.PlanRun; | ||
|
|
||
| public class MyAdvancedKafkaJavaPlan extends PlanRun { | ||
| } | ||
|
|
@@ -56,7 +56,7 @@ Make sure your class extends `PlanRun`. | |
| === "Scala" | ||
|
|
||
| ```scala | ||
| import com.github.pflooky.datacaterer.api.PlanRun | ||
| import io.github.datacatering.datacaterer.api.PlanRun | ||
|
|
||
| class MyAdvancedKafkaPlan extends PlanRun { | ||
| } | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,7 +19,7 @@ follows [OpenLineage API](https://openlineage.io/)). | |
| First, we will clone the data-caterer-example repo which will already have the base project setup required. | ||
|
|
||
| ```shell | ||
| git clone [email protected]:pflooky/data-caterer-example.git | ||
| git clone [email protected]:data-catering/data-caterer-example.git | ||
| ``` | ||
|
|
||
| ### Marquez Setup | ||
|
|
@@ -48,15 +48,15 @@ docker exec marquez-db psql -Upostgres -c 'CREATE DATABASE food_delivery' | |
|
|
||
| Create a new Java or Scala class. | ||
|
|
||
| - Java: `src/main/java/com/github/pflooky/plan/MyAdvancedMetadataSourceJavaPlanRun.java` | ||
| - Scala: `src/main/scala/com/github/pflooky/plan/MyAdvancedMetadataSourcePlanRun.scala` | ||
| - Java: `src/main/java/io/github/datacatering/plan/MyAdvancedMetadataSourceJavaPlanRun.java` | ||
| - Scala: `src/main/scala/io/github/datacatering/plan/MyAdvancedMetadataSourcePlanRun.scala` | ||
|
|
||
| Make sure your class extends `PlanRun`. | ||
|
|
||
| === "Java" | ||
|
|
||
| ```java | ||
| import com.github.pflooky.datacaterer.java.api.PlanRun; | ||
| import io.github.datacatering.datacaterer.java.api.PlanRun; | ||
| ... | ||
|
|
||
| public class MyAdvancedMetadataSourceJavaPlanRun extends PlanRun { | ||
|
|
@@ -70,7 +70,7 @@ Make sure your class extends `PlanRun`. | |
| === "Scala" | ||
|
|
||
| ```scala | ||
| import com.github.pflooky.datacaterer.api.PlanRun | ||
| import io.github.datacatering.datacaterer.api.PlanRun | ||
| ... | ||
|
|
||
| class MyAdvancedMetadataSourcePlanRun extends PlanRun { | ||
|
|
||
Oops, something went wrong.