Skip to content

Commit

Permalink
Update connections and roadmap with ODCS support
Browse files Browse the repository at this point in the history
  • Loading branch information
@pflooky
pflooky committed Jun 11, 2024
1 parent 9b545aa commit 07d1265
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 36 deletions.
63 changes: 32 additions & 31 deletions docs/setup/connection.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,37 +12,38 @@ These configurations can be done via API or from configuration. Examples of both

## Supported Data Connections

| Data Source Type | Data Source | Support | Free |
|------------------|---------------------|-------------------------------------------|-------------------------------------------|
| Cloud Storage | AWS S3 | :white_check_mark: | :white_check_mark: |
| Cloud Storage | Azure Blob Storage | :white_check_mark: | :white_check_mark: |
| Cloud Storage | GCP Cloud Storage | :white_check_mark: | :white_check_mark: |
| Database | Cassandra | :white_check_mark: | :white_check_mark: |
| Database | MySQL | :white_check_mark: | :white_check_mark: |
| Database | Postgres | :white_check_mark: | :white_check_mark: |
| Database | Elasticsearch | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| Database | MongoDB | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| Database | Opensearch | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| File | CSV | :white_check_mark: | :white_check_mark: |
| File | Delta Lake | :white_check_mark: | :white_check_mark: |
| File | Iceberg | :white_check_mark: | :white_check_mark: |
| File | JSON | :white_check_mark: | :white_check_mark: |
| File | ORC | :white_check_mark: | :white_check_mark: |
| File | Parquet | :white_check_mark: | :white_check_mark: |
| File | Hudi | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| HTTP | REST API | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Kafka | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Solace | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | ActiveMQ | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Pulsar | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | RabbitMQ | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Great Expectations | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Marquez | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | OpenMetadata | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | OpenAPI/Swagger | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Amundsen | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Datahub | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Solace Event Portal | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Data Source Type | Data Source | Support | Free |
|------------------|------------------------------------|-------------------------------------------|-------------------------------------------|
| Cloud Storage | AWS S3 | :white_check_mark: | :white_check_mark: |
| Cloud Storage | Azure Blob Storage | :white_check_mark: | :white_check_mark: |
| Cloud Storage | GCP Cloud Storage | :white_check_mark: | :white_check_mark: |
| Database | Cassandra | :white_check_mark: | :white_check_mark: |
| Database | MySQL | :white_check_mark: | :white_check_mark: |
| Database | Postgres | :white_check_mark: | :white_check_mark: |
| Database | Elasticsearch | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| Database | MongoDB | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| Database | Opensearch | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| File | CSV | :white_check_mark: | :white_check_mark: |
| File | Delta Lake | :white_check_mark: | :white_check_mark: |
| File | Iceberg | :white_check_mark: | :white_check_mark: |
| File | JSON | :white_check_mark: | :white_check_mark: |
| File | ORC | :white_check_mark: | :white_check_mark: |
| File | Parquet | :white_check_mark: | :white_check_mark: |
| File | Hudi | :octicons-x-circle-fill-12:{ .red-cross } | :white_check_mark: |
| HTTP | REST API | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Kafka | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Solace | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | ActiveMQ | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | Pulsar | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Messaging | RabbitMQ | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Great Expectations | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Marquez | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | OpenMetadata | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | OpenAPI/Swagger | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Open Data Contract Standard (ODCS) | :white_check_mark: | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Amundsen | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Datahub | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |
| Metadata | Solace Event Portal | :octicons-x-circle-fill-12:{ .red-cross } | :octicons-x-circle-fill-12:{ .red-cross } |

### API

Expand Down
8 changes: 4 additions & 4 deletions docs/setup/guide/data-source/file/iceberg.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ image: "https://data.catering/diagrams/logo/data_catering_logo.svg"

# Iceberg

Creating a data generator for Iceberg. You will have the ability to generate and validate Iceberg tables.
Data testing for Iceberg. You will have the ability to generate and validate Iceberg tables.

## Requirements

- 10 minutes
- 5 minutes
- Git
- Gradle
- Docker
Expand Down Expand Up @@ -100,7 +100,7 @@ Within our class, we can start by defining the connection properties to read/wri
"account.accounts", //table name
"/opt/app/data/customer/iceberg", //warehouse path
"hadoop", //catalog type
"", //catalogUri
"", //catalog uri
Map.of() //additional options
);
```
Expand All @@ -115,7 +115,7 @@ Within our class, we can start by defining the connection properties to read/wri
"account.accounts", //table name
"/opt/app/data/customer/iceberg", //warehouse path
"hadoop", //catalog type
"", //catalogUri
"", //catalog uri
Map() //additional options
)
```
Expand Down
2 changes: 1 addition & 1 deletion docs/use-case/roadmap.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Items below summarise the roadmap of Data Caterer. As each task gets completed,
| Feature | Description | Sub Tasks |
|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Data source support | Batch or real time data sources that can be added to Data Caterer. Support data sources that users want | - AWS, GCP and Azure related data services (:white_check_mark: [cloud storage](../setup/advanced.md#cloud-storage))<br>- :white_check_mark: [Delta Lake](../setup/guide/data-source/file/delta-lake.md)<br>- :white_check_mark: [Iceberg](../setup/guide/data-source/file/iceberg.md)<br>- RabbitMQ<br>- ActiveMQ<br>- MongoDB<br>- [Elasticsearch](https://github.com/data-catering/data-caterer/issues/7)<br>- [Snowflake](https://github.com/data-catering/data-caterer/issues/6)<br>- [Databricks](https://github.com/data-catering/data-caterer/issues/5)<br>- Pulsar |
| Metadata discovery | Allow for schema and data profiling from external metadata sources | - :white_check_mark: [HTTP (OpenAPI spec)](../setup/guide/data-source/http/http.md)<br>- JMS<br>- Read from samples<br>- :white_check_mark: [OpenLineage metadata (Marquez)](../setup/guide/data-source/metadata/marquez.md)<br>- :white_check_mark: [OpenMetadata](../setup/guide/data-source/metadata/open-metadata.md)<br>- [ODCS (Open Data Contract Standard)](https://github.com/data-catering/data-caterer/issues/9)<br>- Amundsen<br>- Datahub<br>- Solace Event Portal<br>- Airflow<br>- [DBT](https://github.com/data-catering/data-caterer/issues/8)<br>- Manually insert create table statement from UI |
| Metadata discovery | Allow for schema and data profiling from external metadata sources | - :white_check_mark: [HTTP (OpenAPI spec)](../setup/guide/data-source/http/http.md)<br>- JMS<br>- Read from samples<br>- :white_check_mark: [OpenLineage metadata (Marquez)](../setup/guide/data-source/metadata/marquez.md)<br>- :white_check_mark: [OpenMetadata](../setup/guide/data-source/metadata/open-metadata.md)<br>- :white_check_mark: [Open Data Contract Standard (ODCS)](https://github.com/data-catering/data-caterer/issues/9)<br>- Amundsen<br>- Datahub<br>- Solace Event Portal<br>- Airflow<br>- [DBT](https://github.com/data-catering/data-caterer/issues/8)<br>- Manually insert create table statement from UI |
| Developer API | Scala/Java interface for developers/testers to create data generation and validation tasks | - :white_check_mark: [Scala](https://github.com/data-catering/data-caterer-example)<br>- :white_check_mark: [Java](https://github.com/data-catering/data-caterer-example) |
| Report generation | Generate a report that summarises the data generation or validation results | - :white_check_mark: [Report for data generated and validation rules](../sample/report/html/index.html) |
| UI portal | Allow users to access a UI to input data generation or validation tasks. Also be able to view report results | - :white_check_mark: [Base UI with create, edit and delete plan, connections and history](../get-started/quick-start.md)<br>- :white_check_mark: [Run on Mac, Linux and Windows](../get-started/quick-start.md)<br>- Metadata stored in database<br>- :white_check_mark: Store data generation/validation run information in file/database<br>- Preview of generated data<br>- Additional dialog to confirm delete and execute plan |
Expand Down

0 comments on commit 07d1265

Please sign in to comment.