From 672caa998582800bd85eb583e5ed5aaef3b397b9 Mon Sep 17 00:00:00 2001 From: Robin Tang Date: Mon, 21 Oct 2024 09:28:13 -0700 Subject: [PATCH 1/2] Update README.md Signed-off-by: Robin Tang --- README.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index b98a7126e..f1eb822fb 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,15 @@ +

+ Artie Transfer +

-

Artie Transfer

-

⚡️ Blazing fast data replication between OLTP and OLAP databases ⚡️

+

⚡️ Blazing fast data replication between OLTP and OLAP databases ⚡️

@@ -11,7 +19,7 @@

-Artie Transfer is a real-time data replication solution for databases and data warehouses/data lakes. +Artie Transfer is a real-time data replication solution for databases and data warehouses/lakes. Typical ETL solutions rely on batched processes or schedulers (i.e. DAGs, Airflow), which means the data in the downstream data warehouse is often several hours to days old. This problem is exacerbated as data volumes grow, as batched processes take increasingly longer to run. @@ -21,11 +29,10 @@ Benefits of Artie Transfer: - Sub-minute data latency: always have access to live production data. - Ease of use: just set up a simple configuration file, and you're good to go! -- Automatic table creation and schema detection: Artie infers schemas and automatically merges changes to downstream destinations. -- Reliability: Artie has automatic retries and processing is idempotent. -- Scalability: handle anywhere from 1GB to 100+ TB of data. -- Monitoring: built-in error reporting along with rich telemetry statistics. - +- Automatic table creation and schema detection: Artie infers schemas and automatically merges changes to downstream destinations. +- Reliability: Artie has automatic retries and processing is idempotent. +- Scalability: handle anywhere from 1GB to 100+ TB of data. +- Monitoring: built-in error reporting along with rich telemetry statistics. Take a look at this [guide](#getting-started) to get started! From 43f3291dcbd706bf63c8faef4a0714c7ccfdb47f Mon Sep 17 00:00:00 2001 From: Robin Tang Date: Mon, 21 Oct 2024 09:30:13 -0700 Subject: [PATCH 2/2] Update README.md Signed-off-by: Robin Tang --- README.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/README.md b/README.md index f1eb822fb..dbeded2ce 100644 --- a/README.md +++ b/README.md @@ -42,18 +42,6 @@ Take a look at this [guide](#getting-started) to get started! -### Pre-requisites - -As you can see from the architecture diagram above, Artie Transfer is a Kafka consumer and expects CDC messages to be in a particular format. - -The optimal set-up looks something like this: -* [Debezium](https://github.com/debezium/debezium) or [Artie Reader](https://github.com/artie-labs/reader) depending on the source -* Kafka - * One Kafka topic per table, such that you can toggle the number of partitions based on throughput. - * The partition key should be the primary key for the table to avoid out-of-order writes at the row level. - -Please see the [supported section](#what-is-currently-supported) on what sources and destinations are supported. - ## Examples To run Artie Transfer's stack locally, please refer to the [examples folder](https://github.com/artie-labs/transfer/tree/master/examples).