From 8541f97d49579f25a9919a1f90562ee18b52ebcd Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Wed, 29 May 2024 00:15:13 +0200 Subject: [PATCH] ETL: Add Apache Hop and Apache NiFi --- docs/integrate/etl/index.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/integrate/etl/index.md b/docs/integrate/etl/index.md index f778bd9..0d84680 100644 --- a/docs/integrate/etl/index.md +++ b/docs/integrate/etl/index.md @@ -40,6 +40,13 @@ Tutorials and resources about configuring the managed variants, Astro and CrateD - [Executable stack: Apache Kafka, Apache Flink, and CrateDB] +## Apache Hop + +- [Using Apache Hop with CrateDB] +- [CrateDB dialect for Apache Hop] +- [CrateDB Apache Hop Bulk Loader transform] + + ## Apache Kafka - [Data Ingestion using Kafka and Kafka Connect] @@ -47,6 +54,11 @@ Tutorials and resources about configuring the managed variants, Astro and CrateD - [Tutorial: Replicating data to CrateDB with Debezium and Kafka] +## Apache NiFi + +- [Connecting to CrateDB from Apache NiFi] + + ## Azure Functions - {ref}`azure-functions` @@ -113,7 +125,10 @@ A demo project which uses SSIS and ODBC to read and write data from CrateDB: [Build a data ingestion pipeline using Kafka, Flink, and CrateDB]: https://dev.to/crate/build-a-data-ingestion-pipeline-using-kafka-flink-and-cratedb-1h5o [Building a hot and cold storage data retention policy in CrateDB with Apache Airflow]: https://community.cratedb.com/t/cratedb-and-apache-airflow-building-a-hot-cold-storage-data-retention-policy/934 [Community Day: Stream processing with Apache Flink and CrateDB]: https://cratedb.com/blog/cratedb-community-day-2nd-edition-summary-and-highlights +[Connecting to CrateDB from Apache NiFi]: https://community.cratedb.com/t/connecting-to-cratedb-from-apache-nifi/647 [CrateDB and Apache Airflow: Building a data ingestion pipeline]: https://community.cratedb.com/t/cratedb-and-apache-airflow-building-a-data-ingestion-pipeline/926 +[CrateDB Apache Hop Bulk Loader transform]: https://hop.apache.org/manual/latest/pipeline/transforms/cratedb-bulkloader.html +[CrateDB dialect for Apache Hop]: https://hop.apache.org/manual/latest/database/databases/cratedb.html [CrateDB's PostgreSQL interface]: inv:crate-reference#interface-postgresql [Data Ingestion using Kafka and Kafka Connect]: https://cratedb.com/docs/crate/howtos/en/latest/integrations/kafka-connect.html [ETL pipeline using Apache Airflow with CrateDB (Source)]: https://github.com/astronomer/astro-cratedb-blogpost @@ -128,6 +143,7 @@ A demo project which uses SSIS and ODBC to read and write data from CrateDB: [Setting up data pipelines with CrateDB and Kestra]: https://community.cratedb.com/t/setting-up-data-pipelines-with-cratedb-and-kestra-io/1400 [Tutorial: Replicating data to CrateDB with Debezium and Kafka]: https://community.cratedb.com/t/replicating-data-to-cratedb-with-debezium-and-kafka/1388 [Updating stock market data automatically with CrateDB and Apache Airflow]: https://community.cratedb.com/t/updating-stock-market-data-automatically-with-cratedb-and-apache-airflow/1304 +[Using Apache Hop with CrateDB]: https://community.cratedb.com/t/using-apache-hop-with-cratedb/1754 [Using dbt with CrateDB]: https://community.cratedb.com/t/using-dbt-with-cratedb/1566 [Using SQL Server Integration Services with CrateDB]: https://github.com/crate/cratedb-examples/tree/main/application/microsoft-ssis [Webinar: How to replicate data from other databases to CrateDB with Debezium and Kafka]: https://cratedb.com/resources/webinars/lp-wb-debezium-kafka