clickhouse credentials integration

docs/products/clickhouse/howto/data-service-integration.rst

@@ -1,86 +1,149 @@
-Manage Aiven for ClickHouse® data service integrations
+Set up Aiven for ClickHouse® data service integrations
 ======================================================
 
-Aiven for ClickHouse supports two types of integrations:
+Connect an Aiven for ClickHouse® service with another Aiven service or external data source to make your data available in the Aiven for ClickHouse service. Depending on your use case, select either the managed-database integration or the managed-credentials integration.
 
-- Regular integrations: Logs, metrics, dataflow/replication, and authentication integrations among your Aiven services and with external applications
-- Data service integrations: Integrations with other Aiven services to use them as data sources
+About data service integrations
+-------------------------------
+
+Depending on objectives and actual objects of the integration, Aiven for ClickHouse supports two types of integrations:
+
+* :doc:`Observability integrations </docs/products/clickhouse/howto/list-integrations>`: connecting to other services (either in-Aiven or external) to expose and process logs and metrics
+* Data service integrations: connecting to other services (either in-Aiven or external) to use them as data sources
+
+In-Aiven integrations vs external integrations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By enabling data service integrations, you create streaming data pipelines across services. Depending on where the services are located, you can have either in-Aiven integrations (between Aiven services) or external integrations (between an Aiven service and an external data source or application).
+
+For integrating with external data sources, Aiven for ClickHouse provides two types of data service integrations:
+
+* Managed databases
+* Managed credentials
+
+.. _managed-credentials-integration:
+
+Managed credentials integration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-This article details how to set up and use data service integrations in Aiven for ClickHouse.
+The managed credentials integration uses the `ClickHouse named collections' <https://clickhouse.com/docs/en/operations/named-collections>`_ logic. It allows storing key-value pairs that are to be used as credentials when connecting to external data sources. With the managed credentials integration enabled, querying data is easiler, quiecker, since you no longer need connections parameters in each query. They are stored and available from in-Aiven credential storage.
+
+On top of integrating credentials, the managed credentials integration allows integrating data from external sources. For that purpose, you create tables using table engines.
 
 .. seealso::
 
-    For information on how to set up and use regular integrations in Aiven for ClickHouse, see intregration guides in :doc:`Integrate your Aiven for ClickHouse® service </docs/products/clickhouse/howto/list-integrations>`.
+   For information on how table engines work in Aiven for ClickHouse services, preview *Engines: database and table* in :doc:`Aiven for ClickHouse® service architecture </docs/products/clickhouse/concepts/service-architecture>`. For the list of table engines available in Aiven for ClickHouse, check :doc:`Supported table engines </docs/products/clickhouse/reference/supported-table-engines>`.
 
-About data service integrations
--------------------------------
+.. _managed-databases-integration:
 
-By enabling data service integrations in Aiven for ClickHouse, you create streaming data pipelines across services. Aiven for ClickHouse supports data service integrations with Aiven for Kafka® and Aiven for PostgreSQL®.
+Managed databases integration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-You can create Aiven for ClickHouse® data service integrations in the `Aiven web console <https://console.aiven.io/>`_.
+The managed databases integration allows using a database engine for handling your external data. When enabled, this type of integration provides you with an automatically created database, where you can have your data ingested.
 
-.. topic:: Integration databases
-
-   When creating integrations in the **Data service integrations** wizard, you can also create integration databases connected to the services you are integrating with.
+.. note::
 
-   If you prefer to create a data service integration without adding integration databases, you can create integration databases for your service any time later. See :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>` for guidance on how to do that.
+   External PostgeSQL data sources (endpoints) support both the managed credentials and the managed databases.
+
+.. seealso::
+
+   For information on how database engines work in Aiven for ClickHouse services, preview *Engines: database and table* in :doc:`Aiven for ClickHouse® service architecture </docs/products/clickhouse/concepts/service-architecture>`. For more information on ClickHouse database engines, check out `Database engines <https://clickhouse.com/docs/en/engines/database-engines>`_.
+
+Limitiations
+------------
+
+* Aiven for ClickHouse supports data service integrations with Apache Kafka® and PostgreSQL®, both in-Aiven and external.
+* You can set up the managed credentials integration with external PostgreSQL and MySQL data stores only.
+* External PostgeSQL data sources (endpoints) support both the managed credentials and the managed databases.
+* External MySQL data sources (endpoints) support the managed credentials integration only.
 
 Prerequisites
 -------------
 
-* Aiven account
-* Access to `Aiven web console <https://console.aiven.io/>`_
+* Aiven organization, project, and Aiven for ClickHouse service
+* Access to the `Aiven Console <https://console.aiven.io/>`_
 
 .. _create-data-service-integration:
 
 Create data service integrations
 --------------------------------
 
-#. Log in to the `Aiven web console <https://console.aiven.io/>`_.
-#. In the **Services** page, select an Aiven for ClickHouse service you want to integrate with a data service.
-#. Select **Get started** from the **Integrate your Aiven for ClickHouse** section in the **Overview** page of your service.
-#. In the **Data service integrations** wizard, select one of the following options:
+#. Log in to the `Aiven Console <https://console.aiven.io/>`_.
+#. On the **Services** page, select an Aiven for ClickHouse service you want to integrate with a data service.
+#. Select **Get started** from the **Integrate your Aiven for ClickHouse** section on the **Overview** page of your service.
+#. In the **Data service integrations** wizard, select an Aiven service or an external endpoint you want to integrate with and continue as instructed in :ref:`Integrate with Aiven services <integrate-aiven-services>` and :ref:`Integrate with external data sources <integrate-external-services>` respectively.
+
+.. _integrate-aiven-services:
+
+Integrate with Aiven services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-   **Option 1: Create a new service and integrate it**
+Integrate with a new service
+''''''''''''''''''''''''''''
 
-   To create an integration with a **new service**: 
+To create an integration with a **new service**: 
 
-   #. Make sure the checkboxes for both service types are unchecked.
-   #. In the **Data service integrations** view, select **Create service**.
-   #. :doc:`Set up the new service </docs/platform/howto/create_new_service>`.
-   #. Come back to your primary service and create an integration to the newly-created service.
-      For that purpose, skip the steps that follow and start over with building your integration using this
-      instruction but now follow the steps below about **integrating with an existing service**.
+#. Make sure the checkboxes for both service types are unchecked.
+#. In the **Data service integrations** view, select **Create service**.
+#. :doc:`Set up the new service </docs/platform/howto/create_new_service>`.
+#. Come back to your primary service and create an integration to the newly-created service.
+   For that purpose, skip the steps that follow and start over with building your integration using this
+   instruction but now follow the steps below about **integrating with an existing service**.
 
-   **Option 2: Use an existing service and integrate it**
+Integrate with an existing service
+''''''''''''''''''''''''''''''''''
 
-   To create an integration with an **existing service**:
+To create an integration with an **existing service**:
 
-   #. Select a service of the chosen type from the list of services available for integration.
-   #. Select **Continue** and proceed to the next step to integrate the database.
-
+#. Select a service of the chosen type from the list of services available for integration.
+#. Select **Continue** and proceed to the next step to integrate the database.
 #. In the **Integration databases** view, select either **Enable without databases** or **Add databases** depending on whether you want to enable your integration with databases:
 
-   - To enable your integration **with** databases:
+.. note::
 
-     #. In the **Integration databases** view, select **Add databases**.
-     #. In the **Add integration databases** section, enter database names and schema names and select **Enable** when ready.
+   If you prefer to create a data service integration without adding integration databases, you can create integration databases for your service any time later. See :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>` for guidance on how to do that.
 
-        You can preview the created databases by selecting **Databases  tables** from the sidebar.
+- To enable your integration **with** databases:
 
-   - To enable your integration **without** databases
+  #. In the **Integration databases** view, select **Add databases**.
+  #. In the **Add integration databases** section, enter database names and schema names and select **Enable** when ready.
 
-     #. In the **Integration databases** view, select **Enable without databases**.
+     You can preview the created databases by selecting **Databases  tables** from the sidebar.
+
+- To enable your integration **without** databases
+
+  #. In the **Integration databases** view, select **Enable without databases**.
 
-        You can preview the created integration by selecting **Overview** from the sidebar.
+     You can preview the created integration by selecting **Overview** from the sidebar.
+
+.. _integrate-external-services:
+
+Integrate with external data sources
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For integration with external data sources, Aiven for ClickHouse offers two methods:
+
+* :ref:`Managed databases <managed-databases-integration>`
+* :ref:`Managed credentials <managed-credentials-integration>`
+
+In the **Data service integrations** wizard, select what type of integration you want to use to connect to your external source and continue as instructed either in :ref:`Integrate using managed databases <integrate-managed-databases>` or in :ref:`Integrate using managed credentials <integrate-managed-credentials>` depending in a chosen method.
+
+.. _integrate-managed-databases:
+
+Integrate using managed databases
+'''''''''''''''''''''''''''''''''
+
+.. _integrate-managed-credentials:
+
+Integrate using managed credentials
+'''''''''''''''''''''''''''''''''''
 
 View data service integrations
 ------------------------------
 
-1. Log in to the `Aiven web console <https://console.aiven.io/>`_.
-2. In the **Services** page, select an Aiven for ClickHouse service you want to check integrations for.
-3. In the **Overview** page of your service, find the **Data service integration** section at the top and discover your integrations grouped according to service types (PostgreSQL or Apache Kafka).
-
+1. Log in to the `Aiven Console <https://console.aiven.io/>`_.
+2. On the **Services** page, select an Aiven for ClickHouse service you want to check integrations for.
+3. On the **Overview** page of your service, find the **Data service integration** section at the top and discover your integrations grouped according to service types (PostgreSQL or Apache Kafka).
 4. Select the meatball menu for a particular service group to preview active data service integrations within that group.
 
 Stop data service integrations