clickhouse credentials integration

_toc.yml

@@ -784,6 +784,8 @@ entries:
             title: Federated queries
           - file: docs/products/clickhouse/concepts/clickhouse-tiered-storage
             title: Tiered storage
+          - file: docs/products/clickhouse/concepts/data-integration-overview
+            title: Data integration
       - file: docs/products/clickhouse/howto
         title: HowTo
         entries:

docs/products/clickhouse/concepts.rst

@@ -26,6 +26,10 @@ Aiven service
         :shadow: md
         :margin: 2 2 0 0
 
+    .. grid-item-card:: :doc:`Data service integration with Aiven for ClickHouse® </docs/products/clickhouse/concepts/data-integration-overview>`
+        :shadow: md
+        :margin: 2 2 0 0
+
 General
 -------
 

docs/products/clickhouse/concepts/data-integration-overview.rst

@@ -0,0 +1,67 @@
+Aiven for ClickHouse® data service integrations
+===============================================
+
+Connect an Aiven for ClickHouse® service with another Aiven service or external data source to make your data available in the Aiven for ClickHouse service. Depending on your use case, select either the managed-database integration or the managed-credentials integration.
+
+About data service integrations
+-------------------------------
+
+Depending on objectives and actual objects of the integration, Aiven for ClickHouse supports two types of integrations:
+
+* :doc:`Observability integrations </docs/products/clickhouse/howto/list-integrations>`: connecting to other services (either in-Aiven or external) to expose and process logs and metrics
+* Data service integrations: connecting to other services (either in-Aiven or external) to use them as data sources
+
+In-Aiven integrations vs external integrations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By enabling data service integrations, you create streaming data pipelines across services. Depending on where the services are located, you can have either in-Aiven integrations (between Aiven services) or external integrations (between an Aiven service and an external data source or application).
+
+For integrating with external data sources, Aiven for ClickHouse provides two types of data service integrations:
+
+* Managed databases
+* Managed credentials
+
+.. _managed-credentials-integration:
+
+Managed credentials integration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The managed credentials integration uses the `ClickHouse named collections' <https://clickhouse.com/docs/en/operations/named-collections>`_ logic. It allows storing key-value pairs that are to be used as credentials when connecting to external data sources. With the managed credentials integration enabled, querying data is easier, quicker, since you no longer need connections parameters in each query. They are stored and available from in-Aiven credential storage.
+
+On top of integrating credentials, the managed credentials integration allows integrating data from external sources. For that purpose, you create tables using table engines.
+
+.. seealso::
+
+   For information on how table engines work in Aiven for ClickHouse services, preview *Engines: database and table* in :doc:`Aiven for ClickHouse® service architecture </docs/products/clickhouse/concepts/service-architecture>`. For the list of table engines available in Aiven for ClickHouse, check :doc:`Supported table engines </docs/products/clickhouse/reference/supported-table-engines>`.
+
+.. _managed-databases-integration:
+
+Managed databases integration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The managed databases integration allows using a database engine for handling your external data. When enabled, this type of integration provides you with an automatically created database, where you can have your data ingested.
+
+.. note::
+
+   External PostgreSQL® data sources (endpoints) support both the managed credentials and the managed databases.
+
+.. seealso::
+
+   For information on how database engines work in Aiven for ClickHouse services, preview *Engines: database and table* in :doc:`Aiven for ClickHouse® service architecture </docs/products/clickhouse/concepts/service-architecture>`. For more information on ClickHouse database engines, check out `Database engines <https://clickhouse.com/docs/en/engines/database-engines>`_.
+
+.. _integration-limitations:
+
+Limitations
+-----------
+
+* Aiven for ClickHouse supports data service integrations with Apache Kafka® and PostgreSQL®, both in-Aiven and external.
+* You can set up the managed credentials integration with external PostgreSQL and MySQL data stores only.
+* External PostgreSQL data sources (endpoints) support both the managed credentials and the managed databases.
+* External MySQL data sources (endpoints) support the managed credentials integration only.
+
+Related pages
+---------------
+
+* :doc:`Set up Aiven for ClickHouse® data service integrations </docs/products/clickhouse/howto/data-service-integration>`
+* :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>`
+* :doc:`Integrate your Aiven for ClickHouse® service </docs/products/clickhouse/howto/list-integrations>`

docs/products/clickhouse/howto/data-service-integration.rst

@@ -1,105 +1,158 @@
-Manage Aiven for ClickHouse® data service integrations
+Set up Aiven for ClickHouse® data service integrations
 ======================================================
 
-Aiven for ClickHouse supports two types of integrations:
-
-- Regular integrations: Logs, metrics, dataflow/replication, and authentication integrations among your Aiven services and with external applications
-- Data service integrations: Integrations with other Aiven services to use them as data sources
-
-This article details how to set up and use data service integrations in Aiven for ClickHouse.
+Connect an Aiven for ClickHouse® service with another Aiven service or external data source to make your data available in the Aiven for ClickHouse service. Depending on your use case, select either the managed-database integration or the managed-credentials integration.
 
 .. seealso::
 
-    For information on how to set up and use regular integrations in Aiven for ClickHouse, see intregration guides in :doc:`Integrate your Aiven for ClickHouse® service </docs/products/clickhouse/howto/list-integrations>`.
-
-About data service integrations
--------------------------------
-
-By enabling data service integrations in Aiven for ClickHouse, you create streaming data pipelines across services. Aiven for ClickHouse supports data service integrations with Aiven for Kafka® and Aiven for PostgreSQL®.
-
-You can create Aiven for ClickHouse® data service integrations in the `Aiven web console <https://console.aiven.io/>`_.
-
-.. topic:: Integration databases
-
-   When creating integrations in the **Data service integrations** wizard, you can also create integration databases connected to the services you are integrating with.
-
-   If you prefer to create a data service integration without adding integration databases, you can create integration databases for your service any time later. See :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>` for guidance on how to do that.
+   For information on data service integraton types and methods available with Aiven for ClickHouse®, check out :doc:`About Aiven for ClickHouse® data service integrations </docs/products/clickhouse/concepts/data-integration-overview>`.
 
 Prerequisites
 -------------
 
-* Aiven account
-* Access to `Aiven web console <https://console.aiven.io/>`_
+* Make sure you are aware of and understand :ref:`Limitations <integration-limitations>`.
+* Make sure you have an Aiven organization, project, and Aiven for ClickHouse service.
+* Make sure you have access to the `Aiven Console <https://console.aiven.io/>`_.
 
 .. _create-data-service-integration:
 
 Create data service integrations
 --------------------------------
 
-#. Log in to the `Aiven web console <https://console.aiven.io/>`_.
-#. In the **Services** page, select an Aiven for ClickHouse service you want to integrate with a data service.
-#. Select **Get started** from the **Integrate your Aiven for ClickHouse** section in the **Overview** page of your service.
-#. In the **Data service integrations** wizard, select one of the following options:
+#. Log in to the `Aiven Console <https://console.aiven.io/>`_.
+#. On the **Services** page, select an Aiven for ClickHouse service you want to integrate with a data service.
+#. From the **Integrate your Aiven for ClickHouse** section on the **Overview** page of your service, select **Get started** (to create your first integration) or the **+** icon (to add another integration).
+
+As a result, the **Data service integrations** wizard opens, showing the **Select data service type** dropdown menu and the list of all data sources available for integration. At this point you need to decide if you want to integrate with an Aiven service or an external endpoint and continue as instructed in either :ref:`Integrate with Aiven services <integrate-aiven-services>` or :ref:`Integrate with external data sources <integrate-external-services>` respectively.
+
+.. _integrate-aiven-services:
+
+Integrate with Aiven services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-   **Option 1: Create a new service and integrate it**
+Integrate with a new Aiven service
+''''''''''''''''''''''''''''''''''
 
-   To create an integration with a **new service**: 
+To create an integration with a new service, take the following steps in the **Data service integrations** wizard:
 
-   #. Make sure the checkboxes for both service types are unchecked.
-   #. In the **Data service integrations** view, select **Create service**.
-   #. :doc:`Set up the new service </docs/platform/howto/create_new_service>`.
-   #. Come back to your primary service and create an integration to the newly-created service.
-      For that purpose, skip the steps that follow and start over with building your integration using this
-      instruction but now follow the steps below about **integrating with an existing service**.
+#. Use the **Select data service type** dropdown menu to select the option for no data service type.
+#. In the **Data service integrations** view, select **Create service**.
+#. :doc:`Set up the new service </docs/platform/howto/create_new_service>`.
+#. Come back to your primary service and create an integration to the newly-created service.
+   For that purpose, skip the steps that follow and start over with building your integration using this
+   instruction but now follow the steps in :ref:`Integrate with an existing Aiven service <integrate-existing-aiven-services>`.
+
+.. _integrate-existing-aiven-services:
 
-   **Option 2: Use an existing service and integrate it**
+Integrate with an existing Aiven service
+''''''''''''''''''''''''''''''''''''''''
 
-   To create an integration with an **existing service**:
+To create an integration with an existing service, take the following steps in the **Data service integrations** wizard:
 
-   #. Select a service of the chosen type from the list of services available for integration.
-   #. Select **Continue** and proceed to the next step to integrate the database.
+#. Select a type of an Aiven service you want to integrate with using the **Select data service type** dropdown menu.
+#. Select a service of the chosen type from the list of services available for integration.
+#. Select **Continue** and proceed to the next step to integrate the database.
+#. Select either **Enable without databases** or **Add databases** depending on whether you want to enable your integration with databases:
 
-#. In the **Integration databases** view, select either **Enable without databases** or **Add databases** depending on whether you want to enable your integration with databases:
+.. note::
 
-   - To enable your integration **with** databases:
+   If you prefer to create a data service integration without adding integration databases, you can create integration databases for your service any time later. See :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>` for guidance on how to do that.
 
-     #. In the **Integration databases** view, select **Add databases**.
-     #. In the **Add integration databases** section, enter database names and schema names and select **Enable** when ready.
+- Enable your integration with databases:
 
-        You can preview the created databases by selecting **Databases  tables** from the sidebar.
+  #. Select **Add databases**.
+  #. Enter database names and schema names and select **Enable**.
 
-   - To enable your integration **without** databases
+     You can preview the created databases by selecting **Databases and tables** from the sidebar.
 
-     #. In the **Integration databases** view, select **Enable without databases**.
+- Enable your integration without databases:
+
+  #. Select **Enable without databases**.
 
-        You can preview the created integration by selecting **Overview** from the sidebar.
+     You can preview the created integration by selecting **Overview** from the sidebar.
+
+.. _integrate-external-services:
+
+Integrate with external data sources
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For integration with external data sources, Aiven for ClickHouse offers two methods:
+
+* :ref:`Managed databases <managed-databases-integration>`
+* :ref:`Managed credentials <managed-credentials-integration>`
+
+To create an integration with an external data source, take the following steps in the **Data service integrations** wizard:
+
+#. Select a type of an external service you want to integrate with using the **Select data service type** dropdown menu.
+#. Select an external service of the chosen type from the list of services available for integration.
+#. Select an integration method: either **Managed databases**  or **Managed credentials**.
+#. Continue as instructed either in :ref:`Integrate using managed databases <integrate-managed-databases>` or in :ref:`Integrate using managed credentials <integrate-managed-credentials>` depending on a chosen method.
+
+.. _integrate-managed-databases:
+
+Integrate using managed databases
+'''''''''''''''''''''''''''''''''
+
+The **Managed databases** integration uses databases engines and, when enabled, automatically creates databases in your Aiven for ClickHouse, where you can ingest your external data.
+
+1. Enable the **Managed databases** integration by selecting **Managed databases** and confirming your choice by selecting **Continue**.
+2. Populate your automatically created databases with your external data using the following query:
+
+.. code:: bash
+
+   SELECT data 
+   FROM ext-postgresql-resource-name.your-pg-table-name
+
+.. _integrate-managed-credentials:
+
+Integrate using managed credentials
+'''''''''''''''''''''''''''''''''''
+
+The **Managed credentials** integration supports storing connection parameters in Aiven and allows you to create tables for your external data.
+
+1. Enable the **Managed credentials** integration by selecting **Managed credentials** and confirming your choice by selecting **Continue**.
+2. Create tables using :doc:`table engines </docs/products/clickhouse/reference/supported-table-engines>`:
+
+.. code:: bash
+
+   SELECT data 
+   FROM postgresql(ext-postgresql-resource-name,
+        database='defaultdb',
+        table='your-pg-table-name')
+
+.. note::
+
+   The connection parameters for your integration are stored in Aiven and automatically seeded in your external data queries.
+
+You can access your credentials storage by passing your external service name in the following query:
+
+.. code:: bash
+
+   SELECT * 
+   FROM postgresql(ext-postgresql-resource-name);
 
 View data service integrations
 ------------------------------
 
-1. Log in to the `Aiven web console <https://console.aiven.io/>`_.
-2. In the **Services** page, select an Aiven for ClickHouse service you want to check integrations for.
-3. In the **Overview** page of your service, find the **Data service integration** section at the top and discover your integrations grouped according to service types (PostgreSQL or Apache Kafka).
-
-4. Select the meatball menu for a particular service group to preview active data service integrations within that group.
+1. Log in to the `Aiven Console <https://console.aiven.io/>`_.
+2. On the **Services** page, select an Aiven for ClickHouse service you want to check integrations for.
+3. On the **Overview** page of your service, find the **Data service integration** section and discover your integrations grouped according to service types.
+4. Select the **>** icon for a particular service group to preview active data service integrations within that group.
 
 Stop data service integrations
 ------------------------------
 
-1. Log in to the `Aiven web console <https://console.aiven.io/>`_.
+1. Log in to the `Aiven Console <https://console.aiven.io/>`_.
 2. In the **Services** page, select an Aiven for ClickHouse service you want to stop integrations for.
-3. In the **Overview** page of your service, find the **Data service integration** section at the top and select the meatball menu for a service group that your unwanted integration belongs to.
-
+3. In the **Overview** page of your service, find the **Data service integration** section and select the **>** icon for a service group that your unwanted integration belongs to.
 4. From the **Active data service integrations** list, select the service integration that you no longer need and select **Disconnect integration**.
+5. Make sure you understand the impact of disconnecting from a service explained in the **Warning** popup, and select **Disconnect integration** if you accept erasing all the databases and configuration information.
 
-5. In the **Warning** popup, study the impact of disconnecting from a service and select **Disconnect integration** if you accept erasing all the databases and configuration information.
-
-.. topic:: Result
-
-   Your integration has been removed along with all the corresponding databases and configuration information.
+Your integration has been terminated, and all the corresponding databases and configuration information has been deleted.
 
 Related pages
 ---------------
 
+* :doc:`About Aiven for ClickHouse® data service integrations </docs/products/clickhouse/concepts/data-integration-overview>`
 * :doc:`Manage Aiven for ClickHouse® integration databases </docs/products/clickhouse/howto/integration-databases>`
 * :doc:`Integrate your Aiven for ClickHouse® service </docs/products/clickhouse/howto/list-integrations>`