From c4ae6307af6328c5b48909ea5a4d6d25ab2ceec7 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Wed, 10 Jan 2024 16:03:48 +0100 Subject: [PATCH 1/6] =?UTF-8?q?Configure=20Aiven=20for=20Apache=20Kafka?= =?UTF-8?q?=C2=AE=20multi-node=20service=20with=20Privatelink=20Prometheus?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- _toc.yml | 1 + .../howto/kafka-prometheus-privatelink.rst | 77 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 docs/products/kafka/howto/kafka-prometheus-privatelink.rst diff --git a/_toc.yml b/_toc.yml index b2fba7fea..3f0e5d45b 100644 --- a/_toc.yml +++ b/_toc.yml @@ -415,6 +415,7 @@ entries: - file: docs/products/kafka/howto/kafka-streams-with-aiven-for-kafka - file: docs/products/kafka/howto/flink-with-aiven-for-kafka - file: docs/products/kafka/howto/datadog-customised-metrics + - file: docs/products/kafka/howto/kafka-prometheus-privatelink - file: docs/products/kafka/howto/ksql-docker title: Use ksqlDB with Aiven for Apache Kafka - file: docs/products/kafka/howto/add-missing-producer-consumer-metrics diff --git a/docs/products/kafka/howto/kafka-prometheus-privatelink.rst b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst new file mode 100644 index 000000000..a6f807800 --- /dev/null +++ b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst @@ -0,0 +1,77 @@ +Configure Aiven for Apache Kafka®-Prometheus via Privatelink +============================================================ + +Ensure secure and efficient monitoring of your Aiven for Apache Kafka® multi-node service with Privatelink Prometheus integration. Privatelink establishes a secure connection between your network and Aiven to keep your Apache Kafka data private and secure. + + +Prerequisites +------------- + +Before you start, ensure you have the following: + +- :doc:`Aiven for Apache Kafka® ` service running. +- :doc:`Prometheus integration ` set up for your Aiven for Apache Kafka for extracting metrics. +- Necessary permissions to modify service configurations. + + +Configuration steps +-------------------- + +Step 1: Basic configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Begin by configuring Prometheus to scrape metrics from your Aiven for Apache Kafka service. This setup involves specifying various parameters for secure and accurate data retrieval. Folowing is an example configuration: + +.. code-block:: yaml + + scrape_configs: + - job_name: aivenmetrics + scheme: https + tls_config: + insecure_skip_verify: true + basic_auth: + username: + password: + http_sd_configs: + - url: + refresh_interval: 120s + tls_config: + insecure_skip_verify: true + basic_auth: + username: + password: + +**Configuration details**: + +- ``job_name``: Identifies the set of targets, e.g., ``aivenmetrics``. +- ``scheme``: Specifies the protocol, typically ``https``. +- ``tls_config``: Manages TLS settings. + + .. note:: + Setting ``insecure_skip_verify: true`` is crucial for Privatelink connections, as it permits Prometheus to disregard TLS certificate validation against host IP addresses, facilitating seamless connectivity. + +- ``basic_auth``: Provides authentication credentials for Kafka service access. +- ``http_sd_configs``: Configures HTTP service discovery. Includes: +- ``url``: The URI for Prometheus Privatelink service access. +- ``refresh_interval``: The frequency of target list refresh, e.g., ``120s``. + +.. note:: + The ``basic_auth`` and ``tls_config`` are specified twice - first for fetching the HTTP SD response and then for the metrics. This duplication is necessary because the same authentication and security settings are used to retrieve the service discovery information and scrape the metrics. + +Step 2: (Optional) Metadata and relabeling +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If your setup involves multiple Privatelink connections, this optional step allows for more precise target management. Use the ``__meta_privatelink_connection_id`` label to filter and distinguish metrics from different Privatelink connections. This approach is beneficial in scenarios requiring separate monitoring of distinct data streams. + +.. code-block:: yaml + + relabel_configs: + - source_labels: [__meta_privatelink_connection_id] + regex: 1 + action: keep + + +Related pages +-------------- + +* :doc:`Aiven for Apache Kafka® metrics available via Prometheus ` \ No newline at end of file From b7da548b661b03b9bbab08d4fa85cf4281ca6b1c Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Wed, 10 Jan 2024 16:22:56 +0100 Subject: [PATCH 2/6] updated redirect file to fix broken link issue --- _redirects | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_redirects b/_redirects index f4a334f00..c2cf402b9 100644 --- a/_redirects +++ b/_redirects @@ -65,6 +65,7 @@ /docs/platform/concepts/service-level-agreement /docs/platform/howto/project-support-center /docs/products/postgresql/reference/list-of-advanced-params /docs/products/postgresql/reference/advanced-params /docs/products/kafka/reference/kstream-data-write-issue https://aiven.io/changelog#06-12-2023 +/docs/tools/api/examples /docs/tools/api # Moved to https://aiven.io/developer @@ -91,7 +92,7 @@ /docs/community/challenge/the-rolling-challenge https://aiven.io/community/ /docs/tools/cli/account/account-authentication-method /docs/tools/cli/account /docs/tools/cli/card /docs/tools/cli/account -/docs/tools/api/examples /docs/tools/api + # Redirect from .index.html to specific page names for landing From 2fe0fe9dbdbbd46dfe24fc1400f93e3a67571c94 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Thu, 11 Jan 2024 12:06:05 +0100 Subject: [PATCH 3/6] incorporated review feedback --- .../howto/integrations/prometheus-metrics.rst | 29 +++++++++++++--- .../howto/kafka-prometheus-privatelink.rst | 33 +++++++++++-------- 2 files changed, 45 insertions(+), 17 deletions(-) diff --git a/docs/platform/howto/integrations/prometheus-metrics.rst b/docs/platform/howto/integrations/prometheus-metrics.rst index fd68a0799..09027f58b 100644 --- a/docs/platform/howto/integrations/prometheus-metrics.rst +++ b/docs/platform/howto/integrations/prometheus-metrics.rst @@ -48,7 +48,8 @@ Aiven offers Prometheus endpoints for your services. To enable this feature, tak At the top of the **Integrations** page, you will see the Prometheus integration listed and status ``active``. -#. From the **Integrations** page, go to the **Overview** page > the **Connection information** section > the **Prometheus** tab. +#. Next, navigate to the service's **Overview** page, and then locate the **Connection information** section. +#. Click on the **Prometheus** tab. #. Copy **Service URI**, and use it in your browser to access the Prometheus dashboard. .. topic:: Result @@ -59,10 +60,29 @@ Aiven offers Prometheus endpoints for your services. To enable this feature, tak There might be a slight delay of approximately one minute before the metrics become available. + Accessing Prometheus in a VPC '''''''''''''''''''''''''''''' -If you use a VPC in your project, to access Prometheus, you need to go to `Aiven Console `_ > your project > the service you want to monitor using Prometheus > the **Service settings** page > the **Cloud and network** section > the actions (**...**) menu > the **More network configurations**. In the **Network configuration** window, select **Add configuration options**, search for property ``public_access.prometheus``, enable it, and select **Save configuration**. +If you use a VPC in your project, follow these steps to access Prometheus: + +1. Access `Aiven Console `_. + +2. Select your project, and select the service you want to monitor using Prometheus. + +4. Click **Service settings** from the sidebar. + +5. In the **Cloud and network** section, click on the actions (**...**) menu. + +6. Choose **More network configurations**. + +7. In the **Network configuration** window, select **Add configuration options**. + +8. Search for the ``public_access.prometheus`` property and enable it. + +10. Click **Save configuration**. + + Configure Prometheus -------------------- @@ -133,7 +153,7 @@ Multi-node services .. note:: - For Aiven services with multiple nodes and a Replica URI, the primary DNS name does not include standby IP addresses. To track those, make sure to include the replica DNS names in the list. If you have ```` as ``public-example.aivencloud.com``, then you will need to add ``public-replica-example.aivencloud.com``. This applies to PostgreSQL®, MySQL®, and Redis®* services. + For Aiven services with multiple nodes and a Replica URI, the primary DNS name does not include standby IP addresses. To track those, make sure to include the replica DNS names in the list. If you have ```` as ``public-example.aivencloud.com``, then you will need to add ``public-replica-example.aivencloud.com``. This applies to PostgreSQL®, MySQL®, Apache Kafka®, and Redis®* services. View full list of metrics '''''''''''''''''''''''''' @@ -148,10 +168,11 @@ You can preview the full list of metrics in :doc:`Prometheus system metrics ` * :doc:`Datadog integration ` +* :doc:`Configure Prometheus for Aiven for Apache Kafka® via Privatelink ` diff --git a/docs/products/kafka/howto/kafka-prometheus-privatelink.rst b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst index a6f807800..14ebad5e2 100644 --- a/docs/products/kafka/howto/kafka-prometheus-privatelink.rst +++ b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst @@ -1,7 +1,7 @@ -Configure Aiven for Apache Kafka®-Prometheus via Privatelink -============================================================ +Configure Prometheus for Aiven for Apache Kafka® using Privatelink +==================================================================== -Ensure secure and efficient monitoring of your Aiven for Apache Kafka® multi-node service with Privatelink Prometheus integration. Privatelink establishes a secure connection between your network and Aiven to keep your Apache Kafka data private and secure. +You can integrate Prometheus with your Aiven for Apache Kafka® service using Privatelink for secure monitoring. This setup uses a Privatelink load balancer, which allows for efficient service discovery of Apache Kafka nodes and enables you to connect to your Aiven for Apache Kafka service using a private endpoint in your network or VPCs. Prerequisites @@ -17,10 +17,10 @@ Before you start, ensure you have the following: Configuration steps -------------------- -Step 1: Basic configuration +Basic configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Begin by configuring Prometheus to scrape metrics from your Aiven for Apache Kafka service. This setup involves specifying various parameters for secure and accurate data retrieval. Folowing is an example configuration: +Begin by configuring Prometheus to scrape metrics from your Aiven for Apache Kafka service. This setup involves specifying various parameters for secure data retrieval. Following is an example configuration: .. code-block:: yaml @@ -48,20 +48,23 @@ Begin by configuring Prometheus to scrape metrics from your Aiven for Apache Kaf - ``tls_config``: Manages TLS settings. .. note:: - Setting ``insecure_skip_verify: true`` is crucial for Privatelink connections, as it permits Prometheus to disregard TLS certificate validation against host IP addresses, facilitating seamless connectivity. + Setting ``insecure_skip_verify: true`` is crucial, as it permits Prometheus to disregard TLS certificate validation against host IP addresses, facilitating seamless connectivity. -- ``basic_auth``: Provides authentication credentials for Kafka service access. -- ``http_sd_configs``: Configures HTTP service discovery. Includes: -- ``url``: The URI for Prometheus Privatelink service access. -- ``refresh_interval``: The frequency of target list refresh, e.g., ``120s``. +- ``basic_auth``: Provides authentication credentials for Apache Kafka service access. +- ``http_sd_configs``: Configures HTTP Service Discovery. Includes: + + - ``url``: The URI for Prometheus Privatelink service access. + - ``refresh_interval``: The frequency of target list refresh, e.g., ``120s``. .. note:: - The ``basic_auth`` and ``tls_config`` are specified twice - first for fetching the HTTP SD response and then for the metrics. This duplication is necessary because the same authentication and security settings are used to retrieve the service discovery information and scrape the metrics. + The ``basic_auth`` and ``tls_config`` are specified twice - first for scraping the HTTP SD response and then to retrieve service metrics. This duplication is necessary because the same authentication and security settings are used to retrieve the service discovery information and scrape the metrics. -Step 2: (Optional) Metadata and relabeling +(Optional) Metadata and relabeling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If your setup involves multiple Privatelink connections, this optional step allows for more precise target management. Use the ``__meta_privatelink_connection_id`` label to filter and distinguish metrics from different Privatelink connections. This approach is beneficial in scenarios requiring separate monitoring of distinct data streams. +If your setup involves multiple Privatelink connections, you can leverage Prometheus's relabeling for better target management. This approach allows you to dynamically modify target label sets before scraping. + +To manage metrics from different Privatelink connections, include the ``__meta_privatelink_connection_id`` label in your configuration. This setup helps categorize and filter relevant metrics for each connection. .. code-block:: yaml @@ -71,6 +74,10 @@ If your setup involves multiple Privatelink connections, this optional step allo action: keep +The ``regex: 1`` in the configuration is a placeholder. Make sure to replace ``1`` with the actual Privatelink connection ID that you wish to monitor. + + + Related pages -------------- From 67eeb7d3eeffd089e3baddcb393ab65e9a2255aa Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Thu, 11 Jan 2024 12:55:14 +0100 Subject: [PATCH 4/6] Fixed list --- .../howto/integrations/prometheus-metrics.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/platform/howto/integrations/prometheus-metrics.rst b/docs/platform/howto/integrations/prometheus-metrics.rst index 09027f58b..8a0d2a776 100644 --- a/docs/platform/howto/integrations/prometheus-metrics.rst +++ b/docs/platform/howto/integrations/prometheus-metrics.rst @@ -70,17 +70,17 @@ If you use a VPC in your project, follow these steps to access Prometheus: 2. Select your project, and select the service you want to monitor using Prometheus. -4. Click **Service settings** from the sidebar. +3. Click **Service settings** from the sidebar. -5. In the **Cloud and network** section, click on the actions (**...**) menu. +4. In the **Cloud and network** section, click on the actions (**...**) menu. -6. Choose **More network configurations**. +5. Choose **More network configurations**. -7. In the **Network configuration** window, select **Add configuration options**. +6. In the **Network configuration** window, select **Add configuration options**. -8. Search for the ``public_access.prometheus`` property and enable it. +7. Search for the ``public_access.prometheus`` property and enable it. -10. Click **Save configuration**. +8. Click **Save configuration**. From 0bef712f0b7307320b1c247b80b923f597b3ce55 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Fri, 12 Jan 2024 09:41:10 +0100 Subject: [PATCH 5/6] Removed cross-link due to build failure --- docs/platform/howto/integrations/prometheus-metrics.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platform/howto/integrations/prometheus-metrics.rst b/docs/platform/howto/integrations/prometheus-metrics.rst index 8a0d2a776..44480dca2 100644 --- a/docs/platform/howto/integrations/prometheus-metrics.rst +++ b/docs/platform/howto/integrations/prometheus-metrics.rst @@ -175,4 +175,4 @@ Learn more about integrations with Aiven: * :doc:`Aiven integrations ` * :doc:`Datadog integration ` -* :doc:`Configure Prometheus for Aiven for Apache Kafka® via Privatelink ` +* Configure Prometheus for Aiven for Apache Kafka® via Privatelink From 887dc7c4a94e90b7b509e9556259ce45ef56f2d8 Mon Sep 17 00:00:00 2001 From: Harshini Rangaswamy Date: Fri, 12 Jan 2024 09:58:17 +0100 Subject: [PATCH 6/6] Fixed cross-link --- docs/products/kafka/howto/kafka-prometheus-privatelink.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/products/kafka/howto/kafka-prometheus-privatelink.rst b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst index 14ebad5e2..e621a5aa9 100644 --- a/docs/products/kafka/howto/kafka-prometheus-privatelink.rst +++ b/docs/products/kafka/howto/kafka-prometheus-privatelink.rst @@ -9,7 +9,7 @@ Prerequisites Before you start, ensure you have the following: -- :doc:`Aiven for Apache Kafka® ` service running. +- :doc:`Aiven for Apache Kafka® ` service running. - :doc:`Prometheus integration ` set up for your Aiven for Apache Kafka for extracting metrics. - Necessary permissions to modify service configurations.