diff --git a/other-examples/collector/singlestore/.env b/other-examples/collector/singlestore/.env deleted file mode 100644 index 0ce7db21..00000000 --- a/other-examples/collector/singlestore/.env +++ /dev/null @@ -1,17 +0,0 @@ -# New Relic API key to authenticate the call. -# docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key -NEW_RELIC_API_KEY= - -# The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed. -# docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/#review-settings -NEW_RELIC_OTLP_ENDPOINT=https://otlp.nr-data.net/ - -# ID for your SingleStore Org -SINGLESTORE_ORG= - -# ID for your SingleStore workspace group ID -SINGLESTORE_WORKSPACE_GROUP= - -# Your SingleStore API Key -# docs: https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key -SINGLESTORE_API_KEY= diff --git a/other-examples/collector/singlestore/README.md b/other-examples/collector/singlestore/README.md index 697d0e57..056a8c80 100644 --- a/other-examples/collector/singlestore/README.md +++ b/other-examples/collector/singlestore/README.md @@ -1,33 +1,79 @@ -# SingleStore OpenTelemetry metrics example setup +# Monitoring Singlestore with OpenTelemetry Collector -This example shows a setup for running a prometheus OpenTelemetry Collector in a docker container to scrape metrics from your SingleStore environment and post them the New Relic OTLP Collector Endpoint. +This simple example demonstrates monitoring [Singlestore](https://www.singlestore.com/) prometheus metrics with the [OpenTelemetry collector](https://opentelemetry.io/docs/collector/), using the [prometheus receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) and sending the data to New Relic via OTLP. -TODO: add link to doc site when available +## Requirements -## Prerequisites - -1. You must have a Docker daemon running. -2. You must have [Docker compose](https://docs.docker.com/compose/) installed. -3. You must have a SingleStore workspace running. +* You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. Docker desktop [includes a standalone Kubernetes server and client](https://docs.docker.com/desktop/kubernetes/) which is useful for local testing. +* [A New Relic account](https://one.newrelic.com/) +* [A New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key) +* [A Singlestore account](https://www.singlestore.com/cloud-trial/) with a [workspace group](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/creating-and-using-workspaces/) +* [A Singlestore API key](https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key) ## Running the example -First, set your environment variables in the `.env` file in this directory. For more information on the individual variables, reference the docs available below. +1. Update the `NEW_RELIC_API_KEY` values `SINGLESTORE_API_KEY` values in [secrets.yaml](./k8s/secrets.yaml) to your New Relic license key, and singlestore API key respectively. See [Singlestore docs](https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key) for obtaining API key / secret. + + ```yaml + # ...omitted for brevity + stringData: + # New Relic API key to authenticate the export requests. + # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key + NEW_RELIC_API_KEY: + # Set your Singlestore API Key. + # docs: https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key + SINGLESTORE_API_KEY: > + ``` + + * Note, be careful to avoid inadvertent secret sharing when modifying `secrets.yaml`. To ignore changes to this file from git, run `git update-index --skip-worktree k8s/secrets.yaml`. + + * If your account is based in the EU, update the `NEW_RELIC_OTLP_ENDPOINT` value in [collector.yaml](./k8s/collector.yaml) the endpoint to: [https://otlp.eu01.nr-data.net](https://otlp.eu01.nr-data.net) + + ```yaml + # ...omitted for brevity + env: + # The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed. + # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol + - name: NEW_RELIC_OTLP_ENDPOINT + value: https://otlp.eu01.nr-data.net + ``` + +2. Set the `SINGLESTORE_ORG_ID` and `SINGLESTORE_WORKSPACE_GROUP_ID` env var values in [collector.yaml](./k8s/collector.yaml). See [Singlestore docs](https://support.singlestore.com/hc/en-us/articles/12396547132564-Workspace-Group-ID-or-Cluster-ID) for details on obtaining org and workspace group ids. + + ```yaml + # ...omitted for brevity + # The Singlestore Org ID. + # docs: https://support.singlestore.com/hc/en-us/articles/12396547132564-Workspace-Group-ID-or-Cluster-ID + - name: SINGLESTORE_ORG_ID + value: + # The Singlestore Workspace Group ID. + # docs: https://support.singlestore.com/hc/en-us/articles/12396547132564-Workspace-Group-ID-or-Cluster-ID + - name: SINGLESTORE_WORKSPACE_GROUP_ID + value: + ``` -Once the variables are set, run the following command from the root directory to start the collector. +3. Run the application with the following command. -```shell -cd ./other-examples/collector/singlstore + ```shell + kubectl apply -f k8s/ + ``` + + * When finished, cleanup resources with the following command. This is also useful to reset if modifying configuration. + + ```shell + kubectl delete -f k8s/ + ``` + +## Viewing your data + +To review your Singlestore data in New Relic, navigate to "New Relic -> Query Your Data". To list the metrics reported, query for: -docker compose up ``` +FROM Metric SELECT uniques(metricName) WHERE otel.library.name = 'otelcol/prometheusreceiver' AND metricName LIKE 'singlestoredb%' +``` + +See [get started with querying](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/) for additional details on querying data in New Relic. -## Local Variable information +## Additional notes -| Variable | Description | Docs | -| -------- | ----------- | ---- | -| **NEW_RELIC_API_KEY** |New Relic Ingest API Key |[API Key docs](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/) | -| **NEW_RELIC_OTLP_ENDPOINT** |Default US OTLP endpoint is `https://otlp.nr-data.net` | [OTLP endpoint config docs](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/get-started/opentelemetry-set-up-your-app/#review-settings) | -| **SINGLESTORE_ORG** | ID of your SingleStore Organization |[SingleStore Organization docs](https://docs.singlestore.com/cloud/user-and-workspace-administration/manage-organizations/)| -| **SINGLESTORE_WORKSPACE_GROUP** | ID of your SingleStore Workspace Group | [SingleStore workspace docs](https://docs.singlestore.com/cloud/getting-started-with-singlestoredb-cloud/about-workspaces/what-is-a-workspace/) | -| **SINGLESTORE_API_KEY** | SingleStore API Key | [SingleStore API Key docs](https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key)| +The prometheus receiver includes `service.name` and `service.instance.id` resource attributes derived from job name and target configured in `.receivers.prometheus.config.scrape_configs`. As documented [here](https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-best-practices-resources/#services), New Relic considers any data with `service.name` as a service despite the fact that not all prometheus data sources are services. As a result, you can find a `singlestore` entity under "New Relic -> All Entities -> Services - OpenTelemetry", although the panels will not contain data because the scraped metrics do not represent APM data. diff --git a/other-examples/collector/singlestore/collector.yaml b/other-examples/collector/singlestore/collector.yaml deleted file mode 100644 index 6881b307..00000000 --- a/other-examples/collector/singlestore/collector.yaml +++ /dev/null @@ -1,26 +0,0 @@ -receivers: - prometheus: - config: - scrape_configs: - - job_name: "singlestore" - scrape_interval: 60s # Do not go any lower than this or you'll hit rate limits - static_configs: - - targets: ["api.singlestore.com"] - scheme: https - bearer_token: "$SINGLESTORE_API_KEY" - metrics_path: /v2/organizations/$SINGLESTORE_ORG/workspaceGroups/$SINGLESTORE_WORKSPACE_GROUP/metrics -processors: - batch: - -exporters: - otlphttp: - endpoint: $NEW_RELIC_OTLP_ENDPOINT - headers: - api-key: $NEW_RELIC_API_KEY - -service: - pipelines: - metrics: - receivers: [prometheus] - processors: [batch] - exporters: [otlphttp] \ No newline at end of file diff --git a/other-examples/collector/singlestore/docker-compose.yaml b/other-examples/collector/singlestore/docker-compose.yaml deleted file mode 100644 index e439f36d..00000000 --- a/other-examples/collector/singlestore/docker-compose.yaml +++ /dev/null @@ -1,15 +0,0 @@ -version: "3.6" - -services: - - otel-collector: - image: otel/opentelemetry-collector-contrib:0.92.0 - command: --config=/etc/otelcol/config.yaml - volumes: - - ./collector.yaml:/etc/otelcol/config.yaml - environment: - - NEW_RELIC_OTLP_ENDPOINT - - NEW_RELIC_API_KEY - - SINGLESTORE_ORG - - SINGLESTORE_WORKSPACE_GROUP - - SINGLESTORE_API_KEY diff --git a/other-examples/collector/singlestore/k8s/collector.yaml b/other-examples/collector/singlestore/k8s/collector.yaml new file mode 100644 index 00000000..8760c20b --- /dev/null +++ b/other-examples/collector/singlestore/k8s/collector.yaml @@ -0,0 +1,91 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: nr-singlestore +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: collector-config + namespace: nr-singlestore + labels: + app.kubernetes.io/name: collector-config +data: + collector-config: | + receivers: + prometheus: + config: + scrape_configs: + - job_name: "singlestore" + scrape_interval: 60s # Do not go any lower than this or you'll hit rate limits + static_configs: + - targets: ["api.singlestore.com"] + scheme: https + bearer_token: "${SINGLESTORE_API_KEY}" + metrics_path: /v2/organizations/${SINGLESTORE_ORG_ID}/workspaceGroups/${SINGLESTORE_WORKSPACE_GROUP_ID}/metrics + + processors: + batch: + + exporters: + otlphttp: + endpoint: ${NEW_RELIC_OTLP_ENDPOINT} + headers: + api-key: ${NEW_RELIC_API_KEY} + + service: + pipelines: + metrics: + receivers: [prometheus] + processors: [batch] + exporters: [otlphttp] +--- +apiVersion: v1 +kind: Pod +metadata: + name: collector + namespace: nr-singlestore + labels: + app.kubernetes.io/name: collector +spec: + containers: + - name: collector + image: otel/opentelemetry-collector-contrib:0.98.0 + env: + # The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed. + # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol + - name: NEW_RELIC_OTLP_ENDPOINT + value: https://otlp.nr-data.net/ + # The New Relic API key used to authenticate export requests. + # Defined in secrets.yaml + - name: NEW_RELIC_API_KEY + valueFrom: + secretKeyRef: + name: nr-singlestore-secret + key: NEW_RELIC_API_KEY + # The Singlestore API key. + # Defined in secrets.yaml + - name: SINGLESTORE_API_KEY + valueFrom: + secretKeyRef: + name: nr-singlestore-secret + key: SINGLESTORE_API_KEY + # The Singlestore Org ID. + # docs: https://support.singlestore.com/hc/en-us/articles/12396547132564-Workspace-Group-ID-or-Cluster-ID + - name: SINGLESTORE_ORG_ID + value: + # The Singlestore Workspace Group ID. + # docs: https://support.singlestore.com/hc/en-us/articles/12396547132564-Workspace-Group-ID-or-Cluster-ID + - name: SINGLESTORE_WORKSPACE_GROUP_ID + value: + volumeMounts: + - name: collector-config-vol + mountPath: /etc/otelcol-contrib + volumes: + - name: collector-config-vol + configMap: + name: collector-config + items: + - key: collector-config + path: config.yaml diff --git a/other-examples/collector/singlestore/k8s/secrets.yaml b/other-examples/collector/singlestore/k8s/secrets.yaml new file mode 100644 index 00000000..6e6bb8ee --- /dev/null +++ b/other-examples/collector/singlestore/k8s/secrets.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Secret +metadata: + name: nr-singlestore-secret + namespace: nr-singlestore +stringData: + # New Relic API key to authenticate the export requests. + # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key + NEW_RELIC_API_KEY: + # Set your Singlestore API Key. + # docs: https://support.singlestore.com/hc/en-us/articles/12396018910228-Creating-Management-API-Key + SINGLESTORE_API_KEY: >