OTEL Demo

This is a demo of how to use Open Telemetry (OTEL) instrumentation for traces and metrics.

Table of contents

• Usage
  • Docker Compose
  • Kubernetes
    • Manual installation
    • Automated installation
    • Testing of the app
• License
• Author

Usage

Docker Compose

Run Grafana Tempo, Grafana Tempo Web UI and the App frontend/backed via Docker Compose:

docker-compose up

Query the main endpoint:

curl http://localhost:8080

Query the metrics endpoint:

curl http://localhost:8080/metrics
curl http://localhost:8888/metrics

Kubernetes

Install local Kubernetes cluster using K3D:

export KUBECONFIG=~/.kube/kind_test1
k3d cluster create test1 -p '80:80@loadbalancer' -p '443:443@loadbalancer'

Manual installation

Add all required Helm repos:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add otel-demo https://jtyr.github.io/otel-demo
helm repo update

Install Kube Prometheus Stack:

cat <<END | helm upgrade --create-namespace --namespace prometheus --values - --install kps prometheus-community/kube-prometheus-stack
fullnameOverride: kps
prometheus:
  ingress:
    enabled: true
    hosts:
      - prometheus.localhost
alertmanager:
  ingress:
    enabled: true
    hosts:
      - alertmanager.localhost
grafana:
  enabled: false
END

Install Grafana:

cat <<END | helm upgrade --create-namespace --namespace grafanalabs --values - --install grafana grafana/grafana
adminPassword: admin
ingress:
  enabled: true
  hosts:
    - grafana.localhost
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        uid: prometheus
        type: prometheus
        url: http://kps-prometheus.prometheus:9090
        access: proxy
        isDefault: true
        editable: true
      - name: Loki
        uid: loki
        type: loki
        url: http://loki:3100
        access: proxy
        jsonData:
          derivedFields:
            - name: "traceID"
              matcherRegex: "traceID=(\\\\w+)"
              url: "\$\${__value.raw}"
              datasourceUid: tempo
        editable: true
      - name: Tempo
        uid: tempo
        type: tempo
        url: http://tempo:16686
        access: proxy
        editable: true
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        options:
          path: /var/lib/grafana/dashboards/default
dashboards:
  default:
    local-dashboard:
      url: https://raw.githubusercontent.com/jtyr/otel-demo/master/files/dashboard.json
END

Install Grafana Tempo:

helm upgrade --create-namespace --namespace grafanalabs --install tempo grafana/tempo

Install Grafana Loki:

cat <<END | helm upgrade --create-namespace --namespace grafanalabs --values - --install loki grafana/loki
tracing:
  jaegerAgentHost: tempo
ingress:
  enabled: true
  hosts:
    - host: loki.localhost
      paths:
        - /
END

Install Grafana Promtail:

cat <<END | helm upgrade --create-namespace --namespace grafanalabs --values - --install promtail grafana/promtail
config:
  lokiAddress: http://loki:3100/loki/api/v1/push
END

Install Fluent Bit:

cat <<END | helm upgrade --create-namespace --namespace grafanalabs --values - --install fluent-bit grafana/fluent-bit
loki:
  serviceName: loki.grafanalabs
END

Install OTEL Demo:

helm upgrade --create-namespace --namespace otel-demo --install otel-demo-backend otel-demo/otel-demo-backend
helm upgrade --create-namespace --namespace otel-demo --install otel-demo-frontend otel-demo/otel-demo-frontend

Automated installation

The above instruction, of how to install all the application onto a Kubernetes cluster, can be automatically managed by Argo CD.

First install the Argo CD:

cat <<END | helm upgrade --create-namespace --namespace argocd --values - --install argo-cd argo/argo-cd
fullnameOverride: argocd
configs:
  secret:
    # Password: admin
    argocdServerAdminPassword: \$2a\$10\$0/WMVu9LJUnTioI.748IxOFhhRX8rXR2OVzSJIsXPdAM8mOLfum1q
server:
  config:
    url: https://argo-cd.localhost
  extraArgs:
    - --insecure
  ingress:
    enabled: true
    hosts:
      - argo-cd.localhost
END

Deploy all the applications:

kubectl apply -f https://raw.githubusercontent.com/jtyr/otel-demo/master/argocd/otel-demo-github.yaml

Open Argo CD server in the web browser and watch how all is installed.

Testing of the app

Open via web browser:

• Grafana dashboard
• OTEL Demo - frontend
• OTEL Demo - frontend metrics

Query logs from command line:

export LOKI_ADDR=http://loki.localhost
logcli query -t '{namespace="otel-demo", instance=~"otel-demo-.*"}'

Check and tune error generation of the backend:

# Query the current value
kubectl run curl \
    --image curlimages/curl \
    --restart=Never \
    --rm \
    --tty \
    --stdin \
    --command -- \
    curl http://otel-demo-backend.otel-demo/api/features/errorGenerator
# Set a new value
# (-d parameter is a number of miliseconds; 0 = generator disabled)
kubectl run curl \
    --image curlimages/curl \
    --restart=Never \
    --rm \
    --tty \
    --stdin \
    --command -- \
    curl -X PUT -d 10 http://otel-demo-backend.otel-demo/api/features/errorGenerator

License

MIT

Author

Jiri Tyr