@aigentsphere/openclaw-otel-observability 0.2.1

package/docs/backends/dynatrace.md

@@ -0,0 +1,168 @@
+# Dynatrace Setup
+
+Send OpenClaw telemetry directly to Dynatrace using OTLP.
+
+## Prerequisites
+
+- Dynatrace environment (SaaS or Managed)
+- API token with ingest permissions
+
+## Create API Token
+
+1. Go to **Settings** → **Access tokens**
+2. Click **Generate new token**
+3. Add scopes:
+   - `metrics.ingest`
+   - `logs.ingest`
+   - `openTelemetryTrace.ingest`
+4. Copy the token (starts with `dt0c01.`)
+
+## Configure OpenClaw
+
+Add to `~/.openclaw/openclaw.json`:
+
+```json
+{
+  "diagnostics": {
+    "enabled": true,
+    "otel": {
+      "enabled": true,
+      "endpoint": "https://{environment-id}.live.dynatrace.com/api/v2/otlp",
+      "headers": {
+        "Authorization": "Api-Token dt0c01.XXXXXXXX"
+      },
+      "serviceName": "openclaw-gateway",
+      "traces": true,
+      "metrics": true,
+      "logs": true
+    }
+  }
+}
+```
+
+Replace:
+- `{environment-id}` — Your Dynatrace environment ID (e.g., `abc12345`)
+- `dt0c01.XXXXXXXX` — Your API token
+
+### Dynatrace Managed
+
+For Dynatrace Managed, use your ActiveGate URL:
+
+```json
+{
+  "diagnostics": {
+    "enabled": true,
+    "otel": {
+      "enabled": true,
+      "endpoint": "https://{your-activegate}/e/{environment-id}/api/v2/otlp",
+      "headers": {
+        "Authorization": "Api-Token dt0c01.XXXXXXXX"
+      },
+      "serviceName": "openclaw-gateway"
+    }
+  }
+}
+```
+
+## Restart Gateway
+
+```bash
+openclaw gateway restart
+```
+
+## Verify in Dynatrace
+
+### Find Your Service
+
+1. Go to **Services** in Dynatrace
+2. Search for `openclaw-gateway`
+3. Click to view service details
+
+### View Traces
+
+1. Go to **Distributed traces**
+2. Filter by service: `openclaw-gateway`
+3. Click a trace to see spans
+
+### View Metrics
+
+1. Go to **Explore** → **Metrics**
+2. Search for `openclaw.`
+3. Available metrics:
+   - `openclaw.tokens` — Token usage
+   - `openclaw.cost.usd` — Cost tracking
+   - `openclaw.run.duration_ms` — Agent run times
+   - `openclaw.message.*` — Message processing
+   - `openclaw.queue.*` — Queue metrics
+
+### Create Dashboard
+
+Create a dashboard with:
+
+```sql
+-- Token usage by model
+timeseries sum(openclaw.tokens), by:{openclaw.model, openclaw.token}
+
+-- Cost over time
+timeseries sum(openclaw.cost.usd), by:{openclaw.model}
+
+-- Agent run duration
+timeseries avg(openclaw.run.duration_ms), by:{openclaw.model}
+```
+
+### View Logs
+
+1. Go to **Logs**
+2. Filter: `dt.entity.service = "openclaw-gateway"`
+3. View log records with severity and attributes
+
+## Example DQL Queries
+
+### Token Usage by Model
+```sql
+fetch spans
+| filter dt.entity.service == "openclaw-gateway"
+| summarize tokens = sum(openclaw.tokens.total), by:{openclaw.model}
+| sort tokens desc
+```
+
+### Average Run Duration
+```sql
+fetch spans
+| filter dt.entity.service == "openclaw-gateway"
+| filter matchesPhrase(span.name, "model")
+| summarize avg_duration = avg(openclaw.run.duration_ms)
+```
+
+### Error Rate
+```sql
+fetch logs
+| filter dt.entity.service == "openclaw-gateway"
+| filter loglevel == "ERROR"
+| summarize count = count()
+```
+
+## Troubleshooting
+
+### No Data in Dynatrace?
+
+1. **Check token permissions**: Ensure all three scopes are enabled
+2. **Verify endpoint URL**: Should be `https://{env-id}.live.dynatrace.com/api/v2/otlp`
+3. **Test connectivity**:
+   ```bash
+   curl -v "https://{env-id}.live.dynatrace.com/api/v2/otlp/v1/traces" \
+     -H "Authorization: Api-Token dt0c01.xxx"
+   ```
+
+### Service Not Appearing?
+
+- Wait 2-5 minutes for service detection
+- Send a few messages to generate telemetry
+- Check Dynatrace logs for ingest errors
+
+### 403 Forbidden?
+
+Token lacks required scopes. Regenerate with all three:
+- `metrics.ingest`
+- `logs.ingest`
+- `openTelemetryTrace.ingest`

package/docs/backends/generic-otlp.md

@@ -0,0 +1,166 @@
+# Generic OTLP Backend
+
+Any backend that supports OTLP (OpenTelemetry Protocol) can receive data from this plugin. Here are configuration snippets for popular backends.
+
+## Honeycomb
+
+```json
+{
+  "config": {
+    "endpoint": "https://api.honeycomb.io",
+    "protocol": "http",
+    "headers": {
+      "x-honeycomb-team": "<YOUR_API_KEY>"
+    }
+  }
+}
+```
+
+## New Relic
+
+```json
+{
+  "config": {
+    "endpoint": "https://otlp.nr-data.net",
+    "protocol": "http",
+    "headers": {
+      "api-key": "<YOUR_INGEST_LICENSE_KEY>"
+    }
+  }
+}
+```
+
+!!! note
+    For EU data centers, use `https://otlp.eu01.nr-data.net`.
+
+## Datadog
+
+Datadog doesn't support direct OTLP ingest — use the OTel Collector with the Datadog exporter.
+
+### Collector Config
+
+```yaml
+exporters:
+  datadog:
+    api:
+      key: "${DD_API_KEY}"
+      site: "datadoghq.com"  # or datadoghq.eu, etc.
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [datadog]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [datadog]
+```
+
+### Plugin Config
+
+Point at the local collector:
+
+```json
+{
+  "config": {
+    "endpoint": "http://localhost:4318"
+  }
+}
+```
+
+## SigNoz
+
+```json
+{
+  "config": {
+    "endpoint": "https://ingest.<region>.signoz.cloud:443",
+    "protocol": "http",
+    "headers": {
+      "signoz-access-token": "<YOUR_SIGNOZ_TOKEN>"
+    }
+  }
+}
+```
+
+Or self-hosted:
+
+```json
+{
+  "config": {
+    "endpoint": "http://<signoz-host>:4318"
+  }
+}
+```
+
+## Jaeger
+
+Jaeger supports OTLP natively since v1.35:
+
+```json
+{
+  "config": {
+    "endpoint": "http://<jaeger-host>:4318",
+    "protocol": "http"
+  }
+}
+```
+
+!!! note
+    Jaeger only supports traces, not metrics or logs via OTLP.
+
+## Splunk
+
+Use the OTel Collector with the Splunk HEC exporter:
+
+### Collector Config
+
+```yaml
+exporters:
+  splunk_hec:
+    token: "${SPLUNK_HEC_TOKEN}"
+    endpoint: "https://<splunk-host>:8088/services/collector"
+    source: "openclaw"
+    sourcetype: "otel"
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [splunk_hec]
+```
+
+## Elastic / Elasticsearch
+
+Use the OTel Collector with the Elasticsearch exporter:
+
+### Collector Config
+
+```yaml
+exporters:
+  elasticsearch:
+    endpoints: ["https://<elastic-host>:9200"]
+    user: "${ELASTIC_USER}"
+    password: "${ELASTIC_PASSWORD}"
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [elasticsearch]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [elasticsearch]
+```
+
+## Custom / Self-Hosted Collector
+
+If your backend isn't listed, you can almost certainly connect it via the OTel Collector. The [contrib distribution](https://github.com/open-telemetry/opentelemetry-collector-contrib) includes exporters for 50+ backends.
+
+1. Find your exporter in the [collector contrib registry](https://opentelemetry.io/ecosystem/registry/?language=collector)
+2. Add it to the collector config
+3. Point the plugin at `http://localhost:4318`

package/docs/backends/grafana.md

@@ -0,0 +1,167 @@
+# Grafana Integration
+
+Export OpenClaw traces to **Grafana Tempo** and metrics to **Grafana Mimir** (or Prometheus) for visualization in Grafana dashboards.
+
+## Grafana Cloud (Direct Export)
+
+### 1. Get Your OTLP Credentials
+
+1. Go to [grafana.com](https://grafana.com) → Your stack → **Connections** → **OpenTelemetry**
+2. Note the OTLP endpoint and generate an API token
+
+### 2. Configure the Plugin
+
+```json
+{
+  "plugins": {
+    "entries": {
+      "otel-observability": {
+        "enabled": true,
+        "config": {
+          "endpoint": "https://otlp-gateway-<region>.grafana.net/otlp",
+          "protocol": "http",
+          "serviceName": "openclaw-gateway",
+          "headers": {
+            "Authorization": "Basic <base64-of-instanceId:apiToken>"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+!!! tip "Creating the Basic auth header"
+    ```bash
+    echo -n "<instanceId>:<apiToken>" | base64
+    ```
+
+## Self-Hosted Grafana Stack
+
+### Docker Compose Addition
+
+Add Tempo and Grafana to the collector setup:
+
+```yaml
+services:
+  otel-collector:
+    # ... existing config ...
+
+  tempo:
+    image: grafana/tempo:latest
+    container_name: openclaw-tempo
+    ports:
+      - "3200:3200"   # Tempo API
+      - "4417:4317"   # OTLP gRPC (Tempo)
+    volumes:
+      - ./collector/tempo-config.yaml:/etc/tempo/config.yaml:ro
+    command: ["-config.file=/etc/tempo/config.yaml"]
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: openclaw-grafana
+    ports:
+      - "3000:3000"
+    environment:
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+    volumes:
+      - grafana-data:/var/lib/grafana
+
+volumes:
+  grafana-data:
+```
+
+### Tempo Configuration
+
+Create `collector/tempo-config.yaml`:
+
+```yaml
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: 0.0.0.0:4317
+
+storage:
+  trace:
+    backend: local
+    local:
+      path: /tmp/tempo/blocks
+    wal:
+      path: /tmp/tempo/wal
+
+metrics_generator:
+  storage:
+    path: /tmp/tempo/generator/wal
+```
+
+### Collector Config (Fan-Out)
+
+Update `collector/otel-collector-config.yaml` to export to both Dynatrace and Tempo:
+
+```yaml
+exporters:
+  otlphttp/dynatrace:
+    endpoint: "${DYNATRACE_ENDPOINT}"
+    headers:
+      Authorization: "Api-Token ${DYNATRACE_API_TOKEN}"
+
+  otlp/tempo:
+    endpoint: "tempo:4317"
+    tls:
+      insecure: true
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [otlphttp/dynatrace, otlp/tempo]
+```
+
+### Grafana Data Source
+
+1. Open Grafana at `http://localhost:3000`
+2. Go to **Configuration** → **Data Sources** → **Add data source**
+3. Select **Tempo**
+4. Set URL: `http://tempo:3200`
+5. Click **Save & Test**
+
+## Grafana Dashboards
+
+### Explore View
+
+1. Go to **Explore**
+2. Select the **Tempo** data source
+3. Search by service name: `openclaw-gateway`
+4. Or search by span name: `openclaw.agent.turn` or `tool.*`
+
+### Example Dashboard Panels
+
+**Token Usage Over Time** (Prometheus/Mimir):
+```promql
+sum(rate(openclaw_llm_tokens_total[5m])) by (model)
+```
+
+**LLM Latency P95** (Prometheus/Mimir):
+```promql
+histogram_quantile(0.95, rate(openclaw_llm_duration_bucket[5m]))
+```
+
+**Tool Call Rate** (Prometheus/Mimir):
+```promql
+sum(rate(openclaw_tool_calls_total[5m])) by (tool_name)
+```
+
+**Error Rate** (Prometheus/Mimir):
+```promql
+sum(rate(openclaw_llm_errors_total[5m])) / sum(rate(openclaw_llm_requests_total[5m])) * 100
+```
+
+!!! note "Metric name format"
+    OTel metrics use dots (`openclaw.llm.tokens.total`) but Prometheus converts them to underscores (`openclaw_llm_tokens_total`).

package/docs/backends/index.md

@@ -0,0 +1,49 @@
+# Backends
+
+The plugin exports telemetry via standard **OTLP** (OpenTelemetry Protocol), which means it works with any OpenTelemetry-compatible backend.
+
+## Supported Backends
+
+| Backend | Direct Export | Via Collector | Guide |
+|---------|:---:|:---:|-------|
+| **Dynatrace** | ✅ | ✅ | [Setup →](dynatrace.md) |
+| **Grafana (Tempo + Mimir)** | ✅ | ✅ | [Setup →](grafana.md) |
+| **Datadog** | ❌ | ✅ | [Generic →](generic-otlp.md) |
+| **Honeycomb** | ✅ | ✅ | [Generic →](generic-otlp.md) |
+| **New Relic** | ✅ | ✅ | [Generic →](generic-otlp.md) |
+| **Splunk** | ❌ | ✅ | [Generic →](generic-otlp.md) |
+| **Jaeger** | ✅ | ✅ | [Generic →](generic-otlp.md) |
+| **SigNoz** | ✅ | ✅ | [Generic →](generic-otlp.md) |
+| **OTel Collector** | — | — | [Setup →](otel-collector.md) |
+
+## Direct Export vs. Collector
+
+### Direct Export
+
+```mermaid
+flowchart LR
+    A[OpenClaw Plugin] -->|OTLP| B[Backend]
+```
+
+- Simpler setup — no extra components
+- Works well for single-backend setups
+- Backend credentials are in the OpenClaw config
+
+### Via OTel Collector (Recommended)
+
+```mermaid
+flowchart LR
+    A[OpenClaw Plugin] -->|OTLP| B[OTel Collector]
+    B -->|OTLP| C[Dynatrace]
+    B -->|OTLP| D[Grafana]
+    B -->|Prometheus| E[Prometheus]
+```
+
+- **Batching & retry** — handles network hiccups gracefully
+- **Processing** — filter, transform, and enrich data before export
+- **Fan-out** — send to multiple backends simultaneously
+- **Decoupled auth** — backend credentials stay on the collector, not in OpenClaw
+- **Sampling** — reduce data volume for high-traffic agents
+
+!!! tip "Recommendation"
+    Use the OTel Collector in production. Use direct export for quick development testing.