openclaw-logfire-observability 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 rita-aga
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,197 @@
+# logfire-observability
+
+Full OpenClaw observability in [Pydantic Logfire](https://logfire.pydantic.dev). Get agent traces, tool calls, metrics, and logs — all in one dashboard.
+
+This setup combines two plugins:
+
+| Plugin | What it sends to Logfire | Source |
+|--------|-------------------------|--------|
+| **logfire-observability** (this plugin) | Agent→tool trace hierarchy with params, results, and parent-child nesting | Custom, ships here |
+| **diagnostics-otel** (built-in) | Metrics (tokens, cost, duration), logs, webhook/queue/session telemetry | Ships with OpenClaw |
+
+Both are configured to export to Logfire via OTLP. Together they give you full coverage.
+
+## Quick start
+
+### 1. Get a Logfire token
+
+1. Go to [logfire.pydantic.dev](https://logfire.pydantic.dev)
+2. Create a project (or use an existing one)
+3. Go to Settings > Write Tokens > Create Token
+4. Copy the `pylf_v1_us_...` token
+
+### 2. Install this plugin
+
+```bash
+# From the repo root
+cd openclaw-plugins/logfire-observability
+npm install
+
+# Link into your OpenClaw instance
+openclaw plugins install -l /path/to/openclaw-plugins/logfire-observability
+```
+
+Or copy the folder to `~/.openclaw/extensions/logfire-observability/` and run `npm install` there.
+
+### 3. Configure both plugins
+
+Add the following to your `openclaw.json` (or `~/.clawdbot/openclaw.json`).
+
+Replace `YOUR_TOKEN_HERE` with your Logfire write token in **both** places:
+
+```json
+{
+  "diagnostics": {
+    "enabled": true,
+    "otel": {
+      "enabled": true,
+      "endpoint": "https://logfire-us.pydantic.dev",
+      "headers": {
+        "Authorization": "Bearer pylf_v1_us_YOUR_TOKEN_HERE"
+      },
+      "serviceName": "openclaw",
+      "traces": true,
+      "metrics": true,
+      "logs": true
+    }
+  },
+  "plugins": {
+    "entries": {
+      "logfire-observability": {
+        "enabled": true,
+        "config": {
+          "logfireToken": "pylf_v1_us_YOUR_TOKEN_HERE"
+        }
+      }
+    }
+  }
+}
+```
+
+Then restart OpenClaw (`sudo systemctl restart clawdbot` or `openclaw restart`).
+
+> **EU region?** Change the endpoint to `https://logfire-eu.pydantic.dev` and set `logfireEndpoint` to `https://logfire-eu.pydantic.dev/v1/traces`.
+
+## What you get in Logfire
+
+### From logfire-observability (this plugin)
+
+Detailed agent execution traces with parent-child nesting:
+
+```
+User message
+  └─ message.received span
+  └─ agent.run span (parent)
+       ├─ tool.web_search span
+       ├─ tool.read_file span
+       └─ tool.send_message span
+```
+
+| Span | Fires when | Key attributes |
+|------|-----------|----------------|
+| `message.received` | Inbound user message | channel, from, content |
+| `agent.run` | LLM call start → end | agent, provider, prompt preview, response, duration, message count |
+| `tool.<name>` | Each tool execution | tool name, params, result, call ID |
+
+All spans include `openclaw.sessionKey` and `openclaw.agent` for filtering.
+
+### From diagnostics-otel (built-in)
+
+Operational metrics, logs, and diagnostic traces:
+
+**Metrics**
+| Metric | Type | What it tracks |
+|--------|------|----------------|
+| `openclaw.tokens` | counter | Token usage by type (input, output, cache, prompt, total) |
+| `openclaw.cost.usd` | counter | Estimated cost per run |
+| `openclaw.run.duration_ms` | histogram | Agent run duration |
+| `openclaw.context.tokens` | histogram | Context window limit vs used |
+| `openclaw.webhook.received` | counter | Inbound webhooks |
+| `openclaw.webhook.duration_ms` | histogram | Webhook processing time |
+| `openclaw.message.queued` / `.processed` | counters | Message throughput |
+| `openclaw.queue.depth` / `.wait_ms` | histograms | Queue health |
+| `openclaw.session.state` / `.stuck` | counters | Session lifecycle |
+| `openclaw.run.attempt` | counter | Run retry tracking |
+
+**Logs** — All OpenClaw logs forwarded to Logfire via OTLP (when `logs: true`).
+
+**Traces** — `model.usage`, `webhook.processed`, `webhook.error`, `message.processed`, `session.stuck` spans.
+
+## Config reference
+
+### logfire-observability (plugin config)
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `logfireToken` | string | **(required)** | Your Logfire project write token |
+| `logfireEndpoint` | string | `https://logfire-us.pydantic.dev/v1/traces` | OTLP trace endpoint |
+| `serviceName` | string | `openclaw` | Service name shown in Logfire |
+| `captureContent` | boolean | `true` | Include message text, LLM responses, tool results |
+| `captureToolParams` | boolean | `true` | Include tool call parameters |
+| `maxAttributeLength` | number | `4096` | Truncate attributes beyond this length |
+
+### diagnostics-otel (top-level diagnostics config)
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `diagnostics.enabled` | boolean | `false` | Enable diagnostics |
+| `diagnostics.otel.enabled` | boolean | `false` | Enable OTLP export |
+| `diagnostics.otel.endpoint` | string | — | OTLP endpoint base URL |
+| `diagnostics.otel.headers` | object | — | Custom headers (use for Logfire auth) |
+| `diagnostics.otel.serviceName` | string | `openclaw` | Service name |
+| `diagnostics.otel.traces` | boolean | `true` | Export traces |
+| `diagnostics.otel.metrics` | boolean | `true` | Export metrics |
+| `diagnostics.otel.logs` | boolean | `false` | Export logs |
+| `diagnostics.otel.sampleRate` | number | `1.0` | Trace sample rate (0.0–1.0) |
+
+## Useful Logfire queries
+
+```sql
+-- Failed agent runs (from logfire-observability)
+SELECT * FROM spans WHERE span_name = 'agent.run' AND attributes->>'openclaw.success' = 'false'
+
+-- Slowest tool calls (from logfire-observability)
+SELECT span_name, duration FROM spans WHERE span_name LIKE 'tool.%' ORDER BY duration DESC LIMIT 20
+
+-- Token usage by model (from diagnostics-otel)
+SELECT attributes->>'openclaw.model', sum(value) FROM metrics WHERE name = 'openclaw.tokens' GROUP BY 1
+
+-- Cost per channel (from diagnostics-otel)
+SELECT attributes->>'openclaw.channel', sum(value) FROM metrics WHERE name = 'openclaw.cost.usd' GROUP BY 1
+
+-- Messages by channel (from logfire-observability)
+SELECT attributes->>'openclaw.channel', count(*) FROM spans WHERE span_name = 'message.received' GROUP BY 1
+```
+
+## Architecture
+
+```
+                          ┌──────────────────────────────────┐
+                          │           Logfire                 │
+                          │   (traces, metrics, logs)         │
+                          └──────────┬───────────────────────┘
+                                     │ OTLP/HTTP
+                          ┌──────────┴───────────────────────┐
+                          │                                    │
+              ┌───────────┴──────────┐         ┌──────────────┴──────────┐
+              │ logfire-observability │         │    diagnostics-otel     │
+              │   (this plugin)      │         │      (built-in)         │
+              ├──────────────────────┤         ├─────────────────────────┤
+              │ agent.run traces     │         │ metrics (tokens, cost)  │
+              │ tool.* child spans   │         │ diagnostic traces       │
+              │ message.received     │         │ log forwarding          │
+              │                      │         │ webhook/queue/session   │
+              └───────────┬──────────┘         └──────────────┬──────────┘
+                          │ api.on() hooks                    │ onDiagnosticEvent()
+                          └───────────┬───────────────────────┘
+                                      │
+                              ┌───────┴────────┐
+                              │    OpenClaw     │
+                              └────────────────┘
+```
+
+The two plugins use different event systems (`api.on()` vs `onDiagnosticEvent()`) and different OTel setups (self-contained provider vs NodeSDK). They don't conflict — `logfire-observability` avoids global OTel registration, sidestepping the module isolation bug where jiti's per-plugin scoping prevents shared TracerProviders.
+
+## Using only this plugin
+
+If you don't need metrics/logs and just want agent traces, you can use `logfire-observability` alone — no need to enable `diagnostics-otel`. The trace hierarchy (agent.run → tool.*) works independently.

package/index.ts

@@ -0,0 +1,263 @@
+import { SpanStatusCode, context as otelContext, trace, type Span, type Context, type Tracer } from "@opentelemetry/api";
+import { NodeTracerProvider, BatchSpanProcessor } from "@opentelemetry/sdk-trace-node";
+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
+import { Resource } from "@opentelemetry/resources";
+import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
+
+const TRACER_NAME = "openclaw.logfire-observability";
+const TRACER_VERSION = "1.0.0";
+
+interface ActiveRun {
+  span: Span;
+  ctx: Context; // OTel context with agent.run span — used to parent tool spans
+  toolSpans: Map<string, Span>; // keyed by toolCallId for parallel-safe lookup
+  toolStack: Span[]; // fallback LIFO for events missing toolCallId
+}
+
+function truncate(str: string | undefined | null, maxLen: number): string {
+  if (!str) return "";
+  return str.length > maxLen ? str.slice(0, maxLen) + "...[truncated]" : str;
+}
+
+function safeStringify(value: unknown, maxLen: number): string {
+  try {
+    const str = typeof value === "string" ? value : JSON.stringify(value);
+    return truncate(str, maxLen);
+  } catch {
+    return "[unstringifiable]";
+  }
+}
+
+function extractTextContent(content: unknown, maxLen: number): string {
+  if (typeof content === "string") return truncate(content, maxLen);
+  if (Array.isArray(content)) {
+    const parts = content
+      .filter((b: any) => b.type === "text")
+      .map((b: any) => b.text ?? "");
+    return truncate(parts.join("\n"), maxLen);
+  }
+  return safeStringify(content, maxLen);
+}
+
+export default {
+  id: "logfire-observability",
+  name: "Logfire Observability",
+  version: TRACER_VERSION,
+
+  register(api: any) {
+    const cfg = api.pluginConfig ?? {};
+
+    // ── Validate config ──────────────────────────────────────────
+    const logfireToken: string | undefined = cfg.logfireToken;
+    if (!logfireToken) {
+      api.logger?.error?.(
+        "logfire-observability: missing logfireToken in config — plugin disabled. " +
+        "Add your Logfire write token to openclaw.json under plugins.entries.logfire-observability.config.logfireToken"
+      );
+      return;
+    }
+
+    const endpoint: string = cfg.logfireEndpoint || "https://logfire-us.pydantic.dev/v1/traces";
+    const serviceName: string = cfg.serviceName || "openclaw";
+    const captureContent: boolean = cfg.captureContent !== false;
+    const captureToolParams: boolean = cfg.captureToolParams !== false;
+    const maxLen: number = typeof cfg.maxAttributeLength === "number" ? cfg.maxAttributeLength : 4096;
+
+    // ── Set up self-contained OTLP exporter ──────────────────────
+    const exporter = new OTLPTraceExporter({
+      url: endpoint,
+      headers: {
+        Authorization: `Bearer ${logfireToken}`,
+      },
+    });
+
+    const provider = new NodeTracerProvider({
+      resource: new Resource({
+        [ATTR_SERVICE_NAME]: serviceName,
+      }),
+      spanProcessors: [new BatchSpanProcessor(exporter)],
+    });
+
+    // Use our own tracer from our own provider — no global registration,
+    // no conflict with diagnostics-otel or any other plugin.
+    const tracer: Tracer = provider.getTracer(TRACER_NAME, TRACER_VERSION);
+
+    api.logger?.info?.(
+      `logfire-observability: initialized (endpoint=${endpoint} service=${serviceName} content=${captureContent} toolParams=${captureToolParams})`
+    );
+
+    // Active agent runs keyed by sessionKey
+    const runs = new Map<string, ActiveRun>();
+
+    // ── message_received ─────────────────────────────────────────
+    api.on("message_received", (event: any, ctx: any) => {
+      const span = tracer.startSpan("message.received", {
+        attributes: {
+          "openclaw.channel": ctx.channelId ?? "unknown",
+          "openclaw.from": event.from ?? "unknown",
+          "openclaw.account": ctx.accountId ?? "unknown",
+          ...(captureContent && event.content
+            ? { "openclaw.message.content": truncate(event.content, maxLen) }
+            : {}),
+        },
+      });
+      span.end();
+    });
+
+    // ── before_agent_start ───────────────────────────────────────
+    api.on("before_agent_start", (event: any, ctx: any) => {
+      const sessionKey = ctx.sessionKey ?? ctx.agentId ?? "unknown";
+
+      // End any stale run for this session
+      const stale = runs.get(sessionKey);
+      if (stale) {
+        endAllToolSpans(stale, true);
+        stale.span.setAttribute("openclaw.incomplete", true);
+        stale.span.end();
+        runs.delete(sessionKey);
+      }
+
+      const attrs: Record<string, string | number | boolean> = {
+        "openclaw.agent": ctx.agentId ?? "unknown",
+        "openclaw.sessionKey": sessionKey,
+        "openclaw.provider": ctx.messageProvider ?? "unknown",
+        "openclaw.prompt.length": event.prompt?.length ?? 0,
+        "openclaw.messages.count": event.messages?.length ?? 0,
+      };
+      if (captureContent && event.prompt) {
+        attrs["openclaw.prompt.preview"] = truncate(event.prompt, 1024);
+      }
+
+      const span = tracer.startSpan("agent.run", { attributes: attrs });
+      // Create an OTel context with this span as the active span.
+      // Tool spans started within this context become children of agent.run.
+      const spanCtx = trace.setSpan(otelContext.active(), span);
+      runs.set(sessionKey, { span, ctx: spanCtx, toolSpans: new Map(), toolStack: [] });
+    });
+
+    // ── before_tool_call ─────────────────────────────────────────
+    api.on("before_tool_call", (event: any, ctx: any) => {
+      const sessionKey = ctx.sessionKey ?? ctx.agentId ?? "unknown";
+      const parent = runs.get(sessionKey);
+
+      const attrs: Record<string, string | number | boolean> = {
+        "openclaw.tool.name": event.toolName,
+        "openclaw.sessionKey": sessionKey,
+        "openclaw.agent": ctx.agentId ?? "unknown",
+      };
+      if (captureToolParams && event.params) {
+        attrs["openclaw.tool.params"] = safeStringify(event.params, maxLen);
+      }
+
+      // Start tool span as a child of the agent.run span via parent context
+      const span = tracer.startSpan(
+        "tool." + event.toolName,
+        { attributes: attrs },
+        parent?.ctx, // passes parent context → makes this a child span
+      );
+
+      if (parent) {
+        // If we have a toolCallId, use the Map for parallel-safe lookup.
+        // Otherwise fall back to the LIFO stack.
+        if (event.toolCallId) {
+          parent.toolSpans.set(event.toolCallId, span);
+        } else {
+          parent.toolStack.push(span);
+        }
+      } else {
+        api.logger?.debug?.(`logfire-observability: tool.${event.toolName} fired without active agent run`);
+        span.end();
+      }
+    });
+
+    // ── tool_result_persist ──────────────────────────────────────
+    api.on("tool_result_persist", (event: any, ctx: any) => {
+      const sessionKey = ctx.sessionKey ?? ctx.agentId ?? "unknown";
+      const parent = runs.get(sessionKey);
+
+      // Look up the matching tool span: prefer Map by toolCallId, fall back to stack
+      let span: Span | undefined;
+      if (event.toolCallId && parent?.toolSpans.has(event.toolCallId)) {
+        span = parent.toolSpans.get(event.toolCallId);
+        parent.toolSpans.delete(event.toolCallId);
+      } else {
+        span = parent?.toolStack.pop();
+      }
+
+      if (span) {
+        if (event.toolName) span.setAttribute("openclaw.tool.name", event.toolName);
+        if (event.toolCallId) span.setAttribute("openclaw.tool.callId", event.toolCallId);
+        if (event.isSynthetic) span.setAttribute("openclaw.tool.synthetic", true);
+
+        if (captureContent && event.message) {
+          const msg = event.message as any;
+          if (msg.content) {
+            span.setAttribute("openclaw.tool.result", extractTextContent(msg.content, maxLen));
+          }
+        }
+        span.end();
+      }
+    });
+
+    // ── agent_end ────────────────────────────────────────────────
+    api.on("agent_end", (event: any, ctx: any) => {
+      const sessionKey = ctx.sessionKey ?? ctx.agentId ?? "unknown";
+      const active = runs.get(sessionKey);
+
+      // End any dangling tool spans
+      if (active) {
+        endAllToolSpans(active, false);
+      }
+
+      const span = active?.span;
+      if (span) {
+        span.setAttribute("openclaw.success", event.success ?? true);
+        if (event.durationMs != null) span.setAttribute("openclaw.durationMs", event.durationMs);
+        if (event.error) {
+          span.setStatus({ code: SpanStatusCode.ERROR, message: event.error });
+          span.setAttribute("openclaw.error", truncate(event.error, maxLen));
+        }
+        if (event.messages?.length) {
+          span.setAttribute("openclaw.messages.total", event.messages.length);
+
+          if (captureContent) {
+            for (let i = event.messages.length - 1; i >= 0; i--) {
+              const msg = (event.messages as any[])[i];
+              if (msg?.role === "assistant") {
+                span.setAttribute("openclaw.response", extractTextContent(msg.content, maxLen));
+                break;
+              }
+            }
+          }
+        }
+        span.end();
+        runs.delete(sessionKey);
+      }
+    });
+
+    // ── Graceful shutdown ────────────────────────────────────────
+    api.on("shutdown", async () => {
+      api.logger?.info?.("logfire-observability: flushing and shutting down...");
+      await provider.forceFlush();
+      await provider.shutdown();
+    });
+
+    /** End all pending tool spans for a run (from both Map and stack). */
+    function endAllToolSpans(run: ActiveRun, markIncomplete: boolean) {
+      for (const ts of run.toolSpans.values()) {
+        if (markIncomplete) ts.setAttribute("openclaw.incomplete", true);
+        ts.end();
+      }
+      run.toolSpans.clear();
+      for (const ts of run.toolStack) {
+        if (markIncomplete) ts.setAttribute("openclaw.incomplete", true);
+        ts.end();
+      }
+      run.toolStack.length = 0;
+    }
+
+    api.logger?.info?.(
+      "logfire-observability: hooks registered (message_received, before_agent_start, before_tool_call, tool_result_persist, agent_end, shutdown)"
+    );
+  },
+};

package/openclaw.plugin.json

@@ -0,0 +1,63 @@
+{
+  "id": "logfire-observability",
+  "name": "Logfire Observability",
+  "description": "Self-contained agent observability — traces messages, LLM calls, and tool executions to Logfire via OpenTelemetry. No dependency on diagnostics-otel.",
+  "version": "1.0.0",
+  "kind": "tool",
+  "uiHints": {
+    "logfireToken": {
+      "label": "Logfire Write Token",
+      "help": "Your Logfire project write token (pylf_v1_us_...)"
+    },
+    "serviceName": {
+      "label": "Service Name",
+      "help": "Service name shown in Logfire traces (default: openclaw)"
+    },
+    "captureContent": {
+      "label": "Capture Content",
+      "help": "Include message text, LLM responses, and tool results in traces"
+    },
+    "captureToolParams": {
+      "label": "Capture Tool Params",
+      "help": "Include tool call parameters in traces"
+    },
+    "maxAttributeLength": {
+      "label": "Max Attribute Length",
+      "help": "Truncate string attributes beyond this length (bytes)"
+    }
+  },
+  "configSchema": {
+    "type": "object",
+    "required": ["logfireToken"],
+    "additionalProperties": false,
+    "properties": {
+      "logfireToken": {
+        "type": "string",
+        "description": "Logfire write token (pylf_v1_us_...)"
+      },
+      "logfireEndpoint": {
+        "type": "string",
+        "default": "https://logfire-us.pydantic.dev/v1/traces",
+        "description": "Logfire OTLP endpoint (override for EU or self-hosted)"
+      },
+      "serviceName": {
+        "type": "string",
+        "default": "openclaw",
+        "description": "Service name for traces"
+      },
+      "captureContent": {
+        "type": "boolean",
+        "default": true
+      },
+      "captureToolParams": {
+        "type": "boolean",
+        "default": true
+      },
+      "maxAttributeLength": {
+        "type": "number",
+        "default": 4096,
+        "minimum": 256
+      }
+    }
+  }
+}

package/package.json

@@ -0,0 +1,41 @@
+{
+  "name": "openclaw-logfire-observability",
+  "version": "1.0.0",
+  "type": "module",
+  "description": "OpenClaw plugin for full observability in Pydantic Logfire — traces agent runs, tool calls, and messages via OpenTelemetry",
+  "keywords": [
+    "openclaw",
+    "openclaw-plugin",
+    "logfire",
+    "pydantic",
+    "observability",
+    "opentelemetry",
+    "otel",
+    "tracing",
+    "agent",
+    "llm"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/rita-aga/openclaw-logfire-observability"
+  },
+  "license": "MIT",
+  "author": "rita-aga",
+  "files": [
+    "index.ts",
+    "openclaw.plugin.json",
+    "README.md"
+  ],
+  "openclaw": {
+    "extensions": [
+      "./index.ts"
+    ]
+  },
+  "dependencies": {
+    "@opentelemetry/api": "^1.9.0",
+    "@opentelemetry/sdk-trace-node": "^1.30.0",
+    "@opentelemetry/exporter-trace-otlp-http": "^0.57.0",
+    "@opentelemetry/resources": "^1.30.0",
+    "@opentelemetry/semantic-conventions": "^1.28.0"
+  }
+}