@turmind/halo 1.2.1 → 1.3.1

package/bundled-docs/design/observability.md

@@ -0,0 +1,214 @@
+# Observability (OpenTelemetry)
+
+## Overview
+
+A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. Principle: zero AWS (or any vendor) code in the server — SigV4 signing, backend endpoints, and resource enrichment all live in the collector, not in halo.
+
+## Configuration
+
+| key | type | default | description |
+|---|---|---|---|
+| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Restart required. |
+| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Restart required. |
+| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Restart required. |
+| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Restart required. |
+
+All four keys are `globalOnly` and take effect on server restart — `otel.ts` reads them once at boot, before anything else touches the OTel API.
+
+**Env-var mapping.** `initObservability()` maps settings onto the standard `OTEL_*` env vars with `??=`, so an operator who already exports them (or `--require`s an external distro) wins:
+
+- `endpoint` → `OTEL_EXPORTER_OTLP_ENDPOINT`
+- `headers` → `OTEL_EXPORTER_OTLP_HEADERS`
+- `service_name` → `OTEL_SERVICE_NAME`
+- always `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` (declarative — the `*-otlp-proto` exporters are hard-wired to it; this just makes an externally-loaded SDK agree on wire protocol)
+
+No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObservability()` returns without registering anything.
+
+**External-SDK detection.** If an external SDK is already registered (e.g. via `node --require` of a vendor distro) — detected by comparing `ProxyTracerProvider.getDelegate()` against a fresh proxy's `NoopTracerProvider` delegate — halo logs `[Observability] external OpenTelemetry SDK already registered — using it, skipping built-in exporters` and skips its own exporters. The `??=` env mapping above still applies first, so the external SDK picks up halo's settings unless it already set its own.
+
+**TLS / HTTPS.** `https://` endpoints work out of the box against CA-signed certs — the exporter picks `node:http` vs `node:https` by URL scheme. Self-signed certs / mTLS go through the standard exporter env vars `OTEL_EXPORTER_OTLP_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_KEY` (read by the exporter itself, not by halo code). Verified: a self-signed sink is rejected with `DEPTH_ZERO_SELF_SIGNED_CERT` unless the cert is trusted — no silent bypass.
+
+**Common pitfall**: pointing `endpoint` at 4317 (the collector's gRPC port) instead of 4318 (OTLP/HTTP). Exports then fail silently — the batch processor swallows the error, at most one diagnostic line, no user-visible failure.
+
+## Architecture
+
+| module | role |
+|---|---|
+| `packages/server/src/observability/otel.ts` (91 lines) | Bootstrap. `initObservability()` / `shutdownObservability()`, the `enabled` gate, `tracer` / `getMeter()` / `otelLogger` / `captureContent()` exports. |
+| `packages/server/src/observability/otel-sdk.ts` (47 lines) | The ONE module that imports the SDK packages and OTLP exporters. |
+| `packages/server/src/observability/genai-spans.ts` (296 lines) | The span model: `beginTurn` / `onAgentEvent` / `recordRetry` / `endTurn`. |
+| `packages/server/src/agents/session-manager.ts` | Calls the four hooks above from the turn loop (see below). |
+| `packages/server/src/logger.ts` | Bridges every `logger.*` call into an OTel LogRecord when `enabled`. |
+| `packages/server/src/index.ts` | `await initObservability()` (L262, before `initLogger()`) and `await shutdownObservability()` (L599, in `gracefulShutdown`). |
+
+**Hook call sites in `session-manager.ts`**: `beginTurn(session, message)` at L1276 (start of `runAgentTurn`), `onAgentEvent(session, event)` at L1316 (inside the agent event loop, one call per `AgentEvent`), `recordRetry(kind)` at the 7 retry catch sites, `endTurn(session, {error?})` at L1547 (the turn's single exit point, success or failure).
+
+**The `enabled` gate.** `otel.ts` exports a single `enabled: boolean`, set once by `initObservability()`. Every hook in `genai-spans.ts` and the `logger.ts` bridge checks it first — an unconfigured server pays nothing beyond one boolean read per hook call.
+
+**Why `otel-sdk.ts` is a dynamic import.** `logger.ts` is shared with the CLI (`halo cli`), which never wants observability. Eagerly importing the SDK module graph would cost ~100ms on every `halo cli` start. `otel.ts` only `import()`s `otel-sdk.ts` after confirming an endpoint is configured, so the unconfigured path (server or CLI) never loads it.
+
+**Why the meter is lazy.** The `@opentelemetry/api` tracer and logger are proxies — they stay no-op until a real provider is registered, then transparently start forwarding, so `tracer` / `otelLogger` can be module-level consts. The metrics API has **no proxy**: a `Meter` fetched via `metrics.getMeter()` before `setGlobalMeterProvider()` runs is a permanent no-op forever after, even once a provider registers later. `getMeter()` in `otel.ts` is therefore a function, not a const, and `genai-spans.ts` calls it lazily on first metric emission (`instruments()`, cached after).
+
+**Dependencies** (`packages/server/package.json`): `@opentelemetry/api` 1.9.1, `api-logs` 0.222.0, `sdk-trace-node` / `sdk-trace-base` / `sdk-metrics` / `sdk-logs` / `resources` 2.11.0, `exporter-trace-otlp-proto` / `exporter-metrics-otlp-proto` / `exporter-logs-otlp-proto` 0.222.0, `semantic-conventions` 1.43.0. No gRPC exporter (see [Rejected alternatives](#rejected-alternatives)).
+
+**Test coverage**: `packages/server/test/observability-genai-spans.test.ts` (7 cases) drives `beginTurn` / `onAgentEvent` / `endTurn` against an in-memory tracer + `InMemorySpanExporter`, asserting the three-level span tree, parent ids, semconv attribute names, and the `capture_content` on/off split.
+
+## Span model
+
+OTel GenAI semantic conventions. Scope `opentelemetry.instrumentation.halo`, version `1`. Hierarchy: **session** (`session.id`, whole conversation) ⊃ **trace** (one turn = one `invoke_agent` span) ⊃ **span**. All spans `kind=INTERNAL`, all carry `gen_ai.system=halo` and `session.id=<halo session id>`.
+
+### `invoke_agent <agentName>`
+
+One per turn — the root span, opened by `beginTurn` and closed by `endTurn`.
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `invoke_agent` | |
+| `gen_ai.agent.name` | ✓ | |
+| `gen_ai.request.model` | ✓ | |
+| `gen_ai.task.input` | | user message text |
+| `gen_ai.task.output` | | final assistant text |
+| `gen_ai.system_instructions` | | system prompt, capped 8 KB — once per turn here, not on every `chat` |
+
+On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`).
+
+### `chat <modelId>`
+
+Child of `invoke_agent`, one per model call — created retroactively on the `usage` event (the agent loop only reports a call ended, with its `durationMs`).
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `chat` | |
+| `gen_ai.request.model` | ✓ | |
+| `gen_ai.usage.input_tokens` | ✓ | |
+| `gen_ai.usage.output_tokens` | ✓ | |
+| `gen_ai.response.finish_reasons` | `["tool_use"]` or `["end_turn"]` | |
+| `gen_ai.input.messages` | | semconv JSON (see below) — **delta only**: messages appended since this turn's previous `chat` span (first call → the user message; later calls → that cycle's `tool_result`s) |
+| `gen_ai.output.messages` | | semconv JSON, only the trailing assistant message |
+
+Semconv message JSON: `[{role, parts:[{type:"text",content}|{type:"tool_call",id,name,arguments}|{type:"tool_call_response",id,result}]}]`.
+
+`gen_ai.input.messages` is incremental on purpose (since 1.3.1): replaying the full history on every model call made a turn's exported bytes O(n²) and the system prompt alone was ~60% of the volume — a 4-turn / 9-call session dropped from 129 KB to 44 KB of content attributes. Nothing is lost: the full conversation for a trace is the concatenation of its `chat` spans' input + output deltas in order (a `TurnState.messageCursor` tracks the boundary; a mid-turn compact that shrinks the history yields an empty delta and re-syncs the cursor). Verified that AgentCore Evaluations scores are unchanged — its evaluators read `invoke_agent`'s `task.*` and `execute_tool`'s `arguments/result`, not the chat messages.
+
+**Note**: `gen_ai.usage.input_tokens` counts only the prompt-cache MISS increment on Bedrock — small values like `2` are normal, not a bug.
+
+### `execute_tool <toolName>`
+
+Child of the `chat` span that requested it, one per tool call — created on the matching `tool_result` event.
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `execute_tool` | |
+| `gen_ai.tool.name` | ✓ | |
+| `gen_ai.tool.call.id` | ✓ | |
+| `gen_ai.tool.call.arguments` | | JSON |
+| `gen_ai.tool.call.result` | | text, empty → `[no output]` |
+
+Tool calls still pending when `endTurn` fires (turn interrupted/aborted mid-cycle) are closed as orphans with status `ERROR` and `halo.tool.orphaned=true`, so no trace is left with a dangling tool-call id.
+
+All content attributes are capped (32 KB general cap, 2 KB per message part, 8 KB for system instructions); over-cap content is truncated with a `…[truncated N chars]` marker, and an over-cap message list drops the oldest messages first with an omission marker.
+
+## Metrics
+
+Exported every 15s via `PeriodicExportingMetricReader`. Same scope as the spans.
+
+| name | kind | unit | attrs |
+|---|---|---|---|
+| `gen_ai.client.token.usage` | histogram | `{token}` | `gen_ai.request.model`, `gen_ai.token.type=input\|output` |
+| `gen_ai.client.operation.duration` | histogram | s | `gen_ai.request.model`, `gen_ai.operation.name=chat` |
+| `halo.tool.duration` | histogram | s | `gen_ai.tool.name` |
+| `halo.turn.duration` | histogram | s | `outcome=ok\|error` |
+| `halo.model.retries` | counter | — | `kind=<retry kind>` |
+
+`halo.model.retries` is incremented by `recordRetry(kind)` from 7 retry sites in `session-manager.ts`: `context_overflow`, `throttle`, `server_error`, `network`, `empty_response`, `corrupted`, `multimodal_4xx`.
+
+## Logs bridge
+
+Every `logger.*` call in `packages/server/src/logger.ts` also emits an OTel LogRecord when `enabled`: severity number `debug=5 / info=9 / warn=13 / error=17`, attribute `halo.module` set to the call's `[ModuleName]` prefix. Same OTLP endpoint, `/v1/logs`.
+
+## Reference: collector → AWS CloudWatch / X-Ray
+
+Verified working, all AWS-specific pieces live in the collector config, not in halo. `otel/opentelemetry-collector-contrib:0.161.0`:
+
+```yaml
+extensions:
+  sigv4auth/xray:
+    region: <region>
+    service: xray
+  sigv4auth/logs:
+    region: <region>
+    service: logs
+
+processors:
+  resource:
+    attributes:
+      - { key: aws.service.type, value: gen_ai_agent, action: insert }
+      - { key: aws.log.group.names, value: /aws/bedrock-agentcore/runtimes/<name>, action: insert }
+
+exporters:
+  otlphttp/xray:
+    endpoint: https://xray.<region>.amazonaws.com/v1/traces
+    auth: { authenticator: sigv4auth/xray }
+  otlphttp/cwlogs:
+    endpoint: https://logs.<region>.amazonaws.com/v1/logs
+    auth: { authenticator: sigv4auth/logs }
+    headers:
+      x-aws-log-group: /aws/bedrock-agentcore/runtimes/<name>
+      x-aws-log-stream: runtime-logs
+  awsemf:
+    namespace: bedrock-agentcore
+```
+
+Prerequisites: CloudWatch Transaction Search must be enabled (spans land in the shared `aws/spans` log group, no resource policy needed); the log group `/aws/bedrock-agentcore/runtimes/<name>` must be created beforehand.
+
+Halo-side settings for this setup: `endpoint=http://localhost:4318`, `service_name=<name>`, `capture_content=true` (AgentCore Evaluations needs content to score against).
+
+## AgentCore Evaluations
+
+Two modes, both read `aws/spans` filtered by data source `serviceNames: ["<service_name>"]`:
+
+1. **Online evaluation config** (console: AgentCore → Evaluations, or `create-online-evaluation-config`): data source `aws/spans` + serviceNames filter, evaluators e.g. `Builtin.Helpfulness` / `Builtin.GoalSuccessRate` / `Builtin.ToolSelectionAccuracy`, results written to `/aws/bedrock-agentcore/evaluations/results/<config>`. Results appear roughly **15 minutes** after the session goes idle (`sessionTimeoutMinutes` idle detection + a processing queue).
+2. **On-demand `aws bedrock-agentcore evaluate`** — synchronous, result only in the response, not persisted. Input: `{"sessionSpans":[<span doc>, …]}`, each span doc the flattened OTLP shape CloudWatch itself writes into `aws/spans`:
+
+```jsonc
+{
+  "resource": { "attributes": { "service.name": "halo" } },
+  "scope": { "name": "opentelemetry.instrumentation.halo", "version": "1" },
+  "traceId": "...", "spanId": "...", "parentSpanId": "...",
+  "name": "chat <modelId>",
+  "kind": "INTERNAL",
+  "startTimeUnixNano": "...", "endTimeUnixNano": "...", "durationNano": "...",
+  "attributes": { "gen_ai.operation.name": "chat", "...": "..." },
+  "status": { "code": "OK" }
+}
+```
+
+Target by evaluator level:
+
+| evaluator level | example | `--evaluation-target` |
+|---|---|---|
+| SESSION | `GoalSuccessRate` | omit — feed the whole session |
+| TRACE | `Helpfulness`, `Correctness` | `{"traceIds":[…]}` |
+| TOOL_CALL | `ToolSelectionAccuracy` | `{"spanIds":[…]}` (passing traceIds → `ValidationException`) |
+
+Verified feeding the input **directly from halo's own OTLP output** (collector `file` exporter → flatten), no CloudWatch involved: `GoalSuccessRate` 1.0, `Helpfulness` 1.0, `ToolSelectionAccuracy` 1.0.
+
+**Contract for custom instrumentation** (official: [supported-frameworks-generic](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks-generic.html), plus 11 probe runs):
+
+- `scope.name` MUST start with `opentelemetry.instrumentation.*` (or `openinference.instrumentation.*`) — a custom scope like `halo.tracing` → `no spans with supported scope` (the error message only lists named frameworks; the generic prefix rule still applies).
+- Every span MUST carry `gen_ai.operation.name` ∈ `{invoke_agent, chat, execute_tool}`; spans without it are skipped (`no spans to evaluate`).
+- Content is read from: `invoke_agent`'s `gen_ai.task.input` / `.task.output` (falls back to the first user / last assistant message on `chat` spans); `chat`'s `gen_ai.input.messages` / `.output.messages` / `gen_ai.system_instructions`; `execute_tool`'s `gen_ai.tool.name` / `.tool.call.arguments` / `.tool.call.result`. Values are stringified, not parsed.
+- `session.id` groups spans into a session — required for SESSION-level evaluators.
+- The parser requires `traceId`, `spanId`, `startTimeUnixNano`, `endTimeUnixNano`. `resource`, `scope.version`, `kind`, `status`, `name`, `durationNano`, `parentSpanId` are all optional — a single-span minimal doc scored fine.
+- Strict on keys, lenient on values: garbage attributes with a correct envelope → rejected; correct keys with any string content → scored.
+- `gen_ai.tool.definitions` (JSON array of tool schemas on `chat` spans) is read by `GoalSuccessRate` to check called tools were declared; halo does not emit it and still scored 1.0 — optional.
+
+The official spec is spread over three places: the CLI/API reference (envelope only — `sessionSpans` is a list of opaque `document`s, 1–20000), the generic-framework page (semantic contract), and the runtime's own error messages (structural field requirements). None of the three alone is sufficient.
+
+References: [CLI reference](https://docs.aws.amazon.com/cli/latest/reference/bedrock-agentcore/evaluate.html) · [API reference](https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_Evaluate.html) · [on-demand guide](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/getting-started-on-demand.html)
+
+## Rejected alternatives
+
+- **Direct CloudWatch / X-Ray SDK calls from halo** (an earlier 2026-08 prototype) — vendor lock-in inside core; replaced by OTel + collector.
+- **ADOT `--require` into the halo process** — pulls in `sdk-node` + gRPC + auto-instrumentations, and its `api-logs` 0.219 is incompatible with halo's 0.222 (0.x minors must match); also couples the process to one vendor's distro. Rejected. The `??=` env mapping + external-SDK detection still let an operator do this if they insist.
+- **gRPC exporter** — an extra native-ish dependency (`@grpc/grpc-js`) for a port (4317) that every collector also serves over plain HTTP (4318).

package/bundled-docs/design/session.md

@@ -181,6 +181,8 @@ A `QueuedMessage` is `{ text, sourceSessionId?, images? }`: `sourceSessionId` is
 
 **runSession loop** (`runSession(sessionId, message)`, `message: string | ContentBlock[]`): runs the opening turn when there is one, then drains. An empty **string** message means "the work is already in `messageQueue`" (the `querySession` idle path) — it skips the opening turn and goes straight to drain. The per-turn reset (fresh `toolCallLog` / `warnedToolHashes`, `interruptRequested = false`) runs before the opening turn; `drainQueue` repeats the same reset before each merged batch so a stale interrupt flag never leaks across turns. Server-only: right before `session.promise = runFn()` it inserts the session id into the run ledger, and in the `finally` (right after `session.promise = null`, before emitting `complete`) it deletes it — see [Run ledger](#run-ledger--restart-nudge-for-interrupted-roots-halo-globalrunsdb) above.
 
+**Observability.** `runAgentTurn` is wrapped by `beginTurn` / `endTurn` from `observability/genai-spans.ts` — one `invoke_agent` span per turn, with `chat` / `execute_tool` child spans created from the `onAgentEvent` hook inside the event loop; see [design/observability.md](observability.md).
+
 **Arrival stamp.** Both paths run the turn through `runAgentTurn`, which prepends `[<ISO-8601 UTC>] ` (e.g. `[2026-09-12T15:17:44.153Z] `) to the first text block of the model-facing input — an image-only input gets a stamp-only text block unshifted. The model has no clock; without this it couldn't tell a reply that came two days later from one that came instantly, or how long a sub-agent report took to land. Stamped **once, before the retry loop**, so every attempt (and the [multimodal 4xx degrade](#multimodal-4xx-degrade) rebuild) reuses the identical input. Agent reports therefore read `[<iso>] (from: session X)\n…` — stamp outermost, no special-casing. Only `agent.messages` changes: the UI `type:'user'` event keeps the raw text (see [Turn matching](#exchange-deletion-soft-ui--hard-raw) for the strip on compare), and the self-compact instruction / `[Conversation Summary]` injection / `tool_result` messages are never stamped.
 
 **Final-text flag.** The agent loop marks each text block `final` when the model call ended for a reason other than `tool_use` (i.e. it is the turn's wrap-up, not the filler emitted before a tool call), and `runAgentTurn`'s `case 'text'` forwards it on the `stream` event as `AgentSessionEvent.final`. Block-send consumers gate on it: the channel responders (wechat / telegram / slack / feishu) append only `final` text, and `halo cli` prints the last root turn's `final` text as its stdout (fallback: that turn's full text). `finalOutput` (used by `tryReportToParent` / `deliverGoalRound`) is the same accumulation on the server side.

package/bundled-docs/design/storage.md

@@ -387,6 +387,10 @@ A value of the form `<<ENV_NAME>>` is replaced at read time with `process.env.EN
 | `general.sandbox.hidden_dirs` | `~/.halo/secrets,~/.aws,~/.ssh,~/.gnupg,~/.docker,~/.config/gh` | bwrap tmpfs overlay (Linux only) |
 | `general.sandbox.hidden_files` | `~/.npmrc,~/.bash_history,~/.gitconfig,~/.git-credentials,~/.netrc,~/.halo/global/evo.db,~/.halo/global/evo.db-wal,~/.halo/global/evo.db-shm,~/.halo/global/cron.db,~/.halo/global/cron.db-wal,~/.halo/global/cron.db-shm,~/.halo/global/runs.db,~/.halo/global/runs.db-wal,~/.halo/global/runs.db-shm` | bwrap /dev/null bind (Linux only) |
 | `general.logging.level` | warn | `debug` / `info` / `warn` / `error` |
+| `general.observability.endpoint` | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`); empty = off. Restart required. See [observability.md](observability.md) |
+| `general.observability.service_name` | `halo` | OTel resource `service.name`. Restart required |
+| `general.observability.headers` | `''` | **Secret.** Extra OTLP request headers, comma-separated `k=v`. Restart required |
+| `general.observability.capture_content` | `false` | Put prompt/completion/tool text on spans. Restart required |
 
 Schema source: [packages/server/src/settings-schema.ts](../../../packages/server/src/settings-schema.ts) `generalSection()`.
 

package/bundled-docs/dev/api.md

@@ -355,6 +355,15 @@ File: `packages/server/src/routes/settings.ts`
 
 PATCH / DELETE walk the dotted key layer by layer, so a segment of `__proto__` / `constructor` / `prototype` is rejected with 400 `Invalid key` — the leaf assignment would otherwise land on `Object.prototype` and pollute every object in the process.
 
+### Observability settings
+
+- `general.observability.endpoint` — OTLP collector base URL (e.g. `http://localhost:4318`); empty = off
+- `general.observability.service_name` — OTel resource `service.name`, default `halo`
+- `general.observability.headers` — extra OTLP request headers, comma-separated `k=v` (secret)
+- `general.observability.capture_content` — put prompt/completion/tool text on spans, default `false`
+
+All four are `globalOnly`, take effect on restart. See [design/observability.md](../design/observability.md).
+
 ## Metrics
 
 File: `packages/server/src/routes/metrics.ts`.

package/bundled-docs/dev/deploy.md

@@ -228,7 +228,27 @@ EXPOSE 9527
 CMD halo setup -y && halo server start
 ```
 
-## 9. Optional: Nginx reverse proxy
+## 9. Observability (OpenTelemetry collector)
+
+Halo exports traces, metrics and logs over OTLP http/protobuf when `general.observability.endpoint` is set — off by default, no vendor code in the server itself. Point it at any OpenTelemetry collector:
+
+```bash
+# settings.yaml (global scope) or via the admin Settings page
+general:
+  observability:
+    endpoint: http://localhost:4318       # collector's OTLP/HTTP port, NOT 4317 (gRPC, unsupported)
+    service_name: halo
+    headers: ''                            # e.g. authorization=Bearer <token> for a hosted backend
+    capture_content: false                 # true to include prompt/completion/tool text on spans
+```
+
+All four keys take effect on restart: `halo server restart`.
+
+For a hosted backend (Honeycomb, Grafana Cloud, etc.) use its `https://` OTLP endpoint and put its auth token in `headers` — TLS with CA-signed certs works out of the box. Self-signed / mTLS collectors need the standard `OTEL_EXPORTER_OTLP_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_KEY` env vars.
+
+For routing into AWS CloudWatch / X-Ray (sigv4auth collector config, AgentCore Evaluations) see [design/observability.md](../design/observability.md).
+
+## 10. Optional: Nginx reverse proxy
 
 ```nginx
 server {

package/bundled-docs/dev/env.md

@@ -128,6 +128,10 @@ settings.yaml only (no env override):
 - `general.sandbox.writable_dirs` (default empty) — dirs bind-mounted read-write inside the bwrap sandbox, for external CLIs that keep local state (e.g. `~/.kiro`); not applied to readonly sessions, scope: global
 - `general.sandbox.hidden_files` (default `~/.npmrc,~/.bash_history,~/.gitconfig,~/.git-credentials,~/.netrc,~/.halo/global/{evo,cron,runs}.db` + their `-wal`/`-shm` files) — bwrap /dev/null binds, scope: global
 - `general.logging.level` (default `warn`) — log level: debug | info | warn | error
+- `general.observability.endpoint` (default `''`) — OTLP collector base URL (e.g. `http://localhost:4318`); empty = off, scope: global, restart required. See [design/observability.md](../design/observability.md)
+- `general.observability.service_name` (default `halo`) — OTel resource `service.name`, scope: global, restart required
+- `general.observability.headers` (default `''`, secret) — extra OTLP request headers, comma-separated `k=v`, scope: global, restart required
+- `general.observability.capture_content` (default `false`) — put prompt/completion/tool text on spans, scope: global, restart required
 
 `<provider-id>.secrets.*` (server-only, hard-rejected by `{{}}` substitution; declared in `models/<provider-id>.yaml` `secrets:`):
 - `aws-bedrock-claude-invoke.secrets.access_key_id` / `.secret_access_key` — AWS / Bedrock credentials

package/bundled-docs/guide/workspace.md

@@ -103,6 +103,7 @@ Common settings:
 - `general.session.max_nesting_depth` — max session nesting depth
 - `general.compact.keep_messages` — recent messages kept intact during compaction
 - `general.logging.level` — `debug` / `info` / `warn` / `error`
+- `general.observability.endpoint` — OTLP collector URL; leave empty to keep observability off (see [design/observability.md](../design/observability.md))
 - `<provider-id>.secrets.api_key` — provider credentials (Kimi / DeepSeek bearer token)
 - `<provider-id>.secrets.access_key_id` / `.secret_access_key` — AWS Bedrock
 - `<skill-id>.params.<key>` — values an agent can inject into its own `shell_exec` via `{{<skill-id>.params.<key>}}`