@turmind/halo 1.2.0 → 1.3.0

package/bundled-docs/design/observability.md

@@ -0,0 +1,212 @@
+# Observability (OpenTelemetry)
+
+## Overview
+
+A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. Principle: zero AWS (or any vendor) code in the server — SigV4 signing, backend endpoints, and resource enrichment all live in the collector, not in halo.
+
+## Configuration
+
+| key | type | default | description |
+|---|---|---|---|
+| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Restart required. |
+| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Restart required. |
+| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Restart required. |
+| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Restart required. |
+
+All four keys are `globalOnly` and take effect on server restart — `otel.ts` reads them once at boot, before anything else touches the OTel API.
+
+**Env-var mapping.** `initObservability()` maps settings onto the standard `OTEL_*` env vars with `??=`, so an operator who already exports them (or `--require`s an external distro) wins:
+
+- `endpoint` → `OTEL_EXPORTER_OTLP_ENDPOINT`
+- `headers` → `OTEL_EXPORTER_OTLP_HEADERS`
+- `service_name` → `OTEL_SERVICE_NAME`
+- always `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` (declarative — the `*-otlp-proto` exporters are hard-wired to it; this just makes an externally-loaded SDK agree on wire protocol)
+
+No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObservability()` returns without registering anything.
+
+**External-SDK detection.** If an external SDK is already registered (e.g. via `node --require` of a vendor distro) — detected by comparing `ProxyTracerProvider.getDelegate()` against a fresh proxy's `NoopTracerProvider` delegate — halo logs `[Observability] external OpenTelemetry SDK already registered — using it, skipping built-in exporters` and skips its own exporters. The `??=` env mapping above still applies first, so the external SDK picks up halo's settings unless it already set its own.
+
+**TLS / HTTPS.** `https://` endpoints work out of the box against CA-signed certs — the exporter picks `node:http` vs `node:https` by URL scheme. Self-signed certs / mTLS go through the standard exporter env vars `OTEL_EXPORTER_OTLP_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_KEY` (read by the exporter itself, not by halo code). Verified: a self-signed sink is rejected with `DEPTH_ZERO_SELF_SIGNED_CERT` unless the cert is trusted — no silent bypass.
+
+**Common pitfall**: pointing `endpoint` at 4317 (the collector's gRPC port) instead of 4318 (OTLP/HTTP). Exports then fail silently — the batch processor swallows the error, at most one diagnostic line, no user-visible failure.
+
+## Architecture
+
+| module | role |
+|---|---|
+| `packages/server/src/observability/otel.ts` (91 lines) | Bootstrap. `initObservability()` / `shutdownObservability()`, the `enabled` gate, `tracer` / `getMeter()` / `otelLogger` / `captureContent()` exports. |
+| `packages/server/src/observability/otel-sdk.ts` (47 lines) | The ONE module that imports the SDK packages and OTLP exporters. |
+| `packages/server/src/observability/genai-spans.ts` (296 lines) | The span model: `beginTurn` / `onAgentEvent` / `recordRetry` / `endTurn`. |
+| `packages/server/src/agents/session-manager.ts` | Calls the four hooks above from the turn loop (see below). |
+| `packages/server/src/logger.ts` | Bridges every `logger.*` call into an OTel LogRecord when `enabled`. |
+| `packages/server/src/index.ts` | `await initObservability()` (L262, before `initLogger()`) and `await shutdownObservability()` (L599, in `gracefulShutdown`). |
+
+**Hook call sites in `session-manager.ts`**: `beginTurn(session, message)` at L1276 (start of `runAgentTurn`), `onAgentEvent(session, event)` at L1316 (inside the agent event loop, one call per `AgentEvent`), `recordRetry(kind)` at the 7 retry catch sites, `endTurn(session, {error?})` at L1547 (the turn's single exit point, success or failure).
+
+**The `enabled` gate.** `otel.ts` exports a single `enabled: boolean`, set once by `initObservability()`. Every hook in `genai-spans.ts` and the `logger.ts` bridge checks it first — an unconfigured server pays nothing beyond one boolean read per hook call.
+
+**Why `otel-sdk.ts` is a dynamic import.** `logger.ts` is shared with the CLI (`halo cli`), which never wants observability. Eagerly importing the SDK module graph would cost ~100ms on every `halo cli` start. `otel.ts` only `import()`s `otel-sdk.ts` after confirming an endpoint is configured, so the unconfigured path (server or CLI) never loads it.
+
+**Why the meter is lazy.** The `@opentelemetry/api` tracer and logger are proxies — they stay no-op until a real provider is registered, then transparently start forwarding, so `tracer` / `otelLogger` can be module-level consts. The metrics API has **no proxy**: a `Meter` fetched via `metrics.getMeter()` before `setGlobalMeterProvider()` runs is a permanent no-op forever after, even once a provider registers later. `getMeter()` in `otel.ts` is therefore a function, not a const, and `genai-spans.ts` calls it lazily on first metric emission (`instruments()`, cached after).
+
+**Dependencies** (`packages/server/package.json`): `@opentelemetry/api` 1.9.1, `api-logs` 0.222.0, `sdk-trace-node` / `sdk-trace-base` / `sdk-metrics` / `sdk-logs` / `resources` 2.11.0, `exporter-trace-otlp-proto` / `exporter-metrics-otlp-proto` / `exporter-logs-otlp-proto` 0.222.0, `semantic-conventions` 1.43.0. No gRPC exporter (see [Rejected alternatives](#rejected-alternatives)).
+
+**Test coverage**: `packages/server/test/observability-genai-spans.test.ts` (7 cases) drives `beginTurn` / `onAgentEvent` / `endTurn` against an in-memory tracer + `InMemorySpanExporter`, asserting the three-level span tree, parent ids, semconv attribute names, and the `capture_content` on/off split.
+
+## Span model
+
+OTel GenAI semantic conventions. Scope `opentelemetry.instrumentation.halo`, version `1`. Hierarchy: **session** (`session.id`, whole conversation) ⊃ **trace** (one turn = one `invoke_agent` span) ⊃ **span**. All spans `kind=INTERNAL`, all carry `gen_ai.system=halo` and `session.id=<halo session id>`.
+
+### `invoke_agent <agentName>`
+
+One per turn — the root span, opened by `beginTurn` and closed by `endTurn`.
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `invoke_agent` | |
+| `gen_ai.agent.name` | ✓ | |
+| `gen_ai.request.model` | ✓ | |
+| `gen_ai.task.input` | | user message text |
+| `gen_ai.task.output` | | final assistant text |
+
+On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`).
+
+### `chat <modelId>`
+
+Child of `invoke_agent`, one per model call — created retroactively on the `usage` event (the agent loop only reports a call ended, with its `durationMs`).
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `chat` | |
+| `gen_ai.request.model` | ✓ | |
+| `gen_ai.usage.input_tokens` | ✓ | |
+| `gen_ai.usage.output_tokens` | ✓ | |
+| `gen_ai.response.finish_reasons` | `["tool_use"]` or `["end_turn"]` | |
+| `gen_ai.input.messages` | | semconv JSON (see below) |
+| `gen_ai.output.messages` | | semconv JSON, only the trailing assistant message |
+| `gen_ai.system_instructions` | | system prompt, capped 8 KB |
+
+Semconv message JSON: `[{role, parts:[{type:"text",content}|{type:"tool_call",id,name,arguments}|{type:"tool_call_response",id,result}]}]`.
+
+**Note**: `gen_ai.usage.input_tokens` counts only the prompt-cache MISS increment on Bedrock — small values like `2` are normal, not a bug.
+
+### `execute_tool <toolName>`
+
+Child of the `chat` span that requested it, one per tool call — created on the matching `tool_result` event.
+
+| attribute | always | capture_content |
+|---|---|---|
+| `gen_ai.operation.name` | `execute_tool` | |
+| `gen_ai.tool.name` | ✓ | |
+| `gen_ai.tool.call.id` | ✓ | |
+| `gen_ai.tool.call.arguments` | | JSON |
+| `gen_ai.tool.call.result` | | text, empty → `[no output]` |
+
+Tool calls still pending when `endTurn` fires (turn interrupted/aborted mid-cycle) are closed as orphans with status `ERROR` and `halo.tool.orphaned=true`, so no trace is left with a dangling tool-call id.
+
+All content attributes are capped (32 KB general cap, 2 KB per message part, 8 KB for system instructions); over-cap content is truncated with a `…[truncated N chars]` marker, and an over-cap message list drops the oldest messages first with an omission marker.
+
+## Metrics
+
+Exported every 15s via `PeriodicExportingMetricReader`. Same scope as the spans.
+
+| name | kind | unit | attrs |
+|---|---|---|---|
+| `gen_ai.client.token.usage` | histogram | `{token}` | `gen_ai.request.model`, `gen_ai.token.type=input\|output` |
+| `gen_ai.client.operation.duration` | histogram | s | `gen_ai.request.model`, `gen_ai.operation.name=chat` |
+| `halo.tool.duration` | histogram | s | `gen_ai.tool.name` |
+| `halo.turn.duration` | histogram | s | `outcome=ok\|error` |
+| `halo.model.retries` | counter | — | `kind=<retry kind>` |
+
+`halo.model.retries` is incremented by `recordRetry(kind)` from 7 retry sites in `session-manager.ts`: `context_overflow`, `throttle`, `server_error`, `network`, `empty_response`, `corrupted`, `multimodal_4xx`.
+
+## Logs bridge
+
+Every `logger.*` call in `packages/server/src/logger.ts` also emits an OTel LogRecord when `enabled`: severity number `debug=5 / info=9 / warn=13 / error=17`, attribute `halo.module` set to the call's `[ModuleName]` prefix. Same OTLP endpoint, `/v1/logs`.
+
+## Reference: collector → AWS CloudWatch / X-Ray
+
+Verified working, all AWS-specific pieces live in the collector config, not in halo. `otel/opentelemetry-collector-contrib:0.161.0`:
+
+```yaml
+extensions:
+  sigv4auth/xray:
+    region: <region>
+    service: xray
+  sigv4auth/logs:
+    region: <region>
+    service: logs
+
+processors:
+  resource:
+    attributes:
+      - { key: aws.service.type, value: gen_ai_agent, action: insert }
+      - { key: aws.log.group.names, value: /aws/bedrock-agentcore/runtimes/<name>, action: insert }
+
+exporters:
+  otlphttp/xray:
+    endpoint: https://xray.<region>.amazonaws.com/v1/traces
+    auth: { authenticator: sigv4auth/xray }
+  otlphttp/cwlogs:
+    endpoint: https://logs.<region>.amazonaws.com/v1/logs
+    auth: { authenticator: sigv4auth/logs }
+    headers:
+      x-aws-log-group: /aws/bedrock-agentcore/runtimes/<name>
+      x-aws-log-stream: runtime-logs
+  awsemf:
+    namespace: bedrock-agentcore
+```
+
+Prerequisites: CloudWatch Transaction Search must be enabled (spans land in the shared `aws/spans` log group, no resource policy needed); the log group `/aws/bedrock-agentcore/runtimes/<name>` must be created beforehand.
+
+Halo-side settings for this setup: `endpoint=http://localhost:4318`, `service_name=<name>`, `capture_content=true` (AgentCore Evaluations needs content to score against).
+
+## AgentCore Evaluations
+
+Two modes, both read `aws/spans` filtered by data source `serviceNames: ["<service_name>"]`:
+
+1. **Online evaluation config** (console: AgentCore → Evaluations, or `create-online-evaluation-config`): data source `aws/spans` + serviceNames filter, evaluators e.g. `Builtin.Helpfulness` / `Builtin.GoalSuccessRate` / `Builtin.ToolSelectionAccuracy`, results written to `/aws/bedrock-agentcore/evaluations/results/<config>`. Results appear roughly **15 minutes** after the session goes idle (`sessionTimeoutMinutes` idle detection + a processing queue).
+2. **On-demand `aws bedrock-agentcore evaluate`** — synchronous, result only in the response, not persisted. Input: `{"sessionSpans":[<span doc>, …]}`, each span doc the flattened OTLP shape CloudWatch itself writes into `aws/spans`:
+
+```jsonc
+{
+  "resource": { "attributes": { "service.name": "halo" } },
+  "scope": { "name": "opentelemetry.instrumentation.halo", "version": "1" },
+  "traceId": "...", "spanId": "...", "parentSpanId": "...",
+  "name": "chat <modelId>",
+  "kind": "INTERNAL",
+  "startTimeUnixNano": "...", "endTimeUnixNano": "...", "durationNano": "...",
+  "attributes": { "gen_ai.operation.name": "chat", "...": "..." },
+  "status": { "code": "OK" }
+}
+```
+
+Target by evaluator level:
+
+| evaluator level | example | `--evaluation-target` |
+|---|---|---|
+| SESSION | `GoalSuccessRate` | omit — feed the whole session |
+| TRACE | `Helpfulness`, `Correctness` | `{"traceIds":[…]}` |
+| TOOL_CALL | `ToolSelectionAccuracy` | `{"spanIds":[…]}` (passing traceIds → `ValidationException`) |
+
+Verified feeding the input **directly from halo's own OTLP output** (collector `file` exporter → flatten), no CloudWatch involved: `GoalSuccessRate` 1.0, `Helpfulness` 1.0, `ToolSelectionAccuracy` 1.0.
+
+**Contract for custom instrumentation** (official: [supported-frameworks-generic](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/supported-frameworks-generic.html), plus 11 probe runs):
+
+- `scope.name` MUST start with `opentelemetry.instrumentation.*` (or `openinference.instrumentation.*`) — a custom scope like `halo.tracing` → `no spans with supported scope` (the error message only lists named frameworks; the generic prefix rule still applies).
+- Every span MUST carry `gen_ai.operation.name` ∈ `{invoke_agent, chat, execute_tool}`; spans without it are skipped (`no spans to evaluate`).
+- Content is read from: `invoke_agent`'s `gen_ai.task.input` / `.task.output` (falls back to the first user / last assistant message on `chat` spans); `chat`'s `gen_ai.input.messages` / `.output.messages` / `gen_ai.system_instructions`; `execute_tool`'s `gen_ai.tool.name` / `.tool.call.arguments` / `.tool.call.result`. Values are stringified, not parsed.
+- `session.id` groups spans into a session — required for SESSION-level evaluators.
+- The parser requires `traceId`, `spanId`, `startTimeUnixNano`, `endTimeUnixNano`. `resource`, `scope.version`, `kind`, `status`, `name`, `durationNano`, `parentSpanId` are all optional — a single-span minimal doc scored fine.
+- Strict on keys, lenient on values: garbage attributes with a correct envelope → rejected; correct keys with any string content → scored.
+- `gen_ai.tool.definitions` (JSON array of tool schemas on `chat` spans) is read by `GoalSuccessRate` to check called tools were declared; halo does not emit it and still scored 1.0 — optional.
+
+The official spec is spread over three places: the CLI/API reference (envelope only — `sessionSpans` is a list of opaque `document`s, 1–20000), the generic-framework page (semantic contract), and the runtime's own error messages (structural field requirements). None of the three alone is sufficient.
+
+References: [CLI reference](https://docs.aws.amazon.com/cli/latest/reference/bedrock-agentcore/evaluate.html) · [API reference](https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_Evaluate.html) · [on-demand guide](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/getting-started-on-demand.html)
+
+## Rejected alternatives
+
+- **Direct CloudWatch / X-Ray SDK calls from halo** (an earlier 2026-08 prototype) — vendor lock-in inside core; replaced by OTel + collector.
+- **ADOT `--require` into the halo process** — pulls in `sdk-node` + gRPC + auto-instrumentations, and its `api-logs` 0.219 is incompatible with halo's 0.222 (0.x minors must match); also couples the process to one vendor's distro. Rejected. The `??=` env mapping + external-SDK detection still let an operator do this if they insist.
+- **gRPC exporter** — an extra native-ish dependency (`@grpc/grpc-js`) for a port (4317) that every collector also serves over plain HTTP (4318).

package/bundled-docs/design/relay.md

@@ -0,0 +1,52 @@
+# Relay — cross-workspace dispatch with auto-report
+
+> Tool reference: [dev/tools.md → Relay tools](../dev/tools.md#relay-tools). Code: `packages/server/src/agents/relay.ts`, wired in `session-manager.ts` (`createRelayTools`, the `deliverRelayReport` finally hook), `session-agent-builder.ts` (opt-in gate), `index.ts` (`setRelayRegistry`).
+
+## Problem
+
+Halo runs many workspaces on one server — one per team / project / credential boundary — and each is a self-contained agent runtime. A "secretary" pattern falls out naturally: one workspace whose agent only knows *which department knows what* and forwards the user's question there. Before relay the only cross-workspace path was the ACP adapter (HTTP + SSE + web-channel tokens, and an `ask-<label>` binding per remote), which is the right tool for a *different server* but heavy for two sqlite files on the same box. Relay is the in-process version: no HTTP, no tokens, no polling — and the result comes back on its own.
+
+## Mechanism
+
+Relay mirrors goal mode's shape exactly, because goal mode had already solved "deliver a session's wrap-up to some *other* session when it's genuinely done":
+
+| | Goal mode | Relay |
+|---|---|---|
+| Back-pointer column | `agent_sessions.goal_session_id` on W | `agent_sessions.reply_to` on the target (JSON `{ workspace, sessionId }`) |
+| Delivery hook | `deliverGoalRound` in `runSession`'s finally | `deliverRelayReport`, same finally, right after it |
+| Quiet gate | root + no active children + empty queue | identical |
+| Tool set | `buildGoalTools` (G only) | `buildRelayTools` (opt-in, full access): send / interrupt / stop / read / list |
+| Reach | same workspace | any workspace via `SessionManagerRegistry` |
+
+**Dispatch** (`relay_send` / `relay_interrupt` share one `dispatch(params, hard)`): resolve the target workspace (`realpathSync`, must contain `.halo/`) → `registry.getOrCreate(wsPath)` gives the foreign `SessionManager` → create the session if missing (`relay_interrupt` refuses instead: nothing to interrupt) → `writeReplyTo(target db, session_id, { workspace: caller ws, sessionId: caller session })` → `appendUserMessage` (UI transcript, raw text) + `sendUserMessage` (model-facing, prefixed `[channel: relay | from: <caller ws>]`). The prefix reuses the channel-tag convention the system prompt already teaches ("don't echo the tag"), so the target agent knows the message is agent-sourced without a new prompt rule.
+
+**Hard interrupt** adds one step: if `sendUserMessage` returned `queued` (target was busy), call `target.interruptSession(session_id)`. Order is enqueue-then-abort, same as `querySession(interrupt=true)` — the aborted turn's finally sees a non-empty queue, so it drains into the message instead of firing a relay report for the half-done turn.
+
+**Delivery** (`deliverRelayReport(host, session)`, runs at *every* root turn end in every workspace): cheap `reply_to` column read first (no-op for the 99.9 % of sessions without one) → subtree-quiet gate → body = `finalOutput || output`, `[RELAY TARGET ABORTED …]` prefix when `turnError` is set (before the cap, so the marker survives) → cap at `limits.autoReportMax` with a `relay_read(...)` pointer → `clearReplyTo` → `registry.getOrCreate(reply_to.workspace)` → `appendUserMessage` + `sendUserMessage` on the caller. Append-then-send is the same pair the run-ledger nudge uses: `sendUserMessage` alone never writes the UI transcript, so the secretary's admin view would show a reply to nothing.
+
+## Invariants and why
+
+- **One dispatch, one report.** `reply_to` is cleared *before* the send. A delivery failure (caller workspace unreachable) can't re-fire on the next turn end, and — more importantly — a user who later chats directly in the department workspace never pings the secretary. The secretary can always re-`relay_send` if a report went missing.
+- **Nested trees report once.** The quiet gate is the same one `tryReportToParent` uses, so a target that fans out three sub-agents delivers exactly one relay report, after the last child bubbles up — not one per child turn end.
+- **Full access only.** `session-agent-builder` injects the set only when `nameSet.has('relay_send') && accessLevel === null`. A sandboxed (`workspace` / `readonly`) session can't be given a tool that opens other workspaces' sqlite and session trees; listing the name in such an agent's yaml is silently ignored.
+- **Name-gated bundle.** Only `relay_send` is recognised in `tools:`; the other four (`relay_interrupt` / `relay_stop` / `relay_read` / `relay_list`) ride along. Prevents a half-configured agent that can send but never read back / stop. The admin picker mirrors this with one chip (`GET /agent-configs/tools` builds it from `buildRelayTools` against a dummy target and rewrites the description to name the full set).
+- **Server only.** `setRelayRegistry` is called once in `index.ts`; the CLI / TUI never call it, so `getRelayRegistry()` is `null` there — every tool returns `{"code": 1, "error": "relay is unavailable in this runtime (server only)"}` and `deliverRelayReport` logs a warning and returns. No CLI code path knows relay exists.
+- **Target-side transcript is normal.** The department workspace's session looks like any other root session in its Sessions tab (user message, agent reply); the only trace of relay is the `reply_to` row while a dispatch is pending. Nothing in the target's system prompt changes.
+- **Same workspace is allowed.** `workspace` is only checked for existence + `.halo/`; passing the caller's own path makes `getOrCreate` return the caller's own `SessionManager`, and everything else is unchanged. In effect relay is a `query_session` without the own-tree scoping — it reaches any **root** session in any workspace, including new ones. Not a hole: the tool is full-access only, and a full-access agent could already `shell_exec` its way into any session file.
+- **Reports come back from roots only.** `deliverRelayReport` fires for `parentId === null`; a sub-session's turn end goes to `tryReportToParent` instead. Dispatching to a `parent>child` id delivers the message but never a report — target roots when you want to hear back. Dispatching to yourself works and terminates (one self-report, `reply_to` cleared) but is pointless.
+
+## Soft vs hard interrupt
+
+`relay_send` to a busy target = queued + soft interrupt (the current tool call finishes, then the message drains as part of one merged turn). That covers follow-ups and corrections — the secretary's default. `relay_interrupt` is for "stop what you're doing *now*": it aborts the in-flight turn (propagates to `shell_exec`, SIGTERMs the process group), then the queued message runs. Idle target → `relay_interrupt` degrades to a plain send (`interrupted: false`). The distinction is the same as `query_session` vs `interrupt_session` inside one workspace; relay just carries it across.
+
+## What it deliberately doesn't do
+
+- **No cross-server reach.** Same `SessionManagerRegistry`, same process. Different server → ACP adapter (`/acp add`).
+- **No session discovery beyond a flat root list.** `relay_list` returns a workspace's root sessions (id / agent / title / status, newest 100) so the secretary can reuse an existing conversation; it doesn't walk trees or search transcripts — that's the admin's job.
+- **No broadcast / fan-out helper.** The secretary calls `relay_send` once per department; the reports come back individually with the workspace + session id in the header, so it can tell them apart without extra bookkeeping.
+- **No ACL beyond full access.** A full-access secretary can reach every workspace on the box — that's the deployment's trust boundary already (it can `shell_exec` into them anyway).
+- **No admin UI for the link.** The pending `reply_to` isn't surfaced anywhere; the secretary's own transcript (dispatch tool call → `[Relay report …]` message) is the audit trail.
+
+## Test
+
+`packages/server/test/relay.test.ts` — stub `RelayTarget` + registry, 11 cases: `deliverRelayReport` no-op without `reply_to`, waits while a child is active, delivers exactly one append + one send with the header then clears, `ABORTED` prefix on `turnError`, ignores sub-sessions; `relay_send` creates + stamps `reply_to` + channel-prefixes the model text, rejects a nonexistent workspace; `relay_interrupt` busy → enqueue then abort + re-stamp, idle → no abort and never creates; `relay_list` roots only with title fallback, and defaults to the caller's own workspace.

package/bundled-docs/design/session.md

@@ -33,13 +33,15 @@ These sessions also do **not** get an `agent_sessions` row in the workspace's `h
   "archivedUserCount": 30,
   "messages": [...],
   "rawMessages": [...],
-  "output": "..."
+  "output": "...",
+  "lastActivityAt": "2026-09-18T..."
 }
 ```
 
 - **messages**: event log format (written by the WS handler) — context / usage / tool_call / tool_result / agent_start/done, with full debug info. Assistant rows persist their tool calls **only** in `contentBlocks` (interleaved with text / thinking); the flat `toolCalls` array is no longer written — it duplicated every tool's input *and* output byte-for-byte (measured: 66 MiB of 340 MiB across 195 sessions). Readers are blocks-first with `toolCalls` as a pure legacy fallback (`messageToolCalls()` in `sessions/session-types.ts`; see [storage.md](storage.md#sessionmessage))
 - **rawMessages**: Bedrock API shape (written by SessionManager `saveAgentState`) — raw user/assistant turns with toolUse/toolResult blocks
-- **output**: accumulated assistant text
+- **output**: accumulated assistant text of the latest turn
+- **lastActivityAt**: ISO time of the latest turn's last text / tool event (null if none) — written alongside `output` by `saveAgentState` so a released session's `get_session_output` can still report liveness
 - **archiveCount / archivedUserCount**: UI-log archiving bookkeeping — absent until the first archive. See [UI-log archiving](#ui-log-archiving)
 
 `saveSessionToFile()` uses read-merge-write so both halves survive. When loading, the event log `messages` takes priority; only when `messages` is empty (e.g. a sub-session tracked only by SessionManager) does `rawMessages` get converted to display format.
@@ -95,7 +97,10 @@ interface AgentSession {
   agentId: string
   agent: ModelRuntime
   description: string
-  output: string
+  output: string                   // all assistant text of the current turn (reset per turn)
+  finalOutput: string              // wrap-up text only (stopReason !== 'tool_use') — feeds auto-report / goal round / relay report
+  lastActivityAt: string | null    // ISO of the turn's latest text / tool_call / tool_result event; null at turn start — surfaced by get_session_output
+  turnError: string | null         // set when the turn died on an unrecoverable error — prefixes the report with an ABORTED marker
   promise: Promise<string> | null  // non-null = running
   abortController: AbortController | null
   messageQueue: QueuedMessage[]    // single unified queue: user→agent AND agent→agent
@@ -142,7 +147,7 @@ Hierarchical encoding: `root_id>child_segment>grandchild_segment`.
 
   **Abnormal-termination marker**: when the reported turn was killed by an unrecoverable error (retry budget exhausted / account-level failure), the session's `turnError` field holds the error text and the auto-report is prefixed with an explicit `[SUB-AGENT ABORTED: … Error: <text> … Re-dispatch with query_session(…) to resume.]` block. Without it the parent LLM consumed mid-turn fragments (or a literal "(no output)") as completed reports — the cron-era "sub-agent reported without finishing" incident, root-caused to a Bedrock h2 hang (`TimeoutError: http2 request did not get a response`, now also in the transient-transport retry list). Prefix, don't suppress: skipping the report would leave the parent waiting forever, the partial trace has diagnostic value, and `stoppedAt` is stamped as usual so the child never shows as falsely running. The marker is prepended *before* the truncation cap so it can never be sliced off. (`deliverGoalRound` has the same `finalOutput || output` shape without an error dimension — known gap, unfixed.)
 
-  The full finally-chain order is `tryReportToParent → deliverGoalRound → releaseSession`: after the parent-report check, goal-bound **root** sessions get their round report delivered to the goal session (fire-and-forget; a no-op for everyone else — see [goal-mode.md](goal-mode.md#the-delivery-point-delivergoalround)), then the session is released.
+  The full finally-chain order is `tryReportToParent → deliverGoalRound → deliverRelayReport → releaseSession`: after the parent-report check, goal-bound **root** sessions get their round report delivered to the goal session (fire-and-forget; a no-op for everyone else — see [goal-mode.md](goal-mode.md#the-delivery-point-delivergoalround)), then a root whose row carries a `reply_to` back-pointer (it was dispatched from another workspace via `relay_send`) gets its wrap-up appended + sent into the caller session in that workspace (same subtree-quiet gate, `reply_to` cleared before sending — see [relay.md](relay.md)), then the session is released. Each hook decides on its own db column (`parent_id` / `goal_session_id` / `reply_to`), so the three never fire for the same session.
 
 - **Sibling-status injection for root** (`siblingStatusSuffix`): `tryReportToParent` early-returns for root (`parentId === null`) since there's no parent to bubble up to — so root never learns whether its *other* children are still running, and the root LLM could wrap up early after consuming just one child's report. When root consumes a child report (`querySession` idle branch + `drainQueue`), a sibling-status line is appended to the message fed to the LLM. "All sub-agents done" requires **both** no sibling running in the DB (`parentId = root AND stoppedAt IS NULL`) **and** an empty in-memory `messageQueue` — a child can be stopped while its report is still queued, so the DB check alone would falsely declare completion. The reporting child needs no identity exclusion: it stamped `stoppedAt` before the report was delivered, so `stoppedAt IS NULL` already excludes it (unless re-dispatched a new task, which clears `stoppedAt` — then it correctly counts as running). The line carries per-child `created` + `last active` timestamps so a capable model can tell a freshly dispatched sibling from an original-batch leftover. (Its `[System @ <iso>]` header used to be the *only* wall-clock signal the model ever saw; since 1.1.6 every turn carries an arrival stamp — see [Message queue and drain](#message-queue-and-drain).) Mid-tier parents are excluded by design — their `tryReportToParent` bubble-up already gates them on a fully-drained subtree.
 
@@ -176,6 +181,8 @@ A `QueuedMessage` is `{ text, sourceSessionId?, images? }`: `sourceSessionId` is
 
 **runSession loop** (`runSession(sessionId, message)`, `message: string | ContentBlock[]`): runs the opening turn when there is one, then drains. An empty **string** message means "the work is already in `messageQueue`" (the `querySession` idle path) — it skips the opening turn and goes straight to drain. The per-turn reset (fresh `toolCallLog` / `warnedToolHashes`, `interruptRequested = false`) runs before the opening turn; `drainQueue` repeats the same reset before each merged batch so a stale interrupt flag never leaks across turns. Server-only: right before `session.promise = runFn()` it inserts the session id into the run ledger, and in the `finally` (right after `session.promise = null`, before emitting `complete`) it deletes it — see [Run ledger](#run-ledger--restart-nudge-for-interrupted-roots-halo-globalrunsdb) above.
 
+**Observability.** `runAgentTurn` is wrapped by `beginTurn` / `endTurn` from `observability/genai-spans.ts` — one `invoke_agent` span per turn, with `chat` / `execute_tool` child spans created from the `onAgentEvent` hook inside the event loop; see [design/observability.md](observability.md).
+
 **Arrival stamp.** Both paths run the turn through `runAgentTurn`, which prepends `[<ISO-8601 UTC>] ` (e.g. `[2026-09-12T15:17:44.153Z] `) to the first text block of the model-facing input — an image-only input gets a stamp-only text block unshifted. The model has no clock; without this it couldn't tell a reply that came two days later from one that came instantly, or how long a sub-agent report took to land. Stamped **once, before the retry loop**, so every attempt (and the [multimodal 4xx degrade](#multimodal-4xx-degrade) rebuild) reuses the identical input. Agent reports therefore read `[<iso>] (from: session X)\n…` — stamp outermost, no special-casing. Only `agent.messages` changes: the UI `type:'user'` event keeps the raw text (see [Turn matching](#exchange-deletion-soft-ui--hard-raw) for the strip on compare), and the self-compact instruction / `[Conversation Summary]` injection / `tool_result` messages are never stamped.
 
 **Final-text flag.** The agent loop marks each text block `final` when the model call ended for a reason other than `tool_use` (i.e. it is the turn's wrap-up, not the filler emitted before a tool call), and `runAgentTurn`'s `case 'text'` forwards it on the `stream` event as `AgentSessionEvent.final`. Block-send consumers gate on it: the channel responders (wechat / telegram / slack / feishu) append only `final` text, and `halo cli` prints the last root turn's `final` text as its stdout (fallback: that turn's full text). `finalOutput` (used by `tryReportToParent` / `deliverGoalRound`) is the same accumulation on the server side.
@@ -257,6 +264,7 @@ Table `agent_sessions` holds metadata only (no runtime state):
 | stopped_at | INTEGER | null = active |
 | archived_at | INTEGER | null = not archived |
 | goal / goal_session_id | TEXT | Goal mode: binding JSON on G's row, back-pointer on W's — see [goal-mode.md](goal-mode.md) |
+| reply_to | TEXT | Relay: JSON `{ workspace, sessionId }` of the caller in another workspace, set by `relay_send`, cleared when the report is delivered — see [relay.md](relay.md) |
 | title / exchange_count / context_tokens / total_output_tokens | TEXT / INTEGER | List-visible metadata mirrored from the session file header; null = row predates the columns |
 
 Status is derived from memory (`promise !== null`) — not stored.

package/bundled-docs/design/storage.md

@@ -114,7 +114,8 @@ Path: `.halo/sessions/{agentId}/{sessionId}.json`
   "archivedUserCount": 30,           // main user turns that moved into segments
   "messages": [SessionMessage],       // UI event-log format (written by WS handler / UIState reducer)
   "rawMessages": [AnthropicMessage],  // raw Bedrock API shape (written by SessionManager saveAgentState); user text blocks start with the `[<iso>] ` arrival stamp — strip it (stripTurnStamp) before matching against `messages`
-  "output": "..."                     // accumulated assistant text from the latest turn
+  "output": "...",                    // accumulated assistant text from the latest turn
+  "lastActivityAt": "ISO8601"         // latest turn's last text / tool event; null if none — read by get_session_output for released sessions
 }
 ```
 
@@ -386,6 +387,10 @@ A value of the form `<<ENV_NAME>>` is replaced at read time with `process.env.EN
 | `general.sandbox.hidden_dirs` | `~/.halo/secrets,~/.aws,~/.ssh,~/.gnupg,~/.docker,~/.config/gh` | bwrap tmpfs overlay (Linux only) |
 | `general.sandbox.hidden_files` | `~/.npmrc,~/.bash_history,~/.gitconfig,~/.git-credentials,~/.netrc,~/.halo/global/evo.db,~/.halo/global/evo.db-wal,~/.halo/global/evo.db-shm,~/.halo/global/cron.db,~/.halo/global/cron.db-wal,~/.halo/global/cron.db-shm,~/.halo/global/runs.db,~/.halo/global/runs.db-wal,~/.halo/global/runs.db-shm` | bwrap /dev/null bind (Linux only) |
 | `general.logging.level` | warn | `debug` / `info` / `warn` / `error` |
+| `general.observability.endpoint` | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`); empty = off. Restart required. See [observability.md](observability.md) |
+| `general.observability.service_name` | `halo` | OTel resource `service.name`. Restart required |
+| `general.observability.headers` | `''` | **Secret.** Extra OTLP request headers, comma-separated `k=v`. Restart required |
+| `general.observability.capture_content` | `false` | Put prompt/completion/tool text on spans. Restart required |
 
 Schema source: [packages/server/src/settings-schema.ts](../../../packages/server/src/settings-schema.ts) `generalSection()`.
 
@@ -566,6 +571,7 @@ The databases hold session metadata indexes and workspace-scoped preferences (e.
 | stopped_at | INTEGER | Stopped timestamp (null = active) |
 | archived_at | INTEGER | Archived timestamp (null = not archived) |
 | goal / goal_session_id | TEXT | Goal mode — binding JSON on the goal session's row, back-pointer on the worker's |
+| reply_to | TEXT | Relay — JSON `{ workspace, sessionId }` of the caller session in another workspace; set by `relay_send`, cleared on report delivery. Added via idempotent ALTER in `db/index.ts`. |
 | title | TEXT | Mirror of the session file's `title` |
 | exchange_count | INTEGER | Mirror: main user turns over the session's lifetime (kept + archived) |
 | context_tokens | INTEGER | Mirror of the file's `contextTokens` |
@@ -628,3 +634,4 @@ All channel types (telegram, web, wechat, slack, feishu) share one table. Common
 | v1+ | 2026-05-13 | Settings i18n + UI polish: compact keys renamed camelCase → snake_case (`keep_messages`, `max_summary_input`, `max_message_slice`, `summarize_timeout_sec`). Added `description_zh` on all leaf nodes and `description`/`description_zh` on branch nodes (replaces `_hint`/`_hint_zh`). Added `secret: true` attribute for masked input. Added Tavily to default params. Admin UI fully i18n'd (agents, skills, settings, nav). |
 | v1+ | 2026-05-13 | Added `packages/cli` — standalone CLI/TUI client with embedded agent loop (no server required). Imports server agent-core via subpath exports. Session prefix: `cli_`. Added `exports` + `typesVersions` to server `package.json`. |
 | v1+ | 2026-08-08 | Session file slimming (no version bump — additive + reader-compatible): assistant `toolCalls` no longer written (`contentBlocks` is the single copy, `toolCalls` stays read-only legacy); files written as compact JSON; UI-log archiving adds header fields `archiveCount` / `archivedUserCount` plus `{sessionId}.arch.{N}.json.gz` segment files; `agent_sessions` gains mirrored `title` / `exchange_count` / `context_tokens` / `total_output_tokens` columns so listing doesn't parse every file. |
+| v1+ | 2026-09-18 | Relay + liveness (additive, no version bump): `agent_sessions.reply_to` column (idempotent ALTER); session file gains `lastActivityAt` beside `output`, read back by `get_session_output` for released sessions. |

package/bundled-docs/dev/api.md

@@ -355,6 +355,15 @@ File: `packages/server/src/routes/settings.ts`
 
 PATCH / DELETE walk the dotted key layer by layer, so a segment of `__proto__` / `constructor` / `prototype` is rejected with 400 `Invalid key` — the leaf assignment would otherwise land on `Object.prototype` and pollute every object in the process.
 
+### Observability settings
+
+- `general.observability.endpoint` — OTLP collector base URL (e.g. `http://localhost:4318`); empty = off
+- `general.observability.service_name` — OTel resource `service.name`, default `halo`
+- `general.observability.headers` — extra OTLP request headers, comma-separated `k=v` (secret)
+- `general.observability.capture_content` — put prompt/completion/tool text on spans, default `false`
+
+All four are `globalOnly`, take effect on restart. See [design/observability.md](../design/observability.md).
+
 ## Metrics
 
 File: `packages/server/src/routes/metrics.ts`.

package/bundled-docs/dev/deploy.md

@@ -204,6 +204,8 @@ This installs the `halo` binary on `$PATH`. Subcommands available:
 4. **Build admin**: `pnpm --filter @turmind/halo-admin build` — verify `admin/out/monaco/vs/loader.js` exists.
 5. Commit, tag `vx.y.z`, push, then publish.
 
+**npm token gotcha**: `npm publish` on this package needs a granular access token created with **"Bypass 2FA"** checked — scope / permission alone yields `403 Two-factor authentication or granular access token with bypass 2fa enabled is required`. `npm whoami` and `npm token list` succeed with a non-bypass token, so neither is a valid pre-flight; check `GET https://registry.npmjs.org/-/npm/v1/tokens` (with the token as bearer) and look for `"bypass_2fa": true` on the token in use before starting a release.
+
 The published package contains a single bundled JS entry (~620 KB), all built-in templates (agents / skills / prompts / models), bundled platform docs, and the admin Web UI static export. Total install footprint ≈ 120 MB after npm dedupes shared deps.
 
 ### Non-interactive (Docker / CI) details
@@ -226,7 +228,27 @@ EXPOSE 9527
 CMD halo setup -y && halo server start
 ```
 
-## 9. Optional: Nginx reverse proxy
+## 9. Observability (OpenTelemetry collector)
+
+Halo exports traces, metrics and logs over OTLP http/protobuf when `general.observability.endpoint` is set — off by default, no vendor code in the server itself. Point it at any OpenTelemetry collector:
+
+```bash
+# settings.yaml (global scope) or via the admin Settings page
+general:
+  observability:
+    endpoint: http://localhost:4318       # collector's OTLP/HTTP port, NOT 4317 (gRPC, unsupported)
+    service_name: halo
+    headers: ''                            # e.g. authorization=Bearer <token> for a hosted backend
+    capture_content: false                 # true to include prompt/completion/tool text on spans
+```
+
+All four keys take effect on restart: `halo server restart`.
+
+For a hosted backend (Honeycomb, Grafana Cloud, etc.) use its `https://` OTLP endpoint and put its auth token in `headers` — TLS with CA-signed certs works out of the box. Self-signed / mTLS collectors need the standard `OTEL_EXPORTER_OTLP_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` / `OTEL_EXPORTER_OTLP_CLIENT_KEY` env vars.
+
+For routing into AWS CloudWatch / X-Ray (sigv4auth collector config, AgentCore Evaluations) see [design/observability.md](../design/observability.md).
+
+## 10. Optional: Nginx reverse proxy
 
 ```nginx
 server {

package/bundled-docs/dev/env.md

@@ -128,6 +128,10 @@ settings.yaml only (no env override):
 - `general.sandbox.writable_dirs` (default empty) — dirs bind-mounted read-write inside the bwrap sandbox, for external CLIs that keep local state (e.g. `~/.kiro`); not applied to readonly sessions, scope: global
 - `general.sandbox.hidden_files` (default `~/.npmrc,~/.bash_history,~/.gitconfig,~/.git-credentials,~/.netrc,~/.halo/global/{evo,cron,runs}.db` + their `-wal`/`-shm` files) — bwrap /dev/null binds, scope: global
 - `general.logging.level` (default `warn`) — log level: debug | info | warn | error
+- `general.observability.endpoint` (default `''`) — OTLP collector base URL (e.g. `http://localhost:4318`); empty = off, scope: global, restart required. See [design/observability.md](../design/observability.md)
+- `general.observability.service_name` (default `halo`) — OTel resource `service.name`, scope: global, restart required
+- `general.observability.headers` (default `''`, secret) — extra OTLP request headers, comma-separated `k=v`, scope: global, restart required
+- `general.observability.capture_content` (default `false`) — put prompt/completion/tool text on spans, scope: global, restart required
 
 `<provider-id>.secrets.*` (server-only, hard-rejected by `{{}}` substitution; declared in `models/<provider-id>.yaml` `secrets:`):
 - `aws-bedrock-claude-invoke.secrets.access_key_id` / `.secret_access_key` — AWS / Bedrock credentials

package/bundled-docs/dev/tools.md

@@ -1,6 +1,6 @@
 # Agent Tool Reference
 
-Agents have two tool categories: workspace tools (files, shell, search) and session tools (managing other sessions). Workspace tools are enabled by name in `agent.yaml`'s `tools` list; the session-tool bundle is granted automatically by a non-empty `team` (see [Session tools](#session-tools)).
+Agents have two tool categories: workspace tools (files, shell, search) and session tools (managing other sessions). Workspace tools are enabled by name in `agent.yaml`'s `tools` list; the session-tool bundle is granted automatically by a non-empty `team` (see [Session tools](#session-tools)). A third, opt-in set — [Relay tools](#relay-tools) — reaches sessions in *other* workspaces on the same server.
 
 ## Workspace tools
 
@@ -183,7 +183,7 @@ Session management tools for agents. **Not enabled by name** — the whole bundl
 
 ### start_session
 
-Start a sub-agent session asynchronously. When the sub-agent finishes, its **wrap-up reply** (the closing summary — not the mid-task progress narration) is auto-delivered back to the caller's conversation. A long summary is **cut head-kept / tail-dropped** at `limits.autoReportMax` (default 8,192 chars, settings `general.limits.auto_report_chars`) with a `[Report truncated: N chars total, showing first M. Use get_session_output("<id>") for the full result.]` marker — so the caller can tell a short answer from a cut-off one, and knows the **tail** is what's missing. Call `get_session_output` for the full untruncated reply. (Before 0.1.5 the auto-report concatenated every text segment of the turn, including mid-task filler.)
+Start a sub-agent session asynchronously. When the sub-agent finishes, its **wrap-up reply** (the closing summary — not the mid-task progress narration) is auto-delivered back to the caller's conversation. A long summary is **cut head-kept / tail-dropped** at `limits.autoReportMax` (default 8,192 chars, settings `general.limits.auto_report_chars`) with a `[Report truncated: N chars total, showing first M. Use get_session_output("<id>") for the full result.]` marker — so the caller can tell a short answer from a cut-off one, and knows the **tail** is what's missing. Call `get_session_output` for the rest — it keeps the tail when it has to cut, so the two are complementary. (Before 0.1.5 the auto-report concatenated every text segment of the turn, including mid-task filler.)
 
 **Arguments**
 
@@ -275,18 +275,22 @@ Abort the current task of a running session. Any queued messages are **not** dro
 
 ### get_session_output
 
-Read the **complete, untruncated** text of an agent session's reply to its **most recent message** — the full response spanning every step taken for that message (one message can drive many steps: narration → tool calls → more narration), which is more than the possibly-cut auto-report. Scoped to that one message's reply, not the session's whole history. Excludes tool calls/results and thinking — those only ever stream to the UI, never into the output.
+Read the text of an agent session's reply to its **most recent message** — the full response spanning every step taken for that message (one message can drive many steps: narration → tool calls → more narration), which is more than the possibly-cut auto-report. Scoped to that one message's reply, not the session's whole history. Excludes tool calls/results and thinking — those only ever stream to the UI, never into the output.
 
 | Arg | Type | Required | Description |
 |---|---|---|---|
 | `session_id` | string | yes | Session to read |
 
+**Output (JSON string)**: `{ "code": 0, "status": "running" | "idle" | "stopped", "output": "...", "last_activity_at": "<ISO>" | null }`. `status` is `running` while a turn is in flight (`promise !== null`), `stopped` when the row carries `stopped_at` (sub-agent reported / user stopped), otherwise `idle`. `last_activity_at` is the wall-clock time of the turn's most recent text / `tool_call` / `tool_result` event (reset to `null` at each turn start) — together with `status` it tells the caller whether a long-running session is still alive or has silently stalled. Unknown / out-of-tree id → `{"code": 1, "error": "session <id> not found"}`.
+
+**Truncation keeps the tail.** When `output` exceeds `limits.toolResultMax` (minus 500 chars of envelope headroom) it is cut **head-first** with a `[Output truncated: N chars total, showing LAST M. Earlier text omitted.]` marker up front — the conclusion lives at the end, and the generic head-keep tool-result cap used to eat exactly that (plus the trailing JSON fields). The auto-report's cut is the opposite (head-kept, see [start_session](#start_session)), so the two are complementary: report shows the opening, `get_session_output` shows the ending. (Before 1.2.0 the tool returned the bare output string with no status and the generic head-keep truncation.)
+
 Implementation: a turn (one `runAgentTurn`, processing one inbound message) accumulates text into **two** per-turn buffers, both reset at the turn's start:
 
 - `session.output` — **all** assistant text of the turn (mid-task filler + wrap-up). This is what `get_session_output` returns and what is persisted to disk.
 - `session.finalOutput` — **only** the wrap-up reply (text emitted when `stopReason !== 'tool_use'`, flagged by the agent-loop `final` event field). This feeds the auto-report to the parent (`tryReportToParent`), falling back to `session.output` when the turn ended without a closing message.
 
-In-memory sessions return `session.output`; released sessions read the `output` field from `.halo/sessions/{agentId}/{sid}.json`. `get_session_output` never truncates — only the auto-report does (see [start_session](#start_session)). (Split introduced in 0.1.5; before that both reads shared one `session.output`.)
+In-memory sessions return `session.output` / `session.lastActivityAt`; released sessions read the `output` / `lastActivityAt` fields from `.halo/sessions/{agentId}/{sid}.json` (both written by `saveAgentState`). (Split introduced in 0.1.5; before that both reads shared one `session.output`.)
 
 ### query_agent
 
@@ -355,12 +359,87 @@ G's **lateral edge**: same name as the standard session tool, different implemen
 
 ### get_session_output (goal-scoped)
 
-Read the full latest-turn output of the worker or any session in the worker's subtree (evidence gathering — round reports are truncated at `limits.autoReportMax`). Scoped to the worker's tree; works regardless of goal status.
+Read the full latest-turn output of the worker or any session in the worker's subtree (evidence gathering — round reports are truncated at `limits.autoReportMax`). Scoped to the worker's tree; works regardless of goal status. Same `{ code, status, output, last_activity_at }` shape and tail-keeping truncation as the standard [get_session_output](#get_session_output) — the goal wrapper only adds the tree check and passes `host.getSessionOutput` through.
 
 | Arg | Type | Required | Description |
 |---|---|---|---|
 | `session_id` | string | yes | Worker session id or a descendant (`worker>child`) id |
 
+## Relay tools
+
+File: `packages/server/src/agents/relay.ts` (`buildRelayTools`). Design notes in [design/relay.md](../design/relay.md).
+
+Cross-**workspace** dispatch on the same server: a "secretary" agent in workspace S hands a message to a session in workspace D and gets the result pushed back when D is done, no polling. Everything is in-process via the server's `SessionManagerRegistry` (`setRelayRegistry` in `index.ts`) — no HTTP, no tokens. The CLI / TUI never set the registry, so there every relay tool returns `{"code": 1, "error": "relay is unavailable in this runtime (server only)"}`.
+
+**Enabling**: opt-in by the single name `relay_send` in `agent.yaml`'s `tools:` — `session-agent-builder` sees that name and injects the whole set (`relay_send` / `relay_interrupt` / `relay_stop` / `relay_read` / `relay_list`), the other four names are not recognised on their own. **Full-access sessions only** (`accessLevel === null`): a `workspace` / `readonly` session listing `relay_send` gets nothing, because the tools reach into other workspaces' sqlite and session trees. The admin Agents tool picker shows one `relay_send` chip whose description names the whole set (`GET /agent-configs/tools` builds it from `buildRelayTools` against a dummy target).
+
+**Common arguments** (every tool except `relay_list`, where it is optional): `workspace` — absolute path of the target workspace on this server (realpath'd; must contain `.halo/`, else `{"code": 1, "error": "not a halo workspace (no .halo/): …"}`); `session_id` — session id inside that workspace.
+
+**Scope**: the target may be the caller's **own** workspace too — there is no "must be different" check, so relay doubles as a `query_session` without the own-tree scoping (any root session, not just the caller's tree). Reports are only delivered for **root** targets (`parentId === null`); a `parent>child` id receives the message but its wrap-up goes to its parent, never back to the caller.
+
+### relay_send
+
+Dispatch a message to a session in another workspace. Creates the session if `session_id` does not exist there (root session, agent `agent_id` or the target workspace's default agent, description `Relay: <first 60 chars>`), stamps the target row's `reply_to` with `{ workspace: <caller ws>, sessionId: <caller session> }`, appends the raw message to the target's UI transcript and sends it prefixed `[channel: relay | from: <caller ws>]\n\n<message>` — so the target agent knows the message came from another workspace's agent, not the admin UI, and doesn't echo the tag. Busy target → the message is **queued + soft interrupt** (same as `query_session` / a user message mid-turn: the current tool finishes, then the queue drains as one merged turn), so follow-ups and corrections ride the same tool. Returns immediately.
+
+| Arg | Type | Required | Description |
+|---|---|---|---|
+| `workspace` | string | yes | Target workspace path |
+| `session_id` | string | yes | Target session id (created if missing) |
+| `message` | string | yes | The message to deliver |
+| `agent_id` | string | no | Agent to create the session with when it does not exist yet; default = the target workspace's default agent |
+
+Returns `{ "code": 0, "workspace": "<realpath>", "session_id": "…", "state": "running" | "queued" }`.
+
+**Report delivery**: when the target root's turn ends **and its subtree is quiet** (no active children in the db, empty message queue — the same gate `tryReportToParent` and `deliverGoalRound` use, so a nested dispatch tree reports exactly once, at the end), `deliverRelayReport` (fourth hook in `runSession`'s finally) appends + sends into the caller session:
+
+```
+[Relay report · workspace <target ws> · session <id>]
+
+<target's finalOutput || output>
+```
+
+Body capped at `limits.autoReportMax` (head-kept) with a `[Report truncated: N chars total. Use relay_read("<ws>", "<id>") for the full text.]` marker. A turn killed by an unrecoverable error is prefixed `[RELAY TARGET ABORTED: … Error: <text> … Re-send with relay_send to let it resume.]` **before** the cap so it can't be sliced off. `reply_to` is cleared **before** sending: one dispatch → one report, a failed delivery can't double-fire on the next turn end, and a user chatting directly in the department workspace afterwards never pings the secretary.
+
+### relay_interrupt
+
+`relay_send` **plus a hard abort** of the target's in-flight turn — `interrupt_session` semantics across workspaces: the message is enqueued first, and if the target was busy (`state === 'queued'`) its turn is aborted immediately (propagates to `shell_exec`, SIGTERMs the process group) so the drain picks the message up now instead of after the current step. Enqueue-then-abort order matters: the finally never sees an empty queue and so never fires a spurious relay report. Idle target → behaves exactly like `relay_send` (`interrupted: false`). Unlike `relay_send` it does **not** create a missing session (`{"code": 1, "error": "session not found"}`) — there is nothing to interrupt. Use it when the target is heading the wrong way and waiting for its current step is not acceptable; for ordinary follow-ups prefer `relay_send`.
+
+| Arg | Type | Required | Description |
+|---|---|---|---|
+| `workspace` | string | yes | Target workspace path |
+| `session_id` | string | yes | Target session id (must exist) |
+| `message` | string | yes | The message the target runs after the abort |
+
+Returns `{ "code": 0, "workspace", "session_id", "state": "running" | "queued", "interrupted": boolean }`.
+
+### relay_stop
+
+Cascades `stopSession` on the target session and its sub-agents. If the target was mid-turn the caller still receives a relay report describing where it was cut off (the stop ends the turn → finally → `deliverRelayReport`, with the partial trace as body).
+
+| Arg | Type | Required | Description |
+|---|---|---|---|
+| `workspace` | string | yes | Target workspace path |
+| `session_id` | string | yes | Target session id (must exist) |
+
+### relay_read
+
+The target workspace's [get_session_output](#get_session_output) — same `{ code, status, output, last_activity_at }` shape and tail-keeping truncation. Use it to check on progress or to fetch the full text after a truncated relay report.
+
+| Arg | Type | Required | Description |
+|---|---|---|---|
+| `workspace` | string | yes | Target workspace path |
+| `session_id` | string | yes | Target session id |
+
+### relay_list
+
+List a workspace's **root** sessions — most recently active first, capped at 100, archived rows excluded — so the caller can find an existing session to `relay_send` into (instead of minting a new id every time) or see what a department is currently working on. `workspace` is optional and defaults to the caller's own workspace, which makes it a `session_list` for roots you don't own (the standard `session_list` only shows your direct children).
+
+| Arg | Type | Required | Description |
+|---|---|---|---|
+| `workspace` | string | no | Workspace path; omit for the current workspace |
+
+Returns `{ "code": 0, "workspace": "<realpath>", "sessions": [{ id, agentId, agentName, title, status, createdAt, updatedAt }], "count": N }`. `title` falls back to `description` like `session_list`; `status` follows the list semantics (`running` when the root itself or any live child is mid-turn, `stopped` when the row is stamped, else `idle`).
+
 ## Self-review tool
 
 ### draft
@@ -391,7 +470,7 @@ skills:
   - code-review    # auto-injects activate_skill
 ```
 
-Tools not listed are not injected. Session/delegation tools do **not** go in `tools:` — they ride on a non-empty `team` (see [Session tools](#session-tools) above). `activate_skill` is auto-injected whenever the YAML lists `skills` (no need to put it in `tools`).
+Tools not listed are not injected. Session/delegation tools do **not** go in `tools:` — they ride on a non-empty `team` (see [Session tools](#session-tools) above). `activate_skill` is auto-injected whenever the YAML lists `skills` (no need to put it in `tools`). The relay set is the one name-gated bundle: listing `relay_send` alone brings `relay_interrupt` / `relay_stop` / `relay_read` / `relay_list` with it, full-access sessions only (see [Relay tools](#relay-tools)).
 
 There is **no implicit default tool set**: `filterTools()` (in `agent-loader.ts`) returns only the tools whose names appear in `agent.yaml`'s `tools:` list. If the field is absent or empty, the agent has zero workspace tools. The admin UI's "Create agent" form scaffolds a fresh agent with an empty `tools: []` for the same reason — fill it in deliberately. The `default` agent's bundled `agent.yaml` lists the common set (`file_read` / `file_write` / `file_edit` / `view_image` / `file_list` / `shell_exec` / `grep` / `glob` / `web_fetch`) that most agents will want, plus `draft` (see Self-review tool above); copy that line if you're starting from scratch.
 

package/bundled-docs/guide/workspace.md

@@ -103,6 +103,7 @@ Common settings:
 - `general.session.max_nesting_depth` — max session nesting depth
 - `general.compact.keep_messages` — recent messages kept intact during compaction
 - `general.logging.level` — `debug` / `info` / `warn` / `error`
+- `general.observability.endpoint` — OTLP collector URL; leave empty to keep observability off (see [design/observability.md](../design/observability.md))
 - `<provider-id>.secrets.api_key` — provider credentials (Kimi / DeepSeek bearer token)
 - `<provider-id>.secrets.access_key_id` / `.secret_access_key` — AWS Bedrock
 - `<skill-id>.params.<key>` — values an agent can inject into its own `shell_exec` via `{{<skill-id>.params.<key>}}`