@turmind/halo 2.1.5 → 2.2.0

package/bundled-docs/design/evolution.md

@@ -24,20 +24,24 @@ Let agents draft and validate improvements to their own prompt surface by learni
    │   (Phase A'/B'/12 for apply: merge+regress+sync) │
    └────────────────────────────────────────────────┘
                             ↓
-         evolution_runs: id → 'awaiting_review'
-                   ↓ (admin approves)
+         L2/L3: three gates (source full · change category · score)
+            ↓ pass                              ↓ fail / below L2
+   wrapper queues the apply itself      evolution_runs: id → 'awaiting_review'
+   (run → 'approved', approved_by=auto)            ↓ (admin approves)
+                            ↓                       ↓
          evolution_applies: id → 'pending'
                             ↓ (ticker spawns apply wrapper)
               wrapper merges patches, regress-tests
-                            ↓ (phase 12 sync to main)
+                            ↓ (phase 12: auto re-check, sync to main)
            evolution_applies: id → 'applied'
                 evolution_runs: id → 'applied'
-                            ↓
+                            ↓ (auto: one chat:system line in the source session)
+                            ↓ (either: one-click rollback → 'rolled_back')
          SessionManager cold-start picks up new prompts
               next user message uses patched rules
 ```
 
-All coordination via global db (`~/.halo/global/evo.db`). Wrapper heartbeats every 60s; ticker requeues a stale `running` row (→ `timeout` once `max_attempts` is spent). Detached process model: server restart doesn't kill in-flight wrappers.
+All coordination via global db (`~/.halo/global/evo.db`). Wrapper heartbeats every 60s, only on its own claim (status + attempt); a heartbeat that finds the row re-claimed stops writing, SIGTERMs the sub-cli in flight (its harness.stop ends the shell_exec groups it started), SIGKILLs what is left after 10s and exits the wrapper, so it can't write on into the run / apply dir the new owner shares; ticker requeues a stale `running` row (→ `timeout` once `max_attempts` is spent). Detached process model: server restart doesn't kill in-flight wrappers.
 
 ## Trigger and Enqueue Flow
 
@@ -58,6 +62,7 @@ Server flow (synchronous, ~10ms):
    - Write `source-snapshot.json` (frozen `session.agent.messages`)
    - Write `tool-flow.md` (tool_result-stripped Markdown view for skimming)
    - Write `meta.json` (metadata: id, triggerKind, sourceSession, userHint, workspacePath, createdAt)
+   - Record the source session's persisted access level in `evolution_runs.source_access` (null → `full`, the session-agent-builder default) — the L2/L3 source gate reads it. A session that was ever handed a cron webhook event (a `cron_events` row with its `session_id`, any state) records `webhook` instead, at any access level
    - Write `evo-context.json` (snapshot of prompt surface at trigger time: assembled system prompt, all prompt files, agent/skill listings)
 4. INSERT `evolution_runs` row with `status='pending'`, `trigger_kind='note'` (internal trigger id keeps the old name; only the user-visible command was renamed to `/evo`).
 5. Reply via `chat:system`: "📝 Queued for evaluation".
@@ -68,7 +73,7 @@ Server flow (synchronous, ~10ms):
 
 Compaction only sets a per-session `compactedThisTurn` flag; `SessionManager.enqueueEvoForCompactedTurn()` consumes it once the compact is done — at the end of `runSession` for auto-compact (mid-turn or turn-end), right after the summary lands for manual `/compact`. A turn that compacts several times therefore enqueues one run.
 
-1. Checks: `evolution.level` = L1, `evolution.triggers.pre_compact` on, root session (`parentId` null).
+1. Checks: `evolution.level` ≥ L1, `evolution.triggers.pre_compact` on, root session (`parentId` null), and the session's `reply_to` is not `{cron}` — a cron run's own turn (a stored job prompt plus what the run fetched, nobody in the loop) teaches nothing and would otherwise pass the source gate as a full session. The flag is still consumed. `/evo` on that session is not affected.
 2. `enqueueEvoRun()` with `trigger='pre-compact'` snapshots the session's current `rawMessages`. Because the enqueue is deferred, that is the already-compacted history (summary + kept tail), not the pre-compact one.
 3. INSERT `evolution_runs` row.
 
@@ -78,7 +83,7 @@ Located: `/packages/server/src/evolution/ticker.ts`
 
 Four jobs per pass:
 
-1. **Mark timeouts**: `running` rows (applies: also `syncing`) with heartbeat older than `runTimeoutMinutes` (`applyTimeoutMinutes` for applies) → back to `pending` while `attempts < maxAttempts`; exhausted rows → terminal `timeout` (a `syncing` apply → `failed`).
+1. **Mark timeouts**: `running` rows (applies: also `syncing`) with heartbeat older than `runTimeoutMinutes` (`applyTimeoutMinutes` for applies) → back to `pending` while the attempts used are below `maxAttempts` (a run counts `attempts - attempts_base`, this round's; an apply has no retry, so `attempts`); exhausted rows → terminal `timeout` (a `syncing` apply → `failed`).
 2. **Start runs**: count current `running`, claim `pending` runs (oldest first, up to slot budget), spawn `evo-wrapper.js --mode=run --id=<X>`.
 3. **Start applies**: same shape, but with per-workspace mutex (only one apply per workspace at a time, since apply reads **current** main to build sandbox).
 4. **Broadcast**: diff in-flight run/apply statuses against the previous tick and push `evolution:run_changed` / `evolution:apply_changed` — wrappers are detached processes and can't reach the server's WS.
@@ -208,19 +213,64 @@ For each source_run_id:
 1. Read `testScenario` from source run's `patch.md`.
 2. Run `halo cli -a <agentId> -n -w <applyDir>/sandbox --access workspace` (testMessage on stdin, same 2700s timer) against merged sandbox.
 3. Spawn `__score__` with regress-mode brief, write `regress/<runId>/score.json`.
-4. Any score with `lint < 50` or `behavior < 50` (or a missing / malformed `score.json`) → regression, abort with `failed`.
+4. Any score with `lint < 50` or `behavior < 50` (or a missing / malformed `score.json`) → regression, abort with `failed`. A previous attempt's `score.json` is deleted before the scorer spawns (run-mode phase C too), so a scorer that dies without writing can't be read as the old attempt's score.
 
 ### Phase 12 — Final Sync (checkpointed)
 
 Four steps, with a checkpoint before the dangerous one:
 
-1. **Preflight** (idempotent): walk sandbox, diff against main, snapshot pre-apply files to `history/apply-<id>/` with `MANIFEST.json`.
-2. **Checkpoint**: set `evolution_applies.status='syncing'`. From here on, if wrapper crashes, ticker recovery resumes at step 3 (skip A'/B'/preflight to avoid re-running LLM phases).
+1. **Preflight** (idempotent): walk sandbox, diff against main, snapshot pre-apply files to `history/apply-<id>/` with `MANIFEST.json`. A fresh claim (`running`) first confirms it still owns the row (heartbeat on `running` at its attempt; refused → exits, nothing written), then rewrites `history/apply-<id>/` from scratch: the row never reached `syncing`, so main is still pre-apply, and anything there came from an earlier attempt or a stale wrapper. A resume (`syncing`) keeps the original.
+   - **Auto re-check** (`approved_by='auto'`, not on resume): the files preflight will publish are classified again against the live surface, and the level is read again (the user may have lowered it since). Blocked → apply `failed` with `auto_recheck:<reason>`, main untouched. See [Auto-apply](#auto-apply-l2--l3).
+2. **Checkpoint** (`markSyncing`): set `evolution_applies.status='syncing'` (a resume only refreshes the heartbeat). From here on, if wrapper crashes, ticker recovery resumes at step 3 (skip A'/B'/preflight to avoid re-running LLM phases).
 3. **Publish** (dangerous): cp each changed file from sandbox to main.
-4. Mark source_runs `applied`, finalize apply row `applied`.
+4. `finalizeApplied`, one transaction: apply row `applied`, its still-`approved` source runs `applied`.
+
+A wrapper's terminal writes (run `finalize` / `markObservation`, apply `finalizeApply`) only land while the row is still `running` at the attempt that wrapper was spawned on: a re-claimed row belongs to the newer wrapper. Phase 12's own writes hold the same condition: the checkpoint lands only on `running` (resume: `syncing`) at this attempt, and refused, the wrapper exits without touching main; `finalizeApplied` lands only on `syncing` at this attempt. A resume whose preflight fails, or a publish that throws, releases the `syncing` row at once (clears started / heartbeat, records the reason in `failure_reason`) for the ticker, which re-resumes it and, once retries run out, fails it with that reason and keeps the run `approved`. A new resume claim clears the reason.
 
 Per-workspace mutex in ticker prevents two applies from racing on the same workspace — the assumption phase 12 leans on.
 
+## Auto-apply (L2 / L3)
+
+Code: `evolution/auto-apply.ts` (pure, no agent-loader import — it runs in the detached wrapper), hook `maybeAutoApply` in `evo-wrapper.ts`, shared enqueue `queue-apply.ts`.
+
+**Where.** At L2+, right after phase C the run-mode wrapper decides and writes `runs/<id>/auto-apply.json`. A pass calls `queueApply({ mode: 'auto', attempts })`: one transaction flips the run `running → approved` (only while it is still `running` at this wrapper's attempt) and inserts a `pending` apply with `approved_by='auto'`. A pass never goes through `awaiting_review`, so it can't race a reviewer. A fail (or below L2) finalizes `awaiting_review` as before. The ticker needs no change — its existing broadcast diff pushes `running → approved` and the new apply.
+
+**Three gates**, checked in this order; the first that fails is the recorded `reason`:
+
+1. **Source access** — the run's `source_access` must be `full`. A `workspace` / `readonly` session's conversation (a channel guest's, say) must not rewrite prompts every session runs on without a reviewer. Rows from before the column read null → never auto-applied. Reason `source_access`. A run recorded `webhook` (its session took cron webhook events — payloads from outside, see enqueue above) never auto-applies either, reason `source_webhook`, whichever trigger enqueued it.
+2. **Change category** — the sandbox is diffed against the live pre-patch surface (workspace file, else global for `agents/` / `skills/` / `prompts/` dirs the workspace doesn't override; byte-equal copies of a global folder drop out). Each changed file is `prompt`, `capability` or `manual`; the strictest wins. Prompt needs L2, capability L3, manual never. Reasons `level`, `no_effective_change`, `category_manual:<rel>(<why>)`.
+3. **Score** — `avg ≥ auto_apply_threshold`, `confidence: high`, `behavior ≥ 70`, `lint = 100`, all four `checks` passed (`patchInSandbox: pass`, `usedInDryRun: yes`, `probe: valid`, `probeLanguage: same`). The behavior floor matters: 100/50/100 averages 83. Reasons `score_missing`, `below_threshold`, `confidence`, `behavior_floor`, `lint_floor`, `checks:<name>`.
+
+| Change | Category |
+|---|---|
+| `INSTRUCTIONS.md`, `INDEX.md`, `agents/<id>/AGENT.md`, `prompts/{bootstrap,all,root}/*.md` | prompt |
+| `SKILL.md` body / `name` / `description` only | prompt |
+| `SKILL.md` `disable-model-invocation` / `user-invocable`; `verbs` with no `requiresAccess` change | capability |
+| `agent.yaml` `description` / `system_prompt` / `context`; `tools` removed or a non-egress tool added; `skills` removed; `model.id` / `maxTokens` / `promptCaching` / `thinking` / `verbosity` | capability |
+| `agent.yaml` `priority` (routing), `team` / `internal` / `name` / unknown keys, `model.provider` / `model.endpoint`, `tools` adding `shell_exec` / `web_fetch` / `relay_send` / `a2a_send` / `node_*`, `skills` adding one | manual |
+| `SKILL.md` `command` (routing: verb gates only apply to object commands `SUBCOMMAND_ROUTES` knows, so a rename drops them), `requiresAccess`, any verb's effective `requiresAccess` (own or object-level fallback; a verb added or removed counts), other frontmatter keys | manual |
+| `USER.md`; any other file in a skill / agent folder; a new agent / skill dir; a deleted file; an internal agent (live `agent.yaml` `internal: true`, or a `__` id); unrecognized paths | manual |
+
+Egress tools and `relay_send` / `node_*` are manual because the dry-run runs `--access workspace`, where they aren't even injected — a score is no evidence about them.
+
+**Phase-12 re-check.** An auto apply is gated again before `markSyncing`: the files preflight will publish are classified against the live surface and the level is re-read (the apply agent may touch files the run didn't; the user may have lowered the level). Publish never deletes, so a workspace file the merge dropped is not a change here; a new override dir that would hide a global file the sandbox lacks still is. The score gate isn't repeated — phase B' already held the merge to the regression floor. Blocked → `auto_recheck:<reason>`, main untouched.
+
+**Failure.** Any failure of an auto apply (merge, regress, re-check, timeout) fails the apply and sends its run back to `awaiting_review` in the same transaction (`failApply`), so a reviewer can approve it by hand. Reviewer applies keep their run `approved`.
+
+**Notice.** Each tick, `notifyAutoApplied` (`evolution/notify.ts`) finds `approved_by='auto' AND status='applied' AND notified_at IS NULL` (db, so restarts neither resend nor miss) and sends one `chat:system` line to the source session: the effectively changed files (MANIFEST minus `unchanged`, > 3 summarised), score, and an observation preview. `notified_at` is stamped even when the session is gone.
+
+## Rollback
+
+`POST /api/evolution/applies/:id/rollback { force? }` (`evolution/rollback.ts`), for reviewer and auto applies alike. Synchronous fs with no await, so the in-process ticker can't interleave. Source of truth is `history/apply-<id>/` on disk.
+
+1. Preconditions (409): apply `applied` (`not_applied`), no `pending` / `running` / `syncing` apply in the workspace (`apply_in_flight`), a readable MANIFEST whose paths stay inside `.halo/` with every `overwrite` backup present (`no_snapshot`).
+2. Per-file state against main now: `clean` (hash = `after`), `reverted` (already back: hash = `before`, or a `new` file that is gone — skipped, not a conflict), `modified`, `missing`, `unverified` (old manifest, no hashes). Any of the last three without `force` → 409 `conflict` with `conflicts[{rel, reason}]`.
+3. Then (conflicts cleared or forced): any `overwrite` backup that doesn't hash to `before` → 409 `backup_mismatch` with `files`, `force` or not — restoring it would write something else. Nothing has been written at this point.
+4. Copy what will be overwritten / removed to `rollback-pre/`, restore `overwrite` files, remove `new` ones, then `rmdir` each `createdDirs` entry deepest first. An emptied `agents/<id>/` (`skills/`, `prompts/`) dir must go: it would override the global one wholesale. A dir someone has since put files in is kept and reported in `keptDirs`; the UI lists them after the rollback, since each still overrides the global agent / skill / prompts of that name.
+5. Write `ROLLBACK.json`; in one transaction the apply and its `applied` source runs become `rolled_back`; broadcast.
+
+`200 { ok, restored, removed, removedDirs, keptDirs }`. A second call answers `not_applied`. Limits: an apply row is archived 14 days after it is terminal and purged 30 days later — no rollback entry after that; deleting the run deletes the snapshot too; rollback restores files, not session behaviour already shown.
+
 ## Data on Disk
 
 ```
@@ -234,6 +284,8 @@ Per-workspace mutex in ticker prevents two applies from racing on the same works
     patch.md                  # __evo_agent__ writes (has testScenario frontmatter)
     .skip.md                  # evo wrote this → no patch proposed
     score.json                # __score__ writes (lint/behavior/scope/avg + checks)
+    auto-apply.json           # L2+ verdict (gates, files, score, observation);
+                              # not an input file, so a retry clears it
     dry-run-output.txt        # stdout from phase B dry-run
     dry-run-fail-<n>.log      # failure logs from dry-run attempts
     sub-cli.log               # tee'd stdout+stderr of every halo cli spawn
@@ -249,8 +301,10 @@ Per-workspace mutex in ticker prevents two applies from racing on the same works
       dry-run-fail.log        # only when the regress dry-run failed
       score.json
   history/apply-<id>/         # rollback snapshot (NOT archived)
-    MANIFEST.json             # which paths were overwritten
-    <files>                   # pre-apply content
+    MANIFEST.json             # files[{rel, kind, before, after, unchanged?}] + createdDirs
+    <files>                   # pre-apply content of `overwrite` files
+    rollback-pre/<files>      # main's content just before a rollback overwrote / removed it
+    ROLLBACK.json             # what the rollback did (per file action, dirs removed / kept)
   archive/
     run-<id>.zip              # zipped after 14 days in terminal state
     apply-<id>.zip
@@ -288,13 +342,28 @@ literals.
 
 Skill content NOT inlined (too large at scale); only listings. Both agents and skills include a `scope` tag (`workspace` | `global` | `builtin`).
 
+`MANIFEST.json` format (written by a fresh claim's preflight, when the sandbox is final and main still pre-apply):
+
+```json
+{
+  "applyId": "…", "sourceRunIds": ["…"], "capturedAt": "<ISO>",
+  "files": [
+    { "rel": "INSTRUCTIONS.md", "kind": "overwrite", "before": "<sha256>", "after": "<sha256>" },
+    { "rel": "agents/helper/agent.yaml", "kind": "new", "before": null, "after": "<sha256>", "unchanged": true }
+  ],
+  "createdDirs": ["agents/helper", "agents"]
+}
+```
+
+`rel` uses native separators. `before` / `after` are sha256 of main before and of what was published; `unchanged` marks a copy byte-equal to the global file it now overrides (completing a new workspace override dir — not an effective change, left out of notices and the admin file list); `createdDirs` are the dirs publish created, deepest first. Manifests from before this format have no hashes / `createdDirs`.
+
 ## Archive and Retention
 
 Located: `/packages/server/src/evolution/archive.ts`
 
 Two-stage lifecycle (both orthogonal to `status`):
 
-- **14 days after terminal status** (applied/rejected/skipped/failed/timeout): zip the run/apply dir to `archive/{run|apply}-<id>.zip`, delete original, set `archived_at = now()`. DB row stays.
+- **14 days after terminal status** (applied/rejected/skipped/failed/timeout/rolled_back): zip the run/apply dir to `archive/{run|apply}-<id>.zip`, delete original, set `archived_at = now()`. DB row stays.
 - **30 days after `archived_at`**: delete zip and DB row outright.
 
 Active rows (pending/running/awaiting_review/approved/syncing) never archived. `history/apply-<id>/` (rollback tree) never archived — kept cheap and discoverable.
@@ -318,6 +387,10 @@ After apply publishes to main, **no explicit session-release step needed**. Sess
 | `/packages/server/src/evolution/spawn.ts` | Real spawner: detached Node child running `evo-wrapper.js` from its own module's directory (`dist/evo-wrapper.js` in the npm bundle — build-bundle's second entry). stdin/stdout ignored, stderr appended to the run / apply log; a non-zero exit logs an `[EvoSpawn]` error. `spawnEvoWrapper(path, mode, id)` takes the wrapper path for tests; the ticker's spawner is still override-able via `setEvoSpawner`. |
 | `/packages/server/src/evolution/evo-wrapper.ts` | Wrapper orchestrator. 3-phase run mode (draft/dry-run/score). Apply mode (merge/regress/publish). Heartbeat every 60s. Handles Windows command-line limits via stdin briefs. |
 | `/packages/server/src/evolution/archive.ts` | Archive job: 14d → zip (pure-JS jszip) + delete, 30d → purge. Runs daily + at boot. Idempotent. |
+| `/packages/server/src/evolution/auto-apply.ts` | L2/L3 decision: live-baseline diff (`collectChanges` / `publishChanges`), per-file classification, the source / category / score gates. Pure; loaded by the wrapper. |
+| `/packages/server/src/evolution/queue-apply.ts` | `queueApply` (reviewer approve + auto, one transaction) and `failApply` (auto failure → run back to `awaiting_review`). |
+| `/packages/server/src/evolution/rollback.ts` | `inspectRollback` / `rollbackApply` from the history MANIFEST; `readPublishedFiles`. |
+| `/packages/server/src/evolution/notify.ts` | `notifyAutoApplied`: one `chat:system` line per landed auto apply, driven by `notified_at`. |
 
 ## Sandbox Model
 
@@ -337,7 +410,8 @@ Session rows, db, logs, memory — all intentionally excluded from sandbox. Avoi
 general:
   language: en-US          # BCP-47; also used for evo/score output language
   evolution:
-    level: L0              # L0=disabled, L1=enabled
+    level: L0              # L0..L3, see below
+    auto_apply_threshold: 80   # 0–100, L2/L3 only; out of range → 400 on save
     max_concurrent_run: 1
     max_concurrent_apply: 1
     run_timeout_minutes: 12
@@ -347,18 +421,21 @@ general:
       pre_compact: true
 ```
 
-L0 = manual drafting only (`/evo`); L1 also enables pre-compact triggering. Defaults conservative; tune if machine handles more.
+L0 = manual drafting only (`/evo`); L1 also enables pre-compact triggering; L2 also auto-applies prompt-only changes that pass the gates; L3 also auto-applies capability changes. The level only decides how runs are enqueued and whether the wrapper queues an apply itself — runs already queued always finish. All evolution keys are global-only. Defaults conservative; tune if machine handles more.
 
 ## Admin UI Integration
 
 Top-level "Evolution" tab shows:
 
-- **List**: evolution_runs + latest apply per run, newest first (cursor-paginated by `created_at`, up to 300 loaded). Filterable by status, plus an `archived` view. Joins runs+applies to show consolidated view.
-- **Detail**: patch.md (raw), score.json, the `.skip.md` reason, a snapshot summary (message count, first user / assistant message), wrapper log + sub-cli log.
+- **List**: evolution_runs + latest apply per run, newest first (cursor-paginated by `created_at`, up to 300 loaded). Filterable by status (incl. `rolled_back`), plus an `archived` view and two client-side filters: `auto` (latest apply `approved_by='auto'`) and `observation` (`has_observation`). Rows carry an AUTO tag, a lightbulb when the draft has an observation, and an "apply failed" mark.
+- **Detail**: patch.md (raw), score.json, the `.skip.md` reason, a snapshot summary (message count, first user / assistant message), wrapper log + sub-cli log. At L2+: the auto-applied banner (level, score, the files the apply effectively changed — `publishedFiles`) or the translated reason it wasn't auto-applied. The Larger observation block (see below), and a **Rollback** button while the latest apply is `applied`.
 - **Approve**: Opens an inline panel, optional reviewer_hint. Flips the run to `approved` **and** inserts the pending `evolution_applies` row **inside one `db.transaction`** — it's a single reviewer decision, and half of it (run flipped, apply missing) strands the run in a status the ticker never advances.
 - **Reject**: Sets status='rejected'.
-- **Retry**: Resets row to pending, requires new hint. The route allows any status except `pending` / `running`; the UI offers it on `awaiting_review`, `failed` / `timeout` / `skipped` / `rejected`, and on an `approved` run whose apply failed.
-- **Manual Delete**: Removes artifacts (live dir + archive zip, whichever exists) + DB row immediately. Blocked on `pending` / `running` / `approved` runs; also deletes the applies the run produced.
+- **Retry**: Resets row to pending, requires new hint. `attempts` is not cleared (wrapper claims compare it, so a zeroed counter could hand a stale wrapper its claim back); `attempts_base = attempts` restarts the retry budget, and the API's `attempts` is `attempts - attempts_base`, this round's count. The route refuses `pending` / `running` / `applied` (roll back first) and an `approved` run unless its latest apply failed / timed out (409); the UI offers it on `awaiting_review`, `failed` / `timeout` / `skipped` / `rejected` / `rolled_back`, and on an `approved` run whose apply failed.
+- **Rollback**: see [Rollback](#rollback). On `conflict` the UI lists the files and asks again before retrying with `force`; `backup_mismatch` is shown as a refusal.
+- **Manual Delete**: Removes artifacts (live dir + archive zip, whichever exists) + DB row immediately. Blocked on `pending` / `running` / `approved` runs; also deletes the applies the run produced — and with them the rollback snapshot (the confirm says so).
+
+**Larger observation.** A draft may end with a `## Larger observation` section (patch.md body or `.skip.md`): something bigger than one prompt patch. The wrapper flags the run `has_observation` after phase A and lifts the section (≤ 2000 chars) into `auto-apply.json`; the detail pane shows it with two prefilled GitHub-issue links — the workspace's github.com origin (if any) and `turmind/halo-agent`. The link only opens github.com's form (nothing is sent until the user submits); the body lists run id, ≤ 20 published files, score and Halo version, cut on grapheme boundaries to fit 8000 URL chars, and a hint warns the text may quote workspace files or the conversation.
 
 Realtime updates via WebSocket (`evolution:run_changed`, `evolution:apply_changed` events). REST mutations broadcast directly; wrapper state changes broadcast at each ticker tick (up to 30s latency).
 
@@ -366,9 +443,9 @@ Realtime updates via WebSocket (`evolution:run_changed`, `evolution:apply_change
 
 **Server restart**: Ticker re-evaluates db on first pass. Old `running` rows timeout naturally; `pending` rows continue to be picked up. Applies in `syncing` state are treated as resume candidates — ticker clears heartbeat and spawns wrapper which sees `status='syncing'` on entry and skips LLM phases.
 
-**Wrapper crash**: Heartbeat expires after `runTimeoutMinutes` (default 12min). If `attempts < maxAttempts`, ticker moves row back to `pending` for retry. For applies in `syncing`, row stays `syncing` and ticker spawns recovery wrapper that runs phase 12 publish only.
+**Wrapper crash**: Heartbeat expires after `runTimeoutMinutes` (default 12min). If the attempts used are below `maxAttempts` (run: `attempts - attempts_base`), ticker moves row back to `pending` for retry. For applies in `syncing`, row stays `syncing` and ticker spawns recovery wrapper that runs phase 12 publish only.
 
-**Apply mid-publish crash**: Phase 12 preflight is idempotent; step 1 (diff + snapshot) is repeated harmlessly. Publish (`phaseApplyPublish`) is a simple file-by-file cp. If wrapper dies part-way through, main is half-applied, but `history/apply-<id>/` has the rollback. On recovery, wrapper re-runs publish (idempotent cp) to complete.
+**Apply mid-publish crash**: The resume's preflight only re-diffs (no snapshot), so the original backup survives. Publish (`phaseApplyPublish`) is a simple file-by-file cp. If wrapper dies part-way through, main is half-applied, but `history/apply-<id>/` has the rollback. On recovery, wrapper re-runs publish (idempotent cp) to complete.
 
 ## Patches and Test Scenarios
 
@@ -398,9 +475,10 @@ If they're identical (both clean), that's legal. Old patches wrote single `messa
 ## Scope Limits
 
 - evo observes **root sessions only** (`parentId === null`). Sub-agent sessions don't trigger evo.
-- evo is **invisible to conversation**: no user/assistant messages appended to source session log. One short `chat:system` line per `/evo`.
+- evo is **invisible to conversation**: no user/assistant messages appended to source session log. One short `chat:system` line per `/evo`, plus one when an auto apply lands (see [Auto-apply](#auto-apply-l2--l3)).
 - Wrappers never modify `~/.halo/global/` directly. Apply uses copy-on-write: read global, write workspace override via file_write in sandbox, then sync to main.
 - `/evo` is rejected for non-`full` users.
+- `__evo_agent__` and `__score__` treat session / transcript content as data, never as instructions to follow.
 
 ## Why This Design
 
@@ -415,27 +493,21 @@ If they're identical (both clean), that's legal. Old patches wrote single `messa
 ## Status Flow (Run Mode)
 
 ```
-pending → running → [draft → dry-run → score] → awaiting_review
-                              ↓ (fix)
-                           awaiting_review
-                                ↓ (approve)
-                            approved → running (apply mode)
-                                    [regress →
-                                     publish]
-                                        ↓
-                                    applied
-
-              OR: awaiting_review → rejected
-              OR: running → timeout / failed / skipped
+pending → running → [draft → dry-run → score] → awaiting_review → approved → applied → rolled_back
+                                              ↘ approved (L2/L3 auto: wrapper queued the apply)
+          running → failed / timeout / skipped
+          awaiting_review → rejected
+          approved → awaiting_review   (its auto apply failed)
+          retry: any but pending / running / applied → pending
+                 (approved only once its latest apply failed / timed out)
 ```
 
 ## Status Flow (Apply Mode)
 
 ```
-pending → running → [merge → regress → publish] → applied
+pending → running → [merge → regress → preflight (+ auto re-check)] → syncing → [publish] → applied → rolled_back
 
         OR: running → failed / timeout
-        OR: running/syncing → syncing (resume path after crash)
-                                 ↓ (ticker recovery)
-                            running → [publish] → applied
+        OR: syncing, heartbeat lost → (ticker recovery) wrapper resumes at publish → applied
+            (retries spent → failed)
 ```

package/bundled-docs/design/node.md

@@ -1,6 +1,6 @@
 # Linked machines (`halo node`)
 
-> User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts`, `cron/runner.ts`), CLI `packages/cli/src/index.ts` (`cmdNode`), admin `features/nodes/`.
+> User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts`), CLI `packages/cli/src/index.ts` (`cmdNode`), Go client `packages/node-go/`, admin `features/nodes/`.
 
 ## Problem
 
@@ -12,13 +12,37 @@ The server's agents work on the server's filesystem. Users also want them to tou
 |---|---|---|
 | Accounts | `nodes/accounts.ts` | Pairing tokens in `~/.halo/secrets/channels/channels.db` `channel_accounts` (`channel_type='node'`, `label` = machine name, `config` = `{ token }`, `access_level` = `'full'`). Name `^[a-z0-9][a-z0-9-]{0,31}$`, unique per workspace (realpath compare) → 409 |
 | Hub | `nodes/hub.ts` | Server side. `noServer` `WebSocketServer` on `/api/nodes/connect`; one live connection per account; request/response correlation, timeouts, cancel, heartbeat; `nodes:changed` broadcast. Process singleton via `setNodeHub` / `getNodeHub` |
-| Tools | `nodes/tools.ts` | `buildNodeTools(workspaceRoot)` → the fixed 8 tools. Dispatch: hub (in-process) → loopback relay (cron child) → `NODES_UNAVAILABLE` |
-| Routes | `nodes/routes.ts` | Admin CRUD (`/api/nodes/accounts`) + the cron relay (`POST /api/nodes/call`) |
+| Tools | `nodes/tools.ts` | `buildNodeTools(workspaceRoot)` → the fixed 8 tools. Dispatch: hub (in-process) → `NODES_UNAVAILABLE` |
+| Routes | `nodes/routes.ts` | Admin CRUD (`/api/nodes/accounts`) |
 | Client | `nodes/node-client.ts` | `runNode()` — the `halo node` process: connect, reconnect with backoff, execute requests |
 | Executor | `nodes/node-exec.ts` + `node-path.ts` | Machine side. Wraps the **same** `createWorkspaceTools(base, { accessLevel: 'full' })` the server's agents use; adds the root check |
+| Go client | `packages/node-go/` | `halo-node`: a static Go binary (linux amd64 / arm64 / armv7, no cgo, ~7 MB) speaking the same protocol v1 with the same flags. It ports the client, the executor and the parts of `workspace-tools.ts` / `sandbox.ts` it needs (the seven ops, rm guard, process-group kill), with JS value, UTF-16 and V8 regex semantics (`internal/jsv`, `internal/jsregex` over regexp2). Ships as release assets via `.github/workflows/node-go.yml`, not npm |
+| Conformance | `packages/server/test/node-go-conformance.test.ts` + `packages/node-go/conformance/` | The TS executor is the oracle: a real hub connects TS `runNode` (`test/helpers/ts-node-main.ts`) and the Go binary side by side, and every result, file and log line is compared. Skipped unless `HALO_NODE_GO_BIN` is set. node-go.yml runs it, along with rm-guard vector `--check` and a 20k regex differential fuzz, on any PR touching either side, so a TS change goes red until Go follows. Accepted differences: [below](#go-client-known-differences-from-the-ts-client) |
 
 Not built in AgentCore mode (`nodeHub = AGENTCORE ? null : …`), the CLI / TUI or the evo sandbox — there the tools exist but return `NODES_UNAVAILABLE`.
 
+### Go client: known differences from the TS client
+
+Accepted gaps between `halo-node` and `halo node`. In the conformance suite D1 is normalized away, D4's prefix is rewritten before comparing, and D5 is a `known` case that also asserts TS still differs; D11 is pinned the same way by `known` vectors in `internal/jsregex/testdata/regex-regress.jsonl`; the rest have no case. The labels are the ones the test files use.
+
+| # | Difference | Why accepted |
+|---|---|---|
+| D1 | The text in `could not connect (…)` is Go's `net` error, not Node's (`dial tcp …: connection refused` vs `connect ECONNREFUSED …`). Also log-only: a frame over 8 MiB from the server is `connection lost (1006 …)` in Go, `(1009 …)` in TS; a reset before the welcome has no `(read ECONNRESET)` | log only |
+| D2 | A multi-byte character split across two `node_run` output chunks: TS decodes each chunk alone (two U+FFFD), Go joins them | Go is right; cases cover single-chunk output |
+| D3 | Killed by an outside signal mid-run, TS waits 500 ms more (`shutdownWithin`), Go doesn't | timing only, same output |
+| D4 | CLI errors start `halo-node:` instead of `halo node:` and HELP names the binary; unknown flags / positional args use Go `flag`'s wording. `--server` values WHATWG still parses but Go's `net/url` doesn't take as a host (`https:/h`, `http:h`, a leading / trailing space) are `Invalid URL` in Go; TS connects | local stderr only; these are typos of `https://h` |
+| D5 | `node_write` content as an array with an array-like plain-object chunk (`[{length:2}]`): TS writes `0000` and succeeds, Go errors `…Received an instance of Object` | off-schema input (case `write D5 array-like object chunk`) |
+| D6, D7 | — (removed: `edit` with `old_string: []` and array `include` in `grep` now match TS) | |
+| D8 | `~/.halo/secrets/{config,settings}.yaml` aren't read: shell timeout only from `HALO_SHELL_TIMEOUT`, grep's 50 matches and the 5 MiB output cap fixed | matters only on a machine that also runs a halo server with those changed |
+| D9 | `mkdir -p` where the parent exists but mkdir keeps failing ENOENT (procfs, e.g. `/proc/sys/x/y.txt`): Node retries forever and TS never replies; Go returns the ENOENT | Go is right; whole-machine writes to procfs only; no case (TS hangs) |
+| D10 | Regex engine limits: V8 throws at `test()` (backtrack stack, `Regular expression too large`), regexp2 matches on; nested nullable quantifiers can backtrack exponentially in regexp2 where V8 prunes. Go caps group nesting at 10000 (`Invalid regular expression: /…/: Stack overflow` at compile, so grep's `Invalid regex pattern` warning; V8 compiles it and throws the same text at `test()` past ~11000 capture / lookaround levels; from 32768 capture groups (`(` × 32768) V8 already refuses at compile with `…: Too many captures` — both warn, the suffix differs). One `test()` over 30s aborts the whole grep / glob with `regex match timed out after 30s on <at> — call aborted …`, `<at>` being `<file>:<line>` for grep's line match and `<file>` for an include / glob match; V8 has no limit, so TS runs into the hub's 120s cancel and keeps spinning. Any other regexp2 match-time failure aborts the same way as `regex engine error on <at> — call aborted` | pathological patterns on both sides; the fuzz only requires Go to compile them |
+| D11 | A capture inside a quantified group isn't reset per iteration: `/^(?:(a)\|b)+\1$/` (also `*`, `{2}`) — `"ab"` is true in V8, false in Go; `"aba"` the reverse. regexp2 keeps the last iteration's capture where ECMAScript clears it | backreference to an alternation's capture under a quantifier only; pinned by `known` vectors |
+| D12 | A lone surrogate escape in a request's JSON (`"\ud83d"`) arrives in Go as U+FFFD (`encoding/json`), in TS as the surrogate: `edit`'s `old_string` and grep's `pattern` can then match a U+FFFD the file decoded to, where TS finds nothing. A NUL byte in a path is `EINVAL: invalid argument, stat '…'` in Go, `ERR_INVALID_ARG_VALUE` (`…without null bytes`) in TS | off-schema input; same outcome (no match / error), different text |
+| D13 | `fs.readFile` limits by bytes, not decoded length: a file over 536870888 bytes (V8's string max) is `Invalid string length` in Go even when it is mostly multi-byte text Node could still hold; over 2 GiB both say `File size (<n>) is greater than 2 GiB`. The cap is on bytes read; decoded, invalid UTF-8 grows to at most 3× (each byte a 3-byte U+FFFD). grep also feeds each line to regexp2 as runes, 4× the line's bytes: the longer one line, the more scratch (a 256 MB single line ≈ 1 GB, past what an armv7 process has) | ≥512 MiB of text through a node tool; byte count is a safe bound |
+| D14 | `node_run`'s stdin is a pipe (FIFO), TS's a socketpair: reads and writes behave the same; only a test like `[ -S /dev/stdin ]` tells them apart | nothing writes to it either way |
+| D15 | Built with `CGO_ENABLED=0`, user lookup reads `/etc/passwd` only, not NSS: for an LDAP / SSSD user the name in the `--allow-shell` log line and the home used when `HOME` is unset can differ (TS asks libc) | log text / an unset `HOME` only |
+| D16 | A `res` goes out in 32 KiB WebSocket fragments, TS sends one frame: the server's `ws` reassembles before parsing, the 8 MiB cap is per message, and pings can interleave so a slow uplink doesn't starve the pong | wire framing only; pinned by the fragment-count proto check |
+
 ## Two modes
 
 - **Folder mode** (`--root <dir>`): `root` is realpath'd at start. Every path argument and node_run's `cwd` must resolve inside it, else the call is refused before any I/O (`"<p>" is outside this machine's shared folder (<root>) — …`; machine log `[Node] refused <op> <path>: outside root`). Relative paths resolve against `root`.
@@ -76,16 +100,9 @@ When `halo node` itself exits — Ctrl-C (`stop`) or a fatal refusal such as a r
 
 Offline vs unknown are distinct errors: `machine "<n>" is offline (…). Do not retry in a loop — tell the user to start it.` vs `unknown machine "<n>" — machines linked to this workspace: a, b (see node_list)`.
 
-## Cron relay
-
-A cron run is a `halo cli` child — no hub in that process. `cron/runner.ts` spawns it with `{ ...cleanChildEnv(), ...getNodeHub()?.relayEnv() }`:
-
-- `HALO_NODE_RELAY_URL` = `http://127.0.0.1:<port>/api/nodes/call`
-- `HALO_NODE_RELAY_TOKEN` = a 32-byte hex secret generated per server boot (never persisted)
-
-The child's node tools POST `{ workspace, node, op, args }` there (`op: 'nodes'` = node_list) with header `x-halo-node-relay`. The route is in `PUBLIC_PATHS` and gated by loopback source **and** a `timingSafeEqual` match on the secret — the secret is the real gate (a same-host reverse proxy is loopback too). The secret is **server-wide, not bound to a workspace**: the body's `workspace` picks the machines, so whoever holds it can reach every workspace's machines. `child-env.ts` strips `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN` / `HALO_NODE_TOKEN` from every other child (shell_exec and the rest of `cleanChildEnv`'s users), so they don't appear in agent subprocess env. That is env hygiene, not a boundary: a full-level command runs unsandboxed (no bwrap) as the same user and can read that user's processes and files — the cron child's environment included — anyway.
+## Cron runs
 
-The relay request carries its own timeout (`max(shell timeout, 120 s) + 30 s`, node:http — fetch's fixed 300 s headers timeout would cut a long node_run first); the hub's per-call timeout normally answers well before it.
+Cron runs execute in the server process since 2.2 ([cron.md](cron.md#execution-runjob)), so their node tools reach the hub directly like any other server-side session. The loopback relay the old `halo cli` child used (`POST /api/nodes/call`, a per-boot secret in `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN`) is gone. `child-env.ts` still strips `HALO_NODE_TOKEN` from every child it builds, so a `halo node`'s own `node_run` commands don't see its pairing token — env hygiene, not a boundary: a full-level command runs unsandboxed as the same user and can read that user's processes and files anyway.
 
 ## Prompt-cache invariant
 
@@ -95,4 +112,4 @@ Cost: the 8 descriptions measure ≈ 460 tokens (Claude CountTokens); with schem
 
 ## Admin
 
-`features/nodes/` adds **Channels → Linked machines** (after Web): a row per token with online dot, name, workspace, `root | Whole machine · shell on/off · os`, token copy and delete. The add dialog takes workspace + name and shows a ready-to-copy `HALO_NODE_TOKEN=<t> halo node --server <origin> --root <folder>` (token in the env, not a flag). Online state comes from the list response, then `nodes:changed` (full snapshot, global broadcast — the channels page is a global view); a WS reconnect drops the pushed snapshot and re-fetches the list. No polling.
+`features/nodes/` adds **Channels → Linked machines** (after Web): a row per token with online dot, name, workspace, `root | Whole machine · shell on/off · os`, token copy and delete. The add dialog takes workspace + name and shows two ready-to-copy commands (token in the env, not a flag). **Linux** is `curl -fsSL <base>/install.sh | sh`, then `PATH="$HOME/.local/bin:$PATH" HALO_NODE_TOKEN=<t> halo-node --server <origin> --root <folder>` — a non-root install lands in `~/.local/bin`, which Debian / Ubuntu / Raspberry Pi OS put on `PATH` only at a login after it exists. **macOS / Windows** runs `HALO_NODE_TOKEN=<t> halo node --server <origin> --root <folder>` (npm). `install.sh` (`packages/node-go/install.sh`, POSIX sh) is uploaded per release with that release's version stamped in. It maps `uname -m`, downloads the binary, checks it against `SHA256SUMS` and runs its `--version`, then installs to `/usr/local/bin` (root) or `~/.local/bin` by a rename in that directory, so a running `halo-node` is replaced without "Text file busy". No token, no service. The base comes from `/api/health`'s `version`: a plain `x.y.z` pins `releases/download/v<x.y.z>`, so the machine and the server come from the same tag; a dev build (`x.y.z-<sha>`) or a failed fetch uses `releases/latest/download`. Online state comes from the list response, then `nodes:changed` (full snapshot, global broadcast — the channels page is a global view); a WS reconnect drops the pushed snapshot and re-fetches the list. No polling.

package/bundled-docs/design/observability.md

@@ -2,20 +2,20 @@
 
 ## Overview
 
-A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. It applies to every process that runs the agent loop: the server, `halo cli` and `halo tui` — and so the cron jobs and evolution runs, which are spawned `halo cli` children. Principle: zero AWS (or any vendor) code in the server — SigV4 signing, backend endpoints, and resource enrichment all live in the collector, not in halo.
+A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. It applies to every process that runs the agent loop: the server (cron jobs included — they run in-process since 2.2), `halo cli` and `halo tui` — and so the evolution runs, which are spawned `halo cli` children. Principle: zero AWS (or any vendor) code in the server — SigV4 signing, backend endpoints, and resource enrichment all live in the collector, not in halo.
 
 ## Configuration
 
 | key | type | default | description |
 |---|---|---|---|
-| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Server: restart required; cron / evolution runs: next run. |
-| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Server: restart required; cron / evolution runs: next run. |
-| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Server: restart required; cron / evolution runs: next run. |
-| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Env `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` overrides it (below). Server: restart required; cron / evolution runs: next run. |
+| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Server (cron runs included): restart required; evolution runs: next run. |
+| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Server (cron runs included): restart required; evolution runs: next run. |
+| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Server (cron runs included): restart required; evolution runs: next run. |
+| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Env `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` overrides it (below). Server (cron runs included): restart required; evolution runs: next run. |
 
-All four keys are `globalOnly`. The server's own exporter switches only on restart — `otel.ts` reads them once at boot, before anything else touches the OTel API. A `halo cli` / `halo tui` process reads them the same way at its own start, so a change reaches the next run — including the cron and evolution children the server spawns, which don't inherit the server's boot-time values (see the child-env rule below). A changed or cleared endpoint therefore reaches their next run without a server restart.
+All four keys are `globalOnly`. The server's own exporter switches only on restart — `otel.ts` reads them once at boot, before anything else touches the OTel API. A `halo cli` / `halo tui` process reads them the same way at its own start, so a change reaches the next run — including the evolution children the server spawns, which don't inherit the server's boot-time values (see the child-env rule below). A changed or cleared endpoint therefore reaches their next run without a server restart.
 
-**Which process exported.** Every process shares `service.name`, so the resource also carries `halo.process` = `server` \| `cli` \| `tui` (`HaloProcess` in `otel.ts`; `initObservability()` defaults to `server`). Cron and evolution children are `halo cli` runs, so they report `cli`; their `session.id` (`cron-<jobId>`, or the `__evo_agent__`-style agent name) tells them apart from a hand-run cli. The startup log line names it: `[Observability] OTLP export → … (service.name=…, halo.process=cli, …)`.
+**Which process exported.** Every process shares `service.name`, so the resource also carries `halo.process` = `server` \| `cli` \| `tui` (`HaloProcess` in `otel.ts`; `initObservability()` defaults to `server`). Evolution children are `halo cli` runs, so they report `cli`; their `session.id` (the `__evo_agent__`-style agent name) tells them apart from a hand-run cli. Cron runs execute in the server since 2.2 and report `server`. The startup log line names it: `[Observability] OTLP export → … (service.name=…, halo.process=cli, …)`.
 
 **Env-var mapping.** `initObservability()` maps settings onto the standard `OTEL_*` env vars with `??=`, so an operator who already exports them (or `--require`s an external distro) wins:
 
@@ -24,11 +24,11 @@ All four keys are `globalOnly`. The server's own exporter switches only on resta
 - `service_name` → `OTEL_SERVICE_NAME`
 - always `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` (declarative — the `*-otlp-proto` exporters are hard-wired to it; this just makes an externally-loaded SDK agree on wire protocol)
 
-`capture_content` goes the other way: it isn't mapped onto an env var; instead `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` (the OTel GenAI instrumentation's own switch), when exported as `true` / `false` / `1` / `0` (any case), beats the setting in both directions (`config.ts` `envBool`); unset or any other value → the setting. Read once at process start, like the rest. This is what lets a container (an AgentCore runtime) turn content capture on or off from its env alone, with no settings edit. A server that exports it passes it on to its cron / evolution children (it isn't in `settingsOtelEnvKeys`).
+`capture_content` goes the other way: it isn't mapped onto an env var; instead `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` (the OTel GenAI instrumentation's own switch), when exported as `true` / `false` / `1` / `0` (any case), beats the setting in both directions (`config.ts` `envBool`); unset or any other value → the setting. Read once at process start, like the rest. This is what lets a container (an AgentCore runtime) turn content capture on or off from its env alone, with no settings edit. A server that exports it passes it on to its evolution children (it isn't in `settingsOtelEnvKeys`).
 
 No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObservability()` returns without registering anything.
 
-**Child env.** Every key the mapping actually set (one that was unset before) is recorded in `settingsOtelEnvKeys` (`child-env.ts`), and `cleanChildEnv()` strips those keys from every child it builds. Inherited, the server's boot-time values would win a cron / evolution `halo cli` child's own `??=`, so a changed endpoint would never arrive and a cleared one would keep exporting until a server restart. Operator-exported `OTEL_*` keys aren't recorded and still pass through ("operator wins" holds in children too). Side effect: terminals, sandbox shells and the Jupyter server — also built from `cleanChildEnv()` — no longer inherit Halo's settings-derived collector config.
+**Child env.** Every key the mapping actually set (one that was unset before) is recorded in `settingsOtelEnvKeys` (`child-env.ts`), and `cleanChildEnv()` strips those keys from every child it builds. Inherited, the server's boot-time values would win an evolution `halo cli` child's own `??=`, so a changed endpoint would never arrive and a cleared one would keep exporting until a server restart. Operator-exported `OTEL_*` keys aren't recorded and still pass through ("operator wins" holds in children too). Side effect: terminals, sandbox shells and the Jupyter server — also built from `cleanChildEnv()` — no longer inherit Halo's settings-derived collector config.
 
 **External-SDK detection.** If an external SDK is already registered (e.g. via `node --require` of a vendor distro) — detected by comparing `ProxyTracerProvider.getDelegate()` against a fresh proxy's `NoopTracerProvider` delegate — halo logs `[Observability] external OpenTelemetry SDK already registered — using it, skipping built-in exporters` and skips its own exporters. The `??=` env mapping above still applies first, so the external SDK picks up halo's settings unless it already set its own.
 
@@ -54,10 +54,10 @@ No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObs
 
 **Why the SDK is loaded by dynamic import.** The SDK module graph costs ~120ms, which every unconfigured `halo` start (`--version` included) would otherwise pay. `otel.ts` only `import()`s `otel-sdk.ts` after confirming an endpoint is configured, and `otel-sdk.ts` itself `import()`s the SDK / exporter packages inside `registerSdk` rather than at the top: the npm bundle (esbuild) inlines `otel-sdk.ts`, which would hoist top-level imports of those external packages into the bundle's static imports and load them on every start. So the unconfigured path (server, `halo cli`, `halo tui`) never loads the SDK.
 
-**Flush on exit (cli / tui).** A short-lived process must push its batched spans / metrics / logs before it goes. `shutdownObservability()` force-flushes and shuts down every provider, is idempotent (a signal can land while the end-of-run flush is in flight), and is **capped at 3 s** (`SHUTDOWN_CAP_MS`) so a dead collector delays an exit but never hangs it; it resolves `false` when the cap fired, and is instant when observability is off. The 3 s sits inside the cron runner's 30 s and the evolution wrapper's 10 s SIGTERM→SIGKILL grace. Three exit paths:
+**Flush on exit (cli / tui).** A short-lived process must push its batched spans / metrics / logs before it goes. `shutdownObservability()` force-flushes and shuts down every provider, is idempotent (a signal can land while the end-of-run flush is in flight), and is **capped at 3 s** (`SHUTDOWN_CAP_MS`) so a dead collector delays an exit but never hangs it; it resolves `false` when the cap fired, and is instant when observability is off. The 3 s sits inside the evolution wrapper's 10 s SIGTERM→SIGKILL grace. Three exit paths:
 - **natural end** (one-shot run done, TUI quit, a thrown error): a `beforeExit` hook registered by `initRuntime` flushes; when the cap fired the exporters' retry timers still hold the event loop, so the hook then calls `process.exit()` itself (stdout has drained by then). `beforeExit` never fires on `process.exit()`.
 - **SIGINT / SIGTERM**: the handlers stop the harness first (ending the turn, so its spans are closed and in the batch), then `exitAfterFlush(130 | 143)` (`harness.ts`) flushes and exits. A second SIGINT still hard-exits at once.
-- Any other `process.exit()` skips the flush. On Windows `child.kill('SIGTERM')` delivers no catchable signal, so a timed-out cron run there loses its last batch.
+- Any other `process.exit()` skips the flush.
 
 **Why the meter is lazy.** The `@opentelemetry/api` tracer and logger are proxies — they stay no-op until a real provider is registered, then transparently start forwarding, so `tracer` / `otelLogger` can be module-level consts. The metrics API has **no proxy**: a `Meter` fetched via `metrics.getMeter()` before `setGlobalMeterProvider()` runs is a permanent no-op forever after, even once a provider registers later. `getMeter()` in `otel.ts` is therefore a function, not a const, and `genai-spans.ts` calls it lazily on first metric emission (`instruments()`, cached after).
 
@@ -82,7 +82,7 @@ One per turn — the root span, opened by `beginTurn` and closed by `endTurn`.
 | `gen_ai.task.output` | | final assistant text |
 | `gen_ai.system_instructions` | | system prompt, capped 8 KB — once per turn here, not on every `chat` |
 
-On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`).
+On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`). A turn that ends in a model refusal (`stop_reason: "refusal"`) counts as an error too: ERROR span with `error.type=_OTHER`, and `halo.turn.duration` records `outcome=error`.
 
 ### `chat <modelId>`
 

package/bundled-docs/design/prompt-system.md

@@ -388,7 +388,7 @@ Schema declared by each package + values stored centrally is the same model VSCo
 ## working_dir behaviour
 
 - `start_session` accepts a `working_dir` argument (absolute or workspace-relative)
-- A root session takes one too: `halo cli` / `halo tui --dir <path>` (a cron job's folder rides on it), same checks; on a resumed session it re-points the stored value via `SessionManager.setWorkingDir` (`--dir .` = back to the root), refused while the session is running
+- A root session takes one too: `halo cli` / `halo tui --dir <path>`, or a cron job's folder (the runner passes it to `createSession` / `setWorkingDir` directly), same checks; on a resumed session it re-points the stored value via `SessionManager.setWorkingDir` (`--dir .` = back to the root), refused while the session is running
 - `/session new` (server-side `execNew`) gives the new session the folder of the issuing session's root, re-validated (gone → root)
 - Default: `workspaceRoot`
 - Enforced inside workspace (path traversal is rejected)

package/bundled-docs/design/relay.md

@@ -22,7 +22,7 @@ Relay rides the same "deliver a session's wrap-up to some *other* session when i
 
 **Hard interrupt** adds one step: if `sendUserMessage` returned `queued` (target was busy), call `target.interruptSession(session_id)`. Order is enqueue-then-abort, same as `querySession(interrupt=true)` — the aborted turn's finally sees a non-empty queue, so it drains into the message instead of firing a relay report for the half-done turn.
 
-**Delivery** (`deliverRelayReport(host, session)`, runs at *every* root turn end in every workspace): cheap `reply_to` column read first (no-op for the 99.9 % of sessions without one) → subtree-quiet gate → body = `finalOutput || output`, `[RELAY TARGET ABORTED …]` prefix when `turnError` is set (before the cap, so the marker survives; for an account error — 401/402/403, bad key, no balance — the closing hint says to fix the target's model configuration and not to resume it with `relay_send`, since a re-send fails the same way) → cap at `limits.autoReportMax` with a `relay_read(...)` pointer → header `[Relay report · workspace <ws> · session <id> · status: completed]` (`status: aborted` when `turnError` is set) → `clearReplyTo` → `registry.getOrCreate(reply_to.workspace)` → `appendUserMessage` + `sendUserMessage` on the caller. Append-then-send is the same pair the run-ledger nudge uses: `sendUserMessage` alone never writes the UI transcript, so the secretary's admin view would show a reply to nothing.
+**Delivery** (`deliverRelayReport(host, session)`, runs at *every* root turn end in every workspace): cheap `reply_to` column read first (no-op for the 99.9 % of sessions without one) → subtree-quiet gate → body = `finalOutput || output`, `[RELAY TARGET ABORTED …]` prefix when `turnError` is set (before the cap, so the marker survives; for an account error — 401/402/403, bad key, no balance — the closing hint says to fix the target's model configuration and not to resume it with `relay_send`, since a re-send fails the same way; for a model refusal — `turnErrorKind === 'refusal'`, the target's history tripped the safety classifier — it says not to resume it and to `relay_send` to a NEW session id with a clean brief instead) → cap at `limits.autoReportMax` with a `relay_read(...)` pointer → header `[Relay report · workspace <ws> · session <id> · status: completed]` (`status: aborted` when `turnError` is set) → `clearReplyTo` → `registry.getOrCreate(reply_to.workspace)` → `appendUserMessage` + `sendUserMessage` on the caller. Append-then-send is the same pair the run-ledger nudge uses: `sendUserMessage` alone never writes the UI transcript, so the secretary's admin view would show a reply to nothing.
 
 ## Invariants and why
 
@@ -30,7 +30,7 @@ Relay rides the same "deliver a session's wrap-up to some *other* session when i
 - **Nested trees report once.** The quiet gate is the same one `tryReportToParent` uses, so a target that fans out three sub-agents delivers exactly one relay report, after the last child bubbles up — not one per child turn end.
 - **Full access only.** `session-agent-builder` injects the set only when `nameSet.has('relay_send') && accessLevel === null`. A sandboxed (`workspace` / `readonly`) session can't be given a tool that opens other workspaces' sqlite and session trees; listing the name in such an agent's yaml is silently ignored.
 - **Name-gated bundle.** Only `relay_send` is recognised in `tools:`; the other four (`relay_interrupt` / `relay_stop` / `relay_read` / `relay_list`) ride along. Prevents a half-configured agent that can send but never read back / stop. The admin picker mirrors this with one chip (`GET /agent-configs/tools` builds it from `buildRelayTools` against a dummy target and rewrites the description to name the full set).
-- **Server only.** `setRelayRegistry` is called once in `index.ts`; the CLI / TUI never call it, so `getRelayRegistry()` is `null` there — every tool returns `{"code": 1, "error": RELAY_UNAVAILABLE}` (`relay.ts`: "relay is not available in the CLI / TUI — it only runs inside halo server", permanent for this runtime, do not retry; send from the admin UI / an IM or Web channel, or open the target with `halo tui -w <workspace path>`) and `deliverRelayReport` logs a warning and returns. No CLI code path knows relay exists. Note the builder does **not** check the registry before injecting, so a CLI / TUI session (and every cron run, which is a `halo cli` child) still *sees* the relay tools when its agent lists `relay_send` — the error is permanent there, not a transient outage; route cross-workspace dispatch through a server-side channel (admin / Web / IM / `halo acp`) instead.
+- **Server only.** `setRelayRegistry` is called once in `index.ts`; the CLI / TUI never call it, so `getRelayRegistry()` is `null` there — every tool returns `{"code": 1, "error": RELAY_UNAVAILABLE}` (`relay.ts`: "relay is not available in the CLI / TUI — it only runs inside halo server", permanent for this runtime, do not retry; send from the admin UI / an IM or Web channel, or open the target with `halo tui -w <workspace path>`) and `deliverRelayReport` logs a warning and returns. No CLI code path knows relay exists. Note the builder does **not** check the registry before injecting, so a CLI / TUI session still *sees* the relay tools when its agent lists `relay_send` — the error is permanent there, not a transient outage; route cross-workspace dispatch through a server-side channel (admin / Web / IM / `halo acp`) instead.
 - **Target-side transcript is normal.** The department workspace's session looks like any other root session in its Sessions tab (user message, agent reply); the only trace of relay is the `reply_to` row while a dispatch is pending. Nothing in the target's system prompt changes.
 - **Same workspace is allowed — except self and ancestors.** `workspace` is only checked for existence + `.halo/`; passing the caller's own path makes `getOrCreate` return the caller's own `SessionManager`, and everything else is unchanged, with one exception: there `relay_send` / `relay_interrupt` / `relay_stop` refuse a `session_id` that is the caller itself or one of its ancestors (`isSelfOrAncestor` from `session-tools.ts`; the tools compare the resolved manager with their own host, so the same id in another workspace is unaffected). A send there stamped `reply_to` with a session inside the target's own tree — a self-report, or an ancestor whose report goes back to its own descendant — and a stop awaited the very turn making the call. In effect relay is a `query_session` without the own-tree scoping — it reaches any **root** session in any workspace, including new ones. Not a hole: the tool is full-access only, and a full-access agent could already `shell_exec` its way into any session file.
 - **Reports come back from roots only.** `deliverRelayReport` fires for `parentId === null`; a sub-session's turn end goes to `tryReportToParent` instead. Dispatching to a `parent>child` id delivers the message but never a report — target roots when you want to hear back. Dispatching to yourself is refused (see above).
@@ -61,7 +61,7 @@ No quiet gate (the session is by definition not done) and **`reply_to` is kept**
 
 ## What it deliberately doesn't do
 
-- **No cross-server reach.** Same `SessionManagerRegistry`, same process. Different server → [A2A](a2a.md), which reuses the same `reply_to` column (`{ a2a: taskId }`), quiet gate and interim doors; so does AgentCore's `POST /invocations` (`{ invocation: id }`, [agentcore.md](agentcore.md#post-invocations)). A2A and relay both write theirs via `takeReplyTo`, which supersedes a pending `{ invocation }` job first.
+- **No cross-server reach.** Same `SessionManagerRegistry`, same process. Different server → [A2A](a2a.md), which reuses the same `reply_to` column (`{ a2a: taskId }`), quiet gate and interim doors; so does AgentCore's `POST /invocations` (`{ invocation: id }`, [agentcore.md](agentcore.md#post-invocations)) and an in-process cron run (`{ cron: runId }`, final only, [cron.md](cron.md#execution-runjob)). A2A and relay both write theirs via `takeReplyTo`, which supersedes a pending `{ invocation }` job or `{ cron }` run first.
 - **No session discovery beyond a flat root list.** `relay_list` returns a workspace's root sessions (id / agent / title / status, newest 100) so the secretary can reuse an existing conversation; it doesn't walk trees or search transcripts — that's the admin's job.
 - **No broadcast / fan-out helper.** The secretary calls `relay_send` once per department; the reports come back individually with the workspace + session id in the header, so it can tell them apart without extra bookkeeping.
 - **No ACL beyond full access.** A full-access secretary can reach every workspace on the box — that's the deployment's trust boundary already (it can `shell_exec` into them anyway).
@@ -69,4 +69,4 @@ No quiet gate (the session is by definition not done) and **`reply_to` is kept**
 
 ## Test
 
-`packages/server/test/relay.test.ts` — stub `RelayTarget` + registry, 19 cases: `deliverRelayReport` no-op without `reply_to`, waits while a child is active, delivers exactly one append + one send with the header then clears, `ABORTED` prefix on `turnError` (and the account-error variant: "fix the model config" instead of the re-send hint), ignores sub-sessions; `relay_send` creates + stamps `reply_to` + channel-prefixes the model text, rejects a nonexistent workspace, returns the permanent error without a registry (CLI / TUI); `relay_interrupt` busy → enqueue then abort + re-stamp, idle → no abort and never creates; `relay_list` roots only with title fallback, and defaults to the caller's own workspace; in the caller's own workspace `relay_send` / `relay_interrupt` / `relay_stop` refuse self and ancestors (no send / abort / stop, `reply_to` untouched) while another session there — or the same id in another workspace — still goes through. `deliverRelayInterim`: no-op without `reply_to`; delivers the interim header and keeps `reply_to`, the later final report still fires once. End-to-end through `drainQueue` / `runSession` (relay interim → final, no kick → no interim, local message → no interim, opening-turn and deferred interims incl. the local-opening-message and quiet-subtree negatives) lives in `test/continue-task.test.ts`.
+`packages/server/test/relay.test.ts` — stub `RelayTarget` + registry, 22 cases: `deliverRelayReport` no-op without `reply_to`, waits while a child is active, delivers exactly one append + one send with the header then clears, `ABORTED` prefix on `turnError` (and the account-error variant: "fix the model config" instead of the re-send hint; the refusal variant: `status: aborted` header, body opening with the marker, "new session id" hint), ignores sub-sessions; `relay_send` creates + stamps `reply_to` + channel-prefixes the model text, rejects a nonexistent workspace, returns the permanent error without a registry (CLI / TUI); `relay_interrupt` busy → enqueue then abort + re-stamp, idle → no abort and never creates; `relay_list` roots only with title fallback, and defaults to the caller's own workspace; in the caller's own workspace `relay_send` / `relay_interrupt` / `relay_stop` refuse self and ancestors (no send / abort / stop, `reply_to` untouched) while another session there — or the same id in another workspace — still goes through. `deliverRelayInterim`: no-op without `reply_to`; delivers the interim header and keeps `reply_to`, the later final report still fires once. End-to-end through `drainQueue` / `runSession` (relay interim → final, no kick → no interim, local message → no interim, opening-turn and deferred interims incl. the local-opening-message and quiet-subtree negatives) lives in `test/continue-task.test.ts`.