@turmind/halo 2.2.2 → 2.2.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/README.md +2 -1
  2. package/admin-out/404.html +1 -1
  3. package/admin-out/_next/static/chunks/1989-9d30cdb1a86e4beb.js +1 -0
  4. package/admin-out/_next/static/chunks/4362.115405260299210d.js +1 -0
  5. package/admin-out/_next/static/chunks/{7613.fe24922a6a839656.js → 7073.4081ed209e6748bd.js} +1 -1
  6. package/admin-out/_next/static/chunks/{7073.9d5f9dd7a77827b7.js → 7613.9c1657bb3b52d1b5.js} +1 -1
  7. package/admin-out/_next/static/chunks/app/{page-10a4afac9cbc27a2.js → page-c9d174a5432b0007.js} +1 -1
  8. package/admin-out/_next/static/chunks/{webpack-9532ec7f1fa9a757.js → webpack-1970c5140e4019b7.js} +1 -1
  9. package/admin-out/_next/static/css/87d4fc049f865389.css +1 -0
  10. package/admin-out/index.html +1 -1
  11. package/admin-out/index.txt +5 -5
  12. package/bundled-docs/design/a2a.md +3 -1
  13. package/bundled-docs/design/architecture.md +1 -1
  14. package/bundled-docs/design/cron.md +1 -1
  15. package/bundled-docs/design/evolution.md +6 -6
  16. package/bundled-docs/design/node.md +18 -39
  17. package/bundled-docs/design/observability.md +2 -2
  18. package/bundled-docs/design/relay.md +1 -1
  19. package/bundled-docs/design/session.md +3 -3
  20. package/bundled-docs/design/storage.md +5 -2
  21. package/bundled-docs/design/stt-local.md +2 -2
  22. package/bundled-docs/design/ws.md +3 -3
  23. package/bundled-docs/dev/api.md +10 -9
  24. package/bundled-docs/dev/deploy.md +1 -1
  25. package/bundled-docs/dev/env.md +2 -1
  26. package/bundled-docs/dev/tools.md +10 -6
  27. package/bundled-docs/guide/README.md +1 -1
  28. package/bundled-docs/guide/cli.md +4 -31
  29. package/bundled-docs/guide/node.md +57 -36
  30. package/bundled-docs/requirements/command.md +1 -1
  31. package/bundled-docs/requirements/settings.md +39 -2
  32. package/dist/evo-wrapper.js +368 -268
  33. package/dist/index.js +2330 -2531
  34. package/package.json +2 -1
  35. package/templates/agents/default/agent.yaml +1 -1
  36. package/templates/models/aws-bedrock-openai.yaml +81 -7
  37. package/templates/skills/cron/SKILL.md +1 -3
  38. package/admin-out/_next/static/chunks/1989-3b2faa2731d973c4.js +0 -1
  39. package/admin-out/_next/static/chunks/4362.9d2ee4f9b6facc49.js +0 -1
  40. package/admin-out/_next/static/css/8fc43fce83e5ce5d.css +0 -1
  41. /package/admin-out/_next/static/{u9pg60rITCj73MBYeXsW1 → vf5W8US5y8B2f1E6zJJw3}/_buildManifest.js +0 -0
  42. /package/admin-out/_next/static/{u9pg60rITCj73MBYeXsW1 → vf5W8US5y8B2f1E6zJJw3}/_ssgManifest.js +0 -0
@@ -1,10 +1,10 @@
1
- # Linked machines (`halo node`)
1
+ # Linked machines (`halo-node`)
2
2
 
3
- > User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts`), CLI `packages/cli/src/index.ts` (`cmdNode`), Go client `packages/node-go/`, admin `features/nodes/`.
3
+ > User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts`), machine client `packages/node-go/`, admin `features/nodes/`.
4
4
 
5
5
  ## Problem
6
6
 
7
- The server's agents work on the server's filesystem. Users also want them to touch files on *their own* machine — a laptop or a box behind NAT the server can't reach. `halo node` is a small outbound client: it dials the server over WebSocket, and the workspace's full-access agents get `node_*` tools that run the local file tools (and, opt-in, `shell_exec`) **on that machine**. No inbound port, no model keys, no `halo setup` on the machine.
7
+ The server's agents work on the server's filesystem. Users also want them to touch files on *their own* machine — a laptop or a box behind NAT the server can't reach. `halo-node` is a small outbound client: it dials the server over WebSocket, and the workspace's full-access agents get `node_*` tools that run the local file tools (and, opt-in, `shell_exec`) **on that machine**. No inbound port, no model keys, no `halo setup` on the machine.
8
8
 
9
9
  ## Pieces
10
10
 
@@ -14,51 +14,28 @@ The server's agents work on the server's filesystem. Users also want them to tou
14
14
  | Hub | `nodes/hub.ts` | Server side. `noServer` `WebSocketServer` on `/api/nodes/connect`; one live connection per account; request/response correlation, timeouts, cancel, heartbeat; `nodes:changed` broadcast. Process singleton via `setNodeHub` / `getNodeHub` |
15
15
  | Tools | `nodes/tools.ts` | `buildNodeTools(workspaceRoot)` → the fixed 8 tools. Dispatch: hub (in-process) → `NODES_UNAVAILABLE` |
16
16
  | Routes | `nodes/routes.ts` | Admin CRUD (`/api/nodes/accounts`) |
17
- | Client | `nodes/node-client.ts` | `runNode()` — the `halo node` process: connect, reconnect with backoff, execute requests |
18
- | Executor | `nodes/node-exec.ts` + `node-path.ts` | Machine side. Wraps the **same** `createWorkspaceTools(base, { accessLevel: 'full' })` the server's agents use; adds the root check |
19
- | Go client | `packages/node-go/` | `halo-node`: a standalone Go binary (Linux amd64 / arm64 / armv7 and Windows amd64, no cgo) speaking the same protocol v1 with the same flags. It ports the client, the executor and the parts of `workspace-tools.ts` / `sandbox.ts` it needs (the seven ops, POSIX rm guard, process-group / Windows Job Object kill), with JS value, UTF-16 and V8 regex semantics (`internal/jsv`, `internal/jsregex` over regexp2). Ships as release assets via `.github/workflows/node-go.yml`, not npm |
20
- | Conformance | `packages/server/test/node-go-conformance.test.ts` + `packages/node-go/conformance/` | The TS executor is the oracle: a real hub connects TS `runNode` (`test/helpers/ts-node-main.ts`) and the Go binary side by side, and every result, file and log line is compared. Skipped unless `HALO_NODE_GO_BIN` is set. node-go.yml runs it, along with rm-guard vector `--check` and a 20k regex differential fuzz, on any PR touching either side, so a TS change goes red until Go follows. Accepted differences: [below](#go-client-known-differences-from-the-ts-client) |
17
+ | Client | `packages/node-go/` | `halo-node`: the machine side, a standalone Go binary (no cgo, cross-compiled for Linux amd64 / arm64 / armv7, macOS arm64 / amd64 and Windows amd64). Connects, reconnects with backoff and runs each request through its own port of the local file tools (the seven ops, POSIX rm guard, process-group / Windows Job Object kill) with JS value, UTF-16 and V8 regex semantics (`internal/jsv`, `internal/jsregex` over regexp2), so results read like the server's own tools. Ships as release assets via `.github/workflows/node-go.yml`, not npm |
21
18
 
22
19
  Not built in AgentCore mode (`nodeHub = AGENTCORE ? null : …`), the CLI / TUI or the evo sandbox — there the tools exist but return `NODES_UNAVAILABLE`.
23
20
 
24
- ### Go client: known differences from the TS client
21
+ ### Platforms
25
22
 
26
- Accepted gaps between `halo-node` and `halo node`. In the conformance suite D1 is normalized away, D4's prefix is rewritten before comparing, and D5 is a `known` case that also asserts TS still differs; D11 is pinned the same way by `known` vectors in `internal/jsregex/testdata/regex-regress.jsonl`; the rest have no case. The labels are the ones the test files use.
23
+ CI (`node-go.yml`) runs `go vet` / `go test` natively on Linux and Windows and `go test` for Linux arm64 / armv7 under qemu (binfmt) — including `cmd/halo-node/lifecycle*_test.go`, which run the binary against a fake hub (fatal 401 / 404 / 426 / 4001, retried 409 / 429 / 4009 / other HTTP, backoff, a running call killed on 4001 / drop / SIGTERM / cancel, a TERM-ignoring command escalated to SIGKILL), and `internal/protocol`'s drift test, which reads `server/src/nodes/protocol.ts` and fails when a mirrored constant (path, headers, close codes, frame / result caps, version, ops) differs — plus the rm-guard vector `--check` (`testgen/gen-rmguard-vectors.mjs`: the vectors must still equal what the server's `sandbox.ts` `assertRmSafe` says) and a 20k-pattern regex differential fuzz against Node's `RegExp` (`testgen/regex-fuzz.mjs`). Accepted regex gaps against V8 are pinned by `known` vectors in `internal/jsregex/testdata/regex-regress.jsonl` (D11: a capture inside a quantified group isn't reset per iteration; D10: regexp2 engine limits and its 30 s per-match timeout, which aborts the grep / glob call). macOS builds are only cross-compiled — no macOS runner, not tested on a Mac. They use the POSIX code paths (process groups, rm guard); `internal/jsv`'s errno table is Linux-only, so on macOS a filesystem error reads `Unknown system error -<n>: …` instead of Node's `ENOENT: no such file or directory, …` text.
27
24
 
28
- | # | Difference | Why accepted |
29
- |---|---|---|
30
- | D1 | The text in `could not connect (…)` is Go's `net` error, not Node's (`dial tcp …: connection refused` vs `connect ECONNREFUSED …`). Also log-only: a frame over 8 MiB from the server is `connection lost (1006 …)` in Go, `(1009 …)` in TS; a reset before the welcome has no `(read ECONNRESET)` | log only |
31
- | D2 | A multi-byte character split across two `node_run` output chunks: TS decodes each chunk alone (two U+FFFD), Go joins them | Go is right; cases cover single-chunk output |
32
- | D3 | Killed by an outside signal mid-run, TS waits 500 ms more (`shutdownWithin`), Go doesn't | timing only, same output |
33
- | D4 | CLI errors start `halo-node:` instead of `halo node:` and HELP names the binary; unknown flags / positional args use Go `flag`'s wording. `--server` values WHATWG still parses but Go's `net/url` doesn't take as a host (`https:/h`, `http:h`, a leading / trailing space) are `Invalid URL` in Go; TS connects | local stderr only; these are typos of `https://h` |
34
- | D5 | `node_write` content as an array with an array-like plain-object chunk (`[{length:2}]`): TS writes `0000` and succeeds, Go errors `…Received an instance of Object` | off-schema input (case `write D5 array-like object chunk`) |
35
- | D6, D7 | — (removed: `edit` with `old_string: []` and array `include` in `grep` now match TS) | |
36
- | D8 | `~/.halo/secrets/{config,settings}.yaml` aren't read: shell timeout only from `HALO_SHELL_TIMEOUT`, grep's 50 matches and the 5 MiB output cap fixed | matters only on a machine that also runs a halo server with those changed |
37
- | D9 | `mkdir -p` where the parent exists but mkdir keeps failing ENOENT (procfs, e.g. `/proc/sys/x/y.txt`): Node retries forever and TS never replies; Go returns the ENOENT | Go is right; whole-machine writes to procfs only; no case (TS hangs) |
38
- | D10 | Regex engine limits: V8 throws at `test()` (backtrack stack, `Regular expression too large`), regexp2 matches on; nested nullable quantifiers can backtrack exponentially in regexp2 where V8 prunes. Go caps group nesting at 10000 (`Invalid regular expression: /…/: Stack overflow` at compile, so grep's `Invalid regex pattern` warning; V8 compiles it and throws the same text at `test()` past ~11000 capture / lookaround levels; from 32768 capture groups (`(` × 32768) V8 already refuses at compile with `…: Too many captures` — both warn, the suffix differs). One `test()` over 30s aborts the whole grep / glob with `regex match timed out after 30s on <at> — call aborted …`, `<at>` being `<file>:<line>` for grep's line match and `<file>` for an include / glob match; V8 has no limit, so TS runs into the hub's 120s cancel and keeps spinning. Any other regexp2 match-time failure aborts the same way as `regex engine error on <at> — call aborted` | pathological patterns on both sides; the fuzz only requires Go to compile them |
39
- | D11 | A capture inside a quantified group isn't reset per iteration: `/^(?:(a)\|b)+\1$/` (also `*`, `{2}`) — `"ab"` is true in V8, false in Go; `"aba"` the reverse. regexp2 keeps the last iteration's capture where ECMAScript clears it | backreference to an alternation's capture under a quantifier only; pinned by `known` vectors |
40
- | D12 | A lone surrogate escape in a request's JSON (`"\ud83d"`) arrives in Go as U+FFFD (`encoding/json`), in TS as the surrogate: `edit`'s `old_string` and grep's `pattern` can then match a U+FFFD the file decoded to, where TS finds nothing. A NUL byte in a path is `EINVAL: invalid argument, stat '…'` in Go, `ERR_INVALID_ARG_VALUE` (`…without null bytes`) in TS | off-schema input; same outcome (no match / error), different text |
41
- | D13 | `fs.readFile` limits by bytes, not decoded length: a file over 536870888 bytes (V8's string max) is `Invalid string length` in Go even when it is mostly multi-byte text Node could still hold; over 2 GiB both say `File size (<n>) is greater than 2 GiB`. The cap is on bytes read; decoded, invalid UTF-8 grows to at most 3× (each byte a 3-byte U+FFFD). grep also feeds each line to regexp2 as runes, 4× the line's bytes: the longer one line, the more scratch (a 256 MB single line ≈ 1 GB, past what an armv7 process has) | ≥512 MiB of text through a node tool; byte count is a safe bound |
42
- | D14 | `node_run`'s stdin is a pipe (FIFO), TS's a socketpair: reads and writes behave the same; only a test like `[ -S /dev/stdin ]` tells them apart | nothing writes to it either way |
43
- | D15 | Built with `CGO_ENABLED=0`, user lookup reads `/etc/passwd` only, not NSS: for an LDAP / SSSD user the name in the `--allow-shell` log line and the home used when `HOME` is unset can differ (TS asks libc) | log text / an unset `HOME` only |
44
- | D16 | A `res` goes out in 32 KiB WebSocket fragments, TS sends one frame: the server's `ws` reassembles before parsing, the 8 MiB cap is per message, and pings can interleave so a slow uplink doesn't starve the pong | wire framing only; pinned by the fragment-count proto check |
45
- | D17 | Windows cancellation / timeout stops the whole Job Object. Go also observes shell exit independently of output EOF and immediately closes the job, terminating background descendants before draining stdout / stderr. TS's win32 `execAsync` kills only cmd.exe, so workers can survive or keep output pipes open. Go preserves `Command timed out after <ms>ms` and `Command failed: <command>` rather than Node's wrapped `chcp …` error text, and decodes failure output as UTF-8 / GBK too (TS only decodes successful output) | Intentional Windows lifecycle / output correctness; native cancel, timeout, shell-exit and decoder tests. The real TS fix belongs in sandbox.ts's win32 exec branch |
46
- | D18 | Windows node paths using `\\?\` / `\\.\` namespace or device prefixes (including slash variants), or any component ending in a dot or space other than `.` / `..`, are refused rather than emulated. The same restrictions apply to resolved paths, including shared roots and junction targets, in folder and whole-machine modes | Ordinary Win32 opens can normalize these spellings to a different object after realpath validation. Go rejects them with a node refusal (the existing outside-root message in folder mode); TS does not. Native read/write regressions cover distinct `inside.` / `inside ` directories beside an outward `inside` junction |
47
-
48
- Windows implementation: the shell starts suspended, is assigned to a `KILL_ON_JOB_CLOSE` job, then resumes. Cancel and timeout call `TerminateJobObject` (no graceful SIGTERM equivalent for hidden cmd.exe), retaining the shared 2 s escalation / 200 ms force-settle ceiling. The byte-capped stdout / stderr streams decode as strict UTF-8, falling back to GBK. Folder checks use case-insensitive Windows `filepath.Rel`, reject other volumes, and resolve the longest existing prefix with `GetFinalPathNameByHandle` (including junctions). A bootstrap cmd.exe gets a new hidden console, runs `chcp 65001`, then starts the command's cmd.exe so it initializes in UTF-8 even without an interactive desktop. The Unicode command passes through `HALO_NODE_COMMAND` in the child environment (delayed expansion in the bootstrap only; the inner `/d /s /c` shell inherits the machine/user `DelayedExpansion` default, like TS), not the bootstrap's legacy-codepage parser. Win32 filesystem errors map through libuv's Node-style errno names. `~/` uses `USERPROFILE`; `~\` stays literal, matching TS.
49
-
50
- CI keeps the Linux TS-oracle conformance / rm-guard vectors unchanged and adds native Windows vet, tests and exe build. The POSIX conformance fixtures (sh commands, chmod, Unix paths and symlink assumptions) are not run on Windows; native Windows tests cover those platform-specific boundaries instead. The POSIX rm-guard tests are build-excluded there because neither client calls the guard on Windows.
25
+ Windows implementation: the shell starts suspended, is assigned to a `KILL_ON_JOB_CLOSE` job, then resumes. Cancel and timeout call `TerminateJobObject` (no graceful SIGTERM equivalent for hidden cmd.exe), retaining the shared 2 s escalation / 200 ms force-settle ceiling. The byte-capped stdout / stderr streams decode as strict UTF-8, falling back to GBK. Folder checks use case-insensitive Windows `filepath.Rel`, reject other volumes, and resolve the longest existing prefix with `GetFinalPathNameByHandle` (including junctions). A bootstrap cmd.exe gets a new hidden console, runs `chcp 65001`, then starts the command's cmd.exe so it initializes in UTF-8 even without an interactive desktop. The Unicode command passes through `HALO_NODE_COMMAND` in the child environment (delayed expansion in the bootstrap only; the inner `/d /s /c` shell inherits the machine/user `DelayedExpansion` default), not the bootstrap's legacy-codepage parser. Win32 filesystem errors map through libuv's Node-style errno names. `~/` uses `USERPROFILE`; `~\` stays literal.
26
+
27
+ The POSIX rm-guard tests are build-excluded on Windows: the guard isn't called there.
51
28
 
52
29
  ## Two modes
53
30
 
54
31
  - **Folder mode** (`--root <dir>`): `root` is realpath'd at start. Every path argument and node_run's `cwd` must resolve inside it, else the call is refused before any I/O (`"<p>" is outside this machine's shared folder (<root>) — …`; machine log `[Node] refused <op> <path>: outside root`). Relative paths resolve against `root`.
55
- - **Whole-machine mode** (no `--root`, or `--root /`): no prefix check — the same reach as a full-access local session, i.e. whatever the machine user can access. Relative paths and node_run's default cwd resolve against `base` = the directory `halo node` was started in. A startup line says so. `node_list` reports the machine as `root: null, wholeMachine: true, base: "<dir>"`; the admin row shows "Whole machine".
32
+ - **Whole-machine mode** (no `--root`, or `--root /`): no prefix check — the same reach as a full-access local session, i.e. whatever the machine user can access. Relative paths and node_run's default cwd resolve against `base` = the directory `halo-node` was started in. A startup line says so. `node_list` reports the machine as `root: null, wholeMachine: true, base: "<dir>"`; the admin row shows "Whole machine".
56
33
 
57
34
  `--allow-shell` is independent of the mode and off by default. Commands run **unsandboxed** as the machine user — in folder mode only their `cwd` is held inside `root`, the command itself can reach anything (v1 accepted trade-off; the startup warning and the guide say so). `{{ns.params.x}}` placeholders are never substituted on the machine (empty namespace set), so no settings secret reaches a command.
58
35
 
59
- ### Path check (`resolveInRoot`)
36
+ ### Path check (`nodepath.InRoot`)
60
37
 
61
- Resolve against root, then walk up with **`lstat`** to the longest existing prefix, `realpath` it, re-append the missing tail, and test containment with `path.relative` (so `/x/root-other` is not inside `/x/root`; drive letters compare right on Windows). `lstat`, not `existsSync`: `existsSync` follows a dangling symlink and reports it missing, so `root/link → /outside/new` would stay unresolved, pass the prefix test, and the write would follow the link out of root. With `lstat` the link is found, its `realpath` fails, and the call is refused. A narrow TOCTOU window (a component swapped for a symlink between check and syscall) remains — same as the local tools.
38
+ Resolve against root, then walk up with **`lstat`** to the longest existing prefix, `realpath` it, re-append the missing tail, and test containment with `filepath.Rel` (so `/x/root-other` is not inside `/x/root`; drive letters compare right on Windows). `lstat`, not a following `stat`: that follows a dangling symlink and reports it missing, so `root/link → /outside/new` would stay unresolved, pass the prefix test, and the write would follow the link out of root. With `lstat` the link is found, its `realpath` fails, and the call is refused. A narrow TOCTOU window (a component swapped for a symlink between check and syscall) remains — same as the local tools.
62
39
 
63
40
  When the checked path equals the plain resolution, the agent's own string is passed through, so results read exactly like the local tool's (`File written: notes/a.md`).
64
41
 
@@ -74,6 +51,8 @@ When the checked path equals the plain resolution, the agent's own string is pas
74
51
  | `x-halo-node-info` | base64url JSON `{ root \| null, base, shell, shellTimeoutMs, os, hostname, version }` |
75
52
  | any `--header` | passed through (auth proxies) |
76
53
 
54
+ Forward proxy: `--proxy <http(s)://[user:pass@]host:port>` is used alone; without it `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` apply (Go's `http.ProxyFromEnvironment`: loopback servers are never proxied). A `wss://` server goes through `CONNECT`, so the token stays inside TLS; `user:pass@` becomes `Proxy-Authorization: Basic` and is never printed (the startup line shows `Redacted()`). Redirects are never followed: a 3xx is `server refused the connection (HTTP 3xx)` and the token never goes to the `Location`.
55
+
77
56
  `verifyClient` answers in this order: **429** IP locked out (`middleware/brute-force.ts`, own `node-token` bucket: 5 bad tokens in 15 min) → **426** protocol mismatch → **401** unknown token (recorded as a failure) → **404** bound workspace gone → **400** info / instance missing or malformed → **409** account already connected from a *different* instance. The path is in no auth middleware; the token is the auth.
78
57
 
79
58
  **Frames** (JSON text, ≤ `MAX_FRAME_BYTES` = 8 MiB both ways):
@@ -87,7 +66,7 @@ node → server { t: 'res', id, ok: true, text } | { t: 'res', id, ok: false, e
87
66
 
88
67
  `args` are the local tool's input minus `node` (node_run: `{ command, cwd }`). Results longer than `NODE_RESULT_MAX_CHARS` (256K) are cut on the machine with `[Content truncated on the machine: N chars total]`; the agent loop's own caps (8000 to the LLM, 64K to the UI) then apply as for any local tool. A request over 8 MiB is refused before sending.
89
68
 
90
- **Close codes**: `4001` token revoked (DELETE in admin) → node exits 1. `4009` replaced: a reconnect from the **same** instance supersedes the old socket (half-open link recovery); a **different** instance is refused (`409` at handshake, or `4009 already connected from another process` if it raced) and keeps retrying with backoff — so starting a second `halo node` with the same token never kicks the first. `1001` server shutting down → reconnect.
69
+ **Close codes**: `4001` token revoked (DELETE in admin) → node exits 1. `4009` replaced: a reconnect from the **same** instance supersedes the old socket (half-open link recovery); a **different** instance is refused (`409` at handshake, or `4009 already connected from another process` if it raced) and keeps retrying with backoff — so starting a second `halo-node` with the same token never kicks the first. `1001` server shutting down → reconnect.
91
70
 
92
71
  **Liveness**: server pings every 20 s and terminates after 2 missed pongs; the node's watchdog terminates after 60 s without any traffic. Either side dropping fails all in-flight calls.
93
72
 
@@ -102,13 +81,13 @@ Results are never replayed. When the link drops mid-call:
102
81
  - server side: each pending call rejects with `machine "<n>" went offline during the call (connection lost). The operation may or may not have completed on the machine — check its state before retrying.`
103
82
  - machine side: every running call is aborted (node_run's process group gets SIGTERM, SIGKILL 2 s later) and the log reads `connection lost (…) — stopped N running call(s); reconnecting in Xs`.
104
83
 
105
- When `halo node` itself exits — Ctrl-C (`stop`) or a fatal refusal such as a revoked token (4001) — the same abort runs, but `done` (and so `process.exit`) waits until the aborted calls have settled, capped at 2.5 s. The SIGKILL follow-up is a timer in the node process and the command's group is detached, so exiting first would leave a TERM-ignoring command running on the machine.
84
+ When `halo-node` itself exits — Ctrl-C or a fatal refusal such as a revoked token (4001) — the same abort runs, but the process exits only after the aborted calls have settled, capped at 2.5 s. The SIGKILL follow-up is a timer in the node process and the command's group is detached, so exiting first would leave a TERM-ignoring command running on the machine.
106
85
 
107
86
  Offline vs unknown are distinct errors: `machine "<n>" is offline (…). Do not retry in a loop — tell the user to start it.` vs `unknown machine "<n>" — machines linked to this workspace: a, b (see node_list)`.
108
87
 
109
88
  ## Cron runs
110
89
 
111
- Cron runs execute in the server process since 2.2 ([cron.md](cron.md#execution-runjob)), so their node tools reach the hub directly like any other server-side session. The loopback relay the old `halo cli` child used (`POST /api/nodes/call`, a per-boot secret in `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN`) is gone. `child-env.ts` still strips `HALO_NODE_TOKEN` from every child it builds, so a `halo node`'s own `node_run` commands don't see its pairing token — env hygiene, not a boundary: a full-level command runs unsandboxed as the same user and can read that user's processes and files anyway.
90
+ Cron runs execute in the server process since 2.2 ([cron.md](cron.md#execution-runjob)), so their node tools reach the hub directly like any other server-side session. The loopback relay the old `halo cli` child used (`POST /api/nodes/call`, a per-boot secret in `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN`) is gone. `child-env.ts` still strips `HALO_NODE_TOKEN` from every child the server builds, and `halo-node` drops it from its own environment at start and from every `node_run` command's — env hygiene, not a boundary: a full-level command runs unsandboxed as the same user and can read that user's processes and files anyway.
112
91
 
113
92
  ## Prompt-cache invariant
114
93
 
@@ -118,4 +97,4 @@ Cost: the 8 descriptions measure ≈ 460 tokens (Claude CountTokens); with schem
118
97
 
119
98
  ## Admin
120
99
 
121
- `features/nodes/` adds **Channels → Linked machines** (after Web): a row per token with online dot, name, workspace, `root | Whole machine · shell on/off · os`, token copy and delete. The add dialog takes workspace + name and shows two ready-to-copy commands (token in the env, not a flag). **Linux** is `curl -fsSL <base>/install.sh | sh`, then `PATH="$HOME/.local/bin:$PATH" HALO_NODE_TOKEN=<t> halo-node --server <origin> --root <folder>` — a non-root install lands in `~/.local/bin`, which Debian / Ubuntu / Raspberry Pi OS put on `PATH` only at a login after it exists. **macOS / Windows** runs `HALO_NODE_TOKEN=<t> halo node --server <origin> --root <folder>` (npm). `install.sh` (`packages/node-go/install.sh`, POSIX sh) is uploaded per release with that release's version stamped in. It maps `uname -m`, downloads the binary, checks it against `SHA256SUMS` and runs its `--version`, then installs to `/usr/local/bin` (root) or `~/.local/bin` by a rename in that directory, so a running `halo-node` is replaced without "Text file busy". No token, no service. The base comes from `/api/health`'s `version`: a plain `x.y.z` pins `releases/download/v<x.y.z>`, so the machine and the server come from the same tag; a dev build (`x.y.z-<sha>`) or a failed fetch uses `releases/latest/download`. Online state comes from the list response, then `nodes:changed` (full snapshot, global broadcast — the channels page is a global view); a WS reconnect drops the pushed snapshot and re-fetches the list. No polling.
100
+ `features/nodes/` adds **Channels → Linked machines** (after Web): a row per token with online dot, name, workspace, `root | Whole machine · shell on/off · os`, token copy and delete. The add dialog takes workspace + name and shows two ready-to-copy commands (token in the env, not a flag). **Linux / macOS** is `curl -fsSL <base>/install.sh | sh`, then `PATH="$HOME/.local/bin:$PATH" HALO_NODE_TOKEN=<t> halo-node --server <origin> --root <folder>` — a non-root install lands in `~/.local/bin`, which Debian / Ubuntu / Raspberry Pi OS put on `PATH` only at a login after it exists. **Windows** (PowerShell) downloads `<base>/halo-node-windows-amd64.exe` with `Invoke-WebRequest`, sets `$env:HALO_NODE_TOKEN` and runs `.\halo-node-windows-amd64.exe --server <origin> --root <folder>`. `install.sh` (`packages/node-go/install.sh`, POSIX sh) is uploaded per release with that release's version stamped in. It maps `uname -s` (Linux / Darwin) and `uname -m` (Darwin: `arm64` → arm64, `x86_64` → amd64), downloads the binary, checks it against `SHA256SUMS` and runs its `--version`, then installs to `/usr/local/bin` (root) or `~/.local/bin` by a rename in that directory, so a running `halo-node` is replaced without "Text file busy". No token, no service. The base comes from `/api/health`'s `version`: a plain `x.y.z` pins `releases/download/v<x.y.z>`, so the machine and the server come from the same tag; a dev build (`x.y.z-<sha>`) or a failed fetch uses `releases/latest/download`. Online state comes from the list response, then `nodes:changed` (full snapshot, global broadcast — the channels page is a global view); a WS reconnect drops the pushed snapshot and re-fetches the list. No polling.
@@ -82,7 +82,7 @@ One per turn — the root span, opened by `beginTurn` and closed by `endTurn`.
82
82
  | `gen_ai.task.output` | | final assistant text |
83
83
  | `gen_ai.system_instructions` | | system prompt, capped 8 KB — once per turn here, not on every `chat` |
84
84
 
85
- On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`). A turn that ends in a model refusal (`stop_reason: "refusal"`) counts as an error too: ERROR span with `error.type=_OTHER`, and `halo.turn.duration` records `outcome=error`.
85
+ On error: status `ERROR` + the error message, plus a low-cardinality `error.type` attribute (parsed `<ErrName>Error/Exception` prefix, else `_OTHER`). A turn that ends in a model refusal (stop reason `refusal` — Anthropic's refusal or an OpenAI-format provider's content-filter stop) counts as an error too: ERROR span with `error.type=_OTHER`, and `halo.turn.duration` records `outcome=error`.
86
86
 
87
87
  ### `chat <modelId>`
88
88
 
@@ -94,7 +94,7 @@ Child of `invoke_agent`, one per model call — created retroactively on the `us
94
94
  | `gen_ai.request.model` | ✓ | |
95
95
  | `gen_ai.usage.input_tokens` | ✓ | |
96
96
  | `gen_ai.usage.output_tokens` | ✓ | |
97
- | `gen_ai.response.finish_reasons` | `["tool_use"]` or `["end_turn"]` | |
97
+ | `gen_ai.response.finish_reasons` | `[<stop reason>]` — that call's own stop reason, already normalized by the provider: `end_turn`, `tool_use`, `max_tokens`, `stop_sequence` or `refusal` (a truncated or refused call is no longer exported as `end_turn`). `refusal` covers Anthropic's `stop_reason: "refusal"` and the OpenAI-format providers' content-filter stops (finish_reason `content_filter` / `sensitive`, Responses `incomplete_details.reason: "content_filter"` or a `refusal` content part); omitted if the call reported none | |
98
98
  | `halo.ttft_ms` | streamed calls only — ms from request start to the first delta (text / thinking / tool-args), same number as the admin `ttft` badge | |
99
99
  | `gen_ai.input.messages` | | semconv JSON (see below) — **delta only**: messages appended since this turn's previous `chat` span (first call → the user message; later calls → that cycle's `tool_result`s) |
100
100
  | `gen_ai.output.messages` | | semconv JSON, only the trailing assistant message |
@@ -22,7 +22,7 @@ Relay rides the same "deliver a session's wrap-up to some *other* session when i
22
22
 
23
23
  **Hard interrupt** adds one step: if `sendUserMessage` returned `queued` (target was busy), call `target.interruptSession(session_id)`. Order is enqueue-then-abort, same as `querySession(interrupt=true)` — the aborted turn's finally sees a non-empty queue, so it drains into the message instead of firing a relay report for the half-done turn.
24
24
 
25
- **Delivery** (`deliverRelayReport(host, session)`, runs at *every* root turn end in every workspace): cheap `reply_to` column read first (no-op for the 99.9 % of sessions without one) → subtree-quiet gate → body = `finalOutput || output`, `[RELAY TARGET ABORTED …]` prefix when `turnError` is set (before the cap, so the marker survives; for an account error — 401/402/403, bad key, no balance — the closing hint says to fix the target's model configuration and not to resume it with `relay_send`, since a re-send fails the same way; for a model refusal — `turnErrorKind === 'refusal'`, the target's history tripped the safety classifier — it says not to resume it and to `relay_send` to a NEW session id with a clean brief instead) → cap at `limits.autoReportMax` with a `relay_read(...)` pointer → header `[Relay report · workspace <ws> · session <id> · status: completed]` (`status: aborted` when `turnError` is set) → `clearReplyTo` → `registry.getOrCreate(reply_to.workspace)` → `appendUserMessage` + `sendUserMessage` on the caller. Append-then-send is the same pair the run-ledger nudge uses: `sendUserMessage` alone never writes the UI transcript, so the secretary's admin view would show a reply to nothing.
25
+ **Delivery** (`deliverRelayReport(host, session)`, runs at *every* root turn end in every workspace): cheap `reply_to` column read first (no-op for the 99.9 % of sessions without one) → subtree-quiet gate → body = `finalOutput || output`, `[RELAY TARGET ABORTED …]` prefix when `turnError` is set (before the cap, so the marker survives; for an account error — 401/402/403, bad key, no balance — the closing hint says to fix the target's model configuration and not to resume it with `relay_send`, since a re-send fails the same way; for a model refusal — `turnErrorKind === 'refusal'`, the target's history tripped the provider's safety / content filter — it says not to resume it and to `relay_send` to a NEW session id with a clean brief instead) → cap at `limits.autoReportMax` with a `relay_read(...)` pointer → header `[Relay report · workspace <ws> · session <id> · status: completed]` (`status: aborted` when `turnError` is set) → `clearReplyTo` → `registry.getOrCreate(reply_to.workspace)` → `appendUserMessage` + `sendUserMessage` on the caller. Append-then-send is the same pair the run-ledger nudge uses: `sendUserMessage` alone never writes the UI transcript, so the secretary's admin view would show a reply to nothing.
26
26
 
27
27
  ## Invariants and why
28
28
 
@@ -156,7 +156,7 @@ Hierarchical encoding: `root_id>child_segment>grandchild_segment`.
156
156
 
157
157
  Both true → sets `stoppedAt` + emits `agent_done` + `querySession`s the result back to the parent. The parent's `querySession` clears its own `stoppedAt`, handles the report, and may trigger its own `tryReportToParent` — bubbling up.
158
158
 
159
- **Abnormal-termination marker**: when the reported turn was killed by an unrecoverable error (retry budget exhausted / account-level failure), the session's `turnError` field holds the error text and the auto-report is prefixed with an explicit `[SUB-AGENT ABORTED: … Error: <text> … Re-dispatch with query_session(…) to resume.]` block. An **account** error (`turnErrorKind === 'account'`: 401/402/403, bad key, no balance) replaces the trailing hint with "this is a model account / credential / balance / permission problem — retrying or re-dispatching will fail the same way; fix the model configuration first (or tell the user), and do not resume it with `query_session` until then"; `RELAY TARGET ABORTED` ([relay.md](relay.md)) and goal mode's `WORKER ABORTED` get the same swap. A **model refusal** (`stop_reason: "refusal"`, see **Refusal stop** under [Resilient execution loop](#resilient-execution-loop)) also sets `turnError` (`Model declined to respond (<category>): <explanation>`) with `turnErrorKind === 'refusal'`, whose hint reads "this session's conversation history tripped the model's safety classifier — resuming it with `query_session` or re-sending will be declined the same way; do not resume it — start a NEW session with start_session and a clean brief instead, rewording the part that tripped the classifier" (relay / goal / A2A word it for their own context). Without it the parent LLM consumed mid-turn fragments (or a literal "(no output)") as completed reports — the cron-era "sub-agent reported without finishing" incident, root-caused to a Bedrock h2 hang (`TimeoutError: http2 request did not get a response`, now also in the transient-transport retry list). Prefix, don't suppress: skipping the report would leave the parent waiting forever, the partial trace has diagnostic value, and `stoppedAt` is stamped as usual so the child never shows as falsely running. The marker is prepended *before* the truncation cap so it can never be sliced off.
159
+ **Abnormal-termination marker**: when the reported turn was killed by an unrecoverable error (retry budget exhausted / account-level failure), the session's `turnError` field holds the error text and the auto-report is prefixed with an explicit `[SUB-AGENT ABORTED: … Error: <text> … Re-dispatch with query_session(…) to resume.]` block. An **account** error (`turnErrorKind === 'account'`: 401/402/403, bad key, no balance) replaces the trailing hint with "this is a model account / credential / balance / permission problem — retrying or re-dispatching will fail the same way; fix the model configuration first (or tell the user), and do not resume it with `query_session` until then"; `RELAY TARGET ABORTED` ([relay.md](relay.md)) and goal mode's `WORKER ABORTED` get the same swap. A **model refusal** (stop reason `refusal` — Anthropic's or an OpenAI-format content-filter stop, see **Refusal stop** under [Resilient execution loop](#resilient-execution-loop)) also sets `turnError` (`Model declined to respond (<category>): <explanation>`) with `turnErrorKind === 'refusal'`, whose hint reads "this session's conversation history tripped the provider's safety / content filter — resuming it with `query_session` or re-sending will be declined the same way; do not resume it — start a NEW session with start_session and a clean brief instead, rewording the part that tripped the filter" (relay / goal / A2A word it for their own context). Without it the parent LLM consumed mid-turn fragments (or a literal "(no output)") as completed reports — the cron-era "sub-agent reported without finishing" incident, root-caused to a Bedrock h2 hang (`TimeoutError: http2 request did not get a response`, now also in the transient-transport retry list). Prefix, don't suppress: skipping the report would leave the parent waiting forever, the partial trace has diagnostic value, and `stoppedAt` is stamped as usual so the child never shows as falsely running. The marker is prepended *before* the truncation cap so it can never be sliced off.
160
160
 
161
161
  The full finally-chain order is `tryReportToParent → deliverGoalRound → deliverRelayReport → releaseSession`. `deliverGoalRound` is a retained legacy hook (goal mode's round delivery): it keys off the row's `goal_session_id` and works as before for any existing goal binding. After the parent-report check, a root whose row carries a `reply_to` back-pointer (it was dispatched from another workspace via `relay_send`) gets its wrap-up appended + sent into the caller session in that workspace (same subtree-quiet gate, `reply_to` cleared before sending — see [relay.md](relay.md)), then the session is released. Each hook decides on its own db column (`parent_id` / `goal_session_id` / `reply_to`), so the three never fire for the same session.
162
162
 
@@ -402,7 +402,7 @@ Frontend network issues don't affect the backend:
402
402
  | **4xx multimodal rejection** (`Multimodal data is corrupted` / `Could not process image`) | Replace all image blocks in history with text placeholders, persist, retry — see [Multimodal 4xx degrade](#multimodal-4xx-degrade) below |
403
403
  | Unrecoverable error | Report to user, stop |
404
404
 
405
- **Refusal stop.** `stop_reason: "refusal"` (Anthropic models on Bedrock invoke + generic anthropic) is an HTTP 200 with `stop_details {category, explanation}` — not an error, so it never enters the retry matrix above. `agent-loop.ts` handles it explicitly: no assistant message is pushed, any partial `tool_use` is **not** executed, the loop ends; `session-manager.ts` emits a `system` event `⚠️ [<agent>] Model declined to respond (<category>): <explanation>` suggesting `/session new`. Previously it fell through the `end_turn` path and looked like a silent stall. It also sets the turn's `turnError` (kind `refusal`), so every abnormal-termination marker sees it — the parent's `[SUB-AGENT ABORTED …]`, `RELAY TARGET ABORTED` / `status: aborted`, goal mode's `WORKER ABORTED`, A2A `FAILED` and an AgentCore invocation's `error` — each with a "start a new session" hint instead of "resume it"; the turn sends no interim report, and a `continue_task` armed in it does not kick (see **Kick position** under [Message queue and drain](#message-queue-and-drain)). Before, a refused sub-agent reported `(no output)` (or the last narration) as a finished result, and the parent kept resuming it into the same refusal.
405
+ **Refusal stop.** `stop_reason: "refusal"` (Anthropic-format providers) is an HTTP 200 with `stop_details {category, explanation}` — not an error, so it never enters the retry matrix above. The OpenAI-format layers map a provider's documented content-filter stop to the same `stopReason: 'refusal'`: `chatCompletionResult` (`openai-chat-format.ts`, every `chat/completions` provider) for finish_reason `content_filter` (OpenAI, DeepSeek, Doubao, Hunyuan TokenHub) or `sensitive` (Zhipu), with `category` = that raw value and no explanation; `MantleAgent` (Responses API) for `incomplete_details.reason: "content_filter"` (category `content_filter`, also with an empty `output[]`, which then skips the `MantleEmptyResponse` retry) or a `refusal` content part (category `refusal`, its text as the explanation). Any other unknown finish value stays `end_turn`. `agent-loop.ts` handles it explicitly: no assistant message is pushed, any partial `tool_use` is **not** executed, the loop ends; `session-manager.ts` emits a `system` event `⚠️ [<agent>] Model declined to respond (<category>): <explanation>` suggesting `/session new`. Previously it fell through the `end_turn` path and looked like a silent stall. It also sets the turn's `turnError` (kind `refusal`), so every abnormal-termination marker sees it — the parent's `[SUB-AGENT ABORTED …]`, `RELAY TARGET ABORTED` / `status: aborted`, goal mode's `WORKER ABORTED`, A2A `FAILED` and an AgentCore invocation's `error` — each with a "start a new session" hint instead of "resume it"; the turn sends no interim report, and a `continue_task` armed in it does not kick (see **Kick position** under [Message queue and drain](#message-queue-and-drain)). Before, a refused sub-agent reported `(no output)` (or the last narration) as a finished result, and the parent kept resuming it into the same refusal.
406
406
 
407
407
  **Classification order: structured signal first, message text last.** The interrupt branch is decided by the attempt's own `AbortController` signal (every interrupt path we own goes through `abortReason()` on it), never by the words `cancelled` / `aborted` in the message — an upstream error body can legitimately read `"The operation was aborted due to …"` (Bedrock), and matching that swallowed a real failure as a user interrupt: silent `break`, no retry, no error event, an empty reply. Likewise the account-level branch trusts `httpStatus` when one was recovered: 401/402/403 is terminal, any *other* status is not, whatever the body says — an OpenAI-compatible 503 whose body reads `"authentication service temporarily unavailable"` used to be classified as a dead key and never retried. The keyword list is only consulted when no status could be extracted. Pinned by `turn-retry-idempotent.test.ts` (seven classification cases).
408
408
 
@@ -509,7 +509,7 @@ Bytes, not an exchange count, because bytes are the actual problem — file size
509
509
 
510
510
  Segments are immutable. Deletion is filesystem-driven: deleting a session lists its directory and removes every file of the session (`ui.jsonl`, `raw.jsonl`, `ui.arch.*`, leftover `.tmp.*`, 1.x `.json` / `arch.*.json.gz`) rather than trusting any count or in-memory map, so crash-orphaned segments go too.
511
511
 
512
- **Read side (scroll-up loading).** The admin loads a session's whole active UI log and fetches segments only when the user scrolls to the top: `GET /api/sessions/logs/:id/archive/:n?projectId=` returns `{ messages }` for one whole segment, gunzipped server-side (`:id` is the encoded full id, so a sub-session's segments are reachable too). The anchor is `archiveCount`, delivered on the `state:snapshot` of **subscribe / reattach only** (one segment count per session open); per-turn snapshots deliberately omit it (`exchange:delete` answers with `exchange:deleted`, no snapshot). The client walks `archiveCount` → 1 and stops at "no earlier messages"; pulled segments are cached (immutable, so never re-requested) and a failed fetch keeps the cursor so the same segment can be retried.
512
+ **Read side (scroll-up loading).** The admin loads a session's whole active UI log and fetches segments only when the user scrolls to the top: `GET /api/sessions/logs/:id/archive/:n?projectId=` returns `{ messages }` for one whole segment, gunzipped server-side (`:id` is the encoded full id, so a sub-session's segments are reachable too), minus its `tool_call` / `tool_result` rows — the same `adminSnapshotRows` filter (`sessions/session-types.ts`) the WS subscribe / reattach snapshot and `GET /api/sessions/logs/:id` use, since the chat renders tool calls from the assistant row's `contentBlocks`. The segment on disk keeps them. The anchor is `archiveCount`, delivered on the `state:snapshot` of **subscribe / reattach only** (one segment count per session open); per-turn snapshots deliberately omit it (`exchange:delete` answers with `exchange:deleted`, no snapshot). The client walks `archiveCount` → 1 and stops at "no earlier messages"; pulled segments are cached (immutable, so never re-requested) and a failed fetch keeps the cursor so the same segment can be retried.
513
513
 
514
514
  **The anchor moves up, never down.** `noteArchiveAnchor` no-ops for an equal or lower count (keeping per-turn snapshots — which omit the field, read as 0 — and ordinary re-subscribes idempotent), but a **higher** count re-anchors. Needed because a turn that ends while the session is open can archive a new segment, and the next subscribe/reattach snapshot both carries the higher count *and* replaces the view with the shrunken active log: with a permanently pinned anchor those just-archived turns were in neither place — gone from the view and below a cursor counting down from the old anchor — showing a hole until the user reopened the session. Re-anchoring restarts the walk instead of extending it (prepends are oldest-first, so a newer segment can't splice onto an already-pulled tail); the re-pull costs no requests because `segmentCache` still holds them. Archived history renders collapsed and **read-only** — no Delete button, which matches `deleteExchange` refusing archived logs.
515
515
 
@@ -441,6 +441,9 @@ A value of the form `<<ENV_NAME>>` is replaced at read time with `process.env.EN
441
441
  | `general.logging.level` | warn | `debug` / `info` / `warn` / `error`. Restart required |
442
442
  | `general.a2a.public_url` | `''` | Global-only. Origin peers use to reach this server for A2A (no trailing slash): the agent card's interface URL and the default push-webhook base for `a2a_send`. Empty = derived from the request host (no outbound dispatch without it or a remote's `push_base`). Env `HALO_A2A_PUBLIC_URL` overrides. See [a2a.md](a2a.md) |
443
443
  | `general.a2a.url_allowlist` | `100.64.0.0/10,*.ts.net` | Global-only. CIDRs / host patterns A2A egress may reach besides public https (push webhooks, remote cards and RPC, inbound image URLs). Private / loopback only when listed; plain http only to listed hosts / ranges; link-local always refused. See [a2a.md](a2a.md#push-delivery-and-url-policy) |
444
+ | `general.outbound_proxy.enabled` | `false` | Global-only. Outbound proxy on/off: proxy env vars for spawned children, an undici `ProxyAgent` for web_fetch and A2A requests to remotes. Read live |
445
+ | `general.outbound_proxy.url` | `''` | Global-only. `http(s)://host:port`, no user-info (PUT / PATCH 400). Empty = off |
446
+ | `general.outbound_proxy.token` | `''` | **Secret**, global-only. `user:password` or a bare token; Basic auth (URL user-info for children, `Proxy-Authorization` in-process). See [requirements/settings.md](../requirements/settings.md#outbound-proxy) |
444
447
  | `general.observability.endpoint` | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`); empty = off. Restart required (server, cron runs included); evolution runs use a new value on their next run. See [observability.md](observability.md) |
445
448
  | `general.observability.service_name` | `halo` | OTel resource `service.name`. Restart required (server, cron included); evolution: next run |
446
449
  | `general.observability.headers` | `''` | **Secret.** Extra OTLP request headers, comma-separated `k=v`. Restart required (server, cron included); evolution: next run |
@@ -671,7 +674,7 @@ All channel types (telegram, web, wechat, slack, feishu, wecom) share one table,
671
674
  | Column | Type | Notes |
672
675
  |---|---|---|
673
676
  | account_id | TEXT PK | Channel-specific ID (e.g. `halo_agent_bot`, `abc-im-bot`, `e718bb7b`) |
674
- | channel_type | TEXT NOT NULL | `'telegram'`, `'web'`, `'wechat'`, `'slack'`, `'feishu'`, `'wecom'`, or `'node'` (a `halo node` pairing token) |
677
+ | channel_type | TEXT NOT NULL | `'telegram'`, `'web'`, `'wechat'`, `'slack'`, `'feishu'`, `'wecom'`, or `'node'` (a `halo-node` pairing token) |
675
678
  | workspace_path | TEXT NOT NULL | Absolute path of the bound workspace |
676
679
  | label | TEXT | User-chosen name |
677
680
  | enabled | INTEGER | 1 = active, 0 = disabled |
@@ -711,7 +714,7 @@ Rows are never pruned.
711
714
  - `schema.sql` / each db's `CREATE_SQL` always describes the FULL current shape, so a fresh db is complete after the CREATEs alone
712
715
  - Every change to an already-existing db gets a numbered slot in that file's ordered migration list — `HALO_MIGRATIONS` (`db/index.ts`), `CRON_MIGRATIONS` (`db/cron-db.ts`), `A2A_MIGRATIONS` (`db/a2a-db.ts`), `EVO_MIGRATIONS` (`db/evo-db.ts`); `channel-db.ts` / `runs-db.ts` currently pass an empty list. `runMigrations(sqlite, list)` (`db/migrate.ts`) runs `list[user_version..]` in order, each in its own transaction, stamping `PRAGMA user_version = i + 1` after each
713
716
  - A fresh db is also at `user_version 0` and runs the whole list, so every slot must be a no-op against the current shape (`addColumnIfMissing`, `CREATE … IF NOT EXISTS`). A CHECK constraint can't be altered in SQLite: the slot rebuilds the table (create `_v2`, copy rows, drop, rename) and skips when `sqlite_master` already shows the new constraint — `HALO_MIGRATIONS` v2 (`disabled_items.scope` admits `'extension'`) is the pattern
714
- - Current slots worth knowing: `HALO_MIGRATIONS` v2 — the `disabled_items` rebuild above (rows kept). `HALO_MIGRATIONS` v3 — `agent_sessions.output TEXT`, `agent_sessions.last_activity_at TEXT` (session store 2.0: the files carry no metadata). `CRON_MIGRATIONS` v5 — `cron_jobs.working_dir TEXT NULL` (the job's folder — the session's `working_dir`), `cron_jobs.stateless INTEGER NOT NULL DEFAULT 0` (1 = fresh session every run), `cron_runs.session_id TEXT NULL` (the session that run used). `CRON_MIGRATIONS` v6 — `cron_runs.workspace_path TEXT NULL` (the workspace the run fired against, where a stateless run's session lives), backfilled from the job row (`UPDATE … SET workspace_path = (SELECT workspace_path FROM cron_jobs …) WHERE workspace_path IS NULL`), so rows whose job is already gone stay NULL; column semantics in [cron.md](cron.md). `CRON_MIGRATIONS` v7 — `cron_jobs.webhook_secret TEXT NULL` (set = webhook-triggered job) plus the `cron_events` table (one row per webhook delivery, `UNIQUE(job_id, delivery_id)`; created by `CREATE_SQL`'s `IF NOT EXISTS`, so the slot itself only adds the column); cron.db at `user_version` 7 after it. `A2A_MIGRATIONS` v1 — `a2a_tasks.result_files TEXT`; v2 — `agentcore_invocations.result`, `result_files`, `superseded_by` (all `TEXT`, for a table an earlier build created without them; a2a.db at `user_version` 2 after it). `EVO_MIGRATIONS` v1 — `evolution_applies.approved_by TEXT NOT NULL DEFAULT 'reviewer'` (`'auto'` for L2/L3), `notified_at INTEGER`, `rolled_back_at INTEGER`; `evolution_runs.has_observation INTEGER NOT NULL DEFAULT 0`, `source_access TEXT` (`full` / `workspace` / `readonly`, or `webhook` for a session that took cron webhook events; null on pre-existing rows → never auto-applied); v2 — `evolution_runs.attempts_base INTEGER NOT NULL DEFAULT 0` (`attempts` at the latest retry; the retry budget counts from it), and v1's adds re-run (idempotent) for dev dbs stamped v1 without `source_access`; see [evolution.md](evolution.md)
717
+ - Current slots worth knowing: `HALO_MIGRATIONS` v2 — the `disabled_items` rebuild above (rows kept). `HALO_MIGRATIONS` v3 — `agent_sessions.output TEXT`, `agent_sessions.last_activity_at TEXT` (session store 2.0: the files carry no metadata). `CRON_MIGRATIONS` v5 — `cron_jobs.working_dir TEXT NULL` (the job's folder — the session's `working_dir`), `cron_jobs.stateless INTEGER NOT NULL DEFAULT 0` (1 = fresh session every run), `cron_runs.session_id TEXT NULL` (the session that run used). `CRON_MIGRATIONS` v6 — `cron_runs.workspace_path TEXT NULL` (the workspace the run fired against, where a stateless run's session lives), backfilled from the job row (`UPDATE … SET workspace_path = (SELECT workspace_path FROM cron_jobs …) WHERE workspace_path IS NULL`), so rows whose job is already gone stay NULL; column semantics in [cron.md](cron.md). `CRON_MIGRATIONS` v7 — `cron_jobs.webhook_secret TEXT NULL` (set = webhook-triggered job) plus the `cron_events` table (one row per webhook delivery, `UNIQUE(job_id, delivery_id)`; created by `CREATE_SQL`'s `IF NOT EXISTS`, so the slot itself only adds the column); cron.db at `user_version` 7 after it. `A2A_MIGRATIONS` v1 — `a2a_tasks.result_files TEXT`; v2 — `agentcore_invocations.result`, `result_files`, `superseded_by` (all `TEXT`, for a table an earlier build created without them; a2a.db at `user_version` 2 after it). `EVO_MIGRATIONS` v1 — `evolution_applies.approved_by TEXT NOT NULL DEFAULT 'reviewer'` (`'auto'` for L2/L3), `notified_at INTEGER`, `rolled_back_at INTEGER`; `evolution_runs.has_observation INTEGER NOT NULL DEFAULT 0`, `source_access TEXT` (`full` / `workspace` / `readonly`, null on pre-existing rows → never auto-applied; `webhook` on rows written before 2.2.3, read as not full); v2 — `evolution_runs.attempts_base INTEGER NOT NULL DEFAULT 0` (`attempts` at the latest retry; the retry budget counts from it), and v1's adds re-run (idempotent) for dev dbs stamped v1 without `source_access`; see [evolution.md](evolution.md)
715
718
  - Append only — never reorder or edit a shipped slot. A db opened by a newer halo (user_version > list length) logs a warning and continues
716
719
  - Do not drop or rename existing columns
717
720
  - New channel types: add rows with a new `channel_type` value + define config shape in the channel's `accounts.ts` adapter
@@ -78,9 +78,9 @@ Per VAD segment, inside the same FIFO job as its decode (so a stream's finals st
78
78
 
79
79
  Thresholds: CAM++ cosine on four.wav, pairwise over its nine reference turns (all ≥ 1 s, each embedding capped at 6 s), was 0.75–0.83 for the same speaker (6 pairs) and 0.03–0.52 between different ones (30 pairs); in the engine every match threshold from 0.45 to 0.7 gave the same labels, so 0.55 sits in the gap. **Acceptance** (sherpa's 4-speaker zh test `four.wav`, 56.9 s, 16 kHz mono, converted to s16le and streamed through `createSttEngine()` with the real models in 100 ms frames, `speakers` on): 8 finals labelled `0 1 2 | 0 | 3 | 3 | 2 | 0`, matching the reference (`A B B C | A | D | D | C | A`, sherpa's two S1 runs fall in one piece here) up to renaming, the first segment split at 7.03 s and 13.78 s into its three speakers' sentences. Total 3.4 s for 57 s of audio (1.8 s without speakers); longest event-loop gap 70 ms.
80
80
 
81
- ### VAD (one per stream)
81
+ ### VAD (one per open stream, pooled)
82
82
 
83
- Silero v4, 512-sample windows at 16 kHz, internal buffer 30 s. Constants in `engine.ts`:
83
+ Silero v4, 512-sample windows at 16 kHz, internal buffer 30 s. Each open stream holds one; on `end()` (after the flush) or `abort()` it is `reset()` and goes back to a per-engine pool that `open()` draws from first, so the pool only grows to the most streams ever open at once and pooled VADs are never freed (~4.6 MB native each). A `new Vad` per stream left that native memory to a GC finalizer the small JS heap rarely runs: +150 MB RSS over 150 streams through the engine (+690 MB for raw new-and-feed), flat with `gc()`. The per-segment recognizer / CAM++ streams are still finalizer-freed (the addon has no release call), ≈ 0.2 MB per decode. Constants in `engine.ts`:
84
84
 
85
85
  | Constant | Value | Why |
86
86
  |---|---|---|
@@ -120,7 +120,7 @@ Sub-agent text is root-tree noise on this wire: the listener is the root's (sub
120
120
 
121
121
  `chat:thinking` / `chat:stream` / `chat:followup` / `agent:tool_call` / `agent:tool_result` additionally carry `replay: true` (plus `sessionId`) when synthesized by the reattach path (never on live events) — see [Reconnect flow](#reconnect-flow) step 6.
122
122
 
123
- `chat:system` producers (`session-manager.ts`'s `stop` event handling): a `max_tokens` stop emits `⚠️ [<agent>] Response truncated: output token limit reached.`; a `refusal` stop (Anthropic `stop_reason: "refusal"`, HTTP 200 — the model declined, not an error) emits `⚠️ [<agent>] Model declined to respond (<category>): <explanation> — …` suggesting `/session new` (see [session.md](session.md#resilient-execution-loop)).
123
+ `chat:system` producers (`session-manager.ts`'s `stop` event handling): a `max_tokens` stop emits `⚠️ [<agent>] Response truncated: output token limit reached.`; a `refusal` stop (Anthropic `stop_reason: "refusal"` or an OpenAI-format content-filter stop such as finish_reason `content_filter`, HTTP 200 — the model or the provider's filter declined, not an error) emits `⚠️ [<agent>] Model declined to respond (<category>): <explanation> — …` suggesting `/session new` (see [session.md](session.md#resilient-execution-loop)).
124
124
 
125
125
  Server-internal flags on `AgentSessionEvent` that are **not** carried into the WS frame: `stream.final` (marks the turn's wrap-up text vs. pre-tool filler — consumed by channel responders and the cli, see [session.md](session.md#message-queue-and-drain)), `stream.streamed` / `thinking.streamed` (the text already went out as deltas — decides whether the whole event is forwarded at all), `system.outOfBand` (a `notifySystem` notice from outside the turn — the evo auto-apply line; still a plain `chat:system` here, only the cron run and `halo cli` leave it out of their printed text). The admin renders every streamed block, so none is needed on the wire.
126
126
 
@@ -128,7 +128,7 @@ Server-internal flags on `AgentSessionEvent` that are **not** carried into the W
128
128
 
129
129
  | Type | Source | Purpose |
130
130
  |---|---|---|
131
- | `state:snapshot` | handler.ts on connect | Initial state (agents, messages, sessionId). On **subscribe / reattach only** it also carries `archiveCount` — the number of committed UI-log archive segments, which the admin's scroll-up loader counts down from (see [Archive anchor](#archive-anchor-in-statesnapshot)) — and `accessLevel` (`'workspace' | 'readonly' | null`, null = full), which seeds the admin's access-level selector. Subscribe includes `accessLevel` only when the session already exists; for a not-yet-created session it's omitted, and the admin leaves the selector alone when the field is absent (so a level picked before the first send isn't reset). The subscribe / reattach rows leave out the log's `tool_call` / `tool_result` rows (`adminSnapshotRows`) — the chat renders each tool call from the assistant row's `contentBlocks`; usage / context rows stay. The on-disk UI log and the other channels keep them. The admin reuses the objects of unchanged rows on replace (`reconcileMessages`), so memoized rows skip re-render |
131
+ | `state:snapshot` | handler.ts on connect | Initial state (agents, messages, sessionId). On **subscribe / reattach only** it also carries `archiveCount` — the number of committed UI-log archive segments, which the admin's scroll-up loader counts down from (see [Archive anchor](#archive-anchor-in-statesnapshot)) — and `accessLevel` (`'workspace' | 'readonly' | null`, null = full), which seeds the admin's access-level selector. Subscribe includes `accessLevel` only when the session already exists; for a not-yet-created session it's omitted, and the admin leaves the selector alone when the field is absent (so a level picked before the first send isn't reset). The subscribe / reattach rows leave out the log's `tool_call` / `tool_result` rows (`adminSnapshotRows`, `sessions/session-types.ts`) — the chat renders each tool call from the assistant row's `contentBlocks`; usage / context rows stay. The admin's REST reads of the same data (`GET /api/sessions/logs/:id` and the archive segment route) apply the same filter, so every admin read path carries the same rows; `messageCount` there still counts the full active log. The on-disk UI log and the other channels keep them. The admin reuses the objects of unchanged rows on replace (`reconcileMessages`), so memoized rows skip re-render |
132
132
  | `exchange:deleted` | `handleExchangeDelete` (this client, when subscribed to the session) | An `exchange:delete` landed — `{sessionId, deletedIds, userOrdinal, userRows}`: `deletedIds` is every soft-deleted row id of the active log (idempotent), `userOrdinal` the turn just deleted, `userRows` the log's user rows without `taskId` (deleted ones included — the count the ordinal runs over). The admin flags those rows plus the `userOrdinal` turn's span (rows this tab created live carry client ids the server log doesn't know) — no log replace. When the tab's own count differs from `userRows` (a turn another connection sent with `localEcho`, a `sendFailed` bubble) it flags the ids only and re-sends `subscribe`, whose snapshot corrects the log |
133
133
  | `chat:ack` | `handleChat` | Chat with `clientMsgId` is persisted in the session log — releases the client's pending-ack entry (see [Chat delivery](#chat-delivery-ack--resend--dedup)) |
134
134
  | `__pong__` | `__ping__` handler | Reply to the client's application-level liveness probe |
@@ -149,7 +149,7 @@ Server-internal flags on `AgentSessionEvent` that are **not** carried into the W
149
149
  | `skills:changed` | `ws/skills-changed.ts` `notifySkillsChanged` (REST `POST` / `DELETE /skills`, `PATCH /skills/:id/toggle`, `/skill disable` · `enable` · `delete`) · `ws/watcher-pool.ts` (the workspace watcher saw `.halo/skills/**` change) | A skill was created / deleted / toggled, or a workspace skill folder changed on disk (an agent's `file_write`, a hand edit). No payload: the admin bumps its skill bus, and the chat slash-command popup, the Skills page and the agent form's skill picker re-fetch. Scoped on the server: a workspace skill or a toggle (the disable flag lives in that workspace's db) goes via `broadcastToWorkspace`, a global skill via `broadcast`. Only real changes emit: a no-op mutation (409 / 404 / already disabled) sends nothing, and the watcher path compares a per-workspace key (folder ids + each SKILL.md's frontmatter), so body / resource edits and the echo of a mutation that already notified stay silent. The watcher debounce is 300 ms on top of the file watcher's own 300 ms flush. `~/.halo/global/skills/` has no watcher, so a hand edit there shows on the Skills page's window-focus refetch. Also bumped on WS reconnect while the Skills page is open. |
150
150
  | `extension:changed` | `extensions/watcher.ts` `rescanAndBroadcast` (broadcast to all clients) | `~/.halo/global/extensions/` changed (install / upgrade / uninstall / on-off switch, from the admin upload, the one-click update, the `extension` skill, or a manual `mv`/`rm`; a switch rewrites `.state.json`, which trips the same watcher) — carries the **full** `ExtensionsSnapshot` (`{extensions, disabled, errors}`: switched on, switched off, refused), not a diff, and only fires when the snapshot key differs from the last one. The admin replaces its extension list wholesale (a missing `disabled`, from an older server, reads as `[]`): the preview registry re-resolves open tabs from `extensions` only (an upgraded extension remounts, or shows a banner if the tab is dirty; an uninstalled or switched-off one falls back to the "no built-in preview" page, or keeps a dirty tab alive with a banner) and the Settings → Extensions panel re-renders. Global, no workspace marker — extensions are server-wide. See [canvas-extensions.md](canvas-extensions.md). |
151
151
  | `updates:changed` | `updates/core-check.ts` · `updates/extension-check.ts` (broadcast to all clients) | An update check finished, or (extensions scope) a watcher rescan changed a cached verdict — `{scope: 'core' \| 'extensions', status}`, `status` = the same `CoreUpdateStatus` / `ExtensionsUpdateStatus` that `GET /api/updates?scope=…` returns. Checks start only from that GET (cache ≥ 24 h old) or `POST /api/updates/check`, never on a timer. The Settings sidebar foot (`core`) and Settings → Extensions (`extensions`) replace their status with it; nothing else listens. Global, no workspace marker. See [canvas-extensions.md → Updates](canvas-extensions.md#updates-updatejson). |
152
- | `nodes:changed` | `nodes/hub.ts` `announce` (broadcast to all clients) | A `halo node` connected or disconnected — carries the **full** list of online machines, `{online: [{accountId, root, shell, os, since}]}` (`root: null` = whole-machine mode), not a diff. The admin's Channels → Linked machines page replaces its online map wholesale and re-fetches the account list on WS reconnect. Global, no workspace marker (the channels page is a global view). See [node.md](node.md). |
152
+ | `nodes:changed` | `nodes/hub.ts` `announce` (broadcast to all clients) | A `halo-node` connected or disconnected — carries the **full** list of online machines, `{online: [{accountId, root, shell, os, since}]}` (`root: null` = whole-machine mode), not a diff. The admin's Channels → Linked machines page replaces its online map wholesale and re-fetches the account list on WS reconnect. Global, no workspace marker (the channels page is a global view). See [node.md](node.md). |
153
153
 
154
154
  ### `error` frames: optional `code`
155
155
 
@@ -130,7 +130,7 @@ Unified session log API — list + read session files across all agents.
130
130
  | Method | Path | Purpose |
131
131
  |---|---|---|
132
132
  | GET | `/api/sessions/logs?projectId=` | List session metadata, keyset-paginated. Default returns each top-level row + all descendants (sidebar tree); `rootOnly=1` returns roots only (chat-panel session tabs). A `projectId` with no `.halo/` (plain or missing directory) returns `{sessions: [], nextCursor: null}` without scaffolding it — a list never turns a directory into a workspace |
133
- | GET | `/api/sessions/logs/:id?projectId=` | Full session log — the **active** file only; archived history is a separate call. `projectId` is required: the row names the session's directory (404 without it) |
133
+ | GET | `/api/sessions/logs/:id?projectId=` | Full session log — the **active** file only, without `tool_call` / `tool_result` rows; archived history is a separate call. `projectId` is required: the row names the session's directory (404 without it) |
134
134
  | GET | `/api/sessions/logs/:id/archive/:n?projectId=` | One archived UI-log segment — see [detail](#get-apisessionslogsidarchivenprojectidabs) |
135
135
  | DELETE | `/api/sessions/logs/:id?projectId=` | Permanently delete the session and all its descendants — log files (with archive segments) and `agent_sessions` rows. Returns `{ok, deleted}` |
136
136
  | PATCH | `/api/sessions/logs/:id?projectId=` | Rename a session (admin-only; body `{title}`) — updates `agent_sessions.title` only (the row is the title's only home; no file is touched — an internal-agent session's title lives in the global internal-sessions row). 400 without `projectId` / `title`, 404 for an unknown id. Accepts any session id (root or sub-agent); the sidebar exposes the rename affordance on every row |
@@ -343,7 +343,7 @@ SSE responses (`/api/web/chat`, `/api/web/subscribe`) carry an SSE comment line
343
343
 
344
344
  See [design/web.md](../design/web.md).
345
345
 
346
- ## Linked machines (`halo node`)
346
+ ## Linked machines (`halo-node`)
347
347
 
348
348
  File: `packages/server/src/nodes/routes.ts` (+ `hub.ts` for the WebSocket). Not mounted in AgentCore mode. Design: [design/node.md](../design/node.md).
349
349
 
@@ -857,7 +857,7 @@ Rows are served from the `agent_sessions` metadata columns (`title` / `exchange_
857
857
 
858
858
  ### GET `/api/sessions/logs/:id?projectId=<abs>`
859
859
 
860
- The session's `agent_sessions` row (metadata) plus its **active** UI log and model history. Older exchanges that archiving moved out are fetched per-segment via [the archive endpoint](#get-apisessionslogsidarchivenprojectidabs). If the UI log is empty and only raw history exists (sub-agent sessions from SessionManager), `convertRawMessages()` transforms it into the display `messages` array on the fly. `projectId` is required: the row names the session's directory (404 without it, or with no row / no UI log). Opening a 1.x session here migrates it; the response then carries the back-filled row.
860
+ The session's `agent_sessions` row (metadata) plus its **active** UI log and model history. Older exchanges that archiving moved out are fetched per-segment via [the archive endpoint](#get-apisessionslogsidarchivenprojectidabs). `messages` leaves out the log's `tool_call` / `tool_result` rows — the same rows as the WS subscribe / reattach snapshot (`adminSnapshotRows`, see [ws.md](../design/ws.md)); each tool call and its output ride on the assistant row's `contentBlocks`. If the UI log is empty and only raw history exists (sub-agent sessions from SessionManager), `convertRawMessages()` transforms it into the display `messages` array on the fly (user / assistant rows only, so nothing to filter). `projectId` is required: the row names the session's directory (404 without it, or with no row / no UI log). Opening a 1.x session here migrates it; the response then carries the back-filled row.
861
861
 
862
862
  ```json
863
863
  // 200
@@ -874,24 +874,24 @@ The session's `agent_sessions` row (metadata) plus its **active** UI log and mod
874
874
  "contextTokens": 5975,
875
875
  "totalOutputTokens": 6058,
876
876
  "archiveCount": 2,
877
- "messages": [ /* SessionMessage[] — see design/storage.md */ ],
877
+ "messages": [ /* SessionMessage[] without tool_call / tool_result rows — see design/storage.md */ ],
878
878
  "rawMessages": [ /* model history */ ],
879
879
  "output": "...", // when set
880
880
  "lastActivityAt": "..." // when set
881
881
  }
882
882
  ```
883
883
 
884
- `messageCount` is the active UI log's length (shrinks on archive); `exchangeCount` is the row's lifetime count, as in the list endpoint. `archiveCount` = committed segments on disk. The 1.x `version` field is gone.
884
+ `messageCount` is the active UI log's full length, `tool_call` / `tool_result` rows included — so it can exceed `messages.length` (shrinks on archive); `exchangeCount` is the row's lifetime count, as in the list endpoint. `archiveCount` = committed segments on disk. The 1.x `version` field is gone.
885
885
 
886
886
  ### GET `/api/sessions/logs/:id/archive/:n?projectId=<abs>`
887
887
 
888
888
  Source: `packages/server/src/routes/session-archive.ts`
889
889
 
890
- One archived UI-log segment, gunzipped server-side. Works for root and sub-sessions alike: a sub-session's segment is addressed by its full id, URL-encoded (`r1>c1` → `r1%3Ec1`). `n` counts up from 1; the client gets its anchor from `archiveCount` on the `state:snapshot` of subscribe/reattach ([ws.md](../design/ws.md)) and walks **down** from it. Segments are immutable once committed, so a client caches what it pulled and never re-requests. Admin-cookie gated (not in `PUBLIC_PATHS`).
890
+ One archived UI-log segment, gunzipped server-side, with the `tool_call` / `tool_result` rows left out (`adminSnapshotRows`, as in the session log above). Works for root and sub-sessions alike: a sub-session's segment is addressed by its full id, URL-encoded (`r1>c1` → `r1%3Ec1`). `n` counts up from 1; the client gets its anchor from `archiveCount` on the `state:snapshot` of subscribe/reattach ([ws.md](../design/ws.md)) and walks **down** from it. Segments are immutable once committed, so a client caches what it pulled and never re-requests. Admin-cookie gated (not in `PUBLIC_PATHS`).
891
891
 
892
892
  ```json
893
893
  // 200
894
- { "messages": [ /* SessionMessage[] — oldest-first within the segment */ ] }
894
+ { "messages": [ /* SessionMessage[] without tool rows — oldest-first within the segment */ ] }
895
895
  ```
896
896
 
897
897
  Failure modes, in the order checked:
@@ -930,6 +930,7 @@ Resolves declared schema (from `models/<id>.yaml` `secrets:`, `skills/<id>/confi
930
930
  "description": "...",
931
931
  "description_zh": "...",
932
932
  "default": "...", // schema-default placeholder; omitted when the schema declares none
933
+ "placeholder": "...", // general.* only: example for an empty input, never a fallback value (e.g. outbound_proxy.url)
933
934
  "secret": true, // UI masks input + value
934
935
  "globalOnly": false, // general.* only: workspace overrides ignored, workspace writes 400
935
936
  "restartRequired": false, // general.* only: read once at boot — UI shows a restart notice after saving
@@ -966,7 +967,7 @@ Resolves declared schema (from `models/<id>.yaml` `secrets:`, `skills/<id>/confi
966
967
  { "ok": true }
967
968
  ```
968
969
 
969
- PUT replaces the full scope; PUT / PATCH / DELETE with `scope: "workspace"` reject a `globalOnly` key with 400 (`<key> is global-only and cannot be set per workspace`) and an unresolvable `projectId` with 404 `Project not found`. DELETE takes `{scope, projectId, key}` and removes the key. PUT / PATCH / DELETE fire the server's `onSettingsChange` listeners (e.g. the sandbox hidden-paths reload); a DELETE of an already-absent key is a no-op — `{ ok: true }`, no file write, no notify. The Settings page uses DELETE for both Reset (current scope removed → falls back to lower scope / unset) and orphan Remove.
970
+ PUT replaces the full scope; PUT / PATCH / DELETE with `scope: "workspace"` reject a `globalOnly` key with 400 (`<key> is global-only and cannot be set per workspace`) and an unresolvable `projectId` with 404 `Project not found`. PUT / PATCH also 400 a value outside a field's `min` / `max`, and a `general.outbound_proxy.url` that isn't `http(s)://` or carries `user:pass@` (`general.outbound_proxy.url: credentials are not allowed in the URL — put them in Token (general.outbound_proxy.token)`; the message never echoes the URL), file untouched. DELETE takes `{scope, projectId, key}` and removes the key. PUT / PATCH / DELETE fire the server's `onSettingsChange` listeners (e.g. the sandbox hidden-paths reload); a DELETE of an already-absent key is a no-op — `{ ok: true }`, no file write, no notify. The Settings page uses DELETE for both Reset (current scope removed → falls back to lower scope / unset) and orphan Remove.
970
971
 
971
972
  ### POST `/api/wechat/login/start`
972
973
 
@@ -1107,6 +1108,6 @@ It does not remove session files. The full delete (rows + JSON + archive segment
1107
1108
  | `listener:released` | `sessionId` | Event listener reclaimed — re-send `subscribe` (see above) |
1108
1109
  | `exchange:deleted` | `sessionId`, `deletedIds`, `userOrdinal`, `userRows` | An `exchange:delete` this connection sent landed: ids of every soft-deleted row of the active log, the ordinal of the turn just deleted, and the log's user-turn count (without `taskId`) that ordinal runs over (the sender's own reply only — no full `state:snapshot` follows) |
1109
1110
  | `skills:changed` | — | A skill was created / deleted / toggled, or a workspace skill folder changed on disk — re-fetch skill lists and `/api/commands`. Workspace-scoped changes reach only that workspace's sockets (see [Skills](#skills)) |
1110
- | `nodes:changed` | `online: [{accountId, root \| null, shell, os, since}]` | A `halo node` connected / disconnected — full snapshot of online machines (global broadcast; `root: null` = whole machine). See [Linked machines](#linked-machines-halo-node) |
1111
+ | `nodes:changed` | `online: [{accountId, root \| null, shell, os, since}]` | A `halo-node` connected / disconnected — full snapshot of online machines (global broadcast; `root: null` = whole machine). See [Linked machines](#linked-machines-halo-node) |
1111
1112
 
1112
1113
  Events with `taskId` set belong to sub-agent turns (nested sessions); events without are the root agent's. Session-scoped frames carry `sessionId` (the root session they belong to); the admin routes each one to the tab holding that session, and a frame without one goes to the tab on screen. Full list in [design/ws.md](../design/ws.md).
@@ -323,7 +323,7 @@ Use only for a new public release; the retained details here are not extra prepa
323
323
  5. **npm**: publish from `packages/cli/dist-pub/` **once**, using the intended dist-tag (`--tag alpha` for an alpha, not `latest`). Confirm `npm view @turmind/halo@x.y.z version` and the intended tag via `npm view @turmind/halo dist-tags.alpha` (or `dist-tags.latest` for stable); both must equal the requested version. Registry replication can lag ~2–3 minutes: wait/recheck, never republish. `dist-pub/` is release-only; desktop staging uses `dist-dev/`. Historical reason: a second v1.3.4 publish put `1.3.4-<sha>` on `latest` for ~90s.
324
324
  6. **GitHub**: `gh release create vx.y.z --notes-from-tag` (or the short CHANGELOG summary); use `--prerelease` for an alpha.
325
325
  7. **Windows exe**: `(cd packages/desktop && HALO_STAGE_FULL=1 CI=true pnpm dist:win)`, then `gh release upload vx.y.z "packages/desktop/dist/Halo Setup x.y.z.exe"`. Verify it appears in `gh release view vx.y.z --json assets`; v1.3.0 and v1.3.1 missed this asset. macOS dmg is the user's step unless requested.
326
- 8. **halo-node binaries**: nothing to build by hand. Publishing the release triggers `.github/workflows/node-go.yml`'s release job (`go vet` + `go test`, then `build.sh`), which uploads `halo-node-linux-{amd64,arm64,armv7}`, `halo-node-windows-amd64.exe`, `SHA256SUMS` and `install.sh` (with the version stamped in; it isn't listed in `SHA256SUMS`). Wait for it to go green, then check that all six appear in `gh release view vx.y.z --json assets` — the admin's Linux connect command runs this tag's `install.sh`. If the job failed, rerun it with `gh workflow run node-go.yml -f tag=vx.y.z` (`workflow_dispatch`; `--clobber` makes reruns safe). The trigger needs the release created by a person (`gh release create` as yourself), because a release created with `GITHUB_TOKEN` doesn't start other workflows.
326
+ 8. **halo-node binaries**: nothing to build by hand. Publishing the release triggers `.github/workflows/node-go.yml`'s release job (`go vet` + `go test`, then `build.sh`), which uploads eight assets: `halo-node-linux-amd64`, `halo-node-linux-arm64`, `halo-node-linux-armv7`, `halo-node-darwin-arm64`, `halo-node-darwin-amd64`, `halo-node-windows-amd64.exe`, `SHA256SUMS` (the six binaries) and `install.sh` (with the version stamped in; it isn't listed in `SHA256SUMS`). Wait for it to go green, then check that all eight appear in `gh release view vx.y.z --json assets` — the admin's Linux / macOS connect command runs this tag's `install.sh`, the Windows one downloads this tag's exe. If the job failed, rerun it with `gh workflow run node-go.yml -f tag=vx.y.z` (`workflow_dispatch`; `--clobber` makes reruns safe). The trigger needs the release created by a person (`gh release create` as yourself), because a release created with `GITHUB_TOKEN` doesn't start other workflows.
327
327
 
328
328
  **npm token gotcha**: `npm publish` on this package needs a granular access token created with **"Bypass 2FA"** checked — scope / permission alone yields `403 Two-factor authentication or granular access token with bypass 2fa enabled is required`. `npm whoami` and `npm token list` succeed with a non-bypass token, so neither is a valid pre-flight; check `GET https://registry.npmjs.org/-/npm/v1/tokens` (with the token as bearer) and look for `"bypass_2fa": true` on the token in use before starting a release.
329
329
 
@@ -122,7 +122,8 @@ Source: `packages/server/src/config.ts`
122
122
  | `HALO_DEFAULT_PROVIDER` | (none) | — | Read by `halo setup --non-interactive`: binds every built-in agent to this provider id (and its default model) and, for each of its env-backed keys whose variable is set, writes the `<<NAME>>` placeholder into `settings.yaml` (Bedrock excepted). An unknown id only prints a warning |
123
123
  | `HALO_CLI` | `halo` (`halo.cmd` on Windows) | — | `halo` cli executable the evolution wrapper spawns — a dev override (cron runs in-process since 2.2 and no longer uses it) |
124
124
  | `HALO_VERSION` | `dev` | — | Version string reported by `GET /api/health` and the cli; stamped by the bundle / desktop launcher, so normally not set by hand |
125
- | `HALO_NODE_TOKEN` | (none) | — | `halo node` only (on the linked machine): pairing token, used when `--token` isn't given — preferred over the flag, which shows up in `ps` / shell history. Stripped from child-process env (see [design/node.md](../design/node.md)) |
125
+ | `HALO_NODE_TOKEN` | (none) | — | `halo-node` only (on the linked machine): pairing token, used when `--token` isn't given — preferred over the flag, which shows up in `ps` / shell history. Stripped from child-process env (see [design/node.md](../design/node.md)) |
126
+ | `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` | (none) | — | `halo-node` only (on the linked machine): forward proxy for an `https://` / `http://` server and the hosts that bypass it (Go `ProxyFromEnvironment`; lowercase names too; `localhost` / loopback never proxied). Ignored when `--proxy <url>` is given — the flag wins (see [guide/node.md](../guide/node.md#through-a-forward-proxy)) |
126
127
  | `HALO_PROMPT` | (none) | — | `plain` makes `halo setup` use numbered-menu prompts instead of the raw-mode TUI |
127
128
  | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | (none) | `settings.yaml general.observability.capture_content` | `true` / `false` / `1` / `0` (any case) overrides `capture_content` either way; unset or any other value → the setting. Read at process start (see [design/observability.md](../design/observability.md)) |
128
129
 
@@ -80,6 +80,8 @@ Run a shell command. Full shell access.
80
80
 
81
81
  Timeout 120 s (`HALO_SHELL_TIMEOUT`). Max output 5 MB. The tool description surfaces the effective timeout to the agent and advises backgrounding (`nohup … &` + log polling) for longer tasks.
82
82
 
83
+ **Outbound proxy.** With Settings → General → `outbound_proxy.enabled` on and a `url` set, every access level's command gets `http_proxy` / `https_proxy` / `HTTP_PROXY` / `HTTPS_PROXY` (credential in the URL's user-info) plus `no_proxy` / `NO_PROXY` (`localhost,127.0.0.1,::1` + the server's own), so curl / git / pip go through it. It is read at each command, no restart. bwrap / Seatbelt sessions get the same vars (in bwrap only the command, not the file tools' helpers). Their network isn't isolated, so a proxy on the host's 127.0.0.1 is reachable. The credential is masked as `***` in the tool result — its URL and base64 forms at any length, the bare token from 4 characters. See [requirements/settings.md → Outbound proxy](../requirements/settings.md#outbound-proxy).
84
+
83
85
  **Server shutdown** (server only — the CLI / TUI never enter shutdown). A command killed by a signal halo didn't send while the server is shutting down (a systemd stop signals the whole cgroup, the command with it) returns `[Command cut off by a server restart — it was still running when the server shut down, so it may or may not have taken effect. Check before re-running it.]` followed by `Output so far:` and the partial stdout (`STDERR:` appended when there is any), instead of `Command failed: …`. A command issued after shutdown began is not run: `Error: command not run — the server is shutting down. Safe to re-issue after the restart if still needed.` Normal exits read as before. See [session.md → Shutdown](../design/session.md#boot-reconcile-of-crash-orphans-reconcileorphansonboot).
84
86
 
85
87
  **Windows output encoding.** The Windows path (`sandbox.ts`) prepends `chcp 65001` so cmd built-ins (`echo`, …) emit UTF-8, then captures raw bytes (`encoding: 'buffer'`) and decodes them strict-UTF-8 with a GBK fallback. This is because native Win32 console tools (`ipconfig`, `systeminfo`, …) ignore `chcp` and still emit the OEM code page (GBK/CP936 on zh-CN); decoding such bytes as UTF-8 produced mojibake. The strict-UTF-8 attempt passes genuine UTF-8 through untouched and only falls back to GBK when the bytes aren't valid UTF-8 (GBK double-byte sequences almost always aren't). mac/Linux are unaffected (UTF-8 throughout).
@@ -120,6 +122,8 @@ HTTP request.
120
122
 
121
123
  Timeout 10 s. Max body 50 KB (truncated if larger). Returns status + content-type + response body.
122
124
 
125
+ With the outbound proxy on (Settings → General → `outbound_proxy.*`), the request goes through it, the credential sent as `Proxy-Authorization`. Hosts in no_proxy (`localhost` / `127.0.0.1` / `::1` + the server's `NO_PROXY`, matched like curl: domain + subdomains, IPs, CIDRs) connect directly, decided per redirect hop. Model provider calls are not affected. A caller-supplied `Proxy-Authorization` header (any case) is refused with a tool error while the proxy is on, no_proxy targets included.
126
+
123
127
  ## Security
124
128
 
125
129
  ### Access level (per-session, dynamic)
@@ -379,7 +383,7 @@ Returns `{ "code": 0, "workspace": "<realpath>", "session_id": "…", "state": "
379
383
  <target's finalOutput || output>
380
384
  ```
381
385
 
382
- Body capped at `limits.autoReportMax` (head-kept) with a `[Report truncated: N chars total. Use relay_read("<ws>", "<id>") for the full text.]` marker. A turn killed by an unrecoverable error carries `· status: aborted]` in the header and is prefixed `[RELAY TARGET ABORTED: … Error: <text> … Re-send with relay_send to let it resume.]` **before** the cap so it can't be sliced off. When the error is an account problem (401/402/403, bad key, no balance), the closing hint instead says it is a model account / credential / balance / permission problem in the target workspace: retrying or re-sending fails the same way, so fix its model configuration (or tell the user to) and do not resume it with `relay_send` until then. For a model refusal (the target's history tripped the safety classifier) it says not to resume it — `relay_send` to a NEW session id with a clean brief, rewording the part that tripped it. `reply_to` is cleared **before** sending: one dispatch → one report, a failed delivery can't double-fire on the next turn end, and a user chatting directly in the department workspace afterwards never pings the secretary.
386
+ Body capped at `limits.autoReportMax` (head-kept) with a `[Report truncated: N chars total. Use relay_read("<ws>", "<id>") for the full text.]` marker. A turn killed by an unrecoverable error carries `· status: aborted]` in the header and is prefixed `[RELAY TARGET ABORTED: … Error: <text> … Re-send with relay_send to let it resume.]` **before** the cap so it can't be sliced off. When the error is an account problem (401/402/403, bad key, no balance), the closing hint instead says it is a model account / credential / balance / permission problem in the target workspace: retrying or re-sending fails the same way, so fix its model configuration (or tell the user to) and do not resume it with `relay_send` until then. For a model refusal (the target's history tripped the provider's safety / content filter) it says not to resume it — `relay_send` to a NEW session id with a clean brief, rewording the part that tripped it. `reply_to` is cleared **before** sending: one dispatch → one report, a failed delivery can't double-fire on the next turn end, and a user chatting directly in the department workspace afterwards never pings the secretary.
383
387
 
384
388
  **Interim report**: if a busy target answers a follow-up `relay_send` in a turn that another turn follows — it called `continue_task` to resume what it was doing, or a further relay message was already queued when the answer ended (also on the opening turn), or the target was waiting on its own sub-agents when asked and the run ended on the answer with a sub-agent still running — the answer is delivered right away as
385
389
 
@@ -530,7 +534,7 @@ Returns `{ "code": 0, "remotes": [{ name, agent, description, skills }], "pendin
530
534
 
531
535
  File: `packages/server/src/nodes/tools.ts` (`buildNodeTools`). Design notes in [design/node.md](../design/node.md); user guide [guide/node.md](../guide/node.md).
532
536
 
533
- Act on the user's **linked machines** — computers running `halo node`, paired with this workspace in Admin → Channels → Linked machines. Each node tool mirrors one local workspace tool and runs **that same tool** on the machine (same arguments, same output), plus a required `node` argument naming the machine:
537
+ Act on the user's **linked machines** — computers running `halo-node`, paired with this workspace in Admin → Channels → Linked machines. Each node tool mirrors one local workspace tool and runs **that same tool** on the machine (same arguments, same output), plus a required `node` argument naming the machine:
534
538
 
535
539
  | Tool | Mirrors | Extra / different args |
536
540
  |---|---|---|
@@ -547,17 +551,17 @@ Act on the user's **linked machines** — computers running `halo node`, paired
547
551
 
548
552
  **Paths**: a machine shares one folder (`--root`; `root` in `node_list`) — relative paths resolve against it, paths that resolve outside it (symlinks included) are refused: `"<p>" is outside this machine's shared folder (<root>) — node tools can only reach paths inside it`. A whole-machine node (`root: null, wholeMachine: true`) has no such check; relative paths resolve against its `base`.
549
553
 
550
- **Runtime**: in `halo server` (cron runs included) calls go over the machine's WebSocket. In the standalone CLI / TUI, the evo sandbox and AgentCore mode the tools are listed but every one — `node_list` included — returns `linked machines are not available in this runtime — only \`halo server\` accepts \`halo node\` connections. This is permanent for this runtime, not a temporary outage: do not retry. …`.
554
+ **Runtime**: in `halo server` (cron runs included) calls go over the machine's WebSocket. In the standalone CLI / TUI, the evo sandbox and AgentCore mode the tools are listed but every one — `node_list` included — returns `linked machines are not available in this runtime — only \`halo server\` accepts \`halo-node\` connections. This is permanent for this runtime, not a temporary outage: do not retry. …`.
551
555
 
552
556
  **Errors** (returned as the tool error text):
553
557
 
554
558
  | Case | Message |
555
559
  |---|---|
556
- | Machine linked but not connected | `machine "<n>" is offline (\`halo node\` is not connected). Do not retry in a loop — tell the user to start it.` |
560
+ | Machine linked but not connected | `machine "<n>" is offline (\`halo-node\` is not connected). Do not retry in a loop — tell the user to start it.` |
557
561
  | Name not linked to this workspace | `unknown machine "<n>" — machines linked to this workspace: <a, b \| (none)> (see node_list)` |
558
562
  | Link dropped mid-call | `machine "<n>" went offline during the call (connection lost). The operation may or may not have completed on the machine — check its state before retrying.` |
559
563
  | No answer in time (120 s; node_run: shell timeout + 15 s) | `machine "<n>" did not answer within <s>s — the call was cancelled; it may or may not have taken effect.` |
560
- | node_run without `--allow-shell` | `node_run is disabled on this machine — the user must restart \`halo node\` with --allow-shell to allow commands` |
564
+ | node_run without `--allow-shell` | `node_run is disabled on this machine — the user must restart \`halo-node\` with --allow-shell to allow commands` |
561
565
  | Request over 8 MB | `request too large to send to the machine (<x> MB > 8 MB limit)` |
562
566
 
563
567
  Results over 256K chars are cut on the machine (`[Content truncated on the machine: N chars total]`); the usual tool-result budget then applies.
@@ -571,7 +575,7 @@ Returns `{ "code": 0, "nodes": [...] }`, one entry per linked machine:
571
575
  - offline: `{ "name": "nas", "online": false }`
572
576
  - online: `{ "name": "mbp", "online": true, "root": "/Users/a/proj", "shell": true, "shellTimeoutSec": 120, "os": "darwin", "since": "<ISO>" }` — whole-machine: `"root": null, "wholeMachine": true, "base": "<start dir>"`; `shellTimeoutSec` only when `shell` is true.
573
577
 
574
- No machines at all → `"nodes": []` plus `"hint": "No machines linked — create a token in Admin → Channels → Linked machines, then run \`halo node\` on the machine."`.
578
+ No machines at all → `"nodes": []` plus `"hint": "No machines linked — create a token in Admin → Channels → Linked machines, then run \`halo-node\` on the machine."`.
575
579
 
576
580
  ### node_run
577
581
 
@@ -26,7 +26,7 @@ Aimed at Halo end-users. Pick the doc that matches your situation.
26
26
 
27
27
  ## Linked machines
28
28
 
29
- [node.md](node.md) — run `halo node` on your own computer so a workspace's agents can work on files there.
29
+ [node.md](node.md) — run `halo-node` on your own computer so a workspace's agents can work on files there.
30
30
 
31
31
  ## Related docs
32
32