@vibelord/lanes 1.3.0 → 1.5.0

package/payload/lanes/docs/RUNBOOK.md

@@ -195,7 +195,7 @@ lanes keys revoke <device>   remove one machine     lanes keys adopt [--dry-run]
 lanes invite <name> --email ADDR   one-time join code (own terminal)   lanes invite list | revoke <id>
 lanes enrol list        open access requests      lanes enrol approve <id> | deny <id>   decide one
 lanes offboard <device> --yes   key, device, view (own terminal)   lanes token revoke <view>   sign a browser out
-lanes onboard <login> --container <name> [--workspace <id> --repo <path>] [--write]   a user's container, dry run first
+lanes onboard <login> --container <name> [--workspace <id> --repo <path>] [--step kimi-notify|grok-build] [--write]   a user's container, dry run first
 systemctl --user status lanes-statusd lanes-sync.timer lanes-ensure.timer lanes-web lanes-enrold
 lanes update --status   the last update and the last check (update/last.json)   lanes update --now   check and install now
 lanes update --rollback the release before again, the one left marked bad   systemctl --user status lanes-update.timer
@@ -493,7 +493,7 @@ lanes studio-host re-key --yes       after the owner revoked the Host: a new mac
   and phone) > the legend ` ⏎ open · / find · x close · ? keys` (phone: ` Swipe to scroll ·
   Tap to open`). The last
   message lasts 3 s (`MESSAGE_S`, measured on the 1 s tick: `Sidebar.expire_message`) or until the next action begins
-  (`Sidebar.begin_action`: open, `x`/`×`, `R`, `P`, `a`/`+`, `u`, `z`, `p`, `h`, `H`, `o`, `/`, `U`, `+`, `r`, `?`, `Tab`, a chord) — never just until
+  (`Sidebar.begin_action`: open, `x`/`×`, `R`, `P`, `a`/`+`, `u`, `z`, `p`, `h`, `H`, `o`, `i`, `/`, `U`, `+`, `r`, `?`, `Tab`, a chord) — never just until
   a key: a cursor move, typing in a field, `Esc`, a focus report, the theme's OSC 11 answer, a wheel tick or a mouse
   release leaves it (DEF-1822; DEF-1815 before it kept it from the focus report only). `?` shows
   the key sheet (`  lanes <current> · view <tree>` first — cockpit v3 V2 — then `sidebar.KEY_SHEET`, 18 lines since R5's
@@ -932,6 +932,7 @@ only normalizes HOW a terminal reaches its lane:
 
 ## Installing agent hooks (opt-in, per container, dry run first)
 
+This section is Claude Code's and Codex's; Kimi Code, Grok Build and grok-cli (1.5.0) are in [Harnesses](#harnesses) below.
 Without hooks the sidebar's agent glyph comes from Claude's session files and Codex's pane title. Hooks make it
 event-driven: Claude Code or Codex runs `~/.lz/bin/lane-signal <status>` on a turn event, which writes
 `~/.lz/state/hooks/<lane id>.json` in the container, and `lanes-statusd` prefers that over the other signals while
@@ -993,6 +994,242 @@ next tool call: after a denied prompt, or an approved call that failed (`Permiss
 write nothing). A lane whose agent crashed loses its sidecar at once (dead pid); one
 whose agent is alive but silent keeps the last word for up to an hour.
 
+## Harnesses
+
+Since 1.5.0 (W12, spec `docs/superpowers/specs/2026-10-15-lanes-harness-observers-design.md`) Lanes reads Kimi Code,
+Grok Build and grok-cli lanes as well as Claude Code and Codex ones; what each status means for the person at the
+board is the user guide's [Harnesses](USERGUIDE.md#harnesses). This section is the operator's: what to install per
+container, how statusd decides, and how to check each harness. Claude Code's and Codex's hooks are in "Installing
+agent hooks" above.
+
+**Per container, once, in order.** Each `hooks install` prints its
+diff first and writes only with `--write`; each changes only its own entries; `--write` refuses a container whose home
+this account cannot write for uid 1000, as for Claude and Codex. Every file is kept as `<file>.bak-<UTC>` before it
+changes, a symbolic link below the container home refuses the file, and in a TOML file only Lanes' own marked block
+(`# >>> lanes <name> >>>` … `# <<< lanes <name> <<<`) is ever rewritten: to undo, delete the block.
+
+1. `lanes install-container <name>` — ships 1.5.0's contract: `lane-signal` (its harness, when no `--harness` or
+   `LANE_HARNESS` names one, read from the parent processes, never assumed `codex`), `lane-mcp` and its editors, and
+   the hook definitions in `~/.lz/hooks/` (`kimi-hooks.toml`, `grok-hooks.json`, `grok-cli-hooks.json`). Unless
+   `--no-mcp`, it also runs `~/.lz/bin/lane mcp install`, which adds the Lanes MCP server for each harness installed
+   there, beside Claude's and Codex's entries (each subsection below names its file).
+2. The hooks of each harness the container has, each with its dry run first: `lanes hooks install --agent kimi
+   --container <name>` ([Kimi Code](#kimi-code)), `--agent grok` ([Grok Build](#grok-build), after Grok Build is
+   installed) and `--agent grok-cli` ([grok-cli](#grok-cli)).
+3. Optional, the operator's call per container: `lanes onboard <login> --container <name> --step kimi-notify --write`
+   (Kimi Code's bell) and `--step grok-build --write` (installs Grok Build).
+4. New agent sessions read the hooks and the MCP entry; a running session keeps what it started with (inferred from
+   how the harnesses read their configuration at start, not measured).
+
+`lanes onboard` runs step 2 by itself for each harness whose home is in the container (`hooks-kimi`, `hooks-grok`,
+`hooks-grok-cli`; `skipped` otherwise); `kimi-notify` and `grok-build` run only when named with `--step`
+(`grok-build` right after the board, before the contract, so the same run writes Grok Build's MCP entry and hooks).
+
+What each event writes (`agent{}` vocabulary):
+
+| Harness | Event | status |
+|---|---|---|
+| Kimi Code | `UserPromptSubmit`, `TurnStarted`, `PreToolUse`, `PostToolUse`, `PermissionResult` | `working` |
+| Kimi Code | `PermissionRequest` | `needs_approval` |
+| Kimi Code | `Stop`, `Interrupt` | `needs_input` |
+| Kimi Code | `StopFailure` / `SessionStart` / `SessionEnd` | `errored` / `idle` / `shell` |
+| Grok Build | `UserPromptSubmit`, `PreToolUse`, `PostToolUse` | `working` |
+| Grok Build | `Notification` (`permission_prompt`) | `needs_approval` |
+| Grok Build | `Stop`, `StopCancelled`, `Notification` (`idle_prompt`) | `needs_input` |
+| Grok Build | `StopFailure` / `SessionStart` / `SessionEnd` | `errored` / `idle` / `shell` |
+| grok-cli | `UserPromptSubmit`, `PreToolUse`, `PostToolUse` | `working` |
+| grok-cli | `Stop` / `StopFailure` / `SessionEnd` | `needs_input` / `errored` / `shell` |
+
+Verified event names (2026-10-15, this host): Kimi Code 2.1.1's `HOOK_EVENT_TYPES` and its strict `[[hooks]]` schema
+(`event`, `matcher?`, `command`, `timeout?` 1–600) in its binary; grok-cli 1.1.7's hook events, its exact-string
+matcher, `sh -c` with its own environment, and its MCP server entry (`{id, label, enabled, transport, command, args,
+env}`, client name `grok-cli-<id>`) in the source its binary carries; Grok Build 1.0.52's events from its source and
+user guide (`docs/superpowers/specs/evidence/2026-10-15-grok-build-measurement.md` §3). Not yet proven on a live,
+logged-in session: that each harness fires these hooks in a lane as the source says.
+
+### The observed harness
+
+Which harness runs in a lane needs nothing installed: statusd OBSERVES it instead of guessing it from the pane title.
+It walks the pane's process tree in the container's `/proc` (`/proc/<pid>/task/*/children` and the `comm` of each
+process, breadth first from the pane's root, at most four levels and 64 processes, no spawn) and publishes
+`agent.harness` as `claude`, `codex`, `kimi`, `grok` (Grok Build), `grok-cli`, `shell` or `unknown`. The first process
+whose `comm` is `claude`, `codex`, `kimi`/`kimi-code` or `grok`, or whose executable holds `/claude/versions/`, names
+it: native Claude Code before 2.1.299 runs with its version as `comm` (`2.1.295`), so its executable names it
+(DEF-1993; the same rule as `lane-signal`'s and `lane_core`'s). A `grok` is Grok Build when its executable is
+`…/downloads/grok-linux-<arch>` (`readlink /proc/<pid>/exe`), any other is grok-cli; one whose executable cannot be read
+is `unknown`. Only shells is `shell`; anything else, a cut tree or an unreadable read is `unknown`. The answer is kept
+per pane pid (`statusd.HARNESS_CACHE`) and walked again when the pane's own children change or the harness process it
+found exited or now runs under another `comm`; `shell` and `unknown` are never kept.
+
+| Harness | Process | Observed without hooks | `source` |
+|---|---|---|---|
+| Claude Code | `claude` (or a native file `…/claude/versions/<version>`) | `~/.claude/sessions/<pid>.json` | `claude-session-file` |
+| Codex | `codex` (under the npm wrapper `node`) | the pane title and the bell | `title` |
+| Kimi Code | `kimi-code` (or `kimi`) | `~/.kimi-code/sessions/…/state.json` + `agents/main/wire.jsonl` | `kimi-session-file` |
+| Grok Build | `grok`, exe `$GROK_HOME/downloads/grok-linux-<arch>` | `$GROK_HOME/active_sessions.json` + the session's `events.jsonl` and `summary.json`, else its pane title | `grok-session-file`, `grok-title` |
+| grok-cli | `grok`, any other exe | nothing: no title, no bell, no pid in its database | — (hooks only) |
+| any of them, nothing else claiming | — | the process itself: `idle`, `done` with the bell | `process` |
+
+The order statusd takes them in: a hook sidecar > a Claude session file > a Kimi session file > the Grok Build
+observer (its session file, else its title) > the Grok Build title on a pane observed as `grok` > the Codex title (only
+where Codex can be) > the observed harness itself; the window bell turns an idle or finished turn into `done`. When
+the walk names a harness, that is the published `harness` whatever signal decided the status. Check it:
+
+```sh
+lanes doctor | grep '  lanes: '        # lanes: claude 50 · codex 20 · kimi 3 · grok 0 · grok-cli 0 · shell 0
+jq '[.lanes[].agent.harness // empty] | group_by(.) | map({(.[0]): length}) | add' ~/.local/state/lanes/status.json
+```
+
+To see what statusd decided for one lane:
+
+```bash
+python3 -c 'import json,sys; a=json.load(open(sys.argv[1]))["lanes"][sys.argv[2]].get("agent"); print(a)' \
+  ~/.local/state/lanes/status.json <lane id>
+```
+
+`harness` wrong: look at the pane's tree the way statusd does, read-only — with `P=/proc/$(cat
+~/.local/state/lanes/containers/<name>.pid)/root/proc` and the lane's `pid` from status.json (the pane's root
+process), `cat $P/<pid>/task/*/children` lists its children, `cat $P/<child>/comm` names each and `readlink
+$P/<child>/exe` its file. A harness more than four levels below the pane is `unknown`; so is a pane whose tree holds
+anything but shells and no harness. Before 1.5.0 a Kimi lane read `codex / idle`: the title rule took any title of two
+or more words for Codex's (DEF-1970).
+
+### Kimi Code
+
+**Observed.** statusd reads the newest session in `~/.kimi-code/sessions/wd_*/` whose `state.json` `cwd` equals the
+kimi process's working directory (`/proc/<pid>/cwd`). Expect `source: process` instead of `kimi-session-file` when two
+kimi lanes run in the same directory (they cannot be told apart, DEF-1971 — give each its own worktree, or install the
+hooks: each lane's sidecar outranks the session file), when the session has written nothing for an hour, or when
+`~/.kimi-code` or a session directory is a symlink (refused, never followed). Kimi rings no bell by default, so its
+finished turn shows `needs_input`, not `done`, until `kimi-notify` (below).
+
+**Hooks** (installed when the container has `~/.kimi-code/`): `lanes hooks install --agent kimi --container <name>`,
+read the diff, then add `--write`. It writes one marker-delimited `hooks` block of `[[hooks]]` into the container's
+`~/.kimi-code/config.toml` (from `~/.lz/hooks/kimi-hooks.toml`: PreToolUse, PostToolUse, PermissionRequest,
+PermissionResult, UserPromptSubmit, TurnStarted, Stop, StopFailure, Interrupt, SessionStart, SessionEnd →
+`~/.lz/bin/lane-signal … --harness kimi`); running it again replaces that block and nothing else, and the rest of the
+file is never rewritten. Another writer's `[[hooks]]` stay — Throttle's session observer and its other entries
+(`throttle setup --install-clients`: provider `throttle-kimi`) included, even when its writer put them inside the
+block: they are moved just below it. A `hooks` that is not `[[hooks]]` tables refuses. Undo: delete the lines between
+the markers. **The `.kimi-code` no-touch doctrine** (`docs/plans/workspace-session-registry-native-resume.md`,
+`src/provider/kimi_device.rs`) is about the Kimi CLI's device identity — its `~/.kimi-code/device_id` slot, which no
+Lanes or Throttle path reads or writes — not about this block, which is ours (spec §3, recorded).
+
+**Bell, MCP, trace.** `lanes onboard`'s `kimi-notify` step sets `notification_condition = "always"` in
+`~/.kimi-code/tui.toml`, in place in Kimi's own `[notifications]` table (its old value kept in a comment), so its bell
+reaches tmux and the observer marks a finished turn `done`. `lane mcp install` writes `mcpServers.lanes` into
+`~/.kimi-code/mcp.json` (the Claude shape; the environment passes through). Every canonical recipe that sets
+`ANTHROPIC_CUSTOM_HEADERS` also sets `KIMI_CODE_CUSTOM_HEADERS='x-throttle-trace: lanes/<ws>/<id>'`: Kimi Code
+blanks the Anthropic variable.
+
+### Grok Build
+
+**Where it lives.** Lanes expects Grok Build (xAI's official `grok`, 1.0.52 measured) under
+**`GROK_HOME=~/.grok-build`**, never the default `~/.grok`: that directory belongs to grok-cli (superagent-ai) in our
+containers, and Grok Build's installer replaces grok-cli's `~/.grok/bin/grok` with a symlink to its own binary. Every
+lane recipe exports `GROK_HOME=/home/coder/.grok-build`. Install it with `lanes onboard <login> --container <name>
+--step grok-build --write`: that runs, in the container as uid 1000, the stable-channel recipe
+(`https://x.ai/cli/stable`, the `.zst` artifact when `zstd` is there, else the plain one; `downloads/grok-linux-x86_64`,
+the relative link `bin/grok`, `[cli] installer = "internal"`). It edits no shell rc file and links nothing into
+`~/.local/bin`. The channel publishes no checksum: the receipt's `warn` row carries the download's size and sha256,
+marked unverified, for the operator to check (DEF-1997). Both CLIs show `grok` as the pane's command; `readlink
+/proc/<pid>/exe` tells them apart — `~/.grok-build/downloads/grok-linux-x86_64` is Grok Build, `~/.grok/bin/grok` is
+grok-cli.
+
+**Observed** (no hook installed; `libexec/harness_grok.py`, ARCHITECTURE "Grok Build and grok-cli"):
+
+| Signal | Read from | Gives |
+|---|---|---|
+| pid → session | `$GROK_HOME/active_sessions.json` (`session_id`, `pid`, `cwd`, `opened_at`) | the session of the pane's grok |
+| state | the last 64 KiB of `$GROK_HOME/sessions/<URL-encoded cwd>/<session id>/events.jsonl` | `working`, `needs_approval`, `needs_input` (`done` with the bell), `errored`, `idle` |
+| name | that session's `summary.json` `generated_title` | the lane's task line |
+| fallback | the pane title (`⚠ Action Required - …`, a braille spinner, `grok`, `<name> - grok`) | the same words, `source: grok-title` |
+
+statusd hands the observer every pane its walk found running Grok Build, every `shell` pane (a Grok-shaped title over
+a tree with no grok left is that harness ended: `grok` / `shell`) and every `unknown` pane whose `grok` could not be
+told apart by its executable (Grok Build only when `active_sessions.json` names its pid), with one title memory per
+container kept between cycles (`statusd.GROK_TITLES`): the `⚠ Action Required` prefix blinks while the pane is
+unfocused, so a busy title right after a prefixed one still reads `needs_approval` (DEF-1982).
+
+**Hooks:** `lanes hooks install --agent grok --container <name>`, then `--write`. It writes
+`~/.grok-build/hooks/lanes.json` (the whole file is Lanes'; Claude Code's JSON shape, which Grok Build reads from every
+`$GROK_HOME/hooks/*.json`: `Notification` as two groups, `permission_prompt` → `needs_approval` and `idle_prompt` →
+`needs_input`, then `Stop`, `StopFailure`, `StopCancelled`, `SessionStart`, `SessionEnd`, `UserPromptSubmit`,
+`PreToolUse`, `PostToolUse` → lane-signal `--harness grok`) and two blocks in `~/.grok-build/config.toml`. `hooks` sets
+`[compat.claude] hooks = false, mcps = false` — without it Grok Build also runs the hooks in `~/.claude/settings.json`,
+which say `--harness claude`, and registers lane-mcp a second time — and `[ui.notifications] condition = "always",
+method = "bel", events = [turn_complete, approval_required, task_complete, agent_error]`, so the window bell is
+reliable inside tmux. `trace` sets `[model."grok-4.6"]` and `[model."grok-4.5"]` `env_http_headers = {
+"x-throttle-trace" = "LANES_TRACE" }` (quoted ids: a bare `[model.grok-4.6]` is the table `grok-4` → `6`, DEF-1995):
+Grok Build's `[models] extra_headers` is global, one trace for every lane, so recipes export `LANES_TRACE` instead. A
+`[compat.claude]` or `[ui.notifications]` table of the person's refuses the install: set the keys there by hand. A
+model they already configure is left out of `trace` and named. The MCP entry is the `mcp` block `lane mcp install`
+writes (`[mcp_servers.lanes]` with `command` only: Grok Build starts a stdio server with its own environment plus
+`GROK_SESSION_ID`, the session a `lanes_signal` report and every hook then name), a file edit rather than `grok mcp
+add`; `mcps = false` comes with the hooks block.
+
+What needs hooks: the sustained-idle backstop (`Notification` `idle_prompt`), why a turn stopped (`StopFailure`,
+`StopCancelled`) and a reliable bell. Not handled: leader mode (`[cli] use_leader = true`; DEF-1983, accepted) — the
+observer answers `unknown` with a note, and statusd then publishes no agent state for the lane until a hook speaks for
+it.
+
+To confirm a Grok Build lane is joined, in its container:
+
+1. `cat ~/.grok-build/active_sessions.json` — an entry whose `pid` is the pane's grok (`tmux display -p
+   '#{pane_pid}'`, then its child named `grok`). No entry while the lane sits at the welcome screen before a session
+   opens; then only the title speaks.
+2. `ls ~/.grok-build/sessions/` — the cwd's directory (the cwd URL-encoded, e.g. `%2Fhome%2Fcoder%2Fsrc%2Fbw-tools`)
+   holding a directory named by that `session_id` with `events.jsonl` and `summary.json`.
+3. `tail -n 3 ~/.grok-build/sessions/<dir>/<session id>/events.jsonl` — the last `type` names the state (`turn_ended`
+   with its `outcome`, a `phase_changed`, a `permission_requested`).
+
+The observer refuses a `GROK_HOME`, session directory or `events.jsonl` that is a symlink, and ignores a stale entry
+(dead pid, a pid that no longer runs grok, or one that started after the entry was opened). The file shapes come from
+Grok Build's source and user guide; no logged-in session has been measured yet, so a first live lane is worth checking
+against the three steps above (DEF-1981). Do not pipe `grok doctor --json` into `head`: 1.0.52 aborts on the closed
+pipe and leaves a `core` file in the cwd.
+
+### grok-cli
+
+grok-cli has no observable signal at all — no title, no bell, no pid in its database — so a grok-cli lane shows a
+state only through its hooks, and never `needs_approval`: it has no approval event (DEF-1980). Hooks (installed when
+the container has `~/.grok/`): `lanes hooks install --agent grok-cli --container <name>`, then `--write`, merges the
+lane-signal groups into the `hooks` of `~/.grok/user-settings.json` (UserPromptSubmit, PreToolUse, PostToolUse, Stop,
+StopFailure, SessionEnd; no matchers: grok-cli compares a matcher to the tool name exactly). Only that member changes;
+the file holds grok-cli's `apiKey`, and a dry run's diff prints any key, token or secret value as `…`. Its MCP entry is
+the `lanes` item of `mcp.servers` in the same file. grok-cli starts MCP servers with only HOME, LOGNAME, PATH, SHELL,
+TERM and USER, so `lane-mcp` reads `LANE_ID`, `LANE_WS`, `TMUX` and `TMUX_PANE` from grok-cli's own
+`/proc/<pid>/environ`, and only when that parent process is a harness (DEF-1992). grok-cli sends no custom headers: its
+requests carry no trace.
+
+### Throttle-side notes
+
+A trace only matters for a harness attached to a Throttle pool; a running lane gets the recipe's trace variables when
+it is next created.
+
+**The fleet's Kimi row after the update.** The 1.5.0 pulse reports the kimi pool (`/pools/kimi/ready`, `/metrics`,
+and `/pools/kimi/status` when tokened), but an update only ships the new `throttle-pulse` file: `lanes
+install-container` starts nothing while the `pulse` session runs, so the 1.4.0 loop keeps running and the sidebar's
+`◆ moonshot` row says `no data` (`lanes doctor`: `kimi ?`) until it is restarted (DEF-2003): `docker exec -u 1000
+<container> tmux -L lanes-pulse kill-session -t pulse`, then `lanes install-container <container>`.
+
+**Throttle-side items (the Throttle provider lane's; filed, not built in Lanes):**
+
+- DEF-2000 — Throttle's managed MCP entry for Kimi is written to `~/.kimi/mcp.json` (`src/cli.rs`
+  `stage_additional_gateway_user_entries` and its remove twin), a path Kimi Code 2.1.1 does not read: it reads
+  `~/.kimi-code/mcp.json`.
+- DEF-2001 — Throttle synthesizes `cc-session/` and `codex-session/` flows for untraced Claude and Codex requests, but
+  no `kimi-session/` flow: Kimi sends no session id on the Anthropic wire. A Kimi lane's requests join their lane only
+  through the `KIMI_CODE_CUSTOM_HEADERS` trace.
+- Grok: Throttle has no xAI pool, so there is no Grok fleet row and no Grok trace join. The paid-API lane and the
+  subscription pool, and their gates (O-XAI-key, O-XAI-capture, O-XAI-legal, O-XAI-oauth), are the review's
+  (`docs/plans/2026-10-14-xai-grok-pool-review.md`).
+- **Operator note, not a defect:** Kimi Code on productbox is not attached to Throttle — no `[pools.kimi]` in its
+  configuration, port 8095 closed — so its requests go to Kimi directly and the fleet's kimi row reads `0/0`.
+  Attaching it is a Throttle step (`throttle accounts add <name> --provider kimi --pool kimi`, then `throttle setup
+  --install-clients`; the [`throttle-kimi-provider`](../../skills/throttle-kimi-provider/SKILL.md) skill), outside
+  Lanes.
+
 ## The agents' MCP entries (`lane-mcp`, on by default)
 
 `lanes install-container <name>` ships `~/.lz/bin/lane-mcp` (with `lane_core.py`) and adds it to both agents in that
@@ -1022,7 +1259,7 @@ hand.
   the file. Same exit 1 and reason as `lanes hooks install` (DEF-1727). A symbolic link at `~/.claude.json`,
   `~/.codex` or `config.toml` is refused too, also by `lane mcp install`: from the host it would resolve outside
   the container. Add the entry to the file it points at by hand.
-- Inside the container: `~/.lz/bin/lane mcp install [--agent claude|codex|all]` prints the diff, `--write` applies it.
+- Inside the container: `~/.lz/bin/lane mcp install [--agent claude|codex|kimi|grok|grok-cli|all]` prints the diff, `--write` applies it (`all`, the default: Claude Code, Codex and each of Kimi Code, Grok Build and grok-cli whose home is there; a harness named on its own is written even without its home).
 - New agent sessions load the server; a running one keeps its tools. Check from a lane with the agent's own MCP
   listing (`/mcp` in Claude Code). The server's log is `~/.lz/s/logs/lane-mcp.log` (one line per request, rotated
   at 1 MiB; never a capture's text, a sent text or a note).
@@ -1211,7 +1448,7 @@ Settings (environment of the installer — `install/services.py` renders them in
 | Variable | Default | Meaning |
 |---|---|---|
 | `LANES_THROTTLE_CONTROL` | `http://127.0.0.1:19181` | control base URL, resolved in the poller's network namespace (see above); `none` sends no request at all (the log fold still runs) |
-| `LANES_THROTTLE_TOKEN_FILE` | unset | file holding a bearer; a loopback daemon without `remote_key` needs none. Never written to `economy.json` or the cursor |
+| `LANES_THROTTLE_TOKEN_FILE` | unset | file holding the control bearer (the daemon's `remote_key`, or the local client token `lanes throttle adopt-token` copies). `/status` and `/sessions` are guarded control routes: without it they answer `HTTP 401`, loopback or not — the control port has no loopback exemption (DEF-1930). Never written to `economy.json` or the cursor |
 | `LANES_THROTTLE_CONTAINER` | unset | the container whose home volume holds the daemon's log; the log dir becomes its `/home/coder/.local/share/throttle` (via `containers.json`) |
 | `LANES_THROTTLE_LOG_DIR` | `~/.local/share/throttle` | where `proxy.log` and `proxy.log.<day>` are read (read only); wins over `LANES_THROTTLE_CONTAINER` |
 
@@ -1224,6 +1461,157 @@ when it recovers; the reason is never written to `economy.json`. `economy.json`
 the next poll re-folds today. For a look without the daemon: `LANES_THROTTLE_CONTROL=none lanes economy --once
 --dry-run --status-file status.json --sessions-file sessions.json` with saved documents.
 
+## Fleet block and Throttle pulse: operator steps (1.4.0)
+
+1.5.0 adds the Kimi pool to the pulse and the block (`◆ moonshot`): after that update restart the pulse once, as
+"Harnesses" above says (DEF-2003); everything else below is unchanged.
+
+The sidebar's fleet block (user guide "Fleet") needs nothing from you to show the host and its containers. Its Throttle
+rows need the pulse running in the daemon's container, and its money and burn figures need one opt-in. In order (the
+reference follows: "Throttle pulse" for the container side, "Fleet facts" for what statusd publishes):
+
+1. **Install 1.4.0.** On an npm-installed server with `lanes-update.timer` on, the publish is the deploy: the update
+   restarts statusd (the new `fleet`/`throttle` keys) and re-runs `lanes install-container` for every container `lanes
+   sync` recorded, and for the daemon's container when sync did not record it, which ships `~/.lz/bin/throttle-pulse`
+   and starts its loop in the daemon's container. Otherwise: `npx @vibelord/lanes@1.4.0 --server --update-only`, or
+   `lanes update --now`.
+2. **Check the pulse.** `lanes doctor` prints `throttle pulse: 4 s old (container <c>), daemon up, tokened: no — lanes
+   throttle adopt-token <c>`. `none in status.json` means no pulse yet: the daemon's container is not one `lanes sync`
+   recorded (not on a board), or it is not the one Lanes picked (the Throttle tab's container) — put
+   `LANES_THROTTLE_CONTAINER=<c>` into the server's `~/.config/lanes/site.conf` (the one place Lanes reads it for the
+   pulse: `lanes install-container`, `lanes doctor` and every update's re-ship, re-runs and rollbacks included; never
+   the environment) and run `lanes install-container <c>` once. A name there (or a Throttle tab) whose container docker
+   no longer knows, or that has no home volume, is skipped by every update with a warning (`WARN  install-container <c>
+   (skipped: docker has no such container)`; the result stays `ok`): correct or remove the line. `stale` (the pulse's
+   `at` more than 90 s old, the bound statusd uses too) means the loop stopped: statusd starts it again within 10
+   minutes ("The re-arm" below, in its journal), or run `lanes install-container <c>`. `daemon down (connection
+   refused)` is the daemon, not the pulse.
+3. **Opt in to money and burn (optional).** `lanes throttle adopt-token <c>` copies the container's own local Throttle
+   client token into `~/.lz/throttle-token`; the pulse then reads the guarded `/pools/<pool>/status` ("The token"
+   below). Every guarded control route needs that bearer, loopback included (DEF-1930): there is no token-free way to
+   read spend. To opt out, delete the file; the next pulse says `tokened: no`.
+4. **The dashboard link (only after an authenticated Grafana).** Put `LANES_FLEET_DASHBOARD_URL=https://<host>[:port]
+   [/path]` into the server's `~/.config/lanes/site.conf` once the fleet-management Grafana (`christian-fleet`) is
+   behind authentication — never before: the link would hand every tailnet viewer the URL. No Lanes command writes
+   it. `lanes doctor` prints the URL, `unset`, or `⚠ … ignored — not an https URL`; a query string, a user part or a
+   fragment is refused too.
+5. **Laptops with a client from before 1.2.0** do not update themselves: their owner runs `npx @vibelord/lanes@1.4.0`
+   once. Newer clients ask `lanes hello --client-version` and install 1.4.0 within 6 h of the server running it.
+
+The pulse only reads (GET, no redirect) and writes one file; it never changes the daemon. Removing it: stop the loop
+as the user it runs as (`docker exec -u 1000 <c> tmux -L lanes-pulse kill-server`) and delete
+`~/.lz/bin/throttle-pulse`, `~/.lz/state/throttle.json` and `~/.lz/throttle-token` in that container; the block's
+Throttle rows then show the stale mark. The pulse is part of the container contract, so the next update ships it and
+starts it in the daemon's container again.
+
+## Throttle pulse
+
+The Throttle daemon binds its own container's `127.0.0.1` (DEF-1525), so the host cannot ask it anything. The pulse
+asks from inside: `~/.lz/bin/throttle-pulse` (Python 3 stdlib), shipped to every workspace container by `lanes
+install-container` with the rest of the contract, runs as a loop **only in the daemon's container** and writes what
+it learns to `~/.lz/state/throttle.json` there, which `lanes-statusd` reads through `/proc/<init>/root` for the
+sidebar's fleet block.
+
+**Which container.** `LANES_THROTTLE_CONTAINER=<name>` in `~/.config/lanes/site.conf`, else the container of the
+registry's Throttle Monitor tab (`tools--throttle-monitor`, the container that hosts most boards' monitor). `lanes
+install-container <that container>` starts the loop; for any other container it ships the file and says
+`throttle pulse: installed, not started here — …`.
+
+**How it runs.** A tmux session `pulse` on the socket `lanes-pulse`, as uid 1000 with `HOME=/home/coder`:
+
+```sh
+docker exec -d -u 1000 -e HOME=/home/coder <container> tmux -L lanes-pulse new -d -s pulse /home/coder/.lz/bin/throttle-pulse --loop
+```
+
+`lanes install-container` asks `tmux -L lanes-pulse has-session -t =pulse` first and starts nothing when the session
+runs (`throttle pulse: already running in …`); a start docker refuses is printed and makes the exit 1, after
+everything else was installed. To restart it after an update, end the session and install again:
+`docker exec -u 1000 <container> tmux -L lanes-pulse kill-session -t pulse`, then `lanes install-container
+<container>`. Its pane holds the journal: `docker exec -u 1000 <container> tmux -L lanes-pulse capture-pane -p -t
+pulse` (one line each time the reason a source failed changes, `ok` when it recovers; never the token).
+
+**What it reads.** Every 10 s, GET only, on `http://127.0.0.1:$LANES_THROTTLE_CONTROL_PORT` (19181 when the
+container's environment does not set it), 3 s timeout, at most 1 MiB per answer, no redirect followed, no proxy:
+
+| Route | Guarded | Gives |
+|---|---|---|
+| `/pools/claude/ready`, `/pools/codex/ready`, `/pools/kimi/ready` | no | configured / usable / exhausted / auth-failed / benched subscriptions, overflow ready (a `503` is a pool that is not ready; its body is read). A daemon with no Kimi account answers the kimi route with nothing configured; one from before its Kimi pool answers `404` (journalled; the other pools are read) |
+| `/metrics` | no | `throttle_pool_pressure_phase`, `throttle_pool_spill_fraction`, `throttle_overflow_in_flight`, `throttle_upstream_429_total` per pool |
+| `/pools/claude/status`, `/pools/codex/status`, `/pools/kimi/status` | yes | per pool: the worst measured 7-day and 5-hour utilization over its subscription accounts, the soonest reset; the overflow spend, enforced daily budget, fraction and alert level — for claude and codex only: Kimi has no paid overflow lane, so `overflow.kimi` stays `null` whatever its status renders |
+
+`cooling` is what is left of `configured` after usable, exhausted, auth-failed, disabled, billing- or
+organization-blocked and unhealthy accounts (the daemon's buckets overlap; floored at 0).
+
+**The token (opt-in, never automatic).** The `status` routes are guarded: the daemon's ControlAuth answers `401`
+to any request without its bearer, loopback included (DEF-1930). Without a token the pulse reads the open routes only
+and the burn and money fields stay `null`. The operator opts in once per daemon container:
+
+```sh
+lanes throttle adopt-token <container>
+```
+
+It copies `env.ANTHROPIC_API_KEY` from that container's own `~/.claude/settings.json` — the LOCAL proxy bearer
+`throttle setup --install-clients` wrote there, which must start `throttle_`; it is never a provider credential — to
+the container's `~/.lz/throttle-token` (0600, by rename, never through a link) and prints `adopted: …` or why not,
+never the value. The pulse picks it up on its next cycle and sends it only to the `status` routes. It uses a
+token file that is not a regular file or is readable by group or other (mode not 0600) as no token, and says so in
+its journal. To withdraw the opt-in, remove the file: `docker exec -u 1000 <container> rm ~/.lz/throttle-token`. After
+`throttle setup` changes the client token, adopt it again.
+
+**How to see it.**
+
+```sh
+docker exec -u 1000 <container> cat /home/coder/.lz/state/throttle.json
+docker exec -u 1000 -e HOME=/home/coder <container> /home/coder/.lz/bin/throttle-pulse --once   # one cycle, printed; exit 1 when the daemon did not answer
+```
+
+`{"v": 1, "at", "up", "tokened", "pools": {"claude", "codex", "kimi"}, "overflow": {"claude", "codex", "kimi"},
+"error"}` (the spec §2.1 shape, with W12's kimi pool; DM-LANES-THROTTLE-PULSE). The file is rewritten only when a value changed or 60 s passed, so `at` is a
+heartbeat: an `at` more than 90 s old means the loop stopped (a healthy pulse is read up to about 84 s after its
+`at`: the 60 s heartbeat lands on the next 10 s cycle, DEF-1941). `up: false` with `error` (`connection refused`,
+`timed out`, `HTTP 500`, …) means no open route answered; the pools then keep their last values. Utilizations and
+fractions are 0–1, rounded to three places; resets are epoch seconds.
+
+## Fleet facts (`fleet` and `throttle` in status.json, W11)
+
+statusd publishes two additive blocks in `~/.local/state/lanes/status.json` that the sidebar's fleet block and
+`lanes doctor` read (they read nothing else for it). No collector, no new service, no new spawn per cycle.
+
+**`fleet` — the host and its biggest containers.** Every `FLEET_INTERVAL` (10 s) statusd reads, from plain files:
+the `cpu` line of `/proc/stat`, `MemTotal`/`MemAvailable` from `/proc/meminfo`, `/proc/sys/fs/inotify/max_user_watches`
+(the ceiling only — watches in use are not counted in the loop), the PSI and load it reads every cycle anyway, and
+per container recorded in `containers.json` its cgroup `memory.current` and `cpu.stat` `usage_usec`. Rates come from
+two samples 10 s apart, so right after a statusd restart `system.cpu.utilization` and every `container.cpu.percent`
+are `null` for one interval. `container.cpu.percent` is of the whole host (`usage_usec` delta ÷ interval ÷ logical
+cpus × 100): 12 % of a 64-cpu host is about 8 cpus busy. `containers` holds the three biggest by memory. Figures are
+quantized (1 %, 0.1 GiB, 0.1 load), so `status.json` is rewritten only when one of them visibly moves or at the 15 s
+heartbeat. On a `critical` host no sample is taken: the last facts stay, `"stale": true`, until the host recovers.
+
+```sh
+jq .fleet ~/.local/state/lanes/status.json            # the facts; "stale": true = not sampled since the host went critical
+jq '.fleet.containers' ~/.local/state/lanes/status.json
+```
+
+**`throttle` — the Throttle pulse's last document.** The pulse ("Throttle pulse" above) writes
+`~/.lz/state/throttle.json` in the daemon's container and statusd copies it: one `stat` per cycle through
+`/proc/<init>/root`, a read only when the file moved, at most 64 KiB, never through a link. statusd names the daemon's
+container as the pulse section's "Which container" says (the tab's container: `jq '.tools'
+~/.local/state/lanes/registry.json`) and looks it up again after a `lanes sync` that changed the registry and every 5
+min (edit site.conf, then wait or `systemctl --user restart lanes-statusd`). No container known = no `throttle` key at
+all. `"stale": true` means the pulse's `at` is more than 90 s old (a healthy pulse rewrites every 60 s and is read up
+to about 84 s after its `at`, so 60–90 s is normal; `lanes doctor` and the sidebar use the same 90 s, DEF-1941), the
+file is absent, or it was refused (a link, a FIFO, more than 64 KiB, no v1 document) — the last values are kept and
+shown dim.
+
+**The re-arm (the only spawn this adds).** When the pulse file is older than 90 s (or missing) while the daemon's
+container runs and the host is not `critical`, statusd — between two cycles, never inside one — runs the start command
+of "How it runs" above, byte for byte (DEF-1940), at most once per 10 min and bounded to 10 s, and journals it
+(`journalctl --user -u lanes-statusd | grep 'throttle pulse'`: the file's age, `re-armed (exit N)` and the argv). It
+starts nothing when `~/.lz/bin/throttle-pulse` is not installed in that container (`lanes install-container <c>` ships
+it), and it never runs on a `critical` host — a pulse that keeps going quiet on a healthy host is a pulse that
+crashes: run it by hand in the container (`~/.lz/bin/throttle-pulse` without `--loop`, as "How to see it" above
+shows) and read its stderr.
+
 ## Push notifications (opt-in)
 
 `lanes-notify.service` sends one push to an [ntfy](https://ntfy.sh) topic when a lane's agent ENTERS needs-you —