agentus 0.1.0 → 0.2.0

package/docs/dev/02-verify.md

@@ -0,0 +1,237 @@
+# Verifying a change
+
+This doc answers: how do I prove a change to Agentus actually works, and which check is the
+right size for which change?
+
+## Pick the cheapest evidence that can fail
+
+Not every change needs a browser. Reach for the smallest check that can actually go red:
+
+| change | evidence |
+|---|---|
+| a pure formatter / decision function | a `*.mts` unit sweep (runs under `npx tsx`, needs no build, no browser) |
+| server logic / a REST contract | a `node scripts/*.mjs` suite against a scratch server |
+| rendered UI behaviour or geometry | a CDP page sweep (`scripts/qa/*.mjs`) |
+| a release | the promote log + the served bundle (see [03-release.md](./03-release.md)) |
+| a doc / config edit | it is done when it is written — no verification run |
+
+A change with no runtime effect is finished when it is written; an unasked-for sweep just
+spends the round.
+
+## Unit suites and smoke suites
+
+### `npm test`
+
+`package.json` maps `test` to `npm run test -ws --if-present`, i.e. "run each workspace's
+own `test` script if it has one". As of this writing **no workspace defines a `test`
+script**, so `npm test` runs nothing. The runnable suites are the scripted ones below —
+run them directly. `TODO(verify)`: this placeholder is expected to gain real suites.
+
+### Server / API smoke suites (each boots its own scratch server on a random port `8900+`)
+
+```bash
+npm run typecheck                                        # 0 errors before any commit
+npm run build                                            # web bundle + shared types
+node scripts/auth-smoke.mjs                              # the lock, both credentials (~48 assertions)
+node scripts/backend-registry-smoke.mjs                  # backend registry, spawn env, health (~42)
+node scripts/workspace-smoke.mjs                         # workspace, file API, shell, attachments (~85)
+node scripts/voice-smoke.mjs                             # settings/theme guards, voice router (~47)
+npm run tls-smoke                                        # the TLS listener, real sockets + real certs
+npm run package-smoke                                    # pack + install into a throwaway HOME + boot
+node scripts/smoke.mjs mock "hello"                      # end-to-end against the mock agent
+```
+
+`backend-registry-smoke` is shaped as three layers so it passes on a machine without the
+operator's CLI: always-on (registry CRUD, spawn env, session health via the in-repo mock),
+local-only checks guarded by "does this command resolve / does that tree exist" and
+announced as `skip`, and a summary line reporting the skip count. A suite that needs a
+local binary must **skip**, never fail.
+
+### `*.mts` unit sweeps (importable TS, no build, no browser)
+
+```bash
+npm run transcript-fold   # tsx scripts/qa/transcript-fold.mts
+npm run store-fold        # tsx scripts/qa/store-fold.mts
+npm run live-stream       # tsx scripts/qa/live-stream.mts   real server frames -> real store
+npm run resume-tail
+npm run rail-recent
+npm run turn-pulse        # the silence->amber decision, with injected `now`
+npm run sentence-units
+```
+
+Plus non-aliased ones run with `npx tsx scripts/qa/<name>.mts`: `accent-text.mts` (the
+accent-as-text derivation, 61 checks), `presence-rule.mts` (`watchingNow`), `open-target.mts`,
+`plan-v2-sweep.mts`, `asr-idle-wedge.mts`. These are the right home for a **time- or
+boundary-driven decision**: extract it into a pure function with an injected `now` and
+assert either side of the boundary (59s vs 60s, `2 < 10 < 100`), because a browser cannot
+be made to wait for a 60-second window reliably.
+
+## The CDP browser sweeps
+
+`scripts/qa/*.mjs` drive a **real browser** over the Chrome DevTools Protocol and read
+computed styles and geometry from the rendered page — never assert on internals. They run
+by hand and are not in CI.
+
+### Start the debug browser
+
+Launch Edge or Chrome with remote debugging on `127.0.0.1:9222`:
+
+```bash
+open -a "Microsoft Edge" --args --remote-debugging-port=9222
+# or: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
+```
+
+### Run a sweep
+
+```bash
+cd worktrees/agentus-dev
+PORT=8901 node scripts/qa/layout-sweep.mjs
+PORT=8901 W=390 H=844 node scripts/qa/popover-sweep.mjs
+```
+
+Each sweep opens its **own tab** (`PUT http://127.0.0.1:9222/json/new?<BASE>`), builds its
+own page state, and defaults to the dev port (`8901`, overridable with `PORT` / `BASE`).
+Sweeps that post REST calls want the QA data dir: `AGENTUS_DATA=/tmp/agentus-qa-account`.
+
+**Rebuild before you sweep a web change.** The dev server serves `packages/web/dist`, so
+after editing web sources run `npm run build` (or `npm run build -w @agentus/web`) first, or
+the sweep measures the previous bundle.
+
+**Page-driving sweeps on `:8901` must log in themselves.** The operator's Edge profile
+holds the cookie for the live port, not the dev one, so an unattended tab renders the login
+wall and every step reads as a 401-induced "product failure". Standard opening:
+`POST /api/auth/login` as `scratch` / `scratch-pass-1` (the `DEV_USER`/`DEV_PASS` defaults),
+then reload. Assert the page is authenticated before measuring.
+
+**Run one browser sweep at a time.** Two sweeps share this box's CPU and the one CDP
+browser; the starved one fails in a cluster of unrelated places. Re-run a red suite alone
+before believing it. And do not run a sweep in the same breath as a branch switch or any
+server-source edit — the dev watcher restarts, the page's WebSocket drops mid-run, and one
+assertion fails at random.
+
+### The exit-code discipline (or node hangs)
+
+End every sweep with:
+
+```js
+ws.close();
+process.exit(fail ? 1 : 0);
+```
+
+A sweep that only sets `process.exitCode` prints its summary and then **hangs forever on
+the open CDP WebSocket** — and a shell chaining two sweeps with `;` never reaches the
+second one. Measured: three sweeps sat "still running" 8–25 minutes after their last check,
+each holding a tab, which then made the next sweep time out. When a suite "hangs after
+passing", look for the missing exit before hunting a product bug.
+
+### Traps worth knowing before you write one
+
+- **`Runtime.evaluate` needs `awaitPromise: true`**, or a `fetch(...)` expression comes back
+  as a serialised pending Promise (`{}`) — truthy, so a step "succeeds" while nothing was
+  created.
+- **`Runtime.evaluate` must return something serialisable.** `returnByValue: true` on an
+  expression that evaluates to a DOM node fails the whole call (`Object reference chain is
+  too long`). Wrap it: `Boolean(document.querySelector('.sidebar'))`.
+- **Enter needs `text:'\r'`.** `Input.dispatchKeyEvent` with only `key:'Enter'` / `code`
+  delivers the event but the app's `keydown` handler never sees it. Add `text:'\r'`.
+- **A trusted click lands where the viewport says.** If the target is scrolled out of the
+  viewport the event is silently swallowed — `scrollIntoView({block:'center'})`, re-measure,
+  assert `top >= 0 && bottom <= innerHeight`, then click.
+- **The oracle comes from the page, not from node.** The cockpit is authenticated, so a
+  node-side `fetch('<BASE>/api/sessions')` returns 401 and every "UI value == server value"
+  comparison silently becomes "== undefined". Fetch from inside the page:
+  `ev("fetch('/api/sessions').then(r=>r.json())")`.
+- **The app consumes `?session=` and strips it** (`state.ts #openTarget` → `replaceState`),
+  so a later `location.search` read is always null — keep the id in a variable. And
+  `Page.reload` wipes injected page helpers; re-inject after every reload.
+- **Never wrap `window.WebSocket`.** Installed via `Page.addScriptToEvaluateOnNewDocument`
+  it replaces the constructor the app uses and the cockpit's own socket never connects.
+  Patch `WebSocket.prototype.send` after the app is up instead.
+- **Token-injection cannot reach the WS handshake.** Injecting the machine token as an HTTP
+  header (via `Fetch.continueRequest` with `headers` as an **array** of `{name,value}`)
+  authenticates REST but not `/ws`, and the app's WS URL carries no token. That is enough
+  to prove REST-fed surfaces on the live instance but **not** WS-fed ones like the session
+  rail's live rows — say which you proved and which you did not.
+
+### Why a visual rule needs an injected fixture row
+
+The dev instance's mock emits **no tool calls for an ordinary prompt** (see
+[01-dev-loop.md](./01-dev-loop.md)). A sweep whose subject is a `.tool-card` will therefore
+never produce one and spends its time failing. When the assertion is about the CSS, inject a
+probe with the **real class names** into the real container, measure it, remove it, and
+assert the count is unchanged (`0 → 0`) so a leaked probe cannot pass:
+
+```js
+const mk = (html) => { const d = document.createElement('div'); d.innerHTML = html;
+  const el = d.firstElementChild; stream.appendChild(el); return el; };
+const row = mk('<div class="msg"><div class="tool-card completed">…</div></div>');
+const box = row.querySelector('.tool-card').getBoundingClientRect();
+row.remove();
+```
+
+Say in the check's label which rows are real and which are probes. An operator who judges a
+layout by eye still needs a screenshot; a probe that only the script saw is not verified.
+
+### Writing a new sweep from an existing one
+
+Copy the closest existing sweep and keep its scaffolding (identical in every file):
+
+1. the CDP client (`send` / `ev` / `until` helpers over a `WebSocket`),
+2. the `check(ok, what, detail)` counter and the pass/fail summary,
+3. `PUT /json/new?<BASE>` to open the tab, `sleep(~2600)` to let the app boot,
+4. login-as-scratch if the surface needs a live socket,
+5. build the page state you need (create a **fresh** mock session rather than adopting one),
+   and reset any preference you depend on (both `localStorage` and the server setting),
+6. `ws.close(); process.exit(fail ? 1 : 0)`.
+
+Prefer a new mock `[trigger]` prompt over editing a shared fixture constant — the mock is
+spawned per session, so a new session picks up the edit with no server restart, and the
+assertion then drives a real agent bubble instead of HTML the sweep injects.
+
+## Table of existing sweeps
+
+The QA directory is large; these are the ones that carry each surface, with what each covers.
+
+| sweep (`scripts/qa/`) | covers |
+|---|---|
+| `layout-sweep.mjs` | rail indent ladder + folder icon + directory colour, accent-neutral split, three-pane widths with real pointer drags, chat column edge alignment, content-width rows (short tool card not stretched), phone 390px — 58 checks |
+| `rail-archive-sweep.mjs` | the 已归档 section: built like 工作空间 (same row shape, folder icons, grouped by directory), a fold flag that survives a reload, and the two sections keeping separate fold state for the same path — 15 checks |
+| `terminal-sweep.mjs` | the workspace panel's terminal is a real pty, not a text box: xterm mounted, a prompt only a tty prints, the shell's own echo of what was typed, an SGR colour surviving to the screen, and a resize that reaches the kernel (`stty size` + SIGWINCH) — 15 checks |
+| `queue-sweep.mjs` | the message queue: enqueue while a turn runs, the row is not in the transcript, the three buttons' order, delete/edit, automatic ordered drain on `turn-end` — 28 checks |
+| `table-sweep.mjs` | table headers sort three ways (numeric, not string), filter, full-screen reader with content rotation on a phone, a file chip in the reply, a relative image — real agent bubble via `[table]` |
+| `rail-contrast-sweep.mjs` | sidebar text contrast computed per element against its actual background, in both palettes with the operator's blue and the built-in amber; the accent-vs-neutral chroma gap |
+| `render-files-sweep.mjs` | local file links / images in a reply (all four forms), the workspace panel previewing and saving by type, the raw endpoint behind auth |
+| `popover-sweep.mjs` | every popover's geometry at both viewports (a popover can pass every DOM text assertion while being off-screen) |
+| `rail-sweep.mjs`, `rail-live-sweep.mjs`, `rail-recent-sweep.mjs` | the session rail: one title per row, row geometry, the ⋯ menu, inline rename landing in DOM *and* API, the phone bottom sheet, rail ordering |
+| `message-actions-sweep.mjs` | per-message copy (text compared character-for-character), fork-on-tail |
+| `permission-sweep.mjs` | a permission request reaches the operator's view (card + dialog), names the file, shows the diff, the answer reaches the agent, ≥32px tap targets |
+| `voice-stop-sweep.mjs`, `call-sweep.mjs` | the speech layer and the full-screen call: deliberate stop vs real failure, the hero's real animation, phases, barge-in, controls on screen at 390×844 |
+| `session-title-sweep.mjs`, `session-restart-sweep.mjs` | naming over API + browser; swapping an agent child under a live slot without it leaving the live list |
+| `transcript-sweep.mjs`, `window-ui-sweep.mjs`, `turn-line-sweep.mjs`, `turn-pulse-sweep.mjs` | transcript rendering/paging/gestures, the usage strip, the turn timer and the silence→amber state |
+| `config-sweep.mjs`, `dismiss-sweep.mjs`, `model-switch-sweep.mjs`, `plan-*-sweep.mjs` | the config/plan/model surfaces |
+| `live-endpoints-check.mjs` | the file endpoints over the machine token; same source runs against live **and** dev |
+| `think-cadence.mjs`, `render-lag-sweep.mjs`, `replay-window.mjs` | streaming cadence, render lag, replay windows |
+
+Unit (`*.mts`) sweeps: `accent-text`, `transcript-fold`, `store-fold`, `live-stream`,
+`resume-tail`, `rail-recent`, `turn-pulse`, `sentence-units`, `presence-rule`, `open-target`,
+`plan-v2-sweep`, `asr-idle-wedge`. There are also `notify-smoke.mts` (the notify contract,
+65 → 72 checks) and `notify-cockpit-e2e.mts` (a real agent turn end-to-end) for the phone
+channel; see [`docs/android-notify-contract.md`](../android-notify-contract.md).
+
+## Assertion discipline
+
+Every sweep assertion must be able to prove **this run** caused the observed state:
+
+- use a water mark (`createdAt >= runStart`) rather than "the newest row in that directory";
+- establish `cwd` for any seeded fixture (`POST /api/sessions` refuses a `cwd` that is not a
+  directory) and assert the fixture exists before asserting behaviour;
+- set what you depend on first — a leftover server preference from an earlier run turns
+  unrelated checks red;
+- judge the effect on the far side ("the button cleared our state" is not "the answer
+  reached the agent" — read the agent's own tool output);
+- when a fix is claimed, re-run the suite whose surface you touched: a change to the harness
+  invalidates every green it produced.
+
+Cross-links: [01-dev-loop.md](./01-dev-loop.md) (ports, mock triggers) ·
+[03-release.md](./03-release.md) (proving a release) ·
+[04-recover.md](./04-recover.md) (stray tabs, leaked processes).

package/docs/dev/03-release.md

@@ -0,0 +1,218 @@
+# Releasing to the live instance
+
+This doc answers: how do I move a verified change from the dev tree onto the live instance
+the operator uses, and how do I undo it?
+
+## The rule that governs every release
+
+`promote` **restarts the live server when server-side code changed**, and a restart SIGTERMs
+every live `hermes acp` child — every session the operator has open, including his current
+conversation. It is a release, not a hot reload. So:
+
+- releases happen only after the sweeps are green and, unless the change is strictly
+  web-only, **only when the operator says so** in that round;
+- a strictly web-only change may be published without asking first — but "web-only" is
+  decided by what `promote` prints, not by which files you remember editing (below).
+
+## Branches
+
+Work happens on a branch cut in the dev tree:
+
+| branch | use |
+|---|---|
+| `feat/<slug>` | a feature |
+| `fix/<slug>` | a fix (e.g. `fix/rail-readable-tokens`) |
+| `run/<slug>` | created **by `promote` on the live tree** — never check it out yourself |
+| `main` | the integration line; **not** what live runs from |
+
+`TODO(verify)`: the project also refers to a `fix/<base>/<desc>` convention; the branches
+observed in the tree are `feat/<slug>` and `fix/<slug>`.
+
+Keep the dev tree on its own branch. `promote` publishes the dev tree's **current HEAD**,
+whatever branch that is — leaving it parked on a stale branch publishes a rollback with no
+warning. `promote` guards against the common case (HEAD strictly behind `origin/main` → it
+refuses; override with `DEV_PROMOTE_FORCE=1` for a deliberate rollback), but it will not
+know that you meant to be on another branch.
+
+## Committing
+
+Commit and push the dev branch **before** promoting: `promote` refuses an uncommitted tree
+(`the dev tree has uncommitted changes — commit them first`), and the pushed branch is what
+makes the round recoverable if the release goes wrong.
+
+**Write the commit message to a file and use `-F`.** A multi-line message with inline
+backticks (`` `--text-dim` ``) or `$`/`!` triggers shell command substitution inside
+`-m "…"`, and the substitutions silently vanish from the message:
+
+```bash
+$EDITOR /tmp/msg.txt          # backticks, $ and ! are literal here
+git commit -F /tmp/msg.txt
+```
+
+## Promoting
+
+```bash
+bash scripts/dev.sh promote [<slug>]
+```
+
+If `<slug>` is omitted, the branch name is used with `/` → `-`. What it does:
+
+1. **Refuses** an uncommitted dev tree, and refuses HEAD-behind-`origin/main` (unless
+   `DEV_PROMOTE_FORCE=1`).
+2. Runs **detached** — because the restart it may cause kills the agent that ran it — and
+   writes its log to `/tmp/agentus-qa-account/promote.log`. Follow it with:
+   ```bash
+   bash scripts/dev.sh promote-log      # follows the log (it hangs until the run ends)
+   ```
+   or read `/tmp/agentus-qa-account/promote.log` directly.
+3. Checks the **live** tree out onto `run/<slug>` at the dev tip and rebuilds the web bundle.
+4. Decides whether the live process must restart, from the diff between the live tree's old
+   tip and the promoted HEAD:
+
+| trigger | files |
+|---|---|
+| server graph | `packages/server/src/**`, `packages/shared/src/**` |
+| start shape | `scripts/start.sh`, `package.json` |
+| stale launcher | the live process is still running under `tsx watch` |
+
+5. If a restart is needed, calls `scripts/relaunch.sh --port 8787 --tree <live>
+   --launcher <live launcher>`, which TERMs the process group, waits for the port to free,
+   starts the launcher, polls `/healthz`, and **asserts the pid changed** (a failure exits
+   non-zero, so "built fine, still serving old code" cannot pass).
+6. Resumes every live session it captured **before** the restart
+   (`POST /api/sessions/<id>/resume`, expected HTTP 200 each).
+
+### Web-only release (no restart)
+
+When nothing in the trigger list moved, `promote` prints `no server-side change` and the
+running server keeps its process. The new bundle is served on reload, so **no session is
+interrupted at all** — the operator only hard-refreshes his tab. The three success criteria
+are all required:
+
+- `no server-side change` printed,
+- the **asset hash moved** (`web asset: <before> -> <after>`; `promote` warns if web sources
+  changed but the hash did not),
+- the **pid did NOT move** (`lsof -ti:8787` unchanged — a moved pid means it took the
+  restart branch and the release was not the kind it claimed to be).
+
+`package.json` is a restart trigger, so **any** edit to it — even one new npm alias for a QA
+sweep — turns a web-only round into a restarting one. When a web change rides on a dev
+branch that also carries unreleased server work, cut the release from the live tip instead:
+
+```bash
+cd worktrees/agentus-dev
+git checkout -B feat/<slug>-live <live-tip>     # live tip = git -C ../agentus rev-parse HEAD
+git cherry-pick <web-commit>
+git diff --name-only <live-tip> HEAD            # MUST list only packages/web/** (+ scripts/qa/**)
+env -u AGENTUS_DATA -u AGENTUS_BASE -u AGENTUS_PORT -u DEV_* bash scripts/dev.sh promote <slug>
+```
+
+Also drop `package.json` back to the live tip's version before promoting (`git checkout
+<live-tip> -- package.json && git commit --amend --no-edit`) so the alias stays out of the
+promoted tree, and run the sweep directly (`npx tsx scripts/qa/x.mts`) where the alias is
+absent.
+
+### Never fire the release from the dev environment
+
+The shell that runs `promote` usually carries the QA instance's variables
+(`AGENTUS_DATA=/tmp/agentus-qa-account`, `AGENTUS_BASE=…:8901`, `DEV_*`), and
+`promote → relaunch.sh → launch.py` inherits them. The launcher now pins its own identity,
+but wrap it anyway to keep the environment (and the log) honest:
+
+```bash
+env -u AGENTUS_DATA -u AGENTUS_BASE -u AGENTUS_PORT \
+    -u DEV_LOG_PATH -u DEV_PROMOTE_DETACHED -u DEV_PROMOTE_SELF -u DEV_PROMOTE_SLUG \
+    bash scripts/dev.sh promote <slug>
+```
+
+## Verifying what actually got deployed (with no password)
+
+The promote log is one leg; prove the *served* instance independently. Everything below
+needs no browser and no operator password.
+
+**1. The machine token.** Each instance mints one at `<DATA_DIR>/auth.token` (0600). The
+server accepts it as `Authorization: Bearer <token>` **or** `?token=<token>`. Read it inside
+a script and never print it — and do not write the literal `Bearer <value>` shape into a
+file, because the write-time secret masker rewrites it to `***` and breaks the syntax.
+Assemble the header at runtime.
+
+```bash
+TOK=$(cat /tmp/agentus-qa-account/auth.token)          # dev;  live: worktrees/agentus/packages/server/.data/auth.token
+curl -s --noproxy '*' -H "Authorization: Bearer $TOK" http://127.0.0.1:8901/api/sessions | head -c 300
+```
+
+**2. The served asset hash.** `GET /api/version` is public and returns the served bundle
+hash. `promote` echoes it; you can also read it directly:
+
+```bash
+curl -s --noproxy '*' http://127.0.0.1:8787/api/version
+```
+
+**3. Grep the served bundle for the new markers.** Fetch the real vite-named file
+(`assets/index-<hash>.js`) and look for a string only the new code has:
+
+```bash
+HASH=$(curl -s --noproxy '*' http://127.0.0.1:8787/api/version | ...)
+curl -s --noproxy '*' "http://127.0.0.1:8787/assets/$HASH.js" | grep -c '<new-marker>'
+```
+
+Two traps: the file is `index-<hash>.js`, so a check that fetches `/assets/<hash>.js`
+misses and hits the SPA fallback (a body opening with `<!doctype` means the wrong URL, not a
+broken build); and `echo "$B" | grep -q` under `set -o pipefail` reports a **hit as a miss**
+(grep exits on first match, `echo` takes SIGPIPE) — use `grep -q "$lit" <<<"$B"` or
+`grep -c` against a saved file.
+
+**4. Rebuild locally and compare the emitted filename.** Stronger than the hash alone and
+needs no credentials: rebuild the same source in the live tree and confirm the produced
+asset filename is **identical** to the one the live instance is serving.
+
+```bash
+cd worktrees/agentus && npm run build
+ls packages/web/dist/assets/          # index-<hash>.js must match what /api/version reports
+```
+
+Same source in the two trees legitimately differs by ~5 KB (the dev build bakes absolute
+source paths into its debug info) and therefore by hash — compare **strings/paths**, not
+sizes, and expect the only differences to be tree-name fragments.
+
+After a **restarting** release, the operator's open tab keeps the old bundle until a hard
+refresh — say so in the hand-off. A web-only release needs only the hard refresh.
+
+## Merging to main
+
+Merging is a separate step from releasing, and is the standing convention for this repo once
+a release is done. `main` is not what live runs from (`promote` puts live on `run/<slug>`),
+so fast-forward the ref without checking `main` out anywhere:
+
+```bash
+cd worktrees/agentus-dev
+git fetch origin
+git push origin <dev-branch>:main          # fast-forward only
+git branch -f main origin/<dev-branch>
+git branch --set-upstream-to=origin/main main
+```
+
+Judge the outcome by identity, not by "the push succeeded": `git ls-remote origin
+refs/heads/main` must equal the commit live is actually serving.
+
+## Rollback
+
+The release path has no separate rollback mechanism — you revert the change and promote
+again:
+
+1. On the dev tree, revert or fix the offending commit.
+2. Promote. If HEAD is now behind `origin/main` and you mean to roll back deliberately,
+   `promote` refuses; override with `DEV_PROMOTE_FORCE=1 bash scripts/dev.sh promote <slug>`.
+   If the release restarted live, the new (reverted) server starts and `promote` resumes the
+   sessions it captured before the restart.
+
+The promote log is the record of what happened and where to look:
+
+- **log file:** `/tmp/agentus-qa-account/promote.log`
+- **follow:** `bash scripts/dev.sh promote-log`
+
+Relay-side incident recovery (a live instance that came up on the wrong store, a slot that
+did not resume) is in [04-recover.md](./04-recover.md).
+
+Cross-links: [01-dev-loop.md](./01-dev-loop.md) · [02-verify.md](./02-verify.md) ·
+[04-recover.md](./04-recover.md).

package/docs/dev/04-recover.md

@@ -0,0 +1,199 @@
+# Recovering when it breaks
+
+This doc answers: the instance is broken or misbehaving — where do I look first, and how do
+I fix it without making the box worse?
+
+## Read the boot log first
+
+The boot log is the authoritative account of what the server resolved. It prints **which
+data directory won and why**, the auth state, and the TLS listener:
+
+```
+data dir: /Users/…/packages/server/.data  (AGENTUS_DATA was set)      ← the dir + the reason
+auth on: user "admin"
+https://0.0.0.0:8443 (self-signed …)                                  ← present = TLS listener up
+tls off: no cert at <DATA_DIR>/tls/cert.pem                           ← absent/off = no TLS
+```
+
+If the directory it names is not the one you expected, stop here — a server serving the
+wrong store looks healthy (200s, valid `/api/version`) while the operator's session list is
+empty. Triage the identity, not the liveness (see "Is it serving the wrong store?" below).
+
+## Ports
+
+| port | who |
+|---|---|
+| `8787` | live HTTP |
+| `8443` | live TLS (public tunnel points here) |
+| `8901` | dev / QA instance |
+| `8648` | **Hermes Studio** — not Agentus, do not touch |
+
+```bash
+lsof -nP -iTCP:8787 -sTCP:LISTEN        # who owns the port (returns the CHILD, not the supervisor)
+ps -o pid,ppid,pgid,command -p <pid>    # its process group and command line
+```
+
+## "Address already in use"
+
+`EADDRINUSE` means another process holds the port — often a previous instance that never
+died, or (this box) an ambient `PORT=8648` pointed at Studio. The Agentus server reads
+`AGENTUS_PORT`, never bare `PORT`, so a script that reads the ambient value will fight for
+Studio's port. Do not just relaunch: identify the current holder, decide whether it is the
+one you want, then stop it through its own launcher.
+
+```bash
+# clean stop of the dev instance (kills the process GROUP on :8901)
+bash scripts/dev.sh stop
+```
+
+For live, stop it only via the release path ([03-release.md](./03-release.md)) — a bare kill
+of the live listener ends the operator's sessions.
+
+## Orphaned agent and shell child processes
+
+Every agent child is spawned detached and its pid recorded, and the server reaps orphans on
+next boot — so a crashed server's leftovers are cleaned up automatically. Shell sessions
+(the workspace terminal) are killed with their socket. Problems appear when a **supervisor**
+(the `tsx watch` wrapper) survives a crash: the listener it spawned may be gone while the
+supervisor sits in a restart loop, or a leaked listener keeps holding the port.
+
+**Never kill by a substring match.** A cleanup filtered on `packages/server/src/index.ts`
+once ended 249 processes in one shot — the operator's own `hermes-studio` checkout has a
+file at that same path, and its server was the only way into the machine. Rules:
+
+1. **Enumerate first and print the full command line and cwd** of every candidate:
+   ```bash
+   ps -eo pid,ppid,pgid,command | grep <pattern>
+   lsof -a -p <pid> -d cwd -Fn      # where it is actually running
+   ```
+2. **Keep the list inside this worktree's path.** The operator's services are never in
+   scope. A `ps | grep` list also contains your own tool shell — exclude the current pid and
+   its ancestors, or an unfiltered kill ends your own command mid-flight.
+3. **Prove the target is its own process group, then kill the group:**
+   ```bash
+   ps -o pid,pgid,sess -p <pid>
+   kill -9 -- -<pgid>              # negative pid = the group
+   ```
+   Under a supervisor the LISTEN pid is the supervisor's child, whose pgid is the
+   supervisor's — killing only the child leaves the supervisor (which respawns and fights
+   for the port). This is why `dev.sh`/`relaunch.sh` kill `-$pgid` and, when `pgid != pid`,
+   the pid too.
+4. **Fix the leak, not the puddle.** New scripts must use the shipped pattern —
+   `spawn(..., { detached: true })` and kill the process group — rather than cleaning up by
+   name afterwards.
+5. **Re-check the neighbours afterwards:** one `curl` per resident service
+   (`:8648` Studio, `:8787`/`:8443` live, `:8901` dev). A cleanup that leaves the box worse
+   than it found it is the whole failure mode.
+
+## The SQLite store, WAL, and the `*.bak` files
+
+State lives in `<DATA_DIR>/agentus.sqlite` (the transcript and every other row), accessed
+through Node's built-in `node:sqlite` (hence Node `>= 22.5`). Files beside it: `credentials.json`,
+`auth.token`, `auth.secret`, `sessions.json`, `settings.json`, `tls/` — several of them
+`0600`. The server creates the directory `0700` on first boot.
+
+**Back up before you touch the store.** A single write against the operator's real data gets
+a backup and a record of the old values:
+
+```bash
+cp -p packages/server/.data/agentus.sqlite \
+      ~/.hermes/cache/scratch/agentus.sqlite.bak-$(date +%H%M%S)
+```
+
+The one migration path in the project that rewrites rows in place — the legacy transcript
+fold — takes its own backup first with `VACUUM INTO <db>.pre-fold.bak` and gates itself on
+`pragma user_version`, so it runs once and is idempotent. If you see a `*.pre-fold.bak`
+beside the store, that fold ran; the original is inside it.
+
+Two cautions that come from real incidents:
+
+- **Two independent writers on one WAL database** corrupted the operator's live `state.db`
+  while the runtime linked a SQLite in the WAL-reset bug range. The fix is the interpreter
+  (link SQLite ≥ 3.51.3), not a symlink, and it belongs to the Hermes Studio runtime, not to
+  this repo. Check it with `hermes doctor` and read the ⚠ lines — a warning is real, not
+  imaginary, but the shipped `hermes doctor` is warn-only. Never add a second writer to the
+  live home on a vulnerable build.
+- `DELETE /api/sessions/<id>` **closes a live session and purges a cold one** — for an
+  already-dead row it deletes the record. So "archive this dead session" is not that route;
+  it takes a direct, backed-up, single-row transaction (`WHERE id=? AND status='error'`), no
+  live row touched, and no restart needed (a cold row is not in the server's in-memory map).
+
+## Is it serving the wrong store?
+
+Symptoms that look like separate bugs but share one cause: the operator's session list came
+back empty, TLS silently turned off (`tls off: no cert at <QA_DIR>/tls/cert.pem`), and a
+running slot was reaped — yet the port answers 200 and `/api/version` looks fine. Cause:
+`relaunch → launch.py` inherited the QA environment. The launcher now pins its own
+`AGENTUS_DATA`/`AGENTUS_PORT` and pops the QA/DEV keys, but verify identity directly:
+
+```bash
+PID=$(lsof -nP -iTCP:8787 -sTCP:LISTEN -t)
+ps eww -p "$PID" | tr ' ' '\n' | grep AGENTUS          # the process's own identity vars
+lsof -p "$PID" | grep sqlite                           # which store it actually opened
+```
+
+Recovery:
+
+```bash
+cd worktrees/agentus
+env -u AGENTUS_DATA -u AGENTUS_BASE -u DEV_LOG_PATH -u DEV_PROMOTE_DETACHED -u DEV_PROMOTE_SELF \
+  bash scripts/relaunch.sh --port 8787 --launcher ~/.hermes/cache/agentus/launch.py
+TOK=$(cat packages/server/.data/auth.token)
+curl -s -X POST -H "Authorization: Bearer $TOK" http://127.0.0.1:8787/api/sessions/<id>/resume
+```
+
+After any restart, put the operator back on the air: `POST /api/sessions/<id>/resume` with the
+machine token → `status: ready` and a fresh pid. **Capture the slot ids to resume before the
+restart** — in the window right after a restart a slot has dropped from `live` into
+`archived` wearing `error`, so a resume loop that reads the current `live` list finds
+nothing. `resume` works on an `archived('error')` row; that is the cold-slot entry point.
+
+## Auth lockout
+
+The login limiter is visible and reversible without a restart, so locking yourself out is a
+one-click problem. Endpoints behind the login:
+
+```bash
+TOK=$(cat <DATA_DIR>/auth.token)
+curl -s -H "Authorization: Bearer $TOK" http://127.0.0.1:8787/api/auth/locked-ips
+curl -s -X DELETE -H "Authorization: Bearer $TOK" http://127.0.0.1:8787/api/auth/locked-ips
+```
+
+In the UI it is Settings → 登录失败锁定 (which IPs are locked, for how long, unlock / unlock
+all). Tunables: `AGENTUS_LOGIN_MAX_FAILS` (default 5) / `AGENTUS_LOGIN_LOCK_MS` (default
+30000). Login sessions are stateless signed cookies indexed at `<DATA_DIR>/sessions.json`
+(0600) — advisory for display, authoritative for existence. A password change invalidates
+every other session; `AGENTUS_AUTH=off` disables the login entirely and the boot log scolds
+you (do not leave it off — the listener is `0.0.0.0` and this cockpit is tunnelled).
+
+## TLS certificate regeneration
+
+The TLS listener runs on a **second port** (8443 live), pointed at by a plain-TCP tunnel.
+Cert material lives at `<DATA_DIR>/tls/`. Regenerate with:
+
+```bash
+scripts/make-cert.sh [name]        # default name: the DDNS name; SAN = that name + localhost + 127.0.0.1
+scripts/make-cert.sh --leaf-only   # rotate the leaf only; installed devices are untouched
+```
+
+`make-cert.sh` issues **two** certs on purpose: a 10-year root (`ca.pem`, what devices install
+once, served at `/cert.crt`) and a 390-day leaf (`cert.pem`/`key.pem`, what the listener
+serves, under Apple's 398-day cap). Keep the SAN to the DDNS name you actually type, not the
+dynamic public IP. It writes into the same directory the server reads from (mirroring the
+server's own resolution in `packages/server/src/index.ts`), and if that resolution and the
+server's ever diverge the listener stays silently off with a cert sitting next to it — after
+regenerating, confirm the boot log still prints `https://…:8443`.
+
+## The process-exit safety net for children
+
+The server installs `SIGINT`/`SIGTERM`/`SIGHUP` handlers that terminate the child agent
+processes, so Ctrl-C (or a deliberate restart) takes its agents with it. The
+`unhandledRejection`/`uncaughtException` guards are installed **only after `listen()`
+succeeds** — installing them earlier would turn a startup failure (`EADDRINUSE`) into a
+zombie that is "alive but not listening". Combined with the on-boot orphan reap, this is
+what keeps a crashed server from leaving runaway children behind. If you find runaway
+children anyway, hunt them by the process-group rule above — never by name.
+
+Cross-links: [01-dev-loop.md](./01-dev-loop.md) (ports, data dirs) ·
+[03-release.md](./03-release.md) (release + rollback) ·
+[02-verify.md](./02-verify.md) (stray CDP tabs, leaked servers).