@telora/daemon 0.24.5 → 0.24.8

package/docs/runbook-tower.md

@@ -0,0 +1,237 @@
+# Runbook: the product tower
+
+The tower is traffic control for a product's armed foci. It changes what
+ARMED means -- from "run now" to "run when cleared" -- and it never arms or
+un-arms anything: arming stays the human go-signal. Decision record:
+`docs/DECISIONS.md` ("The tower: traffic control for armed foci lives at the
+product").
+
+With the tower in `enforce`, an operator can arm several foci of a product,
+set the product's limit to 1, and walk away: they run one at a time, in focus
+rank order.
+
+## The pieces
+
+| Piece | Where | What it is |
+|---|---|---|
+| The limit | `products.tower_concurrency_limit` | How many of the product's foci may hold a live coding or review team at once. `NULL` = no product limit: the host session cap alone bounds it. Set with `telora_product` `update` `towerConcurrencyLimit` (`null` clears it). |
+| The verdict | `product_focuses.tower_verdict` | The tower's verdict on one armed, active focus: `cleared`, or `holding` with exactly one reason and since-when. |
+| The pass | `packages/daemon/src/tower/tower-pass.ts` | One pass per configured product; the SOLE writer of verdicts. |
+| Admission | `packages/daemon/src/tower/tower-admission.ts` | What the spawn passes ask before they start a coding or review team. |
+| The graph | `focus_tower_edges` | Declared `depends_on` / `excludes` edges between two foci of the product, with provenance and evidence. Written over MCP (`telora_product_tower`); shown on the product page's Focus tab. |
+| Landed | `packages/daemon/src/tower/focus-landed.ts` | The ONE definition of "this focus's work has landed", shared with the PRD frontier. |
+| The vocabulary | `@telora/daemon-core` `tower.ts` | Modes, verdict states, reasons (in precedence order), edge kinds and provenance, and the fail-open freshness rule -- one definition for the daemon and the dashboard. |
+
+## The verdict
+
+```
+{ state: 'cleared' | 'holding',
+  reason: null | 'product_limit_reached' | 'waits_on' | 'excluded_by',
+  since: ISO,        -- when this state + reason began (kept across re-confirmations)
+  computedAt: ISO,   -- when the pass last confirmed it
+  armedAt: ISO|null, -- the arm (agent_assigned_at) it was computed under
+  mode: 'observe' | 'enforce',
+  detail: { limit?, occupied?, focusId?, focusName?, evidence? } }
+```
+
+The reasons, and what the dashboard says for each:
+
+| Reason | Dashboard | Means |
+|---|---|---|
+| `product_limit_reached` | Held by tower: product limit reached (N of L running) | The product's free slots went to foci ahead of this one. |
+| `waits_on` | Held by tower: waits on <focus> | A `depends_on` predecessor has not landed (tower graph). |
+| `excluded_by` | Held by tower: excluded by <focus> on <evidence> | An `excludes` edge joins it to a focus that is running or cleared ahead of it (tower graph). |
+
+## The graph
+
+Edges join two foci of ONE product (an edge across products, or from a focus to
+itself, is refused):
+
+- **`depends_on`** (directed): `toFocusId` waits until `fromFocusId` has
+  LANDED. A `depends_on` edge that would close a cycle is refused at the write,
+  and the refusal names the cycle (`UI -> Schema -> API -> UI`). The refusal is
+  atomic: a database trigger serializes each product's `depends_on` writes, so
+  two concurrent writes can never close a cycle between them.
+- **`excludes`** (mutual): the two foci are never cleared together -- e.g. both
+  rewrite the same files, fixtures or migration journal. Stored once per pair.
+
+Every edge carries **provenance** (`declared` -- written by a human or an agent
+over MCP; `inferred` and `observed` are reserved for the follow-on focus that
+draws edges itself), **evidence** (why the edge holds -- required), and who
+wrote it and when. Deleting an edge leaves a tombstone (`deleted_at`).
+
+```
+telora_product_tower { action: "edge_create", productId, fromFocusId, toFocusId,
+                       kind: "depends_on" | "excludes", evidence }
+telora_product_tower { action: "edge_list", productId, focusId? }
+telora_product_tower { action: "edge_delete", towerEdgeId }
+```
+
+## The landed rule
+
+A focus has LANDED -- judged by the daemon, no human close -- when:
+
+1. it has at least one delivery, and
+2. every non-cancelled delivery is `done` (at least one: a focus whose every
+   delivery was cancelled landed nothing), and
+3. its branch tip is contained in the integration branch
+   (`git rev-list --count <integration>..<branch>` reads 0; no checkout) -- or
+   the branch is gone AND the focus's merge record (`focus_merges`, written by
+   the daemon when it merges) names a merge commit integration contains. The
+   daemon removes a focus branch only after merging it; the record is the proof.
+   A branch that is gone with no such record (a hand-deleted branch, a legacy
+   focus that never recorded one) is unproven: unknown, never landed.
+
+The PRD frontier's completion signal is this same rule (`prd/controller-shared.ts`
+`computeCompletedFocusIds`): a frontier overcome clears only when its focus has
+landed. A landed check git cannot answer (integration unreadable, a ref or
+containment read that errored, an unreadable or missing merge record) is UNKNOWN: the tower leaves the dependent focus's verdict unavailable (fail
+open) rather than holding it, and the PRD frontier waits a tick.
+
+## Precedence
+
+A verdict records exactly one reason. When several apply, the first in this
+order wins (daemon-core `TOWER_HOLD_REASONS`):
+
+1. `waits_on` -- a fact about this focus's prerequisites;
+2. `excluded_by` -- a fact about this focus's collisions;
+3. `product_limit_reached` -- a fact about the queue.
+
+A focus held for a graph reason takes no slot. `excluded_by` holds the focus
+reached second, in rank order, while the other runs or is cleared, so two
+excluded foci are never cleared together.
+
+## The slot rule
+
+A focus occupies one of the product's slots **only while it has a live coding
+or review team**. An armed focus with no live team -- waiting on CI, a verify
+hold, an escalation -- occupies nothing, so one stalled focus never holds the
+product's queue.
+
+Each pass, per product:
+
+1. Every focus with a live team is cleared (the tower never stops a running team).
+2. Free slots = limit - live foci (unbounded with no limit).
+3. The other armed foci are walked in **focus rank order** (lowest
+   `priority_rank` first) -- foci the spawn passes last found wanting a team
+   ahead of foci they found idle. A focus that waits on an unlanded
+   predecessor, or is excluded by a focus already running or cleared, holds
+   for that reason; the rest each take a free slot (cleared) until none is
+   left, and the remainder hold `product_limit_reached`.
+
+"Wanting" is CURRENT demand: a focus wants a slot when it could run now if the
+tower and the host cap let it. Before the tower allocates, every ready-focus
+pass refreshes stale agent roles (as the spawn pass does) and probes each
+surfaced focus -- ready, or holding a queued directive -- through the gates its
+spawn pass applies, in the pass's order, bar the tower and the host cap
+(`packages/daemon/src/focus/spawn-demand-probe.ts`): its role, a live team, the
+review budget, an open parking escalation (the guard `spawnFocusTeam` applies),
+a demoted just-torn-down respawn, a queued review whose window cannot be read
+(the ready pass withholds it), the operator-hold park, a review at an approved
+head, a verify hold, a cooldown, no team work, and -- for a queued directive --
+an active row the pass could not read (it cannot spawn without one). So a focus held by any of
+those reserves no slot, even the first time it surfaces; a focus whose gate
+clears is ranked with the waiters on that same pass; a focus that spawned has
+had its demand met; and a focus no longer surfaced wants nothing -- when a team
+exits, its focus reserves nothing and the slot goes to the next wanting focus on
+the same pass. The gates that dispose of, park or release a queued directive (a
+stale stage, a closed review window, a superseded coding directive, the
+operator-hold park and its release, a review at an approved head) run for real
+before allocation: they are idempotent and the pending pass would make the same
+disposal on that poll. A probe that cannot answer keeps a fresh record and
+otherwise fails open (the focus wants a slot), so a read error never demotes a
+higher-ranked focus behind the waiters.
+
+The pass runs at the head of every ready-focus pass -- on daemon start, on
+every 30s poll, on the engine's 60s timer and on a completion trigger; there is
+no production path to the ready pass that skips it, and the passes never overlap
+(`runReadyFocusPassThroughTower`, pinned by `tower/tower-entrypoints.test.ts`) -- so a change to the armed set, the limit, focus rank or a
+focus's team presence is reflected within one poll. It writes only a verdict
+that changed, or one older than half the freshness bound (5 minutes), so a
+running tower's verdicts never go stale. It also clears the verdict of a focus
+that is no longer armed. A pass that throws for one product is logged and never
+stops another product's pass; admission fails open on that product's foci at
+once (see the fail-open rule).
+
+## The three modes: `TELORA_TOWER_MODE`
+
+Declared in `@telora/daemon-core` `env-config.ts`; default **`observe`**.
+
+| Mode | Tower pass | Spawn passes |
+|---|---|---|
+| `off` | Does not run. | Exactly as before the tower: API order, host cap only. |
+| `observe` | Writes verdicts; the dashboard shows them, marked "(observe only)". | Ignore verdicts. Every spawn decision `enforce` would have made differently is logged: `[tower] observe: enforce would hold "<focus>" (<reason> since <t>); decision taken: spawned`. |
+| `enforce` | Writes verdicts. | Start no coding or review team for a focus whose verdict is holding AND fresh; offer ready foci, and foci with a queued (e.g. review) directive, to the host in focus rank order. |
+
+## The fail-open rule
+
+In every mode, only a fresh, explicit holding verdict ever stops a spawn, and
+only while this daemon's tower pass for the focus's product is STANDING: its
+most recent pass for the product succeeded, within the freshness bound. When the
+product's pass fails -- or has not yet succeeded since the daemon started --
+every verdict of that product is unobeyed AT ONCE, however fresh, and the daemon
+logs `[tower] "<focus>": its product's tower pass failed -- failing open`.
+Another product's holds are untouched. A verdict that is
+
+- missing,
+- unreadable,
+- older than the bounded age (`TOWER_VERDICT_MAX_AGE_MS`, 5 minutes), or
+- computed under an earlier arm (its `armedAt` is not the focus's current `agent_assigned_at`)
+
+is NO verdict: the focus spawns exactly as it would with the tower off, the
+daemon logs `[tower] "<focus>": tower verdict unavailable (<why>) -- failing
+open`, and the dashboard names it ("Tower verdict unavailable (stale) --
+spawns as with the tower off"). A tower that stops writing therefore releases
+every hold within five minutes rather than freezing the product.
+
+The dashboard judges freshness on a clock while the page is open, so a hold
+turns into "unavailable" on screen when the spawn pass stops obeying it. Whether
+the product's tower is running at all comes from the daemon heartbeat
+(`metadata.tower` = `{ [productId]: { mode, pass } }`): with the tower running,
+an armed focus with no verdict reads "unavailable" even when no focus has one,
+and a failed pass marks every armed focus of the product "unavailable (the
+product's tower pass failed)"; with `TELORA_TOWER_MODE=off` no tower line shows.
+
+Why admission fails OPEN while the merge gates fail CLOSED: the failures are not
+symmetric. A wrongly blocked merge is visible -- a held delivery, a red gate, a
+named reason. A wrongly held spawn is silent -- a focus that simply never runs,
+which is the daemon's recurring bug class. The tower is in front of every
+spawn, so it must never be able to cause that silently.
+
+## Moving a host from observe to enforce
+
+1. Leave the host in `observe` (the default) and set the product's limit:
+   `telora_product update { productId, towerConcurrencyLimit: 1 }`.
+2. Arm the foci. On the dashboard each armed focus now carries the tower's
+   line; on the host, read the divergences:
+   `telora-daemon logs | grep '\[tower\]'`.
+3. Judge them on real foci: every `observe: enforce would hold` line should be
+   a spawn you would have wanted held, and no focus should show "tower verdict
+   unavailable" while the daemon is healthy.
+4. Set `TELORA_TOWER_MODE=enforce` in the daemon's environment and restart it
+   (drain first if teams are running). Holds now take effect on the next poll.
+5. To back out: `TELORA_TOWER_MODE=observe` (verdicts kept, not obeyed) or
+   `off` (no pass at all) and restart. Clearing the product limit
+   (`towerConcurrencyLimit: null`) releases every `product_limit_reached`
+   hold on the next poll without a restart.
+
+## Reading a stuck focus
+
+- **"Held by tower: product limit reached"** -- look at which foci are running
+  (they hold the slots) and their rank; raise the limit or re-rank if the
+  order is wrong.
+- **"Held by tower: waits on <focus>"** -- the predecessor has not landed: it
+  still has open deliveries, or its branch is not yet merged into integration.
+  If the dependency is wrong, delete the edge (`edge_delete`).
+- **"Held by tower: excluded by <focus> on <evidence>"** -- the other focus is
+  running (or was cleared ahead of this one); this one clears when it stops.
+- **"Tower verdict unavailable"** -- the focus will spawn anyway. Check the
+  daemon log for `[tower] pass failed for product ...`.
+- **"Waiting for a session slot"** -- the tower has cleared it; the HOST cap is
+  full. That is the session cap, not the tower.
+
+## Stated limit
+
+A product served by two daemon hosts would get two towers writing the same
+verdicts, each counting only its own live teams. No product is configured that
+way.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@telora/daemon",
-  "version": "0.24.5",
+  "version": "0.24.8",
   "description": "Agent orchestration daemon for Telora - spawns and manages Claude Code instances",
   "type": "module",
   "bin": {
@@ -20,7 +20,8 @@
     "scripts/telora-daemon-wrapper.sh",
     "build-info.json",
     "README.md",
-    ".env.example"
+    ".env.example",
+    "docs/runbook-tower.md"
   ],
   "scripts": {
     "dev": "tsx watch src/index.ts",
@@ -33,12 +34,14 @@
     "test": "node scripts/run-tests.mjs",
     "stamp-build-info": "node scripts/stamp-build-info.mjs",
     "stamp-build-info:local": "node scripts/stamp-build-info.mjs --empty",
-    "prepublishOnly": "npm run build && npm run stamp-build-info"
+    "copy-shipped-docs": "node scripts/copy-shipped-docs.mjs",
+    "prepublishOnly": "npm run build && npm run stamp-build-info",
+    "prepack": "npm run copy-shipped-docs"
   },
   "//testRunner": "Uses Node.js built-in test runner (node:test) via tsx for TypeScript support. The root package uses Vitest; this is a deliberate choice for the daemon package to avoid framework dependencies. Test files live beside the modules they exercise (owner-directory layout); the runner discovers them recursively.",
   "dependencies": {
-    "@telora/daemon-core": "^0.3.0",
-    "@telora/mcp-products": "^0.23.3",
+    "@telora/daemon-core": "^0.3.1",
+    "@telora/mcp-products": "^0.23.5",
     "commander": "^14.0.3",
     "yaml": "^2.4.0",
     "zod": "^4.3.6"