sphinx-understand 0.0.0-stage → 0.1.0

package/docs/architecture.md

@@ -0,0 +1,196 @@
+# Architecture
+
+Sphinx has one local checkpoint engine and thin Git/agent integrations. It embeds `@earendil-works/pi-coding-agent` 1.2.0 for investigative reviewer sessions and requires Node.js 22.19+. Short-lived detached workers prepare reviews; there is no permanent daemon or Sphinx backend service.
+
+```mermaid
+flowchart LR
+  PC[Commit or branch update] --> B[Debounced background reviewer]
+  B --> PI[Pi reviewer session]
+  PI --> W[Filtered Git tools and project context]
+  W --> PQR[Prepared questions and evidence]
+  G[Git operation] --> P[Deterministic policy]
+  P --> F[Filter paths]
+  F --> S[Exact snapshot]
+  S --> R{Valid completion?}
+  R -->|yes| C[Continue original operation]
+  R -->|no| Q[Save pending checkpoint and wait]
+  Q --> I[Local browser questionnaire]
+  PQR --> I
+  I --> E[Independent evaluator]
+  E -->|misconception| L[Explain and ask new scenario]
+  L --> I
+  E -->|all concepts resolved| A[Record completion]
+  A --> V[Revalidate batch and receipts]
+  V --> C
+  Q -->|unavailable| H[Native or manual fallback]
+```
+
+## Modules
+
+| Module | Responsibility |
+| --- | --- |
+| `config.js` | Defaults, global/repository settings, canonical/legacy policy discovery, private JSON files, credentials |
+| `repo-config.js` | Exclusive creation of repository policy and ignore files, preserving existing files |
+| `ignore.js` | Sensitive-file exclusions and project/global patterns |
+| `git.js` | Deterministic triggers, filtered object/index evidence, snapshot identities |
+| `routing.js` | Line/file/path tiers, depth-dependent question ceilings, and user-selected model routes |
+| `model-presets.js` | Explicit economy presets, per-tier routes, billing descriptions, and reference sources |
+| `evaluator.js` | Fresh CLI evaluator processes, question/rubric schemas, output validation |
+| `reviewer.js` | Pi reviewer integration, exact observation archive, assessment validation, review budgets |
+| `pi-runtime.js` | Isolated Pi SDK session, model-transport bridge, native compaction, cancellation, cleanup |
+| `review-workspace.js` | Filtered immutable file, diff, listing, and search tools |
+| `repo-context.js` | Bounded project background, ignore checks, context identity |
+| `preparation.js` | Detached scheduling, debounce, exact cache matching, failure status |
+| `openrouter.js` | Authenticated structured-output API requests and free-only price limits |
+| `sessions.js` | Durable checkpoints, locks, progress, counted calls and optional caps, expiration |
+| `presentations.js` | Session-bound queues and hook-owned presentation markers, without code or answers |
+| `git-gate.js` | Waiting Git gate, exact snapshot/receipt revalidation, batch ownership, and fallback |
+| `browser-interview.js`, `browser-page.js` | Loopback human questionnaire, background browser opening, recovery, cancellation, and batch reuse |
+| `git-interview.js` | Explicit separate-terminal transport retained for compatibility |
+| `native-check.js` | Direct human-form assessment loop, queue claims, cancellation, and summary-only results |
+| `mcp.js` | Bounded stdio JSON-RPC transport, form elicitation, and lifecycle tool responses |
+| `install.js` | Reversible hook chaining, agent configuration, coverage checks |
+| `agent.js` | SessionStart guidance and Claude shell-session tagging for native questionnaires |
+| `pi-integration.js` | Pi extension installation, native dialogs, and deterministic queued-check presentation |
+| `lifecycle-integrations.js`, `cursor-hook.js` | Cursor/OpenCode installation, browser handoff for queued fallback checks, and status-only client results |
+| `terminal-launch.js`, `terminal-worker.js` | Explicit terminal transport, shared browser/native/terminal reservations, and private worker status |
+| `activation.js`, `activation-worker.js` | Fresh coding-client sessions after installation, prompt-free launch, and startup acknowledgment |
+| `cli.js` | Onboarding, terminal interview, public JSON protocol, hook entry points |
+| `terminal.js`, `help.js`, `prompts.js` | Terminal presentation, contextual command help, and guided selections |
+
+## Repository policy and terminal interaction
+
+Machine-wide settings live in `$SPHINX_HOME/config.json`. A repository's canonical policy is JSON stored in `.sphinxconfig`. If that file is absent, the loader accepts legacy `.sphinx.json`. When both are present, canonical policy wins without merging the legacy file. Repository policy can control triggers, question depth, blocking mode, and exclusions; model routes, authentication, and usage budgets remain machine-wide. Repository exclusions add to machine-wide exclusions.
+
+Reviewer presets write concrete per-tier routes into the global configuration. They are not dynamic model selectors: updating the preset catalog does not migrate saved models or trigger fallback. `sphinx config --preset <id>` and the `--small`, `--medium`, and `--large` flags edit those routes without reinstalling or invoking inference. Model-route changes remain part of assessment identity and invalidate reuse under the previous policy. The internal `openrouter/free` default remains for legacy configuration and checkpoint identity compatibility; new unattended installations require a preset or explicit route rather than accepting it silently.
+
+`sphinx init` guides the user through event, branch, depth, mode, remote, and project-context choices. Branch defaults come from effective configuration rather than the currently checked-out branch. The initializer writes `.sphinxconfig`, `.sphinxignore`, and `.sphinxcontext.md` only when absent. It can copy a legacy policy into the canonical file while retaining the original. Rerunning setup preserves existing files and does not silently apply new flags over them.
+
+Initialization does not install hooks, change global settings, resolve remote Git state, or invoke a model. Machine-wide integration is a separate, once-per-machine operation. A repository with a competing `core.hooksPath` needs the explicit `install --repo` compatibility step.
+
+Human terminal output provides a compact ASCII mark, explanatory selections, and progress/context for assessments. Color and layout adapt to terminal capabilities. Every public command supports its own help through `sphinx <command> --help` and `sphinx help <command>`. `--json` output and hook protocols bypass decorative output; they remain suitable for coding agents, pipes, and tests. `NO_COLOR` disables color; redirected output is uncolored by default. Help and read-only status/configuration commands never invoke an evaluator. Reviewer dependencies are loaded only when an investigation starts, keeping ordinary Git gating and CLI inspection independent of reviewer initialization.
+
+## Human questionnaire transports
+
+A blocking Git hook saves pending assessments and hook-owned presentation requests, then serves a local browser questionnaire while the original Git process waits. On macOS, `open -g` opens the page in the background without activating or minimizing applications. Elsewhere Sphinx prints the local URL instead of guessing window-manager focus behavior. It never automatically opens Terminal for a questionnaire. The batch reuses one page for all destinations. Hook-owned requests are invisible to native after-tool callbacks, which cannot run until the original shell tool finishes. Browser, native, and terminal transports share a per-checkpoint reservation to prevent duplicate questionnaires.
+
+Human answers go straight from the browser to the independent evaluator. Neither questions nor responses enter client hook results, and Git protocol stdin remains untouched. After completion, the gate reloads policy and revalidates every original snapshot and receipt, including previously approved destinations, before returning zero to the same Git process. Existing hooks run once before snapshot collection; commands, refspecs, force flags, stdin, and remotes are never reconstructed or replayed. Git retains its own remote-update concurrency checks.
+
+The HTTP listener binds only to `127.0.0.1` on an ephemeral port and requires a random URL token. Responses are not cached; CSP prevents third-party assets, frames, and form navigation. POST requests require the exact host, origin, token header, current checkpoint/revision, and current question ID. The page renders all source/model text with `textContent`. Questions and feedback are public assessment fields; answer keys, rubrics, and retained source observations are never served. Provider calls stay in the hook process with its normalized Git context, preserving temporary indexes and explicit Git directories without exposing credentials or Git context in launch arguments. SSE carries current state; reconnects do not replay submissions, and monotonic updates prevent a delayed POST response from overwriting a newer question or completion.
+
+Pause aborts active evaluation and releases the waiting operation without approval. Closing a browser tab does not pause the check: the URL can reconnect until the 25-minute deadline. Cancellation, stale content, missing receipts, and the deadline cause nonzero Git exit and cleanup of the hook-owned presentation batch. The per-operation server closes after the gate finishes. CI and evaluator subprocesses do not open pages or begin browser inference. An unavailable browser transport stops Git and queues the native/manual fallback. Codex and Claude Code lifecycle hooks call the local `sphinx_checkpoint` MCP tool only for this fallback queue. Its only arguments are repository context and host/session identifiers: `cwd`, `host`, `hookEvent`, and `hostSession`. It accepts no questions or human answers from the development model.
+
+The controller claims queued work and uses MCP `elicitation/create` to present a flat form. The user can answer, learn, dispute, or pause. Responses return directly to Sphinx and its independent evaluator. Questions, feedback, and responses are not included in the tool result; the development model receives only allowlisted status, assessment IDs, progress, usage counts, and static retry guidance. The client controls how forms appear and may retain its own interaction logs.
+
+The stdio server negotiates protocol `2025-11-25` or legacy `2025-06-18`, supports both modern `elicitation.form` and legacy empty elicitation capabilities, and correlates each form to an active tool request. Unsupported clients leave queued checks untouched and consume no inference. No queued work returns an empty hook response, including on clients without form support. JSON messages and accepted field values are bounded and validated; malformed answers, cancellation, disconnection, or timeouts cannot mark a check complete.
+
+Advertised form support does not guarantee the client will display a request: client approval policies may cancel it. Codex 0.162.1's `never` policy does so unless its host enables full-access form input for a user thread. Sphinx preserves the pending assessment and does not alter the client's approval settings. Terminal checks remain available. Completion guidance retries only blocked Git operations.
+
+A paused or dismissed presentation is consumed while its assessment remains pending. Unrelated shell commands therefore do not reopen it. A new Git attempt queues presentation again and resumes existing progress. Session identifiers let the integration find requests from commands that change directories or use `git -C`; Codex supplies its thread ID, and the Claude SessionStart hook appends a Sphinx session tag through `CLAUDE_ENV_FILE`. Outside native hosts, `sphinx check` provides the terminal interview using the same assessment state.
+
+Pi's development-client integration is a separate local extension from the embedded reviewer runtime. It observes built-in `bash` and `powershell` results, then calls the same questionnaire controller for queued Git requests. `ctx.ui.select` presents Answer, Learn, Dispute, and Pause; `ctx.ui.input` collects the person's response. Only a fixed status message is appended to the shell result. Questions, feedback, and human answers never enter the coding transcript. Interactive or dialog-capable RPC mode is required; noninteractive modes preserve pending work for a terminal check. Pi's built-in shell session ID associates requests across `cd` and `git -C`.
+
+Cursor and OpenCode adapters inspect queued fallback presentation metadata after shell tools and use the same local browser questionnaire. The client hook waits for a status-only result. This fallback cannot resume Git after it has exited; guidance permits a new invocation only after checking that the original operation failed at Sphinx, and forbids repeating waiting or successful operations. Questions, feedback, answer keys, human responses, and browser URLs never pass through the adapter's returned tool result. Cursor receives a context note; OpenCode receives a shell-result note and a TUI toast when available. Session environment tags support cross-directory routing. Cursor user hooks apply only to local sessions.
+
+Private reservation records track the exact assessment, presentation token, and owning process. Active interviews are reused, and dead owners can be recovered. A paused presentation is consumed so unrelated tool events cannot reopen it; a new Git attempt can resume the assessment. Both direct Git and fallback browser interviews have bounded waits. Host shell or remote-connection timeouts may stop Git earlier; a late receipt never revives or replays an exited operation. `sphinx check --id <id>` remains an explicit terminal alternative using the same assessment and receipts. No browser or native transport grants completion by itself.
+
+## Independent investigation and compaction
+
+Version 2 snapshots contain change metadata and pinned Git identities, not a bundled patch. Before the first model request, Sphinx reads up to eight initial diff pages through the filtered immutable workspace, then a bounded file inventory when room remains, all within one quarter of `maxContextBytes`. These local reads consume no model calls. They enter the same exact evidence archive as subsequent tool observations, alongside statistics and `.sphinxcontext.md`. Listed names confer no citation privileges. Header-only or unavailable observations cannot satisfy the changed-code evidence requirement; if no usable evidence can be loaded within these limits, review stops before inference. The agent chooses `read_diff`, `read_file`, `list_files`, and `search` calls for missing ranges and dependencies. Tools filter paths before reading immutable blobs, reject unsafe paths and symlinks, and return bounded pages with explicit continuation information. Unchanged callers, tests, and dependencies are accessible at the pinned revision. Staged checkpoints retain index object identities.
+
+The [Pi SDK](https://pi.dev/docs/latest/sdk) owns the reviewer session, agent/tool loop, and native compaction. Its entries and exact evidence archive are checkpointed privately so interruption does not discard the investigation. Sphinx registers only its filtered immutable workspace tools, supplies explicit project context, and disables ambient resource discovery and general-purpose tools. The session never loads the development transcript, project agent instructions, or unrelated Pi resources. `SPHINX_EVALUATOR=1` prevents recursive Sphinx hooks.
+
+OpenRouter supplies model responses with native function calls. For explicitly selected `anthropic/` models, Sphinx sends `tool_choice: "auto"` on investigation and finalization requests to preserve compatibility with thinking-capable providers. Sphinx still rejects responses without actual tool calls and requires exactly one `finish_review` call during finalization; automatic selection never relaxes those local checks or triggers a transport retry. Other model families retain required or explicitly selected tool choices. See [Anthropic thinking and tool-choice compatibility](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-with-tool-use). The Codex/Claude compatibility routes use fresh isolated CLI invocations as model transports underneath Pi. Structured model responses become Pi tool requests; Pi schedules the registered Sphinx tools and carries the conversation into the next request. These routes reuse CLI authentication without copying secrets into prompts. They do not run a persistent Codex or Claude reviewer session, or delegate isolation to those clients' native tools.
+
+Every request includes bounded project context (maximum 16,000 bytes). Context is descriptive data, not evaluator policy, and its claims must be checked against code. The current project file is hashed into checkpoint identity; an excluded context file is not opened.
+
+Before Pi validates a `finish_review` call, Sphinx normalizes two lossless representation differences: a valid JSON-object string in `assessment` becomes an object, and `summary` or `evidenceIds` nested inside it moves to the envelope only when that outer field is absent. Questions, keys and citations are never rewritten by this step. Duplicate metadata, unknown properties and malformed JSON remain subject to strict rejection. The original native provider message and opaque continuation are retained separately from Pi's normalized arguments. Recorded response replays cover both native tool calls and structured CLI transports without spending another inference request.
+
+Pi's [native compaction](https://pi.dev/docs/latest/compaction) summarizes older conversation history and rebuilds the active context. Sphinx separately archives exact observations with IDs, paths, revisions, and ranges under a fixed byte budget. Compaction can shorten Pi's transcript without changing this archive; a model summary cannot replace code evidence. Final questions must cite inspected paths backed by archived observations. Grading receives the question's selected original evidence, rubric, project context, and verbatim human answer.
+
+New assessments separate their minimal acceptance criteria—one prediction and its causal reason—from the longer answer key and explanation. The independent audit checks both against source and aligns the visible question with the minimum criteria. A question that bundles extra checks must be narrowed, not given hidden requirements. Grading still rejects an unsupported key, but explanatory implementation details do not become additional requirements for the human. Older saved questions remain readable and are assessed against their visible behavioral question. All acceptance criteria and explanatory keys stay private.
+
+Sphinx enforces `maxContextBytes` on serialized review context: prompts, tools, and schemas. Provider transport metadata is outside this count; it is not a byte limit on the complete HTTP body or the whole Git patch. Pi's estimated context threshold leaves reserve space for compaction, while tool pages and the evidence archive have separate bounds. Byte-based token estimates drive Pi's context management, particularly for CLI transports without reliable token reporting; these estimates are not billed usage totals. If the next review context cannot fit, the assessment remains incomplete. Every actual model request, including native compaction, is counted before invocation and checked against any explicitly configured checkpoint cap. Provider retries and cache warming are disabled; an error never triggers a paid-provider fallback. Correctable protocol and question-validation errors return feedback to Pi within the existing allowance. Exact citation evidence is resolved deterministically from observed paths; observation IDs are optional narrowing hints. A separate audit checks all proposed questions against retained evidence before display. Pending audits survive interruptions and resume without another investigation. Audited reviews use five/seven/eight requests as soft planning targets for small/medium/large changes. `agentMaxTurns` and `maxModelCalls` both default to `null`, so those targets never impose a hidden request cap. Explicit numeric `agentMaxTurns` values cap a review attempt; explicit total caps also reserve calls for grading. Correctable finalization and audit errors may continue within those explicit caps, context limits, and the investigation deadline. Provider failures still stop and require a human retry decision. Private audit feedback survives resume without appearing in public errors.
+
+These adapters depend on recent CLI features. Unsupported flags or managed settings that conflict with isolation may cause a failure; Sphinx does not retry in a less isolated mode. Provider adapters need continued compatibility testing.
+
+## Background preparation
+
+Post-commit, post-merge, post-rewrite, and post-checkout dispatchers preserve original hook behavior, then schedule a detached worker without waiting for inference. A 1.5-second default debounce and job identities coalesce repeated work. Local push/upstream tracking metadata predicts the destination and remote base; missing tracking history is a cold-review case, not a reason to fetch or speculate automatically.
+
+Workers call the same session preparation path as foreground checks. Exact source/destination/policy/ignore/context matches reuse questions and evidence. A foreground check waits for an in-progress review instead of duplicating requests. Superseded workers and changed snapshots cannot install valid prepared work. Preparation never opens a questionnaire or creates completion. Failures have safe status messages and require explicit retry. `backgroundPreparation: false` disables scheduling; repositories can opt out but cannot override a machine-wide opt-out.
+
+## State and cost
+
+The existing line, file-count, and consequential-path rules deterministically select a small, medium, or large tier before inference. A consequential path promotes even a tiny patch to large; file count alone does not establish simplicity. That tier selects the configured evaluator route and, together with question depth, the maximum initial question count:
+
+| Tier | Relaxed | Standard | Thorough |
+| --- | --- | --- | --- |
+| Small | 1 | 1 | 1 |
+| Medium | 2 | 2 | 3 |
+| Large | 2 | 3 | 5 |
+
+The evaluator must return at least one question and may use fewer than the ceiling when fewer distinct behaviors are supported. It must not invent extra concerns to fill the budget. Model routes remain independent of depth. `maxOutputTokens` is preserved across all tiers; OpenRouter enforces it, while CLI evaluators receive concise-output instructions and local process limits.
+
+The economy CLI presets select `codex:gpt-6-luna` or `claude:haiku` for all tiers, preserving the cheaper model while effort and review planning targets scale. The paid OpenRouter value preset selects `anthropic/claude-haiku-5.5` for every tier and sets an 8,192-token output ceiling; an explicit output flag takes precedence. Applying the preset preserves the total call limit. The experimental open-weight alternative selects `deepseek/deepseek-v3.2`. Existing routes are never migrated implicitly. See [the evaluation record](../eval/RESULTS.md) for measured results and failures; these starting configurations are not guarantees of assessment quality. OpenRouter routes consume API credits separately from coding-agent plans; Codex and Claude routes consume the selected CLI account's allowance. Pi, Cursor, and OpenCode client integrations do not supply a model transport or reuse those clients' credentials.
+
+OpenRouter request parameters require compatible provider endpoints; model-level capability lists can combine capabilities from different endpoints. The [Haiku endpoint API](https://openrouter.ai/api/v1/models/anthropic/claude-haiku-5.5/endpoints), checked October 15, 2026, lists native tools, structured output and automatic tool selection, but not required/named selection. Reference per-million input/output rates start at $0.10/$0.50 below 100,000 input tokens and $0.50/$2.50 at the higher context tier. These are reference rates, not price ceilings or a reliability evaluation. OpenRouter's default Auto Exacto routing considers tool-call quality; explicitly sorting by price disables that routing step. Sphinx still validates every response and does not turn missing tool calls or invalid output into completion.
+
+Snapshot IDs include repository identity, exact objects/index, destination, effective policy, ignore-policy hash, and project-context hash. Ignored files affect immutable object identity but their contents never enter evidence. Before model calls, tool use, and completion, Sphinx revalidates local source/index and policies. The eventual Git hook compares a fresh destination snapshot to the completion record, so a changed remote base also invalidates reuse.
+
+The policy identity also includes an assessment-contract version. The introduction of minimal acceptance criteria invalidates prepared assessments and receipts from the previous contract; Sphinx does not silently upgrade them or treat them as passing the new assessment. Changing only the call allowance still preserves a checkpoint's identity and usage within the current contract.
+
+`pending → active → passed` is the main state sequence. Wrong answers teach the missed concept and replace its question with a new transfer scenario without advancing progress or adding another concept. Each proposed replacement receives a separate source audit before the grade reaches the session store. That audit sees exact retained evidence and the candidate, without the human answer or grading conversation. It can correct unsupported conditions or reject the replacement; rejection preserves the original pending question and incurred usage. A grade without a replacement costs one request; a proposed replacement costs two, subject to any explicit cap. Retries can exceed the initial question count. Without a configured cap, all calls remain counted but the total alone does not stop the check. Ambiguity remains active. Calls are counted before invocation, including failed attempts; explicit limits are enforced before invoking a provider. Per-session file locks prevent simultaneous grading and duplicate generation; expected question IDs reject stale responses. Cancellation during answer persistence rolls back progress while retaining incurred usage. Receipt readers acquire the same lock so they cannot observe a receipt being rolled back. Private state files use mode `0600`; public JSON and questionnaires deliberately omit private concept labels, rubrics, and full source evidence. Concept labels remain in private assessment state for grading and transfer-question matching because a generated label can reveal the answer.
+
+The source audit can rewrite a provisional replacement that repeats the original question; strict novelty checks run on its final output. For an ambiguous answer, a source audit that cannot support a replacement preserves the feedback with no replacement, leaving the original question pending for clarification. An incorrect answer still requires a valid audited transfer. Provider, cancellation, format and context errors are not swallowed by this optional-replacement path. Neither path can turn ambiguity into approval.
+
+Successful records expire after 24 hours by default. Background investigation is enabled by default and shares the assessment's counted usage and any explicit total cap. Sphinx always bounds serialized review-context bytes, tool-page bytes, and whole-investigation time. With an explicit request cap, it reserves grading calls plus the final synthesis and independent audit. A lone remaining request cannot start a new synthesis that would lack its required audit. Near that explicit cap, native `tool_choice` requests `finish_review` while retaining earlier tool definitions for history compatibility, except for the Anthropic automatic-selection compatibility path above. The transport and reviewer reject any other returned call; the CLI response schema also allows only finalization, and Pi cannot compact away reserved requests. Missing evidence still fails the review. Only a saved pending audit can resume with one available review call; an unfinished investigation needs at least two, plus reserved grading calls when a total cap is set.
+
+`maxModelCalls` is an optional operational ceiling, separate from assessment identity. Its canonical policy-hash value stays at the historical default of 10, and `agentMaxTurns` is normalized to its historical default of 8, preserving those existing checkpoint IDs when request caps change. Older checkpoints originally hashed with other custom limits require a fresh checkpoint. All other policy fields remain bound. Both defaults are `null`, meaning no request limit; numeric saved values remain explicit limits. The session store reloads global request limits under its lock before resume, grading, and every actual model request. A changed total cap replaces the absolute limit and records the change without resetting calls, questions, attempts, review evidence, or completion. Lowering below used calls blocks inference; clearing the cap preserves usage. Prior explicit retry extensions remain until the configured total cap changes. `sphinx config --max-model-calls <1-100|unlimited>` saves or clears the global total cap without reinstalling or invoking a model; `none` is an alias for `unlimited`. Public JSON represents an uncapped `usage.limit` as `null`, and human displays say “no limit.”
+
+When a numeric total cap is configured, for a one-attempt extension without changing that global cap, `check --id <id> --retry-review` and the equivalent human choice in the interactive recovery menu remain available for failed, questionless checkpoints. Under its session lock, it validates the unchanged policy/snapshot, records prior attempt usage, and allows at most `maxModelCalls` additional requests. Total calls remain cumulative, and the public usage limit reflects the extension. Provider errors expose only numeric status and allowlisted error categories, never raw response messages or metadata. With no cap, or when enough calls remain for review plus grading, the interactive menu offers a retry within that allowance; otherwise it offers a new allowance under the explicit cap. Without a cap, a retry reuses saved work and never asks for an allowance extension. Pause is always the default, and every further failure requires a fresh choice. Known setup/auth/context failures and zero-request preflight failures do not offer another spending attempt; stale content or policy stops recovery. The browser page or explicit terminal stays open while the human decides, allowing the original Git hook to remain waiting within its deadline. JSON commands keep their noninteractive failure behavior. Ordinary resuming and background work adopt an explicitly edited global ceiling but never authorize an additional per-attempt allowance; active or passed questionnaires cannot be restarted. Provider escalation remains explicit. A local completion record is a workflow convenience, not a cryptographic attestation.
+
+## Installation
+
+A global Git `core.hooksPath` points to Sphinx dispatchers. Existing hooks are chained with their arguments and input preserved. Unrelated hook protocols are forwarded directly so interactive/streaming hooks are not buffered. Repository-level overrides need `sphinx install --repo` to wrap the repository's effective hook directory.
+
+The installation manifest tracks original settings and the exact files/handlers Sphinx created. Uninstall restores only configuration it still owns; subsequent edits are preserved. `--agents auto` detects Codex, Claude, Pi, Cursor, and OpenCode from executables, configuration directories, or an existing installation record. An explicit comma-separated subset selects integrations; `none` selects Git-only setup.
+
+Interactive setup begins with coding-agent checkboxes, with All supported agents first, then offers economy reviewer presets based on the selected clients. Users can choose paid OpenRouter, free OpenRouter as an explicit best-effort option, or custom per-tier routes. Existing configurations retain their saved routes unless the user selects a replacement. A deliberate paid selection in guided setup supplies consent; unattended `install --preset` and `config --preset` require `--allow-paid` or an existing paid-enabled setting. Fresh unattended installation without a preset or explicit reviewer route stops before installing. Credential setup remains separate, uses hidden input for OpenRouter keys, and makes no model request.
+
+Codex and Claude integrations register a local stdio MCP server and inject guidance at session start. Codex uses a native `PostToolUse` MCP tool hook for `Bash`; Claude uses `PostToolUse` and `PostToolUseFailure` for `Bash`. Pi installs `extensions/sphinx.js` under `PI_CODING_AGENT_DIR` or `~/.pi/agent`; a new session or `/reload` loads it. Cursor adds `sessionStart`, `postToolUse`, and `postToolUseFailure` handlers to `~/.cursor/hooks.json`. OpenCode installs `plugins/sphinx.js` in its configuration directory, honoring `OPENCODE_CONFIG_DIR` or `XDG_CONFIG_HOME`, without rewriting `opencode.json` or JSONC settings.
+
+After configuration succeeds, interactive `sphinx install` opens the selected installed clients by default. `--no-launch` defers activation; `install --yes` opens clients only with explicit `--launch`. `sphinx activate` opens already-configured integrations without reinstalling, and `--agents codex,claude,pi,cursor,opencode` can select a configured subset. `--json` returns launch outcomes without decorative output. Repository-only `install --repo` remains a hook-compatibility operation.
+
+Activation starts fresh Codex, Claude Code, Pi, and OpenCode CLI sessions, or a new Cursor editor window. It supplies no initial model prompt, invokes no Sphinx evaluator, and does not restart existing processes. Missing executables or headless environments produce a focused manual-launch fallback. A successful launch means the client opened, not that its MCP connection, form support, or hook trust has been verified. Native client trust decisions remain with the human; Codex's hook browser must be opened with `/hooks`. Sphinx does not pass a slash command as an initial model prompt or use an undocumented approval deep link.
+
+The Git hook remains the canonical local checkpoint. Client integrations consume already queued work; they do not let a model decide whether a check is required. Evaluator sessions tagged `SPHINX_EVALUATOR=1` skip queueing and client presentation. Tests redirect integration homes and never install into a developer's real configuration.
+
+## References
+
+- [Pi SDK](https://pi.dev/docs/latest/sdk)
+- [Pi compaction](https://pi.dev/docs/latest/compaction)
+- [Pi SDK full-control example](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/examples/sdk/12-full-control.ts)
+- [Pi extensions](https://pi.dev/docs/latest/extensions)
+- [Cursor hooks](https://cursor.com/docs/hooks)
+- [Cursor MCP support](https://cursor.com/docs/mcp)
+- [OpenCode plugins](https://opencode.ai/docs/plugins/)
+- [Git hook protocols](https://git-scm.com/docs/githooks)
+- [Git central hooks path](https://git-scm.com/docs/git-config#Documentation/git-config.txt-corehooksPath)
+- [Codex hooks and trust](https://learn.chatgpt.com/docs/hooks)
+- [Codex CLI reference](https://learn.chatgpt.com/docs/cli/reference)
+- [Claude Code hooks](https://code.claude.com/docs/en/hooks)
+- [Claude Code CLI reference](https://code.claude.com/docs/en/cli-reference)
+- [MCP form elicitation](https://modelcontextprotocol.io/specification/2025-11-25/client/elicitation)
+- [MCP initialization and protocol negotiation](https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle)
+- [MCP stdio transport](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports)
+- [OpenRouter structured outputs](https://openrouter.ai/docs/guides/features/structured-outputs)
+- [OpenRouter tool calling](https://openrouter.ai/docs/guides/features/tool-calling)
+- [OpenRouter provider price limits](https://openrouter.ai/docs/guides/routing/provider-selection)
+- [OpenRouter Auto Exacto routing](https://openrouter.ai/docs/guides/routing/auto-exacto)
+- [Qwen3 Coder Next endpoints](https://openrouter.ai/api/v1/models/qwen/qwen3-coder-next/endpoints)
+- [Qwen3 Coder endpoints](https://openrouter.ai/api/v1/models/qwen/qwen3-coder/endpoints)
+- [DeepSeek V3.2 endpoints](https://openrouter.ai/api/v1/models/deepseek/deepseek-v3.2/endpoints)
+- [Qwen3 Coder Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next)
+- [Codex model configuration](https://learn.chatgpt.com/docs/models)
+- [Claude model configuration](https://code.claude.com/docs/en/model-config)

package/eval/README.md

@@ -0,0 +1,38 @@
+# Reviewer evaluation
+
+The [measured results](RESULTS.md) record the synthetic live runs, failed comparisons, manual quality findings and reported costs.
+
+Run the deterministic baseline:
+
+```sh
+npm run eval -- --out /tmp/sphinx-eval-baseline.json
+```
+
+It creates five temporary, committed Git repositories, verifies their curated facts, and runs Sphinx's real Pi reviewer, immutable workspace, evidence validation and grading pipeline with scripted model responses. It also runs existing missing-evidence, native compaction and explicit recovery regressions. This tests the harness and fixtures; it does not measure model quality. Temporary repositories and their isolated Sphinx homes are removed afterward. No real pending checkpoint or receipt is read or created.
+
+`npm test` and `npm run check` preload a test-only network guard into Node and its spawned children. It rejects outbound fetch/HTTP/socket connections while permitting loopback and local IPC fixtures, preventing a partial model mock from silently reaching a real provider. Explicit live evaluations use the separate `npm run eval -- --live` entry point. The guard is not an OS sandbox for arbitrary non-Node executables.
+
+The cases cover scoped card styling versus unchanged toolbar controls, process-local retry deduplication including a failure-handling bug, tenant authorization, environment override precedence, and retry behavior across twelve changed handlers. Each includes committed secret files excluded by built-in rules and `.sphinxignore`, plus dirty working-tree content. Distinct canaries and excluded paths are checked in memory against every outbound model payload and retained evidence.
+
+Run a bounded live probe:
+
+```sh
+npm run eval -- --live --case cosmetic --model openrouter/free \
+  --max-calls 20 --max-cost 0.10 --out /tmp/sphinx-eval-cosmetic.json
+```
+
+Without `--model`, live mode uses the configured global OpenRouter tier routes. Paid routes require `--allow-paid` for this invocation, even if the product config permits them. The saved OpenRouter key or `OPENROUTER_API_KEY` is passed only in memory into the isolated fixture environment. Credentials and raw HTTP headers are never saved in reports. Use `--case retry,auth` to select cases, or `--probes generated` to focus on generated-question grading while iterating. Live requests pause for 1 second after a response by default; `--delay-ms` adjusts that interval. The first provider/authentication/transport failure stops the run and identifies probes that were not attempted. Failure telemetry includes the provider name, error type, a bounded credential-redacted message, and a valid numeric/date `Retry-After` value when returned.
+
+The run-wide request limit covers generation, Pi compaction, grading and failed requests; there are no automatic retries or route changes. `--max-cost` uses provider-reported USD cost, including failed responses when reported. A completed request can exceed the threshold; the next request is blocked. Missing cost telemetry also blocks the next request when this option is set. This is a stop condition, not a guaranteed billing cap.
+
+`--max-output-tokens 8192` can raise the per-request output allowance for a reasoning-heavy model in this evaluation only (allowed range: 512–16384). The report records this limit, including the model's reasoning allowance; it does not change saved product configuration. Transfer questions receive the production independent source audit too, and that extra request counts toward the same run-wide budget.
+
+Artifacts include generated questions and private rubrics **for synthetic fixtures only**, exact retained evidence, curated answer probes, model verdicts, transfer questions, request counts, workspace reads, compactions, token counts and reported cost. They also retain the latest in-memory reviewer checkpoint and last proposed assessment, including after rejection, plus live audit and grading prompt/result/error transcripts, including rejected structured responses. Environment/config objects, raw HTTP headers, and opaque thinking fields are excluded; the supplied key is redacted from every report string. Generated questions are tested with independently authored source-grounded answers and misconceptions. For recognized explicit numeric `BATCH_SIZE` assignments, probes adapt to the visible question using the fixture's executable precedence rule; they never read the model's answer key. Recognized forms now include `BATCH_SIZE=5` and “BATCH_SIZE is set to 5”; detected unsupported assignments are marked uncovered. The latter form was added after the frozen V20 failure, without another live run. Inspect the probe's input scenario before classifying a mismatch; natural-language coverage is not universal. Canonical questions additionally test unsupported premises, a contradictory answer key, and disputes. Both invalid-key probes require ambiguity rather than awarding completion for correcting an invalid question. The retry case checks that identifying the actual failure-handling bug counts as understanding. Generation includes the production fresh question audit, charged within the same run-wide limit.
+
+Concept coverage checks inspect the visible question, not private rubric text. Literal recall and missing causal/boundary prompts are flagged. These checks are explicit heuristics, not another model's opinion. Inspect flagged questions and rubrics before deciding whether a mismatch is a product bug or an incomplete fixture oracle. A passing fixture run is not a claim that arbitrary reviews are correct. The output report is refreshed after each case so a long run can be inspected before completion. For iteration, retain each report, change one behavior, rerun the same case and route, and compare the question, evidence, verdicts and total cost. `--json` prints the machine-readable report; a failed probe produces exit code 1, while `needs-review` is surfaced separately.
+
+The CSS fixture checks selector/markup facts without a browser render. Its question tests propagation to an unchanged surface, not numeric CSS recall. Its oracle therefore does not claim to verify computed layout or visual appearance.
+
+New questions carry private `acceptance.prediction` and `acceptance.causalReason` fields, separate from their fuller source-derived explanation. The curated fixtures author these minimum requirements independently, preserve the existing human answer probes, and still test contradictory full keys. Live reports retain the criteria so manual review can check both fairness and whether an essential asked behavior was omitted; matching a model's own criteria alone is not proof of a good assessment.
+
+[Known false negatives](fixtures/known-false-negatives.json) preserve the exact V18 styling and V20 authorization questions, criteria, sufficient answers, observed ambiguous verdicts and synthetic source definitions. These are unresolved regressions, not passing tests. To replay one, select its `source.caseId` from `CASES`, overlay the saved `source.definition`, and call `createFixture(fixture, config)`. Read immutable evidence with `createReviewWorkspace(snapshot, config, { env })` and `canonicalEvidence(snapshot, record.source.requiredPaths, workspace)`, then build a review bound to that new `snapshot.id` and call `evaluator.gradeAnswer({ snapshot, question: record.question, answer: record.answer, review, config, env, beforeCall: budget.beforeCall })`. Use the existing metered evaluator transport and an explicit live budget; do not reuse an old temporary snapshot ID, re-audit to an easier question, or replace the saved human answer. This replay is not part of the default deterministic command.

package/eval/RESULTS.md

@@ -0,0 +1,205 @@
+# Evaluation results — 2026-10-15
+
+These are development measurements on five small, synthetic Git repositories, not a certification of arbitrary code reviews. The source changed between the earlier runs; the model comparisons are not a controlled benchmark. No real pending checkpoint, human answer, or receipt was used. Saved model routes and output limits were not changed by these evaluations.
+
+The latest frozen-source coverage, V19/V20, generated seven questions across all five fixtures. Twelve of fourteen generated-question verdict probes matched; no canonical probes were rerun in this coverage. Fresh styling, retry and multifile checks passed independent source review. Authorization still rejected a sufficient explanation for not naming a specific code condition. Configuration exposed an uncovered fixture input phrasing and a malformed grade that validation rejected. The result is **not an all-green semantic evaluation**, despite all 560 guarded unit/integration tests passing.
+
+Explicit minimum acceptance criteria improved the fresh styling question, but did not establish consistently fair grading. The failed legacy styling replays remain below, as do the current authorization failure and earlier model/provider failures. No tested route is established here as reliably correct for arbitrary repositories or answers.
+
+## Reproduce
+
+```sh
+npm ci
+npm run eval -- --out /tmp/sphinx-eval-scripted.json
+npm run eval -- --live --case cosmetic --probes generated \
+  --model anthropic/claude-haiku-5.5 --allow-paid \
+  --max-calls 10 --max-cost 0.04 --max-output-tokens 8192 \
+  --delay-ms 1000 --skip-regressions --out /tmp/sphinx-eval-live-cosmetic.json
+npm run eval -- --live --case retry,auth,config,multifile --probes generated \
+  --model anthropic/claude-haiku-5.5 --allow-paid \
+  --max-calls 35 --max-cost 0.15 --max-output-tokens 8192 \
+  --delay-ms 1000 --skip-regressions --out /tmp/sphinx-eval-live-remaining.json
+```
+
+Live inference requires an existing OpenRouter key. Each invocation stops on a provider/transport failure; it does not automatically retry or change the selected model. The cost stop uses reported cost, may be exceeded by the final completed request, and blocks further requests if cost is missing. These are evaluation limits; they do not extend a product checkpoint's allowance.
+
+The V11/V12 live measurements used the working tree based on `34bd7da9790b4110ec744256800ba1b847224303`, including uncommitted reviewer, evidence, audit, and eval changes. Pi was pinned to 1.2.0. The model-facing source loaded by V11 and V12 was the same. A separate presentation fix removed private concept labels from human questionnaires; a later transport diagnostic fix does not change model prompts. The transfer-distance fix described below was tested separately after V12 loaded its baseline. This commit alone therefore does not reproduce the worktree. The SHA-256 of the ordered path/content pairs for `src/evaluator.js`, `src/reviewer.js`, `src/pi-runtime.js`, `src/review-workspace.js`, `eval/cases.js`, `eval/harness.js`, and `eval/run.js` was `39bfce2b350afeb2f5d32bd42f6c9649e3bc9e2ad20e09e34aac7dbf33bcb39b` (UTF-8 path, NUL, file bytes, NUL for each).
+
+## Deterministic baseline
+
+`/tmp/sphinx-eval-scripted-final.json` passed all five fixtures and 35 verdict probes: 25 canonical probes plus 10 probes against generated questions. It made 59 scripted requests, including independent question and transfer audits. This exercises the real Pi/workspace/schema/grading pipeline with fake responses; it establishes harness behavior, not model quality. The related eval unit suite passed 14 tests. Seven selected regressions passed, covering omitted/invented evidence IDs, retained changed diffs, unread citations, native Pi compaction and explicit recovery. Earlier V11 artifacts ran only four of those regressions because a renamed evidence test no longer matched the runner's filter; the corrected final baseline retains the original artifacts.
+
+The fixtures verify scoped CSS propagation, retry deduplication after failure, tenant isolation, environment/default precedence, and retry limits across twelve handlers. Executable source checks establish the non-CSS answers. CSS checks establish selector/markup relationships, not rendered layout. Tracked ignored files and dirty working-tree canaries were absent from every inspected outbound payload and retained evidence.
+
+After the normalization and neutral-question wording changes, the complete guarded unit/integration suite passed 545 tests with no skips. The focused reviewer/recovery/evaluator run passed 83 tests, including recorded synthetic envelopes replayed through real Pi and immutable temporary Git repositories.
+
+## Final-source GLM measurements
+
+V11 used a full five-case run with `--max-calls 100 --max-cost 0.25 --max-output-tokens 8192 --delay-ms 3000`. A transport failure stopped it before authorization assessment generation. V12 was a separately authorized run of the remaining cases using the same limits and `--case auth,config,multifile`; this was not an automatic retry. Both used `z-ai/glm-5.3-flash`, with the output allowance including reasoning.
+
+| Case / run | Questions | Assessment requests | Canonical verdict probes | Generated verdict probes | Total requests | Reported USD |
+|---|---:|---:|---:|---:|---:|---:|
+| Scoped styling / V11 | 1 | 3 | 5/5 | 2/2 | 14 | 0.023045310 |
+| Retry deduplication / V11 | 1 | 4 | 5/5 | 2/2 | 15 | 0.020155650 |
+| Authorization / V11 | none | 1 failed | unrun | unrun | 1 | missing |
+| Authorization / V12 | 2 | 3 | 5/5 | 4/4 | 17 | 0.026317280 |
+| Configuration / V12 | 1 | 3 | 5/5 | 2/2 | 14 | 0.016660060 |
+| Twelve handlers / V12 | 3 | 8 | 5/5 | 5/6 | 25 | 0.037822720 |
+
+Assessment requests include investigation, synthesis and the independent assessment audit. Total requests also include grading and a fresh audit for every proposed replacement question. Workspace reads are local operations and are not model requests. Verdict probe counts are separate from manual question-quality review.
+
+Across the completed five-fixture coverage, eight generated questions received 16 generated-question probes; 15 completed as expected and one failed validation. All 25 canonical verdict probes matched. V11 plus V12 used 21 successful assessment requests, one failed assessment request, and 86 total requests: $0.124001020 reported, with cost telemetry for 85 requests. No live compaction occurred in these cases; compaction coverage is deterministic.
+
+Manual review accepted the styling question's propagation boundary and the authorization questions' tenant boundary and indistinguishable denial/missing responses. Their automatic concept-coverage flags were vocabulary false positives; raw flags were retained. The retry questions and audited replacement keys were source-correct. One rubric mentioned process restart although the visible question did not ask it. A separate, independently written minimal-answer probe omitted restart information and was accepted as correct: one request, $0.000912350, no replacement question.
+
+Known limitations remain: one styling feedback sentence was incomplete despite a complete structured response; an authorization replacement mostly renamed the original scenario rather than testing substantially different application of the concept. The configuration question correctly tested default/override precedence, but its wording gave away that the override was unaffected. Its wrong-answer replacements tested a useful new boundary: a set-but-empty value becomes zero instead of using the default. Passing these verdict probes does not prove grading fairness for all shorter answers, semantic correctness of every future audit, or sufficiently distinct transfer questions.
+
+The multifile failure was retained, not converted to a pass. The independent replacement audit changed a valid ID to `permanent-id?`, which the existing validator rejected before accepting the grade or replacement. Manual inspection found a second defect in that rejected candidate: it described `await run()` without the operation argument required by the inspected handler, yet predicted three side effects from that operation. A deterministic reproduction gives zero counter increments and a `TypeError`; the audit missed the incomplete call. Its claim of no measurable delay also exceeded the source's narrower guarantee of no intentional backoff. The main exhaustion question's lack of an explicit causal-explanation request remains a quality flag. Thus the final live matrix is not all green, even though the accepted source keys and 40 completed verdict checks were consistent with the fixtures.
+
+V11's authorization request returned HTTP 200 headers but no parsed completion or cost telemetry after approximately 180 seconds. Sphinx failed closed and issued no receipt. A deterministic aborted-body reproduction confirmed that the old transport diagnostic classified its own deadline as invalid/oversized JSON; the missing completion is not evidence of a semantic model failure. No timeout was extended for the next run.
+
+## Focused transfer-distance follow-up
+
+The label-only authorization transfer exposed missing input to its audit: the audit did not receive the previous public question. A narrow change now supplies that question and asks for a changed condition, ordering, boundary or prediction rather than renamed labels. This was evaluated separately; the V12 process continued using its already-loaded baseline.
+
+Two authorized audit-only replay requests used the saved synthetic authorization evidence and candidate. At 8192 tokens, the first exhausted its output allowance, including 7675 reasoning tokens. At 16384, the second produced a source-correct new unauthenticated-caller scenario, but changed the private concept string and was rejected by the existing exact-concept validator. Neither replay produced an accepted product result. Together they cost $0.008992400; the second report's usage is cumulative and must not be added to the first. These observations support neither a claim that 8192 prevents truncation nor a claim that the transfer-distance fix passed a complete live flow.
+
+Afterward, the output schema constrained the replacement's private concept to its existing value, matching the validator. The ID schema likewise now advertises its existing 1–80 character and allowed-character rules. Guarded evaluator/eval tests passed 45 cases. These schema fixes do not establish that the missed operation argument was semantically repaired.
+
+## Bounded Haiku compatibility comparison
+
+The rejected multifile key motivated one separate comparison, without changing the fixture or oracle:
+
+```sh
+npm run eval -- --live --case multifile \
+  --model anthropic/claude-haiku-5.5 --allow-paid --max-calls 35 \
+  --max-cost 0.10 --max-output-tokens 8192 --delay-ms 3000 \
+  --out /tmp/sphinx-eval-live-multifile-haiku-v13.json
+```
+
+OpenRouter rejected the first request with HTTP 404 because no endpoint supported the requested `tool_choice` value. No assessment or grading ran; no cost was reported, and Sphinx did not retry. This is evidence of route incompatibility for this request, not evidence about Haiku's question quality. The process loaded the concept constraint but started before the ID schema parity change. Its same-files source fingerprint was `d133d4d15db043ee66effb38fa46f9d0413d1308997b8fc9f9f27f23d045de76`.
+
+After the transport used Anthropic's supported automatic tool selection while retaining Sphinx's local final-tool enforcement, a separately authorized five-case V14 run used the same model, 8192 output tokens, a 100-request/$0.15 stop, and 1000ms spacing:
+
+```sh
+npm run eval -- --live --model anthropic/claude-haiku-5.5 --allow-paid \
+  --max-calls 100 --max-cost 0.15 --max-output-tokens 8192 \
+  --delay-ms 1000 --out /tmp/sphinx-eval-live-matrix-haiku-v14.json
+```
+
+| V14 case | Assessment requests | Accepted questions | Verdict probes | Total requests |
+|---|---:|---:|---:|---:|
+| Scoped styling | 4, failed | none | unrun | 4 |
+| Retry deduplication | 4, failed | none | unrun | 4 |
+| Authorization | 6 | 1 | 7/7 | 17 |
+| Configuration | 4, failed | none | unrun | 4 |
+| Twelve handlers | 5 | 2 | 9/9 | 19 |
+
+All 48 responses explicitly reported zero cost; this records that run's telemetry and does not establish free model pricing. The 16 verdict probes comprise ten canonical and six generated probes. Independent inspection found all three accepted main keys and nine accepted replacement keys source-correct, including the missing-tenant-field boundary and retry exhaustion/recovery. Auth's prompt was somewhat leading; the unaccepted styling/configuration candidates also supplied part of their answers. One multifile replacement overlapped a different initial question. Those quality limitations remain separate from the successful verdict counts.
+
+The three smaller cases failed on response shape before their audits: their first finish payload placed `summary` inside the `assessment` object rather than beside it. Subsequent repair attempts emitted malformed JSON strings and exhausted the bounded review allowance. A lossless normalizer was added afterward for both transports: it decodes only valid JSON-object strings and moves known metadata only when the outer field is absent. It preserves question/evidence values, unknown fields, conflicting metadata and raw native continuation; strict validation still rejects bad JSON and unread citations. Recorded first-payload fixtures exercise this fix without another live request. V14 remains failed and was not rewritten as a pass.
+
+V14's source sidecar is `sphinx-eval-live-matrix-haiku-v14-source.json`; it adds `src/openrouter.js` to the ordered fingerprint inputs listed above. Its fingerprint is `0037ff2cfb1ac5e7cae8e05976dc43ec154dba0ba04f6bf25396c95a07d8cdf2`. The normalization change was not loaded in this run.
+
+## Post-normalization Haiku matrix
+
+V15 repeated the full V14 command with output `/tmp/sphinx-eval-live-matrix-haiku-v15.json`, after lossless finish-envelope normalization and neutral-question wording. Model-facing source was frozen throughout the run. The ordered source fingerprint added `src/reviewer-response.js` and was `d803245a77cdbcacc240df9fd016b42b81c4c382c2dea81dbe210752a5449224`; the full path/hash list is in the corresponding `-source.json` sidecar.
+
+| V15 case | Assessment requests | Questions | Canonical verdict probes | Generated verdict probes | Total requests |
+|---|---:|---:|---:|---:|---:|
+| Scoped styling | 3 | 1 | 5/5 | 1/2 | 15 |
+| Retry deduplication | 3 | 1 | 5/5 | 2/2 | 14 |
+| Authorization | 2 | 2 | 4/5 | 3/4 | 15 |
+| Configuration | 3 | 1 | 3/5 | 2/2 | 16 |
+| Twelve handlers | 4 | 2 | 5/5 | 3/4 | 18 |
+
+All five assessments were accepted, using 15 assessment requests. The complete run used 78 requests; all 78 explicitly reported zero cost. There was no provider failure, truncation, or automatic route change. Seven deterministic regressions also passed. This is **not** an all-green quality result: 33 of 39 verdict probes matched, and manual inspection found problems beyond verdict counts.
+
+The styling question copied CSS and markup into a numeric-padding question and required exact pixels and specificity notation. Its key was source-correct, but it rejected the correct qualitative explanation of affected card controls and unchanged toolbar controls. Four of its five replacements repeated this numeric focus. This violates the intended behavioral-understanding scope; adding “why” to property arithmetic does not fix it.
+
+Authorization had two product failures: a correct equivalent explanation of the shared bodyless 404 response was withheld for not naming the literal combined `if`; and a corrected invalid-key replacement repeated the original prompt, so novelty validation rejected it before the source auditor could rewrite it. The replacement already had a fresh ID, so changing IDs would not repair the issue. The other authorization question asked whether lookup occurred for an unauthenticated caller; the broad fixture answer omitted that ordering but was nevertheless accepted. This is a separate false positive and probe-coverage weakness, not a reason to accept incomplete predictions.
+
+Two configuration failures were **fixture evidence defects**: canonical grading supplied only HEAD bodies, while its precommitted key claimed the default had increased. The source-correct generated question and both generated probes passed. The question still tested a narrower default/fallback case than the more useful default-versus-override comparison. The multifile generated failure was also a **probe-coverage gap**: the question asked for a second-call successful return, while the fixed answer described the cap and generic return-on-success without explicitly predicting that scenario. Withholding a pass was defensible. Its wrong-answer feedback did incorrectly say “exactly three” attempts while discussing a scenario that succeeds on the second; the incorrect verdict itself was justified.
+
+Independent review found all seven accepted main keys and 24 accepted replacement keys source-correct. Retry behavior was consistently graded; one replacement omitted an explicit request for the reason, although the questionnaire's shared instructions request it. These findings do not erase the styling and grading failures.
+
+After the frozen run, the harness was corrected to include actual filtered immutable diffs alongside HEAD dependencies for canonical probes. Source-executed fixture checks now establish unauthenticated early return before lookup and a transient retry resolving with the successful second result. The revised probes do not derive answers from model keys. Sixteen guarded eval tests passed; the V15 report remains unchanged. Focused product fixes and follow-up measurements are reported separately rather than relabeling V15.
+
+## Focused Haiku follow-up
+
+V16 replayed the exact saved styling candidate through the production assessment audit, then graded the same qualitative correct answer and misconception. It also checked equivalent authorization reasoning, complete and omitted lookup ordering, an invalid-key replacement, configuration probes with actual diff evidence, and transient retry probes with the independently completed answer. A scripted investigator supplied the saved candidate; its two turns were not live requests. The captured provisional invalid-key grade was likewise replayed, with only its existing source audit live. All other grading was live on fresh isolated immutable fixtures.
+
+Ten of eleven focused checks passed using 15 actual requests, each explicitly reporting zero cost. Summed check duration was 107 seconds. The 11 checks include one assessment audit and ten grading/replacement checks; they are not eleven independently generated assessments. The source fingerprint was `6993ace15095bdd328d3c95ec8e82f8ce6ea5d2a1bbed602ac59282f90223f8c`.
+
+The audit removed numeric arithmetic from the styling question, but Haiku still withheld a pass from the sufficient answer: “new padding” on card buttons plus unchanged horizontal padding establishes a vertical change, and the answer gave the ancestor cause and unchanged toolbar boundary. Its redundant clarification transfer was rejected by the audit. This is a remaining semantic false negative, not a successful fairness check; the conditional fresh styling-generation run was not launched.
+
+Authorization's equivalent explanation was accepted, a complete early-return explanation passed, and omission of the explicitly requested ordering remained ambiguous. The invalid-key replacement reached the existing audit, which rewrote it into a source-correct base/head comparison; the verdict stayed ambiguous. Both configuration and transient retry checks passed. Their audited replacements were source-correct and distinct. A retry citation reason still loosely described the attempt cap as exactly three; the feedback itself correctly discussed the second-call success.
+
+An initial temporary replay-wrapper mistake bound rebuilt source observations to the old synthetic snapshot ID. Eight remaining probes failed locally before any requests; the untouched initial artifact retains those zero-call errors. The corrected wrapper continued only the unrun probes and carried forward all five already-used styling requests. `/tmp/sphinx-eval-focused-haiku-v16.json` is cumulative and must not be added to its `-initial.json` artifact. This correction did not retry a model request or erase the styling failure.
+
+After this measurement, an optional replacement rejected by an insufficient-evidence audit no longer turns an ambiguous grade into a provider-style failure: it keeps the original question pending with clarification feedback. Incorrect answers still require an audited transfer; provider, cancellation, context and malformed-output errors still propagate. This recovery change has deterministic coverage and does not make the false-negative answer correct. The revised scripted baseline passed five fixtures, 35 verdict probes and seven regressions with 59 fake requests; the full guarded suite before that final recovery change passed 550 tests.
+
+## Stronger-model fairness falsification
+
+V17 used `anthropic/claude-sonnet-5.5` with the exact saved V16 qualitative styling question, key and unchanged sufficient answer. It did not first rewrite the question to an easier one. The model returned ambiguous, requiring explicit vertical-change wording and higher-specificity terminology despite the answer establishing new card padding, unchanged horizontal padding, the ancestor scope and unchanged toolbar. This reproduced the material false negative.
+
+The predetermined stop condition ended the comparison after that first request: 4.6 seconds, one explicitly reported zero-cost response. The planned misconception, authorization and five-fixture generation checks were **not run**. This provides no evidence that Sonnet passes the broader matrix or that increasing model cost solves the grading-design problem. The report is `/tmp/sphinx-eval-focused-sonnet-v17.json`, with source fingerprint `413f07e509a00922375faf3176aee438ce01ed8ba5dbc7b96af71643a0c3ceae`. No further route comparison followed this result.
+
+## Explicit minimum-criteria replay
+
+The next change separates private minimum acceptance into `acceptance.prediction` and `acceptance.causalReason`. New questions and replacement audits must produce both; fuller `expectedAnswer` and `rationale` remain independently source-checked explanations, not extra pass requirements. Grading receives those explanations in a separate `sourceKey`. Existing questions remain readable. Curated eval criteria were authored from source without changing human probes, and 17 eval tests passed. The scripted acceptance baseline passed all five fixtures and 35 verdict probes using 59 fake requests.
+
+V18 source-audited the exact legacy V16 question through the same audit wrapper used by production generation. The audit added source-correct criteria without requiring specificity terminology. Haiku then still marked the unchanged sufficient answer ambiguous for not explicitly saying “vertical”, despite new card padding plus unchanged horizontal padding establishing that result. The criteria separation therefore has structural test coverage but **did not pass this live fairness regression**.
+
+The run stopped after two requests and 21.5 seconds, both with explicit zero-cost telemetry. No misconception, fresh styling generation or remaining-case matrix ran because their prerequisite failed. The report is `/tmp/sphinx-eval-acceptance-haiku-v18.json`; source fingerprint `9bbcff04e1281c1178ce44f14def6f2c7e26b739abd632489060009cf7b65eaf`. The ambiguous/null result leaves the checkpoint pending and grants no approval; safe noncompletion is separate from fair grading.
+
+## Fresh generation with minimum criteria
+
+V19 separately tested fresh styling generation, rather than auditing the old compound candidate again. It used the first live command above. After independent review accepted that case, V20 used the second command for the four remaining fixtures. Both used `anthropic/claude-haiku-5.5`, 8192 output tokens and the same frozen source. No models, prompts or human probes changed between them. The assessment policy now includes contract version 3, so production questions and receipts prepared under the earlier contract are not silently reused.
+
+| Case / run | Questions | Assessment requests | Generated verdict probes | Total requests |
+|---|---:|---:|---:|---:|
+| Scoped styling / V19 | 1 | 3 | 2/2 | 6 |
+| Retry deduplication / V20 | 1 | 3 | 2/2 | 6 |
+| Authorization / V20 | 2 | 3 | 3/4 | 9 |
+| Configuration / V20 | 1 | 3 | 1/2 | 6 |
+| Twelve handlers / V20 | 2 | 4 | 4/4 | 10 |
+
+The two runs used 37 actual requests: 16 for assessments, 14 for grading and seven for replacement audits. All 37 explicitly reported zero cost; that is provider telemetry, not a claim that the route is free. V19 took 40.1 seconds and V20 took 241.3 seconds. There was no provider failure, truncation, live compaction or automatic retry. The source fingerprint adds `src/config.js` to the nine V15 inputs and is `d7d13eeafc213420d9f2d999c11a98751613e015eb5556a826920e98eb514661`. Corresponding source sidecars retain each file hash.
+
+Fresh styling tested one propagation boundary: which of a card button and a toolbar button receives the changed vertical padding, and why. Its minimum criteria required the affected/unchanged surfaces and the card ancestor relationship, without numeric padding, specificity notation or an extra horizontal-axis checklist. The same unchanged qualitative answer rejected in V18 was accepted; the global-class misconception was rejected. The audited transfer placed a card inside a toolbar and correctly tested the resulting ancestor match. This is evidence for that fresh question, not a successful replay of V18.
+
+The authorization failure is a current product false negative. The answer explained that another tenant gets a bodyless 404 “just like a missing id,” establishing the requested response equivalence. Grading still demanded an explicit “no” and the literal combined-condition mechanism. The source-audited acceptance criteria themselves unnecessarily named a single combined condition. Both question keys and accepted replacements were source-correct; source correctness did not make that sufficient answer fairly graded.
+
+The configuration question used “BATCH_SIZE is set to 5.” The independently authored probe adapter recognized `BATCH_SIZE=number` but not this phrasing, and supplied its original example with 7. That is an uncovered scenario, so this result does not establish how the model would grade a sufficient answer to the stated input. Separately, the raw response called the omitted scenario prediction ambiguous and acknowledged no material misconception, yet emitted `incorrect` with no required transfer. Strict validation rejected that malformed result. The report remains failed; neither failure was relabeled as a pass or rerun with a tailored answer. The wrong-precedence probe was rejected correctly, and its unset-environment transfer was source-correct.
+
+After the frozen run, the fixture adapter was extended to parse the visible “is set to” numeric assignment and mark other detected unsupported assignment forms uncovered. It still uses only the independently implemented numeric precedence rule, never a model key or criterion. This is a fixture correction with no further live measurement; V20's source fingerprint, answers and failure remain unchanged.
+
+Retry and multifile predictions, minimum criteria and accepted replacement keys were independently checked against source. The retry transfer changes failure to success and tests the duplicate call's uncached `undefined` return. Multifile checks cover second-call success and final rejection after three total attempts; replacements test an unreachable fourth-call success and a hypothetical limit of two. The latter is a modest parameter-change transfer, but its count affects actual retry behavior. All seven accepted main keys and seven accepted replacement keys in V19/V20 were source-correct on manual review. Styling, authorization-denial and multifile-final-failure keyword flags were vocabulary false positives; their raw flags remain.
+
+This coverage is generated-only. Earlier canonical results are retained under their original source versions, and no complete canonical matrix was rerun after contract version 3. The latest scripted acceptance baseline passed five fixtures, 35 verdict probes and seven selected regressions using 59 fake requests; eval unit tests passed 17 cases. The final guarded suite passed 560 tests with no skips. Those deterministic checks verify implementation behavior and privacy boundaries; they do not override the live semantic failures.
+
+## Earlier failures were retained
+
+| Run | Model | Requests | Reported USD | Observations |
+|---|---|---:|---:|---|
+| Initial cosmetic iterations | Qwen Coder/Next and DeepSeek | 21 | 0.011807669 | Contradictory selector key, unsupported-premise false pass, weak numeric questions, rejected audits, and 429s. |
+| V4 | DeepSeek V3.2 | 50 | 0.073801164 | Invalid-key grading and feedback failures; run-wide allowance left the final case untested. |
+| V5 | DeepSeek V3.2 | 17 | 0.022592460 | Cosmetic numeric recall and an audit that confused distinct retry branches. |
+| V6 | Qwen3 Coder Next | 5 | 0.002084040 | Provider 429; no successful matrix. |
+| V7 | DeepSeek V3.2 | 9 | 0.014308000 | Investigation/finalization exhausted the normal review ceiling. |
+| V8 | DeepSeek V3.2 | 26 | 0.016196685 | Valid questions were marked ambiguous when the human answer was wrong; contradictory retry key. Stopped after two cases. |
+| V9 | Qwen3 Coder | 48 | 0.035690180 | Weak/redundant authorization questions despite 11/11 verdicts; wrong answers sometimes invalidated sound keys; final case timed out. |
+| V10 | GLM 5.3 Flash, 4096 tokens | 25 | 0.033687670 | Two cases had 7/7 verdicts each, but manual review found a replacement key that omitted a possible callback throw. Authorization audit hit the output cap, including 3736 reasoning tokens. |
+
+The V9 generated configuration question used `BATCH_SIZE=15` while its original fixed probe answered `=7`. That mismatch was a fixture coverage defect, not proven bad grading. Subsequent probes derive explicit numeric inputs from the visible question and the independently implemented loader rule, never from a model's answer key. The separate V9 canonical wrong-answer-to-ambiguous failure remains a product failure.
+
+The observed defects led to bounded source inventory/diff seeding, a single repair window within the existing allowance, source-constrained citations, source-first question-validity grading, and independent audits of replacement questions. These fixes did not retroactively change earlier results. No model's assertion that its own question is good was used as the quality oracle.
+
+## Artifacts and accounting
+
+Raw reports remain local under `/tmp`, not in the repository. They contain only synthetic source, questions and rubrics, and sanitized diagnostics; no credentials. The two confirmed sufficient-answer failures from V18 and V20 are also retained in [a small regression fixture](fixtures/known-false-negatives.json), including exact source definitions, question criteria, answers and provenance. Main reports are `sphinx-eval-scripted-final.json`, `sphinx-eval-live-matrix-glm-v11.json`, `sphinx-eval-live-remaining-glm-v12.json`, and `sphinx-eval-retry-fairness-v11.json`. The focused audit reports are `sphinx-eval-auth-transfer-replay-v12.json` and `sphinx-eval-auth-transfer-replay-16384-v12.json`; the compatibility comparison is `sphinx-eval-live-multifile-haiku-v13.json`. Earlier comparisons use `sphinx-eval-live-matrix-deepseek-v4.json`, `sphinx-eval-live-small-deepseek-v5.json`, `sphinx-eval-live-matrix-qwen-v6.json`, `sphinx-eval-live-small-deepseek-v7.json`, `sphinx-eval-live-matrix-deepseek-v8.json`, `sphinx-eval-live-matrix-qwen-coder-v9.json`, and `sphinx-eval-live-matrix-glm-v10.json`. The initial cosmetic reports are `sphinx-eval-live-cosmetic.json`, `sphinx-eval-live-cosmetic-v2.json`, `sphinx-eval-live-cosmetic-v3.json`, `sphinx-eval-live-cosmetic-qwen-coder.json`, and `sphinx-eval-live-cosmetic-deepseek.json`.
+
+The latest reports are `sphinx-eval-live-cosmetic-haiku-v19.json` and `sphinx-eval-live-remaining-haiku-v20.json`, with matching `-source.json` sidecars. The acceptance baseline is `sphinx-eval-scripted-acceptance.json`. Focused reports V16–V18 remain `sphinx-eval-focused-haiku-v16.json`, `sphinx-eval-focused-sonnet-v17.json` and `sphinx-eval-acceptance-haiku-v18.json`; V15 is `sphinx-eval-live-matrix-haiku-v15.json`.
+
+All listed exploratory live runs through V20, the fairness probe and the two audit replays total **472 actual requests and $0.344073638 reported**, with cost telemetry for 465 requests and seven missing reports. The V12 remaining-case run alone used 56 requests and $0.080800060, all reported. Missing telemetry is not assumed to mean a free request; reported totals are not a billing reconciliation. Scripted requests are excluded from these totals.