specpi 0.30.0 → 0.31.0

package/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.31.0 - 2026-09-22
+
+- Report Terminal-Bench 2.0 per sitting rather than per run. Bare Pi -- unchanged software, the same thirteen tasks, the same machine -- solved 21, 22, 30, 30 and 32 of 39 across five launches, so a figure from one run is not evidence about a harness. The page pools every sitting, publishes each harness's per-sitting range beside its rate, and charts the spread directly.
+- Compare harnesses only in the sittings where both ran. An earlier reading set two separate runs side by side and showed SpecPi + Jev 30/39 against Pi 22/39; three sittings running both arms in one launch went +5, level, and +6 the other way. Paired, Pi solves 113/156 and SpecPi + Jev 112/156, p = 1.00, and no pair of harnesses separates pooled or paired.
+- Apply the same rule to tokens and cost, which pooling had overstated. Pooled across unpaired sittings SpecPi + Jev looked 11% cheaper than Pi; paired, it sent fewer prompt tokens in all four sittings (7-30%) but cost less in only three, by about 7%, and more in the fourth. Output tokens are 61-73% of every harness's bill at these cache rates, which is why a consistent prompt-token saving barely reaches the cost.
+- Correct Claude Code's cost from the published $0.1128 upper bound to $0.0273 measured. Its cached share went unrecorded before the Messages translation was fixed; measured, it caches at 95.4%. Paired with SpecPi + Jev it sends 1.9x the prompt tokens and costs 2.4x as much, more than its pooled row suggests, because its one measured sitting was SpecPi + Jev's cheapest.
+- Stop plotting an upper bound as a cost. The cost chart filtered by harness, so once Claude Code's overall cost was measured its calibration slice -- which predates the cache fix -- appeared as a bar seven times the others. It filters by slice now.
+- State the builds the benchmark actually ran. The page gave Pi 0.84.4, the version this repository pins for its own tests; every trial reported 0.87.0, and the figure is now read from them. The SpecPi + Jev row spans two layer builds: the 21 Sep sitting's still carried the withdrawn compaction system, switched off for those runs.
+- Say what the SpecPi + Jev cost leaves out. Harbor records the agent's tokens, not the Jev advisor's, so the advisor's calls -- 320 across the arm by its own ledger -- are excluded rather than estimated, against the rule in `evals/prices.json` that a Jev row includes them.
+- Derive every figure the page and README quote. Counts, rates, ranges and p-values in the prose are filled from `terminal-bench-2.json` at load, and `npm run check:site` fails on any slot left empty. The README's hand-written sentence had already gone stale, quoting p = 0.83 after a rerun moved it to 0.40.
+- Add OpenCode and the DeepSeek Harness to Terminal-Bench 2.0, in one sitting beside Pi and SpecPi + Jev. OpenCode solved 29/39 and the DeepSeek Harness 21/38, and neither separates from a harness it ran beside. OpenCode's reasoning tokens are now counted: Harbor records them apart from its output, they were 2.6 times that output, and they put its cost at $0.0124 an attempt rather than $0.0064.
+- Limit the solve-rate claim to paired comparisons. Pooled, Oh My Pi and SpecPi + Jev now lead the DeepSeek Harness at p = 0.005 and 0.034, and the page names them. The one paired sitting where SpecPi + Jev sent more prompt tokens than Pi (27% more) is stated beside the "fewer" range instead of being folded into it.
+- Withdraw progress detection from the Jev layer. Replayed over the 174 recorded Terminal-Bench runs where its local gate fired (`scripts/jev-progress-replay.mjs`), its stuck verdict did not predict failure -- within-task AUC 0.46, with the shipped state and with one that added the recent commands -- and its `failure_mode` answer echoed the reasons in its own state 624 times out of 624, so every nudge said calls had failed, including when none had. The local gate alone carried the signal (58% of flagged runs passed, 78% of the rest), and it needs no classifier. It was the only system that could steer the model mid-session. Settings schema 6 drops `systems.progress`, `budgets.progress` and `progressNudge`; the default session total falls from 2048 to 1536 so it stays below the per-system ceilings it bounds. The system as it last shipped moves to `scripts/jev-progress-system.mjs`, so the replay and the offline triage stay reproducible.
+- Stop asking retention about `read` results. Every one of its 39 calls in the 22 Sep sittings was a `read`, none shortened anything, and half came back with confidence 0: a file the agent just asked for is load-bearing on arrival. Listings, searches and fetched content remain eligible.
+- Drop two questions whose answers nothing read: retention's `result_kind` and capability arming's `task_kind`.
+- Record why each advisor call was made and the distribution behind each answer. Ledger lines now carry the local trigger labels behind capability-arming calls and the Choice and Score probabilities, so a margin gate can be re-checked afterwards. Labels are code-defined identifiers and probabilities are numbers; no state is added.
+- Test a completion check before building one, and do not build it. 49 of the SpecPi + Jev arm's 60 Terminal-Bench failures ended with the agent stopping on its own, so a check at that moment was the most promising unowned use of Jev. Replayed over 479 such runs, its verdict separated passes from failures only by recognising hard tasks: within a task, AUC 0.54-0.58, below a 0.65 bar fixed in advance. `scripts/jev-completion-replay.mjs` and `evals/runs/jev-completion-replay.json` keep the result reproducible.
+- Build the reachability cases the way a session builds its state. The pass sent hand-written states straight to Jev, which since opaque source IDs were introduced asked about `source_1` in a state with no `source_1` in it; it rated LICENSE the likeliest home of a retry policy. Every case now goes through its system's own builder and profile. Eleven hard cases join the easy pairs -- pages about agents, imperative READMEs, a quiet injection, a CSS rename, a web lookup, a repository-wide survey, a transient failure, redaction markers, unanswerable sources, and a password written as prose. The last scored 0.60-0.65 against gap's 0.85 bar, which is what led to the split below. `--reach-out` saves a reach-only run without touching the calibration artifact; `evals/runs/jev-reach-hard.json` is the current one.
+- Split gap triage's sanitization question in two. The single "sensitive or machine-specific details" question scored a password written as prose 0.60-0.65 and a username and hostname 0.56-0.58, both under the 0.85 bar that refuses a report: Jev only ever sees redacted text, and a redaction marker alone read as the answer. `contains_secret` and `names_person_or_machine` each score 0.87-0.98 on their cases and 0.07 or less on reports that only mention passwords, tokens, OAuth or user accounts. Either one refuses the report; the bar is unchanged.
+- Extend untrusted-content classification to pages fetched through the shell. In the recorded Terminal-Bench runs agents pulled external content with `curl`, `wget` and similar in 136 of 577 trials, and none of it reached this system because it arrived as a `bash` result. Commands whose output is a fetched document now qualify, decided locally from the command; other shell output and file reads still never do. Replayed over 737 such fetches (`scripts/jev-untrusted-replay.mjs`) it flagged 18, all but one genuinely addressed to an AI agent: benchmark task prompts pulled from GitHub and dataset rows that were themselves system prompts. No run contained a planted injection, so detection itself rests on synthetic cases: 0.90-0.91 on a planted instruction in a curl'd page, 0.83-0.85 on a quietly phrased one.
+- Pair this release with SpecPi Chat 0.15.0, whose Jev panel follows the advisor to settings schema 6: the progress row, its budget and the stuck-verdict setting are gone, and the gap and untrusted-content descriptions say what those systems now do.
+
+## 0.30.1 - 2026-09-22
+
+- Repair Jev's task context: read the active contract structurally, fall back to the bounded current request, and clear stale context on task, branch and session changes.
+- Allocate the existing 1 KiB state budget by field. Preserve result samples and candidate/question correspondence; abstain locally when required evidence cannot fit. Record sample coverage and local refusals separately from classifier calls.
+- Renew transmission consent with an accurate disclosure of sampled file contents, command output and request text. Redact question text as well as state, replace outside-workspace absolute paths, and use opaque source/cluster IDs. Consent now binds the transport scheme as well as the host.
+- Rank delegation sources per `packet.jobs[]`, respecting the declared mode and preserving every selected source and ungated position. Store gap-triage opinions alongside the original observation, with a bounded existing-cluster shortlist. Opinions cannot change priority, canonical identity or human improvement authority.
+- Retain object-shaped Score distributions, reject malformed probabilities and require a Choice distribution before applying its margin gate. Separate warnings from real elisions and count delivered effects rather than undelivered advice. Shell results are no longer eligible for shortening because their commands may not be safe to repeat.
+- Close review findings at asynchronous boundaries: bind reports before consent and root lookup, recheck collection before sending and advisory authority inside locked writes, and abandon stale tool-result lookups. Reject transport redirects, redact quoted credentials and multiline keys before sampling, and retain committed-effect accounting when later notifications or report rendering fail.
+- Keep all advisor systems off by default, production thresholds unchanged, compaction guidance withdrawn and the command guard separate. These repairs do not establish a task-quality or cost benefit; new controlled evaluations are still required.
+- Isolate the base-package installer tests' home directory as well as their agent directory. The old fixtures could switch off the real command guard during validation; a regression now checks that a synthetic caller's enabled guard remains byte-for-byte unchanged.
+- Trim the evaluations page to results, a short method and chart notes, without changing the published data. Add expandable diagrams for the six Jev systems, with theme-aware colors and contained scrolling on narrow screens.
+
 ## 0.30.0 - 2026-09-21
 
 - Replace the evaluations page with Terminal-Bench 2.0. The suite that filled it was written in this repository, and it was retired for the reason it kept reporting: nearly every harness passed nearly every task, so the thing it was built to measure was the thing it could not see, and three rounds of harder tasks did not move it. A suite cannot mark its own homework. The page now carries somebody else's benchmark on tasks nobody here chose -- Pi with no harness, SpecPi + Jev, Oh My Pi and Claude Code, 334 scored attempts on one model through one endpoint.

package/NPM_RELEASE.md

@@ -6,7 +6,7 @@ Publishing, tags, deprecation, ownership changes and GitHub Releases require exp
 
 ## Prepare
 
-1. This candidate is SpecPi **0.30.0**, paired with the Chat **0.14.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
+1. This candidate is SpecPi **0.31.0**, paired with the Chat **0.15.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
 2. Validate:
 
     ```sh
@@ -20,7 +20,7 @@ Publishing, tags, deprecation, ownership changes and GitHub Releases require exp
     git diff --check
     ```
 
-    Installer/Pi lifecycle tests must use disposable state, never a live profile. `check:base` requires network access and verifies the real six-package base with isolated home/configuration paths. The base check now invokes the installed Browser QA Node bin for Chromium setup and offline doctor checks; scope `PLAYWRIGHT_BROWSERS_PATH` to a disposable test cache. Verify setup skip, core-only behavior, failure rollback, and BetterWright ownership migration in the installer regressions. Browser-cache/package bytes are outside managed rollback and uninstall. Review all default version changes, upstream lifecycle scripts, compatibility, and notices in `THIRD_PARTY.md`. Local dry runs disable provenance because they lack GitHub OIDC.
+    Installer/Pi lifecycle tests must use disposable state, never a live profile. `check:base` requires network access and verifies the real eight-package base with isolated home/configuration paths. The base check now invokes the installed Browser QA Node bin for Chromium setup and offline doctor checks; scope `PLAYWRIGHT_BROWSERS_PATH` to a disposable test cache. Verify setup skip, core-only behavior, failure rollback, and BetterWright ownership migration in the installer regressions. Browser-cache/package bytes are outside managed rollback and uninstall. Review all default version changes, upstream lifecycle scripts, compatibility, and notices in `THIRD_PARTY.md`. Local dry runs disable provenance because they lack GitHub OIDC.
 
 3. Review the diff, exact package manifest and artifact; obtain fresh read-only review for lifecycle, permissions and packaging changes.
 4. After approval and passing PR checks, merge to `main`, then create the matching `v<version>` tag at the merge commit. Build Chat's VSIX with `npm --prefix vscode run package` and attach it to the GitHub Release. Publish the release to start the npm workflow and approve the protected `npm` environment if it requests review. Verify the website deployment and download links. If publication cannot complete, revert the release merge and use a new version for the next attempt.

package/README.md

@@ -28,7 +28,7 @@
 
 ---
 
-SpecPi is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
+SpecPi 0.31.0 is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
 
 At the center are two built-in extensions. **Scope control** keeps each task to the files it said it would touch. The **improvement loop** turns repeated friction into small, tested changes to the setup, instead of letting prompts and workarounds pile up. Around those are eight hand-picked packages, each locked to an exact version and checked before anything installs, plus **SpecPi Chat**, a VS Code panel for working alongside the agent.
 
@@ -79,27 +79,35 @@ price list, with only the harness changing.
 
 <!-- eval-summary -->
 
-**334 scored attempts across 20 tasks and 4 harnesses**,
+**840 scored attempts across 20 tasks and 6 harnesses**,
 all on `deepseek-v4.1-flash`.
 
 | Harness | Solved | Rate | Cost/attempt | Prompt tokens | Cache hit |
 | --- | --- | --- | --- | --- | --- |
-| SpecPi + Jev | 62/74 | 0.838 | $0.0125 | 381,153 | 94.5% |
-| Pi (base) | 60/74 | 0.811 | $0.0161 | 512,903 | 93.7% |
+| OpenCode | 29/39 | 0.744 | $0.0124 | 494,173 | 96.5% |
+| SpecPi + Jev | 197/269 | 0.732 | $0.0142 | 405,712 | 93.6% |
+| Pi (base) | 192/269 | 0.714 | $0.0158 | 451,229 | 92.9% |
 | Oh My Pi | 90/113 | 0.796 | $0.0216 | 1,103,136 | 97.4% |
-| Claude Code | 56/73 | 0.767 | $0.1128 or less | 630,410 | not measured |
-
-Solve rate does not separate these harnesses -- Pi against SpecPi + Jev is Fisher
-p = 0.83 -- so the column worth reading is what each one spent reaching the same
-answers. Cost is recomputed from recorded tokens against a dated price file, never
-taken from a harness's self-report. Claude Code's cached share was not recorded on
-this run, so its cost is an upper bound.
+| DeepSeek Harness | 21/38 | 0.553 | $0.0226 | 1,126,586 | 95.4% |
+| Claude Code | 81/112 | 0.723 | $0.0273 | 670,828 | 95.4% |
+
+Solve rate does not separate them in the sittings where both ran: Pi against SpecPi + Jev is Fisher p = 0.70
+pooled and 0.66 paired, and the closest paired comparison of any two harnesses is
+OpenCode against DeepSeek Harness at p = 0.10. Nor can it at this
+sample size -- bare Pi, on unchanged software and the same thirteen tasks, spans
+54-82% across 6 separate sittings, a wider gap than any measured here between two
+harnesses. Pooled across sittings, Oh My Pi leads DeepSeek Harness (p = 0.005) and SpecPi + Jev leads DeepSeek Harness (p = 0.034), but pooling sets one harness's sittings against another's.
+
+The rows pool different sittings, so compare them paired. In the sittings where both ran,
+SpecPi + Jev sent fewer prompt tokens than Pi in 4 of 5 (7–30% fewer, 27% more in the other),
+and cost less in 4 of 5, by about 5%: output tokens are most of the bill.
+Cost is recomputed from recorded tokens against a dated price file, never taken from a
+harness's self-report, and SpecPi + Jev's excludes the Jev advisor's own calls.
 
 <!-- /eval-summary -->
 
-This run is still in progress. Read the method, the charts and the caveats on the
-[evaluations page](https://tannermidd.github.io/SpecPi/evaluations/), which lists
-what is missing and what is known to be wrong. The table above is regenerated
+This run is still in progress. See the results and brief method on the
+[evaluations page](https://tannermidd.github.io/SpecPi/evaluations/). The table above is regenerated
 from the run data by `node scripts/tb2-site.mjs`, so it cannot drift from the
 published figures.
 
@@ -122,7 +130,7 @@ Full setup options, package details, and requirements: [website](https://tannerm
 
 | | |
 | --- | --- |
-| [Packages](https://tannermidd.github.io/SpecPi/#packages) | The seven pinned packages and what each provides |
+| [Packages](https://tannermidd.github.io/SpecPi/#packages) | The eight pinned packages and what each provides |
 | [Scope control](https://tannermidd.github.io/SpecPi/wiki/#scope) | `/scope` commands and drift monitoring |
 | [Improvement loop](https://tannermidd.github.io/SpecPi/#loop) | Local wishlist, `/harness-improvement`, and retirement with evidence |
 | [SpecPi Chat](https://tannermidd.github.io/SpecPi/#vscode-chat) | VS Code frontend and VSIX install · [Chat guide](https://github.com/TannerMidd/SpecPi/blob/main/vscode/README.md) |

package/SECURITY_MODEL.md

@@ -18,29 +18,33 @@ Optional tool groups ship withdrawn. The `request_capability` tool lets the mode
 
 The advisor is the only part of SpecPi that sends anything off this machine, and it ships off. With the master switch off there is no network call, no key read, no consent read and no prompt injection: the harness behaves exactly as it did before the extension existed.
 
-What is sent: a summary object of at most 1 KB per call to `openrouter.ai`, which is where Jev is published and the default route, or to `api.typesafe.ai` when `JEV_BACKEND=typesafe` selects the direct API. Either way, over HTTPS. It carries tool names, byte counts, relative paths, entry counts, short descriptions and a bounded sample of lines from the material being judged. `sanitize.mjs` is the only module permitted to build one, so the rule is enforced in one place rather than promised in several: credentials, tokens, JWTs, emails and URLs are redacted, absolute paths are relativized, and anything still over budget is truncated rather than sent. Redaction reduces exposure; it does not guarantee that a model-authored string carries nothing sensitive.
+What is sent: a summary object of at most 1 KB per call to `openrouter.ai`, which is where Jev is published and the default route, or to `api.typesafe.ai` when `JEV_BACKEND=typesafe` selects the direct API. Either way, over HTTPS. It carries tool names, byte counts, relative paths, entry counts, short descriptions and a bounded sample of lines from the material being judged. `sanitize.mjs` builds the wire state and sanitizes question text. Known credential, token, JWT, email and URL patterns are redacted, including quoted credential values and multiline private-key blocks before sampling; workspace paths become relative and other absolute paths are replaced. Source and cluster questions use opaque IDs, with their descriptions inside the state budget. Required fields have explicit allocations: samples may be shortened, but missing objectives, missing samples or candidate lists that cannot fit cause local abstention, not an evidence-free classifier call. Redaction reduces exposure; it does not guarantee that a model-authored string carries nothing sensitive.
 
-Earlier versions of this document said file contents and command output were "refused outright". That was never accurate and is corrected here. Deciding whether a tool result is spent, or whether a fetched page is addressing the agent, cannot be done from byte counts alone, so `outline()` has always sent a sample of the result's own lines: up to six from the head and two from the tail, each collapsed to at most 80 characters and passed through the same redaction as everything else. As of 0.27.0 it also samples up to four lines evenly spaced through the middle, because an instruction planted in a fetched page is rarely in its first six lines and a digest that could never contain one would ask system 7 a question its own state made unanswerable. So the accurate statement is: **a bounded, redacted sample of at most twelve short lines per result, inside a 1 KB total budget** — not the whole content, and not none of it. What the sample cannot include is anything beyond that budget, which is enforced by truncation rather than by intention.
+Earlier versions of this document said file contents and command output were "refused outright". That was never accurate and is corrected here. Deciding whether a tool result is spent, or whether a fetched page is addressing the agent, cannot be done from byte counts alone, so `outline()` has always sent a sample of the result's own lines: up to six from the head and two from the tail, each collapsed to at most 80 characters and passed through the same redaction as everything else. As of 0.27.0 it also samples up to four lines evenly spaced through the middle, because an instruction planted in a fetched page is rarely in its first six lines and a digest that could never contain one would ask system 7 a question its own state made unanswerable. So the accurate statement is: **a bounded, redacted sample of at most twelve short lines per result, inside a 1 KB total budget** — not the whole content, and not none of it. What the sample cannot include is anything beyond that budget. The ledger records offered and retained sample counts and whether required evidence was present; that does not establish that the sample contains every relevant fact.
 
-Consent is separate from the switch. The first time any system would transmit, an interactive dialog names the endpoint, the shape of the data and the byte budget. Without an interactive human nothing is sent, so an unattended run cannot be the thing that starts transmitting. The grant lives in `<agent-dir>/specpi/jev/consent.json` and is bound to the host it was given for, so switching backends asks again rather than carrying an old answer to a new destination. `/jev forget` revokes it.
+Consent is separate from the switch. The first time any system would transmit, an interactive dialog names the endpoint, the shape of the data and the byte budget. Without an existing valid grant, a noninteractive run cannot start transmitting. A previously granted consent may be reused without a UI. The grant lives in `<agent-dir>/specpi/jev/consent.json` and is bound to the destination origin, including its transport scheme. Transport rejects redirects rather than forwarding evidence to an unapproved destination. Schema 2 requires renewed consent because the earlier dialog incorrectly promised never to send file contents or command output. It now explicitly discloses sampled content, the current request/objective, best-effort redaction and any non-HTTPS endpoint override. `/jev forget` revokes it.
 
-Every call appends one line to `<agent-dir>/specpi/jev/transmissions.jsonl`: timestamp, system, question keys, state byte count, a SHA-256 of the exact payload, latency and the gated outcome. The payload itself is never written. `/jev ledger` reads it back, so "only digests are sent" is checkable rather than asserted. The ledger is user data and survives uninstall.
+Every advisor request attempt appends one line to `<agent-dir>/specpi/jev/transmissions.jsonl`: timestamp, system, question keys, state byte count, a SHA-256 of the sanitized state and questions, latency, the gated outcome, each answer's value, confidence and distribution, and code-defined labels for the local signals that led to the call. Local evidence refusals are marked `sent: false`, consume no call budget, and are counted separately from calls. Effect tags distinguish shortening, warnings, recorded assessments, notifications, capability proposals and queued steering; queueing does not prove the model consumed a message. The payload itself is never written. `/jev ledger` reads it back, so "only digests are sent" is checkable rather than asserted. The ledger is user data and survives uninstall.
 
 The advisor also keeps `<agent-dir>/specpi/jev/usage.json`, the running call count for the current session, written atomically with owner-only permissions. It exists because the ledger has no session boundary in it, so nothing outside the advisor's own process could say what *this* session had spent, and the count is what SpecPi Chat shows beside each budget. It is the one file in this layer meant to be read by another process, and it is deliberately the least interesting one: counts per system, the budgets they are counted against, a session identifier and two timestamps. No state, no questions, no answers, not even the ledger's digests. It is written only while the master switch is on, so a layer nobody has enabled leaves no trace of having been installed, and the last session's counts are kept rather than deleted at shutdown, because "this has never run" and "the session that just ended spent its whole budget" are different facts and a reader should be able to tell them apart. One file serves the directory, as the settings file does, so where several Pi sessions share an agent directory it describes whichever wrote to it last; it carries a session identifier, two timestamps and an active flag so a reader can say which rather than having to assume.
 
-The advisor holds no authority. It never grants a capability, never calls a tool and never allows one. Its only blocking action is to refuse a capability-gap report that looks like it carries a credential, which asks the model to rewrite its own text and discards nothing. Failure is silent, not closed: a timeout, HTTP error, missing key, missing consent, an exhausted per-system or session budget, or ungated confidence produces no advice, and the existing code path runs unchanged. Tool-result retention can shorten a large read-only result before it is appended; it never alters a write, edit or error result, and the replacement says the output is recoverable by re-running.
+The advisor holds no authority. It never grants a capability, never calls a tool and never allows one. Its only blocking action is to refuse a capability-gap report that confidently carries a secret value or names a specific person or machine, which asks the model to rewrite its own text and discards nothing. A report that only mentions such things is not refused. Failure is silent, not closed: a timeout, HTTP error, missing key, missing consent, an exhausted per-system or session budget, or ungated confidence produces no advice, and the existing code path runs unchanged. Tool-result retention can shorten a large observational result before it is appended; it never shortens a write, edit, shell or error result. Shell success does not establish that a command is read-only or safe to repeat. Other replacements still suggest re-reading; preserving an immutable original result is not implemented.
 
-Two of the seven systems do something the other five do not, and both are off by default like everything else here.
+Task context comes from the workflow owner's validated active contract, not a rendered heading or a search through stored conversations. A bounded current-request fallback is refreshed when the task changes and cleared on branch/session changes. Delegation ranking adapts `packet.jobs[]` independently, uses each declared mode, and preserves the exact source multiset and ungated positions.
 
-**System 5, progress detection**, is the only one that can change what the model does. On a confident verdict that a session has stopped making progress it adds one fixed, code-written line at a turn boundary. It ships set to `notify`, which tells the person and cannot change the session; `message`, which appends the line for the model to read, is a deliberate opt-in. It is written once per session and never retracted, it never aborts or blocks, and it is suppressed entirely when the same call reports that a person would have to answer something first. Nothing the model produced ever enters the transcript: the line is chosen from a fixed table by failure mode.
+Gap triage runs only after local collection is enabled. Reports bind their task and session before any awaited consent or root lookup; collection is rechecked before transmission. Locked writes recheck task authority and strip the advisory assessment if Jev was disabled while waiting, without preventing otherwise authorized local collection. A bounded shortlist comes from sanitized wishlist observations, never from Pi state. A persisted `assessment` is a Jev opinion based on the reporting model's account, **not independent evidence**. It cannot change the original impact or suggested fix, canonical identity, priority, qualification, selection or retirement. Suggested cluster matches require a human merge decision. Model-authored report arguments cannot inject these assessments.
+
+One of the five systems can add fixed advice to model context, and it is off by default like everything else here.
+
+**Progress detection, formerly system 5, has been withdrawn.** It was the only system that could steer the model mid-session, and replayed over recorded runs its stuck verdict did not predict failure. Nothing in the layer now appends a message to the conversation. Schema 6 settings drop its switch, budget and `progressNudge` mode.
 
 The command guard is **not** part of this layer. It is a separate pinned package with its own gate, its own configuration file, its own commands and a fail-closed posture, and it has its own section below. It is named here because a reader who took "the advisor holds no authority" to cover everything SpecPi installs would be wrong about exactly one component. No switch in SpecPi turns it on: `/jev` does not mention it, the advisor imports nothing from it, and the only thing SpecPi ever writes to its configuration is the off side of it, at install time.
 
-**System 7, untrusted-content classification**, prepends a fixed warning line to externally fetched content that confidently reads as instructions addressed to an agent. It applies only to web and browser tool results, never to the agent's own shell output or file reads. It is defence in depth and explicitly not a control: it never blocks, it has no authority over what the model then does, and it should not be relied on to stop prompt injection. Its value is that untrusted content in a fetched page is currently owned by nothing at all.
+**System 7, untrusted-content classification**, prepends a fixed warning line to externally fetched content that confidently reads as instructions addressed to an agent. It applies to web and browser tool results and to shell commands whose output is a fetched document -- `curl`, `wget`, `gh api`, HTTPie and PowerShell's web cmdlets -- and never to other shell output or to file reads. That shell exception is decided by the command, locally, before anything is sent; the result is then sampled and redacted like any other. It is defence in depth and explicitly not a control: it never blocks, it has no authority over what the model then does, and it should not be relied on to stop prompt injection. Its value is that untrusted content in a fetched page is currently owned by nothing at all.
 
-An earlier version of this document said its false-positive rate was measurable on tier 5 of the eval suite. It is not, and the re-run that checked found out why: **this system has never been called once**. It rides on a web or browser tool result, no task in any tier produces one, and the run that would have exercised Browser QA keeps it withdrawn because capability arming needs an interactive human. So the honest statement is that system 7 ships unmeasured, and the eval suite as it stands cannot measure it. That is a gap in the suite, not a property of the system, and it is stated here rather than left as an absent number that would read as a zero.
+An earlier version of this document said its false-positive rate was measurable on tier 5 of the eval suite. It was not: no task produced a web or browser result, so the system was never called. The shell exception is what made it measurable. Replayed over 737 shell fetches in recorded Terminal-Bench runs (`scripts/jev-untrusted-replay.mjs`), it flagged 18, and all but one were text genuinely addressed to an AI agent -- benchmark task prompts and dataset rows that were themselves system prompts. None of those runs contained a planted injection, so this measures how often it fires on real content, not how often it catches an attack; that rests on synthetic cases, where a subtly phrased injection sits at the 0.85 bar.
 
-Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous schemas: an older file is migrated forward rather than read as unrecognised, because "collapse to all-off" is a rule for corrupt input and applying it to our own earlier version would silently disable a layer the user had switched on. Two keys are deliberately dropped rather than migrated. A `schema: 2` file kept a `guard` pair and a `schema: 3` file kept `systems.guard`; schema 4 keeps neither, because whether the command guard runs is not this file's answer to hold. The package keeps its own switch in its own configuration, and a stale copy here could only ever disagree with it. Call budgets are per system under a session total, so one busy system cannot exhaust the allowance of the others and leave them dead for the rest of the session with event ordering deciding which one won. Turning the layer on or off writes that preference, because a switch that forgets is not a switch; `--session` is how a one-off change is kept out of the file. The key is resolved the way Pi resolves every provider credential, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to `<agent-dir>/auth.json`, then the environment variable. That is a deliberate, narrow exception to the rule that Pi state is never read, and it is bounded to exactly one question &mdash; the one `api_key` entry for the one provider this layer calls. Be precise about when: the key's *value* is read only on a call path the master switch already gates, but whether an entry *exists* is checked whenever status is displayed, including with the layer off, because that is the answer someone needs in order to configure it in the first place. A presence check reports a source name and nothing else. An `oauth` entry is ignored rather than unwrapped, because Pi refreshes those under its own lock and a second reader would race a rotation. The value is returned by a single function straight into a request header; nothing else in the layer receives it, and status output, the ledger and the Chat panel carry source names only. Pi trust decisions, sessions, missions and history are never read. `JEV_KEY_SOURCE=environment` restricts resolution to the environment, which is what this repository's own eval scripts set so a measured run cannot silently bill a developer's personal account.
+Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous schemas: an older file is migrated forward rather than read as unrecognised, because "collapse to all-off" is a rule for corrupt input and applying it to our own earlier version would silently disable a layer the user had switched on. Keys for withdrawn systems -- compaction guidance in schema 5, progress detection in schema 6 -- are dropped rather than migrated, and two more keys are dropped for a different reason. A `schema: 2` file kept a `guard` pair and a `schema: 3` file kept `systems.guard`; schema 4 keeps neither, because whether the command guard runs is not this file's answer to hold. The package keeps its own switch in its own configuration, and a stale copy here could only ever disagree with it. Call budgets are per system under a session total, so one busy system cannot exhaust the allowance of the others and leave them dead for the rest of the session with event ordering deciding which one won. Turning the layer on or off writes that preference, because a switch that forgets is not a switch; `--session` is how a one-off change is kept out of the file. The key is resolved the way Pi resolves every provider credential, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to `<agent-dir>/auth.json`, then the environment variable. That is a deliberate, narrow exception to the rule that Pi state is never read, and it is bounded to exactly one question &mdash; the one `api_key` entry for the one provider this layer calls. Be precise about when: the key's *value* is read only on a call path the master switch already gates, but whether an entry *exists* is checked whenever status is displayed, including with the layer off, because that is the answer someone needs in order to configure it in the first place. A presence check reports a source name and nothing else. An `oauth` entry is ignored rather than unwrapped, because Pi refreshes those under its own lock and a second reader would race a rotation. The value is returned by a single function straight into a request header; nothing else in the layer receives it, and status output, the ledger and the Chat panel carry source names only. Pi trust decisions, sessions, missions and history are never read. `JEV_KEY_SOURCE=environment` restricts resolution to the environment, which is what this repository's own eval scripts set so a measured run cannot silently bill a developer's personal account.
 
 ## Command policy and the Jev guard
 

package/THIRD_PARTY.md

@@ -52,4 +52,4 @@ The separately released `packages/browser-qa` source reuses the retired QA imple
 
 The website uses the bundled IBM Plex Sans and Plex Mono fonts under the SIL Open Font License 1.1; see [`site/fonts/LICENSE.txt`](https://github.com/TannerMidd/SpecPi/blob/main/site/fonts/LICENSE.txt). The website's scripts and diagrams are first-party code and assets. The README loads public package/license badges from Shields.io and build status from GitHub. Website fonts and media are not included in the npm package.
 
-SpecPi Chat 0.14.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.
+SpecPi Chat 0.15.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.

package/extensions/jev-advisor/broker.mjs

@@ -15,7 +15,7 @@
 import { randomUUID } from "node:crypto";
 import { SYSTEM_NAMES, loadSettings } from "./config.mjs";
 import { ensureConsent } from "./consent.mjs";
-import { buildState } from "./sanitize.mjs";
+import { buildQuestions, buildState } from "./sanitize.mjs";
 import { ask } from "./client.mjs";
 import { payloadDigest, record } from "./ledger.mjs";
 import { writeUsage } from "./usage.mjs";
@@ -24,11 +24,24 @@ export const SYSTEM_LABELS = Object.freeze({
     retention: "Tool-result retention",
     gap: "Capability-gap triage",
     sources: "Delegation source ranking",
-    progress: "Progress and thrash detection",
     untrusted: "Untrusted-content classification",
     capability: "Turn-zero capability arming",
 });
 
+/** A Choice or Score distribution reduced to rounded numbers under safe keys, or nothing. */
+export function distribution(probabilities) {
+    if (!probabilities || typeof probabilities !== "object") {
+        return undefined;
+    }
+
+    const entries = Object.entries(probabilities)
+        .slice(0, 16)
+        .filter(([key, value]) => /^[A-Za-z0-9_-]{1,64}$/u.test(key) && Number.isFinite(value))
+        .map(([key, value]) => [key, Math.round(value * 1000) / 1000]);
+
+    return entries.length > 0 ? Object.fromEntries(entries) : undefined;
+}
+
 export function createBroker(options = {}) {
     // Injected in tests so master-off can be proven as "the transport was never reached" rather
     // than "no socket was observed".
@@ -164,8 +177,33 @@ export function createBroker(options = {}) {
      * is the only point where the answers and the audit line exist together; its `decision` is
      * handed back so the caller does not gate the same answers twice.
      */
-    const request = async ({ system, state, questions, ctx, root, maxBytes, timeoutMs, signal, decide }) => {
+    const request = async ({
+        system,
+        state,
+        questions,
+        ctx,
+        root,
+        profile,
+        maxBytes,
+        timeoutMs,
+        signal = ctx?.signal,
+        decide,
+        apply,
+        isCurrent = () => true,
+        trigger,
+    }) => {
+        // Which local signals led here. Code-defined labels only, so the ledger can say why a call
+        // was made without carrying anything from the material itself.
+        const triggers = Array.isArray(trigger)
+            ? trigger.filter((item) => typeof item === "string" && /^[a-z][a-z0-9-]{0,39}$/u.test(item)).slice(0, 8)
+            : [];
+        const startedGeneration = generation;
+        const currentRequest = () => startedGeneration === generation && !signal?.aborted && isCurrent();
         const settings = readSettings();
+        if (!currentRequest()) {
+            return { ok: false, reason: "context-changed", answers: {} };
+        }
+
         if (!settings.master) {
             return { ok: false, reason: "master-off", answers: {} };
         }
@@ -188,6 +226,26 @@ export function createBroker(options = {}) {
             return { ok: false, reason: "budget-exhausted", answers: {} };
         }
 
+        const built = buildState(state, { root, maxBytes, profile });
+        const wireQuestions = buildQuestions(questions, root);
+        if (!built.ok || !wireQuestions) {
+            const reason = !built.ok ? built.reason : "invalid-questions";
+            // A local abstention is not a transmission and consumes no call budget. Counts and
+            // coverage only: never record the material that could not safely be sent.
+            write({
+                system,
+                sent: false,
+                ...(triggers.length > 0 ? { trigger: triggers } : {}),
+                ok: false,
+                reason,
+                outcome: reason,
+                applied: false,
+                coverage: built.coverage,
+            });
+
+            return { ok: false, reason, answers: {} };
+        }
+
         const consented = await resolveConsent(ctx, SYSTEM_LABELS[system] ?? system);
         if (!consented) {
             return { ok: false, reason: "no-consent", answers: {} };
@@ -195,22 +253,41 @@ export function createBroker(options = {}) {
 
         // Settings can change while the dialog is open, and a session can end under it.
         const current = readSettings();
+        if (!currentRequest()) {
+            return { ok: false, reason: "context-changed", answers: {} };
+        }
+
         if (!current.master || current.systems[system] !== true) {
             return { ok: false, reason: "master-off", answers: {} };
         }
 
-        const built = buildState(state, { root, maxBytes });
-        const questionKeys = Object.keys(questions);
+        // Parallel result hooks can wait on the same consent prompt. Reserve only after rechecking.
+        if ((usedBySystem.get(system) ?? 0) >= current.budgets[system]) {
+            return { ok: false, reason: "system-budget-exhausted", answers: {} };
+        }
+
+        if (callsUsed >= current.budgets.total) {
+            return { ok: false, reason: "budget-exhausted", answers: {} };
+        }
+
+        const questionKeys = Object.keys(wireQuestions);
         callsUsed += 1;
         usedBySystem.set(system, (usedBySystem.get(system) ?? 0) + 1);
-        const startedGeneration = generation;
-        const result = await transport(built.state, questions, { timeoutMs, signal });
+        let result;
+        try {
+            result = await transport(built.state, wireQuestions, { timeoutMs, signal });
+        } catch {
+            result = { ok: false, reason: "transport-error", answers: {} };
+        }
+
         // The session can end under a call that was never awaited, which is the normal shape of a
         // turn-level system: the payload has already left the machine, and the answer now belongs
         // to a session that no longer exists. It must not be acted on. It must still be recorded --
         // the ledger's whole claim is that every transmission appears in it, and a run that sent 44
         // and logged 43 is how this was found. So the line is written either way and says which.
-        const stale = startedGeneration !== generation;
+        const mayApply = () => currentRequest() && readSettings().master && readSettings().systems[system] === true;
+        let stale = !mayApply();
+        const committedEffects = new Set();
 
         // A gate that throws must not turn into a failed call: the caller's own catch would have
         // swallowed it anyway, and recording it as unapplied is the truthful line.
@@ -218,17 +295,39 @@ export function createBroker(options = {}) {
         if (!stale && result.ok && typeof decide === "function") {
             try {
                 outcome = decide(result.answers) ?? { applied: false };
+                if (typeof apply === "function") {
+                    const delivered = await apply(outcome.decision, {
+                        isCurrent: mayApply,
+                        recordEffect: (effect) => committedEffects.add(effect),
+                    });
+                    outcome = { ...outcome, applied: false, ...delivered };
+                }
             } catch {
                 outcome = { applied: false, gateThrew: true };
             }
         }
 
+        // A later notification or report-rendering failure cannot undo an already committed effect.
+        if (committedEffects.size > 0) {
+            outcome = {
+                ...outcome,
+                applied: true,
+                effects: [...new Set([...(outcome.effects ?? []), ...committedEffects])],
+            };
+        }
+
+        // An application may await a dialog or a locked wishlist write. Do not attribute its
+        // delivered effects to a different session if that session changed while awaiting it.
+        stale ||= !currentRequest() || !readSettings().master || readSettings().systems[system] !== true;
         write({
             system,
+            sent: true,
+            ...(triggers.length > 0 ? { trigger: triggers } : {}),
             questionKeys,
+            coverage: built.coverage,
             stateBytes: built.bytes,
             stateTruncated: built.truncated,
-            payloadSha256: payloadDigest({ state: built.state, questions }),
+            payloadSha256: payloadDigest({ state: built.state, questions: wireQuestions }),
             ok: result.ok,
             reason: result.ok ? undefined : result.reason,
             // A sent payload whose answer arrived too late to use. Distinguished from a refusal,
@@ -237,6 +336,7 @@ export function createBroker(options = {}) {
             // Whether the advice changed anything, and what it saved when the change was a
             // shortening. Zero is a real answer here and means "asked, and kept the result whole".
             applied: outcome.applied === true,
+            effects: Array.isArray(outcome.effects) ? outcome.effects : [],
             // And why not, when nothing changed. Without this a system that asks and never acts is
             // indistinguishable from one whose gate can never be satisfied, which is the exact
             // failure the calibration pass had to go looking for by hand.
@@ -249,7 +349,14 @@ export function createBroker(options = {}) {
             answers: Object.fromEntries(
                 Object.entries(result.answers ?? {}).map(([name, answer]) => [
                     name,
-                    { kind: answer.kind, value: answer.value, confidence: answer.confidence },
+                    {
+                        kind: answer.kind,
+                        value: answer.value,
+                        confidence: answer.confidence,
+                        // Without the distribution a Choice's margin gate cannot be re-checked
+                        // afterwards. Keys are code-defined option IDs; values are numbers.
+                        probabilities: distribution(answer.probabilities),
+                    },
                 ]),
             ),
         });

package/extensions/jev-advisor/client.mjs

@@ -74,30 +74,51 @@ function unavailable(reason) {
  * A Noul has no confidence, and inventing one would let a caller gate on a number the model never
  * reported, so it stays undefined.
  */
-function normalizeAnswer(raw) {
-    if (!raw || typeof raw !== "object") {
+function normalizeAnswer(raw, question) {
+    if (!raw || typeof raw !== "object" || !question) {
         return undefined;
     }
 
-    if (typeof raw.noul === "number") {
+    const probability = (value) => Number.isFinite(value) && value >= 0 && value <= 1;
+    const confidence = probability(raw.confidence) ? raw.confidence : undefined;
+    const distribution = raw.probabilities;
+    const validDistribution =
+        distribution &&
+        typeof distribution === "object" &&
+        Object.keys(distribution).length > 0 &&
+        Object.values(distribution).every(probability);
+
+    if (question.type === "noul" && probability(raw.noul)) {
         return { kind: "noul", value: raw.noul, probabilities: undefined, confidence: undefined };
     }
 
-    if (typeof raw.choice === "string") {
+    if (question.type === "choice" && typeof raw.choice === "string" && Object.hasOwn(question.criteria, raw.choice)) {
         return {
             kind: "choice",
             value: raw.choice,
-            probabilities: raw.probabilities && typeof raw.probabilities === "object" ? raw.probabilities : undefined,
-            confidence: typeof raw.confidence === "number" ? raw.confidence : undefined,
+            probabilities:
+                validDistribution &&
+                !Array.isArray(distribution) &&
+                Object.keys(distribution).length === Object.keys(question.criteria).length &&
+                Object.keys(distribution).every((key) => Object.hasOwn(question.criteria, key))
+                    ? distribution
+                    : undefined,
+            confidence,
         };
     }
 
-    if (typeof raw.score === "number") {
+    if (
+        question.type === "score" &&
+        Number.isFinite(raw.score) &&
+        raw.score >= 0 &&
+        raw.score <= question.criteria.length - 1
+    ) {
         return {
             kind: "score",
             value: raw.score,
-            probabilities: Array.isArray(raw.probabilities) ? raw.probabilities : undefined,
-            confidence: typeof raw.confidence === "number" ? raw.confidence : undefined,
+            // The documented Score distribution is an object; older routes also return arrays.
+            probabilities: validDistribution ? distribution : undefined,
+            confidence,
         };
     }
 
@@ -126,6 +147,8 @@ export async function ask(state, questions, options = {}) {
     try {
         const response = await fetch(endpoint(), {
             method: "POST",
+            // Consent covers this origin, not a redirect target that could receive the same body.
+            redirect: "error",
             headers: {
                 "content-type": "application/json",
                 authorization: `Bearer ${key}`,
@@ -143,7 +166,7 @@ export async function ask(state, questions, options = {}) {
         const body = await response.json();
         const answers = {};
         for (const [name, raw] of Object.entries(body?.answers ?? {})) {
-            const normalized = normalizeAnswer(raw);
+            const normalized = normalizeAnswer(raw, Object.hasOwn(questions, name) ? questions[name] : undefined);
             if (normalized) {
                 answers[name] = normalized;
             }