specpi 0.30.1 → 0.32.0

package/CHANGELOG.md

@@ -1,6 +1,39 @@
 # Changelog
 
-## Unreleased
+## 0.32.0 - 2026-09-23
+
+- Send less on every request. SpecPi's first request was 14,022 characters against Pi's 5,521, and that overhead rides every turn and is never removed by compaction. Measured again with all eight pins, the interactive default is now 11,119. A headless session drops the two tools below as well, which the measured parts put at about 8,300: SpecPi's overhead falls from 8,501 characters to about 2,800.
+- Trim the installed working agreement from about 2,400 characters to 1,300, keeping the rules that shape ordinary work, including the restraint that wins `sanitize-git-repo` on Terminal-Bench. The `/scope` rule moves into the scope notice SpecPi already adds while scope is active. The wishlist and credential rules already live in the improvement skill, and the web and browser rule now lives in `request_capability`'s own description.
+- Offer `report_capability_gap` only where a report could land: collection on, or undecided in an interactive session, where its first report asks for consent. With collection off, or undecided in a headless session, it could only ever answer "Not recorded". `/wishlist on` and `/wishlist off` switch it with collection. Its four guidance lines, which repeated its description, are now one.
+- Offer `request_capability` only to an interactive session with a requestable group installed. Headless, it could only refuse. Its description is shorter too.
+- Keep the improvement skill out of the model's skill list. It only ever runs from a `/harness-improvement` selection, so the kickoff now names its file, and Pi still lists it as a command.
+- Repair `scripts/measure-context.mjs`, which failed on 0.31.0: it expected two first-party extensions and authoring tools that no longer ship by default. It now records where each Pi row's instruction characters go, section by section. The published record and charts are re-measured; Oh My Pi runs again (18.2.5), so its row is no longer carried forward.
+- Give tier 6's Pi-family harnesses compaction settings that fit the declared window. Pi's defaults compact at 7,616 tokens of a 24,000 window and try to keep 20,000, so earlier runs compacted almost every turn; plain SpecPi compacted 17-19 times on the marathon task against Pi's 5-7, because its fixed prompt took most of what was left. Runs from before this setting are not comparable.
+- Note on the evaluations page that `configure-git-webserver` measures a cleanup habit. Its checker's push never succeeded for any harness, so every pass came from a test file left in the web root. That also qualifies a claim in 0.31.0: SpecPi + Jev's self-stopped failures outnumber Pi's only because of this task. Without it they are level, 31 against 32.
+- Consider a background-task system and do not build one. In the Terminal-Bench transcripts agents already start servers with `nohup` or `&`, tool timeouts are about one in eighteen attempts, and the one task that spends minutes blocked in commands (`tune-mjcf`) times out equally in both arms. Both earlier background-task integrations were removed, and a new one would add tool schema to every request.
+- SpecPi Chat 0.15.0 needs no change: it runs Pi interactively, so both tools stay available to it, and its command menu still lists the improvement skill.
+
+## 0.31.0 - 2026-09-22
+
+- Report Terminal-Bench 2.0 per sitting rather than per run. Bare Pi -- unchanged software, the same thirteen tasks, the same machine -- solved 21, 22, 30, 30 and 32 of 39 across five launches, so a figure from one run is not evidence about a harness. The page pools every sitting, publishes each harness's per-sitting range beside its rate, and charts the spread directly.
+- Compare harnesses only in the sittings where both ran. An earlier reading set two separate runs side by side and showed SpecPi + Jev 30/39 against Pi 22/39; three sittings running both arms in one launch went +5, level, and +6 the other way. Paired, Pi solves 113/156 and SpecPi + Jev 112/156, p = 1.00, and no pair of harnesses separates pooled or paired.
+- Apply the same rule to tokens and cost, which pooling had overstated. Pooled across unpaired sittings SpecPi + Jev looked 11% cheaper than Pi; paired, it sent fewer prompt tokens in all four sittings (7-30%) but cost less in only three, by about 7%, and more in the fourth. Output tokens are 61-73% of every harness's bill at these cache rates, which is why a consistent prompt-token saving barely reaches the cost.
+- Correct Claude Code's cost from the published $0.1128 upper bound to $0.0273 measured. Its cached share went unrecorded before the Messages translation was fixed; measured, it caches at 95.4%. Paired with SpecPi + Jev it sends 1.9x the prompt tokens and costs 2.4x as much, more than its pooled row suggests, because its one measured sitting was SpecPi + Jev's cheapest.
+- Stop plotting an upper bound as a cost. The cost chart filtered by harness, so once Claude Code's overall cost was measured its calibration slice -- which predates the cache fix -- appeared as a bar seven times the others. It filters by slice now.
+- State the builds the benchmark actually ran. The page gave Pi 0.84.4, the version this repository pins for its own tests; every trial reported 0.87.0, and the figure is now read from them. The SpecPi + Jev row spans two layer builds: the 21 Sep sitting's still carried the withdrawn compaction system, switched off for those runs.
+- Say what the SpecPi + Jev cost leaves out. Harbor records the agent's tokens, not the Jev advisor's, so the advisor's calls -- 320 across the arm by its own ledger -- are excluded rather than estimated, against the rule in `evals/prices.json` that a Jev row includes them.
+- Derive every figure the page and README quote. Counts, rates, ranges and p-values in the prose are filled from `terminal-bench-2.json` at load, and `npm run check:site` fails on any slot left empty. The README's hand-written sentence had already gone stale, quoting p = 0.83 after a rerun moved it to 0.40.
+- Add OpenCode and the DeepSeek Harness to Terminal-Bench 2.0, in one sitting beside Pi and SpecPi + Jev. OpenCode solved 29/39 and the DeepSeek Harness 21/38, and neither separates from a harness it ran beside. OpenCode's reasoning tokens are now counted: Harbor records them apart from its output, they were 2.6 times that output, and they put its cost at $0.0124 an attempt rather than $0.0064.
+- Limit the solve-rate claim to paired comparisons. Pooled, Oh My Pi and SpecPi + Jev now lead the DeepSeek Harness at p = 0.005 and 0.034, and the page names them. The one paired sitting where SpecPi + Jev sent more prompt tokens than Pi (27% more) is stated beside the "fewer" range instead of being folded into it.
+- Withdraw progress detection from the Jev layer. Replayed over the 174 recorded Terminal-Bench runs where its local gate fired (`scripts/jev-progress-replay.mjs`), its stuck verdict did not predict failure -- within-task AUC 0.46, with the shipped state and with one that added the recent commands -- and its `failure_mode` answer echoed the reasons in its own state 624 times out of 624, so every nudge said calls had failed, including when none had. The local gate alone carried the signal (58% of flagged runs passed, 78% of the rest), and it needs no classifier. It was the only system that could steer the model mid-session. Settings schema 6 drops `systems.progress`, `budgets.progress` and `progressNudge`; the default session total falls from 2048 to 1536 so it stays below the per-system ceilings it bounds. The system as it last shipped moves to `scripts/jev-progress-system.mjs`, so the replay and the offline triage stay reproducible.
+- Stop asking retention about `read` results. Every one of its 39 calls in the 22 Sep sittings was a `read`, none shortened anything, and half came back with confidence 0: a file the agent just asked for is load-bearing on arrival. Listings, searches and fetched content remain eligible.
+- Drop two questions whose answers nothing read: retention's `result_kind` and capability arming's `task_kind`.
+- Record why each advisor call was made and the distribution behind each answer. Ledger lines now carry the local trigger labels behind capability-arming calls and the Choice and Score probabilities, so a margin gate can be re-checked afterwards. Labels are code-defined identifiers and probabilities are numbers; no state is added.
+- Test a completion check before building one, and do not build it. 49 of the SpecPi + Jev arm's 60 Terminal-Bench failures ended with the agent stopping on its own, so a check at that moment was the most promising unowned use of Jev. Replayed over 479 such runs, its verdict separated passes from failures only by recognising hard tasks: within a task, AUC 0.54-0.58, below a 0.65 bar fixed in advance. `scripts/jev-completion-replay.mjs` and `evals/runs/jev-completion-replay.json` keep the result reproducible.
+- Build the reachability cases the way a session builds its state. The pass sent hand-written states straight to Jev, which since opaque source IDs were introduced asked about `source_1` in a state with no `source_1` in it; it rated LICENSE the likeliest home of a retry policy. Every case now goes through its system's own builder and profile. Eleven hard cases join the easy pairs -- pages about agents, imperative READMEs, a quiet injection, a CSS rename, a web lookup, a repository-wide survey, a transient failure, redaction markers, unanswerable sources, and a password written as prose. The last scored 0.60-0.65 against gap's 0.85 bar, which is what led to the split below. `--reach-out` saves a reach-only run without touching the calibration artifact; `evals/runs/jev-reach-hard.json` is the current one.
+- Split gap triage's sanitization question in two. The single "sensitive or machine-specific details" question scored a password written as prose 0.60-0.65 and a username and hostname 0.56-0.58, both under the 0.85 bar that refuses a report: Jev only ever sees redacted text, and a redaction marker alone read as the answer. `contains_secret` and `names_person_or_machine` each score 0.87-0.98 on their cases and 0.07 or less on reports that only mention passwords, tokens, OAuth or user accounts. Either one refuses the report; the bar is unchanged.
+- Extend untrusted-content classification to pages fetched through the shell. In the recorded Terminal-Bench runs agents pulled external content with `curl`, `wget` and similar in 136 of 577 trials, and none of it reached this system because it arrived as a `bash` result. Commands whose output is a fetched document now qualify, decided locally from the command; other shell output and file reads still never do. Replayed over 737 such fetches (`scripts/jev-untrusted-replay.mjs`) it flagged 18, all but one genuinely addressed to an AI agent: benchmark task prompts pulled from GitHub and dataset rows that were themselves system prompts. No run contained a planted injection, so detection itself rests on synthetic cases: 0.90-0.91 on a planted instruction in a curl'd page, 0.83-0.85 on a quietly phrased one.
+- Pair this release with SpecPi Chat 0.15.0, whose Jev panel follows the advisor to settings schema 6: the progress row, its budget and the stuck-verdict setting are gone, and the gap and untrusted-content descriptions say what those systems now do.
 
 ## 0.30.1 - 2026-09-22
 

package/NPM_RELEASE.md

@@ -6,7 +6,7 @@ Publishing, tags, deprecation, ownership changes and GitHub Releases require exp
 
 ## Prepare
 
-1. This candidate is SpecPi **0.30.1**, paired with the Chat **0.14.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
+1. This candidate is SpecPi **0.32.0**, paired with the Chat **0.15.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
 2. Validate:
 
     ```sh

package/README.md

@@ -28,7 +28,7 @@
 
 ---
 
-SpecPi 0.30.1 is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
+SpecPi 0.32.0 is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
 
 At the center are two built-in extensions. **Scope control** keeps each task to the files it said it would touch. The **improvement loop** turns repeated friction into small, tested changes to the setup, instead of letting prompts and workarounds pile up. Around those are eight hand-picked packages, each locked to an exact version and checked before anything installs, plus **SpecPi Chat**, a VS Code panel for working alongside the agent.
 
@@ -44,7 +44,7 @@ Everything it touches is written down, versioned, and easy to undo.
 
 ## Measured context
 
-This chart shows first-call context from a clean install: all eight pinned packages, the working agreement, and the skills Pi finds. It has not been re-measured since the command guard returned to a pinned package rather than a native system. "Enabled" means browser QA, delegation, and web access are switched on, with no goal, scope, or improvement selection active.
+This chart shows first-call context from a clean install: all eight pinned packages, the working agreement, and the skills Pi finds, in an interactive session with wishlist collection undecided. A headless session, or one with collection off, also leaves out the gap report and capability request tools. "Enabled" means browser QA, delegation, and web access are switched on, with no goal, scope, or improvement selection active.
 
 The solid rows are measured by us, from the request each setup actually sends through one local test provider. That includes OpenCode, the DeepSeek Harness, and Oh My Pi, all measured as installed. The faded Codex CLI and Claude Code rows come from HarnessTax's published numbers, measured under their own setup. Treat those as a rough reference, not a head-to-head test. These are character counts. They say nothing about tokens, cost, or how well each tool does the job. The research page [breaks down the enabled setup by feature](https://tannermidd.github.io/SpecPi/research/#specpi), so you can see what each switch costs on its own.
 
@@ -52,7 +52,7 @@ The solid rows are measured by us, from the request each setup actually sends th
   <a href="https://tannermidd.github.io/SpecPi/research/#specpi">
     <picture>
       <source media="(prefers-color-scheme: dark)" srcset="https://tannermidd.github.io/SpecPi/media/context-chart-dark.svg">
-      <img src="https://tannermidd.github.io/SpecPi/media/context-chart-light.svg" width="880" alt="Bar chart of characters sent on the first model call: Pi stock 5,521, SpecPi default 15,069, OpenCode 31,043, DeepSeek Harness 31,743, SpecPi enabled 40,203, Codex CLI 41,616, Oh My Pi 65,816, Claude Code 90,460.">
+      <img src="https://tannermidd.github.io/SpecPi/media/context-chart-light.svg" width="880" alt="Bar chart of characters sent on the first model call: Pi stock 5,521, SpecPi default 11,119, OpenCode 31,043, DeepSeek Harness 31,743, SpecPi enabled 35,767, Codex CLI 41,616, Oh My Pi 66,708, Claude Code 90,460.">
     </picture>
   </a>
 </p>
@@ -64,7 +64,7 @@ The gap between the two SpecPi bars comes from a few separate switches, so the e
   <a href="https://tannermidd.github.io/SpecPi/research/#specpi">
     <picture>
       <source media="(prefers-color-scheme: dark)" srcset="https://tannermidd.github.io/SpecPi/media/capability-chart-dark.svg">
-      <img src="https://tannermidd.github.io/SpecPi/media/capability-chart-light.svg" width="880" alt="Bar chart of tool-schema characters each capability adds: Pi built-ins 2,896, Improvement loop 4,214, Goals 1,315, Browser QA 8,046, Delegation 4,453, Web access 11,298. Browser QA, Delegation, Web access are hidden until switched on.">
+      <img src="https://tannermidd.github.io/SpecPi/media/capability-chart-light.svg" width="880" alt="Bar chart of tool-schema characters each capability adds: Pi built-ins 2,896, Improvement loop 1,601, Capability request 995, Goals 1,315, Browser QA 8,234, Delegation 4,453, Web access 11,298. Browser QA, Delegation, Web access are hidden until switched on.">
     </picture>
   </a>
 </p>
@@ -79,21 +79,30 @@ price list, with only the harness changing.
 
 <!-- eval-summary -->
 
-**334 scored attempts across 20 tasks and 4 harnesses**,
+**840 scored attempts across 20 tasks and 6 harnesses**,
 all on `deepseek-v4.1-flash`.
 
 | Harness | Solved | Rate | Cost/attempt | Prompt tokens | Cache hit |
 | --- | --- | --- | --- | --- | --- |
-| SpecPi + Jev | 62/74 | 0.838 | $0.0125 | 381,153 | 94.5% |
-| Pi (base) | 60/74 | 0.811 | $0.0161 | 512,903 | 93.7% |
+| OpenCode | 29/39 | 0.744 | $0.0124 | 494,173 | 96.5% |
+| SpecPi + Jev | 197/269 | 0.732 | $0.0142 | 405,712 | 93.6% |
+| Pi (base) | 192/269 | 0.714 | $0.0158 | 451,229 | 92.9% |
 | Oh My Pi | 90/113 | 0.796 | $0.0216 | 1,103,136 | 97.4% |
-| Claude Code | 56/73 | 0.767 | $0.1128 or less | 630,410 | not measured |
-
-Solve rate does not separate these harnesses -- Pi against SpecPi + Jev is Fisher
-p = 0.83 -- so the column worth reading is what each one spent reaching the same
-answers. Cost is recomputed from recorded tokens against a dated price file, never
-taken from a harness's self-report. Claude Code's cached share was not recorded on
-this run, so its cost is an upper bound.
+| DeepSeek Harness | 21/38 | 0.553 | $0.0226 | 1,126,586 | 95.4% |
+| Claude Code | 81/112 | 0.723 | $0.0273 | 670,828 | 95.4% |
+
+Solve rate does not separate them in the sittings where both ran: Pi against SpecPi + Jev is Fisher p = 0.70
+pooled and 0.66 paired, and the closest paired comparison of any two harnesses is
+OpenCode against DeepSeek Harness at p = 0.10. Nor can it at this
+sample size -- bare Pi, on unchanged software and the same thirteen tasks, spans
+54-82% across 6 separate sittings, a wider gap than any measured here between two
+harnesses. Pooled across sittings, Oh My Pi leads DeepSeek Harness (p = 0.005) and SpecPi + Jev leads DeepSeek Harness (p = 0.034), but pooling sets one harness's sittings against another's.
+
+The rows pool different sittings, so compare them paired. In the sittings where both ran,
+SpecPi + Jev sent fewer prompt tokens than Pi in 4 of 5 (7–30% fewer, 27% more in the other),
+and cost less in 4 of 5, by about 5%: output tokens are most of the bill.
+Cost is recomputed from recorded tokens against a dated price file, never taken from a
+harness's self-report, and SpecPi + Jev's excludes the Jev advisor's own calls.
 
 <!-- /eval-summary -->
 

package/SECURITY_MODEL.md

@@ -10,7 +10,7 @@ Scope records use Pi's current session branch. Restoring a branch does not creat
 
 ## Capability requests
 
-Optional tool groups ship withdrawn. The `request_capability` tool lets the model name a withdrawn group instead of silently working around it; it never grants one. Activation requires a human decision, either accepted in the moment or recorded earlier as a standing grant, so the tool refuses in headless sessions and a declined prompt leaves the session unchanged. Activation is additive, offers only the named group's tools, and lasts for the session: it writes no startup preference, and the next session starts from the saved preference as before. Delegation is not requestable, because its own package requires a human command to bind a model. Acceptance offers the tools; it is not a per-call permission, and the Permission System continues to govern what those tools may do.
+Optional tool groups ship withdrawn. The `request_capability` tool lets the model name a withdrawn group instead of silently working around it; it never grants one. Activation requires a human decision, either accepted in the moment or recorded earlier as a standing grant, so the tool is not offered in headless sessions or when no requestable group is installed, refuses if called without an interactive user, and a declined prompt leaves the session unchanged. Activation is additive, offers only the named group's tools, and lasts for the session: it writes no startup preference, and the next session starts from the saved preference as before. Delegation is not requestable, because its own package requires a human command to bind a model. Acceptance offers the tools; it is not a per-call permission, and the Permission System continues to govern what those tools may do.
 
 `/capability allow <name>` records a standing grant so that group is offered without a prompt, and `/capability ask <name>` returns it to prompting. Recording either requires an interactive human command, and a grant that fires still announces itself in the session. Grants live in `<agent-dir>/specpi/capabilities/settings.json`, written atomically with owner-only permissions; links and irregular files are refused, and an unreadable file, a foreign schema or an unknown capability name reads as no grant. A standing grant removes the prompt, not the requirement for a human: headless sessions are still refused, so a granted capability cannot be activated by an unattended run. It also does not offer the group at startup — the model must still ask, and the group stays withdrawn until it does.
 
@@ -24,27 +24,27 @@ Earlier versions of this document said file contents and command output were "re
 
 Consent is separate from the switch. The first time any system would transmit, an interactive dialog names the endpoint, the shape of the data and the byte budget. Without an existing valid grant, a noninteractive run cannot start transmitting. A previously granted consent may be reused without a UI. The grant lives in `<agent-dir>/specpi/jev/consent.json` and is bound to the destination origin, including its transport scheme. Transport rejects redirects rather than forwarding evidence to an unapproved destination. Schema 2 requires renewed consent because the earlier dialog incorrectly promised never to send file contents or command output. It now explicitly discloses sampled content, the current request/objective, best-effort redaction and any non-HTTPS endpoint override. `/jev forget` revokes it.
 
-Every advisor request attempt appends one line to `<agent-dir>/specpi/jev/transmissions.jsonl`: timestamp, system, question keys, state byte count, a SHA-256 of the sanitized state and questions, latency and the gated outcome. Local evidence refusals are marked `sent: false`, consume no call budget, and are counted separately from calls. Effect tags distinguish shortening, warnings, recorded assessments, notifications, capability proposals and queued steering; queueing does not prove the model consumed a message. The payload itself is never written. `/jev ledger` reads it back, so "only digests are sent" is checkable rather than asserted. The ledger is user data and survives uninstall.
+Every advisor request attempt appends one line to `<agent-dir>/specpi/jev/transmissions.jsonl`: timestamp, system, question keys, state byte count, a SHA-256 of the sanitized state and questions, latency, the gated outcome, each answer's value, confidence and distribution, and code-defined labels for the local signals that led to the call. Local evidence refusals are marked `sent: false`, consume no call budget, and are counted separately from calls. Effect tags distinguish shortening, warnings, recorded assessments, notifications, capability proposals and queued steering; queueing does not prove the model consumed a message. The payload itself is never written. `/jev ledger` reads it back, so "only digests are sent" is checkable rather than asserted. The ledger is user data and survives uninstall.
 
 The advisor also keeps `<agent-dir>/specpi/jev/usage.json`, the running call count for the current session, written atomically with owner-only permissions. It exists because the ledger has no session boundary in it, so nothing outside the advisor's own process could say what *this* session had spent, and the count is what SpecPi Chat shows beside each budget. It is the one file in this layer meant to be read by another process, and it is deliberately the least interesting one: counts per system, the budgets they are counted against, a session identifier and two timestamps. No state, no questions, no answers, not even the ledger's digests. It is written only while the master switch is on, so a layer nobody has enabled leaves no trace of having been installed, and the last session's counts are kept rather than deleted at shutdown, because "this has never run" and "the session that just ended spent its whole budget" are different facts and a reader should be able to tell them apart. One file serves the directory, as the settings file does, so where several Pi sessions share an agent directory it describes whichever wrote to it last; it carries a session identifier, two timestamps and an active flag so a reader can say which rather than having to assume.
 
-The advisor holds no authority. It never grants a capability, never calls a tool and never allows one. Its only blocking action is to refuse a capability-gap report that looks like it carries a credential, which asks the model to rewrite its own text and discards nothing. Failure is silent, not closed: a timeout, HTTP error, missing key, missing consent, an exhausted per-system or session budget, or ungated confidence produces no advice, and the existing code path runs unchanged. Tool-result retention can shorten a large observational result before it is appended; it never shortens a write, edit, shell or error result. Shell success does not establish that a command is read-only or safe to repeat. Other replacements still suggest re-reading; preserving an immutable original result is not implemented.
+The advisor holds no authority. It never grants a capability, never calls a tool and never allows one. Its only blocking action is to refuse a capability-gap report that confidently carries a secret value or names a specific person or machine, which asks the model to rewrite its own text and discards nothing. A report that only mentions such things is not refused. Failure is silent, not closed: a timeout, HTTP error, missing key, missing consent, an exhausted per-system or session budget, or ungated confidence produces no advice, and the existing code path runs unchanged. Tool-result retention can shorten a large observational result before it is appended; it never shortens a write, edit, shell or error result. Shell success does not establish that a command is read-only or safe to repeat. Other replacements still suggest re-reading; preserving an immutable original result is not implemented.
 
 Task context comes from the workflow owner's validated active contract, not a rendered heading or a search through stored conversations. A bounded current-request fallback is refreshed when the task changes and cleared on branch/session changes. Delegation ranking adapts `packet.jobs[]` independently, uses each declared mode, and preserves the exact source multiset and ungated positions.
 
 Gap triage runs only after local collection is enabled. Reports bind their task and session before any awaited consent or root lookup; collection is rechecked before transmission. Locked writes recheck task authority and strip the advisory assessment if Jev was disabled while waiting, without preventing otherwise authorized local collection. A bounded shortlist comes from sanitized wishlist observations, never from Pi state. A persisted `assessment` is a Jev opinion based on the reporting model's account, **not independent evidence**. It cannot change the original impact or suggested fix, canonical identity, priority, qualification, selection or retirement. Suggested cluster matches require a human merge decision. Model-authored report arguments cannot inject these assessments.
 
-Two of the six systems can add fixed advice to model context, and both are off by default like everything else here.
+One of the five systems can add fixed advice to model context, and it is off by default like everything else here.
 
-**System 5, progress detection**, is the only one that can change what the model does. On a confident verdict that a session has stopped making progress it adds one fixed, code-written line at a turn boundary. It ships set to `notify`, which tells the person and cannot change the session; `message`, which appends the line for the model to read, is a deliberate opt-in. It is written once per session and never retracted, it never aborts or blocks, and it is suppressed entirely when the same call reports that a person would have to answer something first. Nothing the model produced ever enters the transcript: the line is chosen from a fixed table by failure mode.
+**Progress detection, formerly system 5, has been withdrawn.** It was the only system that could steer the model mid-session, and replayed over recorded runs its stuck verdict did not predict failure. Nothing in the layer now appends a message to the conversation. Schema 6 settings drop its switch, budget and `progressNudge` mode.
 
 The command guard is **not** part of this layer. It is a separate pinned package with its own gate, its own configuration file, its own commands and a fail-closed posture, and it has its own section below. It is named here because a reader who took "the advisor holds no authority" to cover everything SpecPi installs would be wrong about exactly one component. No switch in SpecPi turns it on: `/jev` does not mention it, the advisor imports nothing from it, and the only thing SpecPi ever writes to its configuration is the off side of it, at install time.
 
-**System 7, untrusted-content classification**, prepends a fixed warning line to externally fetched content that confidently reads as instructions addressed to an agent. It applies only to web and browser tool results, never to the agent's own shell output or file reads. It is defence in depth and explicitly not a control: it never blocks, it has no authority over what the model then does, and it should not be relied on to stop prompt injection. Its value is that untrusted content in a fetched page is currently owned by nothing at all.
+**System 7, untrusted-content classification**, prepends a fixed warning line to externally fetched content that confidently reads as instructions addressed to an agent. It applies to web and browser tool results and to shell commands whose output is a fetched document -- `curl`, `wget`, `gh api`, HTTPie and PowerShell's web cmdlets -- and never to other shell output or to file reads. That shell exception is decided by the command, locally, before anything is sent; the result is then sampled and redacted like any other. It is defence in depth and explicitly not a control: it never blocks, it has no authority over what the model then does, and it should not be relied on to stop prompt injection. Its value is that untrusted content in a fetched page is currently owned by nothing at all.
 
-An earlier version of this document said its false-positive rate was measurable on tier 5 of the eval suite. It is not, and the re-run that checked found out why: **this system has never been called once**. It rides on a web or browser tool result, no task in any tier produces one, and the run that would have exercised Browser QA keeps it withdrawn because capability arming needs an interactive human. So the honest statement is that system 7 ships unmeasured, and the eval suite as it stands cannot measure it. That is a gap in the suite, not a property of the system, and it is stated here rather than left as an absent number that would read as a zero.
+An earlier version of this document said its false-positive rate was measurable on tier 5 of the eval suite. It was not: no task produced a web or browser result, so the system was never called. The shell exception is what made it measurable. Replayed over 737 shell fetches in recorded Terminal-Bench runs (`scripts/jev-untrusted-replay.mjs`), it flagged 18, and all but one were text genuinely addressed to an AI agent -- benchmark task prompts and dataset rows that were themselves system prompts. None of those runs contained a planted injection, so this measures how often it fires on real content, not how often it catches an attack; that rests on synthetic cases, where a subtly phrased injection sits at the 0.85 bar.
 
-Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous schemas: an older file is migrated forward rather than read as unrecognised, because "collapse to all-off" is a rule for corrupt input and applying it to our own earlier version would silently disable a layer the user had switched on. Two keys are deliberately dropped rather than migrated. A `schema: 2` file kept a `guard` pair and a `schema: 3` file kept `systems.guard`; schema 4 keeps neither, because whether the command guard runs is not this file's answer to hold. The package keeps its own switch in its own configuration, and a stale copy here could only ever disagree with it. Call budgets are per system under a session total, so one busy system cannot exhaust the allowance of the others and leave them dead for the rest of the session with event ordering deciding which one won. Turning the layer on or off writes that preference, because a switch that forgets is not a switch; `--session` is how a one-off change is kept out of the file. The key is resolved the way Pi resolves every provider credential, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to `<agent-dir>/auth.json`, then the environment variable. That is a deliberate, narrow exception to the rule that Pi state is never read, and it is bounded to exactly one question &mdash; the one `api_key` entry for the one provider this layer calls. Be precise about when: the key's *value* is read only on a call path the master switch already gates, but whether an entry *exists* is checked whenever status is displayed, including with the layer off, because that is the answer someone needs in order to configure it in the first place. A presence check reports a source name and nothing else. An `oauth` entry is ignored rather than unwrapped, because Pi refreshes those under its own lock and a second reader would race a rotation. The value is returned by a single function straight into a request header; nothing else in the layer receives it, and status output, the ledger and the Chat panel carry source names only. Pi trust decisions, sessions, missions and history are never read. `JEV_KEY_SOURCE=environment` restricts resolution to the environment, which is what this repository's own eval scripts set so a measured run cannot silently bill a developer's personal account.
+Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous schemas: an older file is migrated forward rather than read as unrecognised, because "collapse to all-off" is a rule for corrupt input and applying it to our own earlier version would silently disable a layer the user had switched on. Keys for withdrawn systems -- compaction guidance in schema 5, progress detection in schema 6 -- are dropped rather than migrated, and two more keys are dropped for a different reason. A `schema: 2` file kept a `guard` pair and a `schema: 3` file kept `systems.guard`; schema 4 keeps neither, because whether the command guard runs is not this file's answer to hold. The package keeps its own switch in its own configuration, and a stale copy here could only ever disagree with it. Call budgets are per system under a session total, so one busy system cannot exhaust the allowance of the others and leave them dead for the rest of the session with event ordering deciding which one won. Turning the layer on or off writes that preference, because a switch that forgets is not a switch; `--session` is how a one-off change is kept out of the file. The key is resolved the way Pi resolves every provider credential, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to `<agent-dir>/auth.json`, then the environment variable. That is a deliberate, narrow exception to the rule that Pi state is never read, and it is bounded to exactly one question &mdash; the one `api_key` entry for the one provider this layer calls. Be precise about when: the key's *value* is read only on a call path the master switch already gates, but whether an entry *exists* is checked whenever status is displayed, including with the layer off, because that is the answer someone needs in order to configure it in the first place. A presence check reports a source name and nothing else. An `oauth` entry is ignored rather than unwrapped, because Pi refreshes those under its own lock and a second reader would race a rotation. The value is returned by a single function straight into a request header; nothing else in the layer receives it, and status output, the ledger and the Chat panel carry source names only. Pi trust decisions, sessions, missions and history are never read. `JEV_KEY_SOURCE=environment` restricts resolution to the environment, which is what this repository's own eval scripts set so a measured run cannot silently bill a developer's personal account.
 
 ## Command policy and the Jev guard
 
@@ -62,7 +62,7 @@ As of 0.3.0 the guard resolves its key the way the advisor does and the way Pi r
 
 ## Improvement authority and evidence
 
-Collection is off by default. Enabling it permits sanitized gap observations, not implementation. Only an exact human `/harness-improvement` selection authorizes a wishlist-sourced change. The selected contract is bound to the gap, source checkout, session, and selection generation. Web access tools (`web_search`, `source_check`, `fetch_content`, `get_search_content`) are hidden until a human offers them, through `/webaccess on` or by accepting a `request_capability` prompt.
+Collection is off by default. Enabling it permits sanitized gap observations, not implementation. The observation tool is offered only where a report could land: collection on, or undecided in an interactive session, where its first report asks for consent. The improvement skill is kept out of the model's skill list and is named by path only in the `/harness-improvement` kickoff. Only an exact human `/harness-improvement` selection authorizes a wishlist-sourced change. The selected contract is bound to the gap, source checkout, session, and selection generation. Web access tools (`web_search`, `source_check`, `fetch_content`, `get_search_content`) are hidden until a human offers them, through `/webaccess on` or by accepting a `request_capability` prompt.
 
 Retirement requires source registry integration, unchanged verification policy, a matching contract, bounded source snapshots, `npm run check`, and closed registered validators. Receipts distinguish machine-observed gates from model-reported acceptance evidence. Stale selections, changed source, missing evidence, and failed checks reject retirement. A validator proves only the behavior it exercises; the human remains responsible for accepting the result. The loop never commits, publishes, or installs a resulting change automatically.
 

package/THIRD_PARTY.md

@@ -52,4 +52,4 @@ The separately released `packages/browser-qa` source reuses the retired QA imple
 
 The website uses the bundled IBM Plex Sans and Plex Mono fonts under the SIL Open Font License 1.1; see [`site/fonts/LICENSE.txt`](https://github.com/TannerMidd/SpecPi/blob/main/site/fonts/LICENSE.txt). The website's scripts and diagrams are first-party code and assets. The README loads public package/license badges from Shields.io and build status from GitHub. Website fonts and media are not included in the npm package.
 
-SpecPi Chat 0.14.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.
+SpecPi Chat 0.15.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.

package/extensions/jev-advisor/broker.mjs

@@ -24,11 +24,24 @@ export const SYSTEM_LABELS = Object.freeze({
     retention: "Tool-result retention",
     gap: "Capability-gap triage",
     sources: "Delegation source ranking",
-    progress: "Progress and thrash detection",
     untrusted: "Untrusted-content classification",
     capability: "Turn-zero capability arming",
 });
 
+/** A Choice or Score distribution reduced to rounded numbers under safe keys, or nothing. */
+export function distribution(probabilities) {
+    if (!probabilities || typeof probabilities !== "object") {
+        return undefined;
+    }
+
+    const entries = Object.entries(probabilities)
+        .slice(0, 16)
+        .filter(([key, value]) => /^[A-Za-z0-9_-]{1,64}$/u.test(key) && Number.isFinite(value))
+        .map(([key, value]) => [key, Math.round(value * 1000) / 1000]);
+
+    return entries.length > 0 ? Object.fromEntries(entries) : undefined;
+}
+
 export function createBroker(options = {}) {
     // Injected in tests so master-off can be proven as "the transport was never reached" rather
     // than "no socket was observed".
@@ -177,7 +190,13 @@ export function createBroker(options = {}) {
         decide,
         apply,
         isCurrent = () => true,
+        trigger,
     }) => {
+        // Which local signals led here. Code-defined labels only, so the ledger can say why a call
+        // was made without carrying anything from the material itself.
+        const triggers = Array.isArray(trigger)
+            ? trigger.filter((item) => typeof item === "string" && /^[a-z][a-z0-9-]{0,39}$/u.test(item)).slice(0, 8)
+            : [];
         const startedGeneration = generation;
         const currentRequest = () => startedGeneration === generation && !signal?.aborted && isCurrent();
         const settings = readSettings();
@@ -216,6 +235,7 @@ export function createBroker(options = {}) {
             write({
                 system,
                 sent: false,
+                ...(triggers.length > 0 ? { trigger: triggers } : {}),
                 ok: false,
                 reason,
                 outcome: reason,
@@ -302,6 +322,7 @@ export function createBroker(options = {}) {
         write({
             system,
             sent: true,
+            ...(triggers.length > 0 ? { trigger: triggers } : {}),
             questionKeys,
             coverage: built.coverage,
             stateBytes: built.bytes,
@@ -328,7 +349,14 @@ export function createBroker(options = {}) {
             answers: Object.fromEntries(
                 Object.entries(result.answers ?? {}).map(([name, answer]) => [
                     name,
-                    { kind: answer.kind, value: answer.value, confidence: answer.confidence },
+                    {
+                        kind: answer.kind,
+                        value: answer.value,
+                        confidence: answer.confidence,
+                        // Without the distribution a Choice's margin gate cannot be re-checked
+                        // afterwards. Keys are code-defined option IDs; values are numbers.
+                        probabilities: distribution(answer.probabilities),
+                    },
                 ]),
             ),
         });

package/extensions/jev-advisor/config.mjs

@@ -18,26 +18,7 @@ import path from "node:path";
 import { randomUUID } from "node:crypto";
 
 /** Systems that may run inside a session. Offline scripts are not gated here. */
-export const SYSTEM_NAMES = Object.freeze(["retention", "gap", "sources", "progress", "untrusted", "capability"]);
-
-/**
- * What a confident stuck verdict is allowed to do. `notify` tells the person and cannot be wrong in
- * a way that costs anything; `message` appends a fixed line the model reads before its next
- * request, which changes behaviour.
- *
- * It ships on `notify`. Not because the gate cannot tell the cases apart -- it demonstrably can: on
- * the recorded fixtures a session repeating one failing call scores 0.89 for stuck with the mode at
- * 0.99 confidence, and a session working steadily scores 0.30 and reports "unknown" below the gate.
- * The missing number is the false-positive rate on real sessions, and the two things that bear on it
- * point the other way: running the same taxonomy over the 24 recorded failures left 14 of them
- * ungated, and a wrong nudge costs a turn, which is the exact quantity this system exists to save.
- *
- * So the condition for changing this default is a measurement, not an opinion, and the eval suite
- * is where it comes from: `--harness=specpi-jev` sets `message` and discloses it, because a
- * notification in a headless run reaches nobody and would measure the cost of the system with none
- * of its effect.
- */
-export const NUDGE_MODES = Object.freeze(["notify", "message"]);
+export const SYSTEM_NAMES = Object.freeze(["retention", "gap", "sources", "untrusted", "capability"]);
 
 const MAX_SETTINGS_BYTES = 4096;
 const MAX_CALL_BUDGET = 1024;
@@ -49,7 +30,7 @@ const MAX_TOTAL_BUDGET = 2048;
  * session, and which one won would be decided by event ordering rather than by anyone's policy.
  *
  * So the ceiling is two-level: each system gets its own, and the total is a real constraint because
- * it is deliberately less than their sum -- 2048 against 2274. That relationship is the invariant,
+ * it is deliberately less than their sum -- 1536 against 1570. That relationship is the invariant,
  * not either number: raising the total without raising the per-system ceilings would leave a total
  * no combination of systems could ever reach, which is a limit that reads as a limit and is not
  * one. `tests/jev-advisor.test.mjs` pins the inequality so a future change to one has to consider
@@ -61,7 +42,7 @@ const MAX_TOTAL_BUDGET = 2048;
 export const DEFAULT_BUDGETS = Object.freeze({
     // A backstop, not a working limit, and the number says which. Measured, a full tier-3 task -- a
     // 120-step repair chain over about 25 model requests -- spends 4 to 7 calls, and the busiest
-    // attempt ever recorded spent 12. A session would have to run for days before 2048 bound
+    // attempt ever recorded spent 12. A session would have to run for days before 1536 bound
     // anything a person was actually doing, which is the point: the ceiling should only ever be hit
     // by a loop, and hitting it should therefore be information rather than an inconvenience.
     //
@@ -74,22 +55,23 @@ export const DEFAULT_BUDGETS = Object.freeze({
     // to compact several times spends in the hundreds, so the backstop sat close enough to real
     // use to be reachable by a session that was working correctly.
     //
-    // Cost is not what these are for. A call is about $0.00003, so the whole total is about six
+    // Cost is not what these are for. A call is about $0.00003, so the whole total is about five
     // cents. They bound two things that do not get cheaper with scale: how much digest leaves the
     // machine for a third party, at up to 1 KB a call, and how much awaited latency a runaway loop
-    // can add before something stops it. Two megabytes of digest and an announced stop is the shape
+    // can add before something stops it. About a megabyte and a half of digest and an announced stop is the shape
     // of the trade.
-    total: 2048,
+    //
+    // Lowered from 2048 to 1536 when progress detection was withdrawn. That system was the only one
+    // that could ask every turn, and its 704 left the per-system ceilings summing to 1570 -- below
+    // 2048, so the total could never have bound. 1536 is still far above anything a working session
+    // has spent.
+    total: 1536,
     // The per-system numbers move with the total, because a total the per-system ceilings can never
     // add up to is not a constraint at all -- see below. They are scaled rather than re-derived:
     // each one's rationale is a firing frequency, and none of those frequencies changed.
     retention: 832,
     gap: 192,
     sources: 128,
-    // Turn-level, but gated behind local signals and a four-turn cooldown, so it only spends on
-    // sessions that already look wrong. The ceiling is what stops a genuinely thrashing session
-    // from spending the total on being told it is thrashing.
-    progress: 704,
     // Usually free: when retention is on, system 7's question rides the call retention was already
     // making against the same state. This ceiling only binds when retention is off, or when the
     // fetched result is too small for retention to be interested in it.
@@ -135,12 +117,11 @@ export function regularFile(file, label) {
 /** Every unknown shape collapses to the same all-off default rather than a partial enable. */
 export function defaultSettings() {
     return {
-        schema: 5,
+        schema: 6,
         master: false,
         startup: false,
         systems: Object.fromEntries(SYSTEM_NAMES.map((name) => [name, false])),
         budgets: { ...DEFAULT_BUDGETS },
-        progressNudge: "notify",
     };
 }
 
@@ -220,6 +201,25 @@ function migrateToFive(raw) {
     return { ...raw, schema: 5, systems, budgets };
 }
 
+/**
+ * Schema 6 removes progress detection, withdrawn on a replay rather than a solve-rate result.
+ *
+ * Over 174 recorded Terminal-Bench 2 runs where its local gate fired, the stuck verdict did not
+ * predict failure -- within-task AUC 0.46, with the shipped state and with one that added the recent
+ * commands -- and its failure-mode answer echoed its own input 624 times out of 624. A system that
+ * can steer a model has to show it can tell when to; this one could not.
+ *
+ * `systems.progress`, `budgets.progress` and `progressNudge` go, for the reason the compaction
+ * keys went: the hooks they gated no longer exist.
+ */
+function migrateToSix(raw) {
+    const { progress: _system, ...systems } = raw?.systems ?? {};
+    const { progress: _budget, ...budgets } = raw?.budgets ?? {};
+    const { progressNudge: _nudge, ...rest } = raw ?? {};
+
+    return { ...rest, schema: 6, systems, budgets };
+}
+
 /**
  * What the advisor will read, given a settings object, without writing it anywhere.
  *
@@ -237,20 +237,20 @@ export function normalizeSettings(raw) {
 function normalize(raw) {
     const one = raw?.schema === 1 ? migrate(raw) : raw;
     const four = one?.schema === 2 || one?.schema === 3 ? migrateToFour(one) : one;
-    const source = four?.schema === 4 ? migrateToFive(four) : four;
-    if (source?.schema !== 5) {
+    const five = four?.schema === 4 ? migrateToFive(four) : four;
+    const source = five?.schema === 5 ? migrateToSix(five) : five;
+    if (source?.schema !== 6) {
         return defaultSettings();
     }
 
     const systems = Object.fromEntries(SYSTEM_NAMES.map((name) => [name, source.systems?.[name] === true]));
 
     return {
-        schema: 5,
+        schema: 6,
         master: source.master === true,
         startup: source.startup === true,
         systems,
         budgets: normalizeBudgets(source.budgets),
-        progressNudge: NUDGE_MODES.includes(source.progressNudge) ? source.progressNudge : "notify",
     };
 }
 
@@ -283,7 +283,7 @@ export function writeFileAtomic(file, contents) {
 export function saveSettings(settings) {
     // A caller handing back an older shape is migrated rather than reset, so a round trip through
     // an old reader cannot quietly disable the layer.
-    const next = normalize([1, 2, 3, 4].includes(settings?.schema) ? settings : { ...settings, schema: 5 });
+    const next = normalize([1, 2, 3, 4, 5].includes(settings?.schema) ? settings : { ...settings, schema: 6 });
     const file = settingsFile();
     if (fs.existsSync(file)) {
         regularFile(file, "Jev settings");

package/extensions/jev-advisor/gate.mjs

@@ -95,19 +95,12 @@ export const THRESHOLDS = Object.freeze({
         boundary: 0.3,
         choiceConfidence: 0.75,
         margin: 0.2,
-        // Measured 0.96 on a report carrying a machine-specific path and 0.04 on a clean one.
-        high: 0.85,
-        low: 0.15,
-    }),
-    progress: Object.freeze({
-        scoreConfidence: 0.6,
-        boundary: 0.3,
-        // The strictest Choice gate in the layer, because it is the only system whose action can
-        // change what the model does next. Running the same taxonomy over the 24 recorded failures
-        // left 14 of them below this bar, which is the intended behaviour: silence is the correct
-        // answer to a session whose trouble is not yet legible.
-        choiceConfidence: 0.8,
-        margin: 0.25,
+        // The first measurement (0.96 on a machine-specific path) sent the path unredacted, which
+        // production never does. Built as a session builds it, the old single question scored a
+        // password written as prose 0.60-0.65 -- under this bar, on the case it existed for. The bar
+        // stayed; the question was split in two (`questions/gap.mjs`), and each written secret or
+        // identity now scores 0.87-0.98 against 0.07 or less for a report that only mentions one
+        // (`evals/runs/jev-reach-hard.json`).
         high: 0.85,
         low: 0.15,
     }),