specpi 0.28.0 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +45 -0
- package/NPM_RELEASE.md +1 -1
- package/README.md +22 -23
- package/SECURITY_MODEL.md +14 -12
- package/THIRD_PARTY.md +6 -5
- package/extensions/jev-advisor/broker.mjs +2 -4
- package/extensions/jev-advisor/config.mjs +91 -62
- package/extensions/jev-advisor/gate.mjs +5 -30
- package/extensions/jev-advisor/index.ts +8 -260
- package/extensions/jev-advisor/key-source.mjs +1 -1
- package/extensions/jev-advisor/layer.mjs +8 -31
- package/package.json +2 -1
- package/scripts/jev-guard.mjs +154 -0
- package/scripts/packages.mjs +0 -56
- package/scripts/specpi.mjs +24 -52
- package/templates/settings.json +2 -1
- package/extensions/jev-advisor/questions/compaction.mjs +0 -153
- package/extensions/jev-advisor/questions/guard.mjs +0 -168
- package/extensions/jev-advisor/risk.mjs +0 -442
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,50 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.30.0 - 2026-09-21
|
|
4
|
+
|
|
5
|
+
- Replace the evaluations page with Terminal-Bench 2.0. The suite that filled it was written in this repository, and it was retired for the reason it kept reporting: nearly every harness passed nearly every task, so the thing it was built to measure was the thing it could not see, and three rounds of harder tasks did not move it. A suite cannot mark its own homework. The page now carries somebody else's benchmark on tasks nobody here chose -- Pi with no harness, SpecPi + Jev, Oh My Pi and Claude Code, 334 scored attempts on one model through one endpoint.
|
|
6
|
+
- Report the result the outside benchmark actually gives, which is that it does not separate the harnesses either. Solve rate spans 0.767 to 0.838 across the four arms; Pi against SpecPi + Jev is Fisher p = 0.83, and exactly 30/39 each on the slice the comparison rests on. That is the second suite to find no difference in capability, and this one's tasks were not chosen here and are failed about a fifth of the time, so the usual explanation -- that the tasks are too easy -- does not cover it.
|
|
7
|
+
- Make spend the column the page is about, because it is the one that separates. SpecPi + Jev reaches the same solve rate as bare Pi on 26% fewer prompt tokens and 23% less cost per attempt, and cost per attempt spans 1.7x across the three rows whose cached share could be measured. Score is the control here rather than the finding.
|
|
8
|
+
- Publish the run unfinished, and say so in the first paragraph. Terminal-Bench 2.0's hard split and a SWE-Gym slice have not run, plain SpecPi is not an arm so nothing separates the harness from the advisory layer, and Oh My Pi's widened slice holds six attempts per task rather than three because two launches of the same script raced into one output directory. Section 07 of the page lists each of these rather than leaving the reader to infer them from a table that looks complete.
|
|
9
|
+
- Publish Claude Code's cached share as absent rather than as zero. Its traffic crosses the Messages/chat-completions translation, which recorded no cached tokens at all on these runs, and a zero would have read as "Claude Code destroys cache efficiency" -- which the run does not show, since Pi and SpecPi see 94% on the same endpoint in the same sitting. Its cost is published as an upper bound for the same reason, because the cached share is what would discount it.
|
|
10
|
+
- Map Anthropic's cache tokens onto the pair Anthropic reports them as. `scripts/eval-anthropic.mjs` was passing the provider's prompt total through as `input_tokens` and emitting no `cache_read_input_tokens`, but Anthropic counts cache reads outside the input total rather than inside it, so the two have to split the prompt. This is the fault above, fixed; the rows it spoiled predate it and are labelled rather than silently re-priced, because the tokens are not recoverable from an archived aggregate.
|
|
11
|
+
- Count auxiliary calls as spend. `modelRequests()` answers how many turns an attempt took, and the token totals were reading the same filtered list, so the session-title call Claude Code makes on every attempt cost nothing in any report. Billing and conversation counting are separate questions now: totals and pricing cover the auxiliary calls, while the turn series and the per-turn figures stay conversation-only, because an auxiliary is not a turn.
|
|
12
|
+
- Fail a Responses reply the provider reported as failed. `scripts/eval-responses.mjs` inspected only `incomplete_details`, so an HTTP 200 stream carrying `response.failed` was assembled into an ordinary reply and returned with a `stop` finish reason -- a provider failure arriving at the harness as a short answer, which is the one shape that cannot be told from the harness giving up. The proxy returns 502 with the provider's own code and message instead. `max_output_tokens` stays a finish reason rather than a failure, because that turn happened and its text is real.
|
|
13
|
+
- Admit a row to the controlled-window comparison only when the window was enforced. `scripts/tier6-metrics.mjs` accepted any attempt that had compacted at all, which is evidence that the harness compacts and not that it compacted at the declared window: an 80,000-token attempt with one compaction was being compared against attempts held to 24,000. The bound is measured rather than picked -- at that window, enforced attempts peak at 1.58x it and provider-default attempts start at 1.89x -- and it excludes rather than admits when it is unsure.
|
|
14
|
+
- Let an eval run drop one part of the SpecPi layer. `SPECPI_EVAL_ABLATE` takes `agents`, `tools` and `goal`, combinable, so a run can strip the harness to Pi's exact tool surface and measure what that alone accounts for. It was built to test a specific claim of mine -- that SpecPi's four extra tools explained a collapse on one model -- and it refuted it, which is the use an ablation is for.
|
|
15
|
+
- Delete the retired suite's publishers rather than leaving them aimed at sections that no longer exist. `scripts/tb-metrics.mjs`, `scripts/tb-site.mjs` and `scripts/tier6-site.mjs` go with the three data files they wrote. `scripts/eval-site.mjs` keeps the part with other callers -- reading run reports, and the chart and table renderers the new page draws with -- and loses the page-specific rendering it no longer has a page for. `scripts/tier6-metrics.mjs` still runs, writing beside the runs it describes instead of into a published directory.
|
|
16
|
+
|
|
17
|
+
- Add Claude Code to the eval suite as `--harness=claude-code`. It is the first harness that speaks the Anthropic Messages API, and the provider the whole suite runs against is chat-completions only, so measuring it at all meant translating both directions. `scripts/eval-anthropic.mjs` does that and nothing else.
|
|
18
|
+
- Translate at the edge rather than throughout. The request becomes a chat-completions request before the proxy records anything, and the reply is converted back only on the way out, so `summarizeRequest`, `toolOutcomeOf`, `extractUsage`, `extractToolCalls` and the cost pipeline all keep reading the one shape they have always read. The alternative -- teaching each of them a second shape -- is six places to drift instead of one place to read.
|
|
19
|
+
- Rebuild the stream rather than piping it. `startProxy` already buffers the whole upstream reply before answering, so there is no streaming fidelity to lose: the OpenAI deltas are collected into one finished message and re-emitted as a well-formed Anthropic event sequence. `tests/eval-anthropic.test.mjs` covers the cases where a wrong conversion would be scored as the harness failing the task rather than as the suite losing its work -- several tool results in one turn, a failed tool result, fragmented tool arguments, and arguments that never parse.
|
|
20
|
+
- Read no Anthropic credential and spend none. Claude Code normally authenticates against a stored subscription login, so a run against a real configuration could fall back to it and bill a subscription for an eval. The row uses a disposable `CLAUDE_CONFIG_DIR` with no login to fall back on, points `ANTHROPIC_BASE_URL` at the logging proxy, and the proxy discards the client's token and sends `EVAL_FORWARD_KEY` upstream. Runs bill the same OpenCode Go subscription as every other harness; the probe confirms it, reporting `apiKeySource: none`.
|
|
21
|
+
- Answer the reachability probe instead of recording it. Claude Code sends `HEAD /api/hello` before it starts. It carries no body and names no model, so recording it added a turn with no tokens, and forwarding it posted an empty payload to the provider's completions endpoint and billed for the reply.
|
|
22
|
+
- Keep the session-title call out of the turn count. A Messages request carrying no tool schema is the harness titling its own session, which is the rule `conversationSummary` has always used to find the real first call; it is now tagged so its spend stays in the cost column while it stops inflating turns. Claude Code is the first proxy harness to make one -- OpenCode makes one too but reports its own per-step usage rather than going through this log. Uncorrected, a single-prompt attempt read as three turns instead of one.
|
|
23
|
+
- Stop losing a finished attempt to a directory that will not delete. On Windows a directory stays locked while any process holds it as a working directory, and a harness that leaves a helper alive for a moment after its own exit keeps the attempt's workspace locked with it. The attempt's result is already computed by then, so throwing discarded a real measurement to report a temporary file the operating system cleans up anyway. It now retries for longer, then says so and moves on.
|
|
24
|
+
- Note what this row cannot do. Claude Code exposes no context-window setting -- its compaction triggers off the model's own window -- so it cannot be held to a tier's declared window, and tier 6 reads its attempts as unwindowed the way it did for Codex and OpenCode before those learned to read one.
|
|
25
|
+
- Withdraw compaction guidance from the Jev layer. It was the layer's only system that had ever applied a verdict at scale, and the only measurement of it says it cost solve rate. Tier 6 ran the pair comparison twice, months apart on two versions: SpecPi 0.28.0 gave 5/8 against plain SpecPi's 8/8 on the long-session task, and 0.29.0 gave 3/8 against 6/8. Pooled that is 8/16 against 14/16, Fisher exact p = 0.054. The shorter task showed nothing either time. `extensions/jev-advisor/questions/compaction.mjs`, both hooks that used it, its gate thresholds and its budget are gone, and the managed set drops with them.
|
|
26
|
+
- Attribute the loss before removing the thing. Over those attempts `compaction` accounted for 139 of the 142 verdicts the layer applied; `retention` applied nothing across 287 calls and `progress` applied three. Whatever separated the two arms was almost entirely this one system's doing, which is what made a removal defensible on a comparison that never reached significance.
|
|
27
|
+
- Say plainly what this does not establish. p = 0.054 is not significance. A within-arm check of whether heavier compaction predicts failure reaches only p = 0.251. The mechanism was never isolated: the system could not delete anything -- it appended one sentence to the summariser's prompt, leaving Pi's own cut point and token budget alone -- so any harm ran through a summary that kept the wrong material, most likely because the sentence told the summariser to *prioritise* one kind of content on a task built so that a fact from the first turn must survive. What carried the decision was three weak signals agreeing and the asymmetry of the bet, not a result.
|
|
28
|
+
- Take the `/tree` branch labels with it. They rode the compaction switch by design, on the reasoning that someone who has decided the advisor may steer a summary has decided that once, and the same `decide()` that built the harmful sentence also fed the branch-summary path. Nothing measured the labels either way; they are removed because the machinery they shared is.
|
|
29
|
+
- Bump the settings schema to 5 and drop both `systems.compaction` and `budgets.compaction` on the way through, the way schema 4 dropped the guard keys. Nothing anyone had switched on is being switched off in any meaningful sense: the hooks those keys gated no longer exist, so a retained preference could only describe a system that cannot run. The panel in SpecPi Chat drops the row with it, and a test pins the panel's shape to the advisor's so the two cannot drift.
|
|
30
|
+
- Keep the runs that led here in the record rather than deleting them. The tier-6 tables carried the compaction row and both pair comparisons until this release retired that page; they remain in this repository's history, and the Jev page points at this changelog for the comparison and the p-values. A measurement that led to a removal is the reason the removal can be defended, so it outlives the page it was published on.
|
|
31
|
+
- Grade tier 1 and 2 answers by content rather than by the line endings they were written with. A file written through a Windows shell redirection ends its lines CRLF and one written through a file tool ends them LF; six checkers compared bytes, so they failed correct work for the route it took. Five of the six tier 1-2 failures in the published DeepSeek run were this, all on Codex, which published Codex at 30/37 and a 0.742 score against the 35/37 and 0.862 the regrade measures. Files a task says not to touch stay byte-exact, because rewriting one with different endings is still rewriting it.
|
|
32
|
+
- Supersede eval cells rather than pooling them. Each attempt's workspace is discarded, so a cell cannot be re-graded, only re-run; the site now keeps one cell per harness, task and tier and takes the last report that measured it. Without that rule the regrade above would have been averaged with the readings it corrects.
|
|
33
|
+
- Reach a provider on the wire it actually speaks. Muse Spark serves only the Responses API and returns 503 "Endpoint is unavailable" to a chat-completions request, as gpt-5.6-luna and grok-4.6 do; Codex already spoke Responses and reached it unaided, so the model was measurable on one row out of eight. `scripts/eval-responses.mjs` translates at the edge, the seam `eval-anthropic.mjs` already uses, and `EVAL_FORWARD_WIRE` declares the wire rather than probing for it, so latency is not doubled and the wire stays a property of the run. The report records which wire was used and the method says so, because seven rows having been translated is a material fact about how they were measured.
|
|
34
|
+
- Mint an OpenCode session only for the endpoint that routes on one. The gate was on forwarding rather than on the endpoint, so a run against any other provider spent an OpenCode call per attempt, billed its tokens to the run, and needed a login the run was not otherwise using, to produce an id the provider ignores. The published method claimed every forwarded run minted one, which was never true of a direct provider.
|
|
35
|
+
## 0.29.0 - 2026-09-20
|
|
36
|
+
|
|
37
|
+
- Return the command guard to `specpi-jev-guard`, and leave it there. 0.28.0 built the guard into the layer as its eighth system, for three stated reasons that were properties of the package as it stood: a global configuration file with no session scope, a key read from the environment only, and a fail-closed posture. Two of the three are fixed upstream. Guard 0.3.0 ships `/jev-guard setup | on | off [--global] | check | model | backend`, so it has a session scope and a switch of its own, and it resolves Pi's saved login before the environment, so `/login openrouter` serves it exactly as it serves the advisor. What remains is fail-closed, which is a disclosed trade rather than a defect. `extensions/jev-advisor/risk.mjs` and `questions/guard.mjs` are gone, the package is pinned again, and the base is eight.
|
|
38
|
+
- Keep the guard out of the Jev layer entirely, rather than half in it. There is no `/jev guard` command, no `systems.guard`, no `guard` pair in the layer's settings, no guard row in the Chat panel and no guard line in `/jev status`; the advisor imports nothing from it, and a test walks `extensions/jev-advisor/` to keep it that way. Two switches over one gate is what made the package's own shape a problem the first time, and adding a second one back would have reintroduced it with the roles swapped. The package owns its switch; SpecPi owns whether the package is installed and that it arrives off.
|
|
39
|
+
- Move the seam to `scripts/jev-guard.mjs`, because it is an installer concern and nothing else. It was a managed resource copied into every agent directory, where after this change nothing would have loaded it. The managed set drops from 34 files to 33, and the seam ships with the CLI instead.
|
|
40
|
+
- Write the inert posture at install and update, and nowhere else. The old seam re-asserted it at every session start, which is what made a `--global` save through the package's own command fail to survive a restart; it does not run at session time now, so the package's switch stays the package's between installer runs. The install-time write is still unconditional, because an install carrying a stale `enabled: true` from before the package was last unpinned would otherwise arm a fail-closed gate the moment it came back. That is the one case where establishing the default takes something away, so the run reports it by name and says which command restores it.
|
|
41
|
+
- Drop both older guard keys in schema 4 rather than migrating either. Schema 2 kept `guard: { enabled, startup }` and schema 3 kept `systems.guard`; neither was ever the authority over whether the guard runs, so keeping a copy could only produce a second answer that disagreed with the package's own file. Nothing about anyone's guard changes: a schema-2 preference had already been written through to that file by the last session that read it, and schema 3 shipped with the package unpinned.
|
|
42
|
+
- Drop the package-retirement machinery with the package it existed for. `retiredPackages`, `removeRetiredPackages` and `retireBasePackages` unpinned `specpi-jev-guard` by name on every install and update, including under `--skip-package-install`, and `specpi doctor` failed on an entry that was still configured. With the package pinned again all three would strip the entry SpecPi itself writes, so they are gone rather than filtered. The transaction's watched set returns to its narrower condition, which is correct again now that nothing writes settings outside it.
|
|
43
|
+
- Pin the command guard at 0.4.0 and show its counter in SpecPi Chat. The package publishes one status line per session under the key `jev-guard` -- classifier calls, blocked calls once there are any, and the last verdict -- and Chat already stored it, as a raw key/value row inside the collapsed generic runtime list behind up to twenty-four other widgets. It is a first-class readout now: a chip in the session footer beside the token and cache counts, showing the leading count, turning amber once something has been blocked, with the package's whole line in the tooltip. The key joins the reserved set so a busy session cannot crowd it out. Chat renders the string and does nothing else with it -- no recount from the session records, no reformatting, and no control, because the guard is configured in Pi and not here. The chip is absent exactly when the guard is not gating, since the package clears its own line when it is switched off or its audit display is.
|
|
44
|
+
- Leave `auditDisplay` to the package. It defaults to `status`, which is what publishes the counter, and asserting it in the install-time seam would take a display preference away from the user to guarantee a readout in one frontend.
|
|
45
|
+
- Raise the session call ceiling from 512 to 2048, and the per-system ceilings with it. 512 was sized the way 120 was before it -- against sessions shorter than the ones the layer now runs in. The long-session evals are what showed it: a session long enough to compact several times spends in the hundreds, so a backstop meant to be reachable only by a loop had drifted close enough to real use to stop a session that was working correctly. The per-system numbers move too, because the total is only a constraint while the per-system ceilings can add up past it: 2048 against a sum of 2322, scaled from the same firing frequencies rather than re-derived. `capability` stays at 2, alone, because it is bounded by construction rather than by frequency -- it asks once a session and then never again. A test pins the inequality rather than the literals, so the next change to either number has to consider the other. The whole ceiling is about six cents and two megabytes of digest.
|
|
46
|
+
- Take the guard out of the eval row it was never going to measure. `eval-harnesses.mjs` asked for `guard: true` and verified the advisor read it back, which now throws rather than silently measuring less than it asked for — the check working as designed. The disposable homes do not install the package, so the row does not claim it. The tier-6 and Terminal-Bench tables keep their recorded guard rows, labelled as the native guard they measured, because deleting a measurement is not the same as correcting one.
|
|
47
|
+
|
|
3
48
|
## 0.28.0 - 2026-09-18
|
|
4
49
|
|
|
5
50
|
- Read the key Pi already has. The layer resolved its credential from `OPENROUTER_API_KEY` and nothing else, while Pi had long since stored an OpenRouter key where it stores every provider credential — the `openrouter` entry that `/login openrouter` writes to `auth.json`. So a person who had logged in, whose every model call was working, was told `key: missing` by the one component that had opted out of Pi's credential store, and there was no interface anywhere that would have explained the gap. The advisor now follows Pi's own documented resolution order, the store first and then the environment, which is what makes `/login openrouter` the answer to "how do I configure the API key" rather than a second thing to set up. `/jev status` lists every source and marks the one in force, because "missing" with nothing to act on is what sent people looking for a key field that does not exist.
|
package/NPM_RELEASE.md
CHANGED
|
@@ -6,7 +6,7 @@ Publishing, tags, deprecation, ownership changes and GitHub Releases require exp
|
|
|
6
6
|
|
|
7
7
|
## Prepare
|
|
8
8
|
|
|
9
|
-
1. This candidate is SpecPi **0.
|
|
9
|
+
1. This candidate is SpecPi **0.30.0**, paired with the Chat **0.14.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
|
|
10
10
|
2. Validate:
|
|
11
11
|
|
|
12
12
|
```sh
|
package/README.md
CHANGED
|
@@ -44,7 +44,7 @@ Everything it touches is written down, versioned, and easy to undo.
|
|
|
44
44
|
|
|
45
45
|
## Measured context
|
|
46
46
|
|
|
47
|
-
This chart shows first-call context from a clean install: all
|
|
47
|
+
This chart shows first-call context from a clean install: all eight pinned packages, the working agreement, and the skills Pi finds. It has not been re-measured since the command guard returned to a pinned package rather than a native system. "Enabled" means browser QA, delegation, and web access are switched on, with no goal, scope, or improvement selection active.
|
|
48
48
|
|
|
49
49
|
The solid rows are measured by us, from the request each setup actually sends through one local test provider. That includes OpenCode, the DeepSeek Harness, and Oh My Pi, all measured as installed. The faded Codex CLI and Claude Code rows come from HarnessTax's published numbers, measured under their own setup. Treat those as a rough reference, not a head-to-head test. These are character counts. They say nothing about tokens, cost, or how well each tool does the job. The research page [breaks down the enabled setup by feature](https://tannermidd.github.io/SpecPi/research/#specpi), so you can see what each switch costs on its own.
|
|
50
50
|
|
|
@@ -73,36 +73,35 @@ The gap between the two SpecPi bars comes from a few separate switches, so the e
|
|
|
73
73
|
## Harness evaluations
|
|
74
74
|
|
|
75
75
|
The chart above counts characters. It says nothing about what a harness costs to
|
|
76
|
-
actually use, or whether it finishes the job. That is what the
|
|
77
|
-
|
|
78
|
-
harness changing.
|
|
76
|
+
actually use, or whether it finishes the job. That is what the evals are for:
|
|
77
|
+
[Terminal-Bench 2.0](https://www.tbench.ai/), the same model and the same frozen
|
|
78
|
+
price list, with only the harness changing.
|
|
79
79
|
|
|
80
80
|
<!-- eval-summary -->
|
|
81
81
|
|
|
82
|
-
**
|
|
82
|
+
**334 scored attempts across 20 tasks and 4 harnesses**,
|
|
83
|
+
all on `deepseek-v4.1-flash`.
|
|
83
84
|
|
|
84
|
-
| Harness | Solved | Cost/attempt | Prompt tokens |
|
|
85
|
-
| --- | --- | --- | --- | --- |
|
|
86
|
-
|
|
|
87
|
-
|
|
|
88
|
-
|
|
|
89
|
-
|
|
|
90
|
-
| Codex CLI | 30/37 | $0.0067 | 242,031 | 38,148 |
|
|
91
|
-
| Oh My Pi | 35/37 | $0.0070 | 257,783 | 62,612 |
|
|
92
|
-
| DeepSeek Harness | 35/37 | $0.0078 | 281,676 | 31,750 |
|
|
85
|
+
| Harness | Solved | Rate | Cost/attempt | Prompt tokens | Cache hit |
|
|
86
|
+
| --- | --- | --- | --- | --- | --- |
|
|
87
|
+
| SpecPi + Jev | 62/74 | 0.838 | $0.0125 | 381,153 | 94.5% |
|
|
88
|
+
| Pi (base) | 60/74 | 0.811 | $0.0161 | 512,903 | 93.7% |
|
|
89
|
+
| Oh My Pi | 90/113 | 0.796 | $0.0216 | 1,103,136 | 97.4% |
|
|
90
|
+
| Claude Code | 56/73 | 0.767 | $0.1128 or less | 630,410 | not measured |
|
|
93
91
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
92
|
+
Solve rate does not separate these harnesses -- Pi against SpecPi + Jev is Fisher
|
|
93
|
+
p = 0.83 -- so the column worth reading is what each one spent reaching the same
|
|
94
|
+
answers. Cost is recomputed from recorded tokens against a dated price file, never
|
|
95
|
+
taken from a harness's self-report. Claude Code's cached share was not recorded on
|
|
96
|
+
this run, so its cost is an upper bound.
|
|
98
97
|
|
|
99
98
|
<!-- /eval-summary -->
|
|
100
99
|
|
|
101
|
-
Read the method, the charts and the caveats on the
|
|
102
|
-
[evaluations page](https://tannermidd.github.io/SpecPi/evaluations/)
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
100
|
+
This run is still in progress. Read the method, the charts and the caveats on the
|
|
101
|
+
[evaluations page](https://tannermidd.github.io/SpecPi/evaluations/), which lists
|
|
102
|
+
what is missing and what is known to be wrong. The table above is regenerated
|
|
103
|
+
from the run data by `node scripts/tb2-site.mjs`, so it cannot drift from the
|
|
104
|
+
published figures.
|
|
106
105
|
|
|
107
106
|
## Install
|
|
108
107
|
|
package/SECURITY_MODEL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Security model
|
|
2
2
|
|
|
3
|
-
SpecPi provides scope monitoring, an explicit harness improvement loop and an optional Jev advisor, and installs
|
|
3
|
+
SpecPi provides scope monitoring, an explicit harness improvement loop and an optional Jev advisor, and installs eight pinned packages as its default base. Extensions run as trusted code with Pi's permissions. Scope is not an OS sandbox or a general command guard. Use OS isolation for hostile code.
|
|
4
4
|
|
|
5
5
|
## Scope monitoring
|
|
6
6
|
|
|
@@ -30,29 +30,31 @@ The advisor also keeps `<agent-dir>/specpi/jev/usage.json`, the running call cou
|
|
|
30
30
|
|
|
31
31
|
The advisor holds no authority. It never grants a capability, never calls a tool and never allows one. Its only blocking action is to refuse a capability-gap report that looks like it carries a credential, which asks the model to rewrite its own text and discards nothing. Failure is silent, not closed: a timeout, HTTP error, missing key, missing consent, an exhausted per-system or session budget, or ungated confidence produces no advice, and the existing code path runs unchanged. Tool-result retention can shorten a large read-only result before it is appended; it never alters a write, edit or error result, and the replacement says the output is recoverable by re-running.
|
|
32
32
|
|
|
33
|
-
|
|
33
|
+
Two of the seven systems do something the other five do not, and both are off by default like everything else here.
|
|
34
34
|
|
|
35
35
|
**System 5, progress detection**, is the only one that can change what the model does. On a confident verdict that a session has stopped making progress it adds one fixed, code-written line at a turn boundary. It ships set to `notify`, which tells the person and cannot change the session; `message`, which appends the line for the model to read, is a deliberate opt-in. It is written once per session and never retracted, it never aborts or blocks, and it is suppressed entirely when the same call reports that a person would have to answer something first. Nothing the model produced ever enters the transcript: the line is chosen from a fixed table by failure mode.
|
|
36
36
|
|
|
37
|
-
**
|
|
37
|
+
The command guard is **not** part of this layer. It is a separate pinned package with its own gate, its own configuration file, its own commands and a fail-closed posture, and it has its own section below. It is named here because a reader who took "the advisor holds no authority" to cover everything SpecPi installs would be wrong about exactly one component. No switch in SpecPi turns it on: `/jev` does not mention it, the advisor imports nothing from it, and the only thing SpecPi ever writes to its configuration is the off side of it, at install time.
|
|
38
38
|
|
|
39
39
|
**System 7, untrusted-content classification**, prepends a fixed warning line to externally fetched content that confidently reads as instructions addressed to an agent. It applies only to web and browser tool results, never to the agent's own shell output or file reads. It is defence in depth and explicitly not a control: it never blocks, it has no authority over what the model then does, and it should not be relied on to stop prompt injection. Its value is that untrusted content in a fetched page is currently owned by nothing at all.
|
|
40
40
|
|
|
41
41
|
An earlier version of this document said its false-positive rate was measurable on tier 5 of the eval suite. It is not, and the re-run that checked found out why: **this system has never been called once**. It rides on a web or browser tool result, no task in any tier produces one, and the run that would have exercised Browser QA keeps it withdrawn because capability arming needs an interactive human. So the honest statement is that system 7 ships unmeasured, and the eval suite as it stands cannot measure it. That is a gap in the suite, not a property of the system, and it is stated here rather than left as an absent number that would read as a zero.
|
|
42
42
|
|
|
43
|
-
Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous
|
|
43
|
+
Settings live in `<agent-dir>/specpi/jev/settings.json`, written atomically with owner-only permissions. Links, irregular files, oversize files, an unrecognised schema and unparseable contents all read as off. The one exception is the layer's own previous schemas: an older file is migrated forward rather than read as unrecognised, because "collapse to all-off" is a rule for corrupt input and applying it to our own earlier version would silently disable a layer the user had switched on. Two keys are deliberately dropped rather than migrated. A `schema: 2` file kept a `guard` pair and a `schema: 3` file kept `systems.guard`; schema 4 keeps neither, because whether the command guard runs is not this file's answer to hold. The package keeps its own switch in its own configuration, and a stale copy here could only ever disagree with it. Call budgets are per system under a session total, so one busy system cannot exhaust the allowance of the others and leave them dead for the rest of the session with event ordering deciding which one won. Turning the layer on or off writes that preference, because a switch that forgets is not a switch; `--session` is how a one-off change is kept out of the file. The key is resolved the way Pi resolves every provider credential, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to `<agent-dir>/auth.json`, then the environment variable. That is a deliberate, narrow exception to the rule that Pi state is never read, and it is bounded to exactly one question — the one `api_key` entry for the one provider this layer calls. Be precise about when: the key's *value* is read only on a call path the master switch already gates, but whether an entry *exists* is checked whenever status is displayed, including with the layer off, because that is the answer someone needs in order to configure it in the first place. A presence check reports a source name and nothing else. An `oauth` entry is ignored rather than unwrapped, because Pi refreshes those under its own lock and a second reader would race a rotation. The value is returned by a single function straight into a request header; nothing else in the layer receives it, and status output, the ledger and the Chat panel carry source names only. Pi trust decisions, sessions, missions and history are never read. `JEV_KEY_SOURCE=environment` restricts resolution to the environment, which is what this repository's own eval scripts set so a measured run cannot silently bill a developer's personal account.
|
|
44
44
|
|
|
45
45
|
## Command policy and the Jev guard
|
|
46
46
|
|
|
47
|
-
|
|
47
|
+
`specpi-jev-guard` is pinned in the base set but ships **inert**. Its own default is `enabled: true`, so left alone a fresh install would begin gating shell and file calls through a third-party service immediately; SpecPi writes `enabled: false`, and the package's own `/jev-guard setup` is how a human opts in — no SpecPi command does. `@gotgenes/pi-permission-system` stays pinned and, while the guard is off, decides every tool call exactly as it did before the guard existed.
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
Be precise about what the guard does once it is on, because it cannot be configured away. The guard is **fail-closed by design**: with no key, an unreachable endpoint, an answer it cannot parse, or a middle-band verdict in a session with no UI, it blocks the call and reports why. There is no setting that hands the decision back to the permission system instead — `uncertain` chooses between asking and denying the middle band, and only its `allow` value lets that band through unattended. So switching it on accepts that an outage stops gated work until it is switched off again. That trade is the user's to make, which is why SpecPi only ever writes the off side of it and why turning it on takes the package's own `/jev-guard` command.
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
SpecPi asserts only three fields in `<home>/.pi/jev-guard.json` and merges them into whatever is already there, so a user's own thresholds, safe-command globs and protected paths survive: `enabled: false`, `uncertain: "ask"` so a middle-band verdict asks a human wherever there is one to ask, and `backend: "openrouter"` so the guard and the advisor resolve the same credential. Off is a written configuration, not an absence of one. SpecPi writes only the global file; a project-local override under a workspace's `.pi/` remains the user's to make.
|
|
52
52
|
|
|
53
|
-
|
|
53
|
+
That write happens at install and update, and nowhere else. It deliberately does not repeat at session start, so the package's own `/jev-guard on [--global]` is the only switch and a saved choice survives every restart. It does repeat on the next installer run, unconditionally: an install left holding a stale `enabled: true` from before the package was last unpinned would otherwise arm a fail-closed gate the moment the package came back. That is the one case where establishing the default takes something away from someone, so the run reports it by name rather than doing it quietly.
|
|
54
54
|
|
|
55
|
-
|
|
55
|
+
As of 0.3.0 the guard resolves its key the way the advisor does and the way Pi resolves every provider credential: Pi's saved login first, then the environment. One `/login openrouter` therefore serves both without either component knowing about the other. `/jev-guard` reports its own state, including which backend and whether a key is present; `/jev status` does not, because the guard is not part of the layer it describes. Neither the guard nor the advisor is an OS sandbox.
|
|
56
|
+
|
|
57
|
+
0.4.0 publishes a per-session counter as a Pi status item under the key `jev-guard` — the number of classifier calls, the number of blocked calls once there are any, and the last verdict. SpecPi Chat renders that line in its session footer and does nothing else with it: it does not recount from the session records, reformat the text, or offer a control. The key is absent whenever the guard is off or its `auditDisplay` is, so the readout appears exactly when the guard is gating. Every judged call is also written to Pi's own session file by the package, which is the durable audit trail and is not SpecPi's to manage.
|
|
56
58
|
|
|
57
59
|
## Improvement authority and evidence
|
|
58
60
|
|
|
@@ -66,11 +68,11 @@ Wishlist records remain local under `<agent-dir>/specpi/`. Sanitization and salt
|
|
|
66
68
|
|
|
67
69
|
Package acquisition requests exact npm dependency saves through the child process environment and verifies the installed top-level versions before completing the transaction. A mismatch fails the operation and triggers managed-state rollback. Upstream transitive dependency ranges remain outside this pinning guarantee.
|
|
68
70
|
|
|
69
|
-
`plan` is read-only. Installation, updates, and removal require confirmation or `--yes`. SpecPi manages its two first-party extension families, improvement skill, manifest, AGENTS marker block, and the
|
|
71
|
+
`plan` is read-only. Installation, updates, and removal require confirmation or `--yes`. SpecPi manages its two first-party extension families, improvement skill, manifest, AGENTS marker block, and the eight package entries listed in `templates/settings.json`. Normal install/update runs `pi install npm:<name>@<pin>` for every default package. Only package entries are merged; existing resource filters and unrelated settings are retained. `--skip-package-install` allows a core-only install or preserves an already configured base during update, skipping Chromium setup too. Confirmed install/update otherwise invokes the installed Browser QA Node bin with bounded subprocesses to download Chromium and check readiness. `--skip-browser-install` skips that setup only; doctor still checks readiness. SpecPi never passes `--with-deps`, auto-installs OS libraries, or installs/removes Bun. No new shell profile integration is installed.
|
|
70
72
|
|
|
71
73
|
Managed configuration and files are locked and backed up before mutation; first-party writes are atomic and checksum-tracked. Failure restores the saved configuration and first-party files. Package acquisition runs upstream package-manager scripts and may leave downloads, dependency changes, or external script effects even after configuration rollback. Those effects, including downloaded browser-cache bytes, are outside SpecPi's transaction and survive uninstall. Updates require `--force` before replacing modified retained resources. Retired resources are backed up before deactivation; pre-install files are restored where ownership records identify them. Old runtime directories are moved into backups without inspecting their contents. Backups and private evidence remain after uninstall and can contain sensitive local material.
|
|
72
74
|
|
|
73
|
-
Legacy migration restores only recorded settings ownership, preserves differing user values, and removes only the SpecPi shell marker block before applying the new base. The installer does not enumerate or modify authentication, provider credential stores, trust, sessions, missions, history, or unrelated private evidence. Normal updates deliberately reapply the default package pins, retaining the original entry for removal.
|
|
75
|
+
Legacy migration restores only recorded settings ownership, preserves differing user values, and removes only the SpecPi shell marker block before applying the new base. The installer does not enumerate or modify authentication, provider credential stores, trust, sessions, missions, history, or unrelated private evidence. Normal updates deliberately reapply the default package pins, retaining the original entry for removal. Uninstall restores only package entries that still match the last installed value; user edits are preserved. Downloaded packages and tools are not deleted. Resources that SpecPi never owned require separate human management.
|
|
74
76
|
|
|
75
77
|
## Browser QA package
|
|
76
78
|
|
|
@@ -86,7 +88,7 @@ The optional **Destructive guard** preset replaces the complete global configura
|
|
|
86
88
|
|
|
87
89
|
## Upstream package boundary
|
|
88
90
|
|
|
89
|
-
The
|
|
91
|
+
The eight packages add their own extensions, tools, prompts, skills, network connections, filesystem operations, and subprocesses under their upstream defaults. They are not confined by the improvement loop's selection requirement. Permission System owns tool policies; SpecPi does not inject a duplicate guard or claim its coverage. Delegation starts real Pi child sessions in the same process tree, restricted to a frozen source snapshot and three read-only tools; experiments run the user's own `git` and create worktrees on disk. Neither is an OS sandbox. See [delegation](packages/delegation/SECURITY.md) and [experiments](packages/experiments/SECURITY.md). `pi-lens` and `pi-background-tasks` (including its Anthropic provider wrapper) are no longer part of the default base. Normal updates remove only unchanged entries originally added by SpecPi; pre-existing or modified entries and downloaded bytes remain. `--skip-package-install` preserves the old base. Restart Pi and Chat connections to unload retired extensions. Independently retained installations remain trusted upstream code; retained Pi Lens can still apply configured formatting/autofixes. Web access and usage reporting can contact services and use credentials through their upstream implementations. SpecPi's local-only wishlist collection policy does not describe all activity of those packages.
|
|
90
92
|
|
|
91
93
|
Top-level versions are pinned; upstream transitive dependency ranges are not frozen by SpecPi. `doctor` reads configured pins and installed package metadata, then invokes the installed Browser QA bin for real offline rendering, pixel-comparison, and accessibility checks when the managed base includes it. Doctor never downloads a browser, and missing Chromium or OS libraries fail with recovery guidance. It does not validate provider access or browser readiness for independently retained BetterWright. Core-only installations do not run browser checks. `check:base` acquires the packages, checks combined resource loading and Chat RPC startup, and exercises upstream approval, denial, and cancellation with synthetic context. It uses temporary home/configuration directories and sends no model prompt. The base check also exercises Node-only Chromium setup and offline Browser QA readiness; authenticated services, OS isolation, and every upstream tool's behavior remain outside that check. See [THIRD_PARTY.md](THIRD_PARTY.md) for sources and compatibility limits.
|
|
92
94
|
|
package/THIRD_PARTY.md
CHANGED
|
@@ -11,7 +11,7 @@ Development formatting uses Prettier **3.9.6** (MIT), ESLint **10.9.1** (MIT), `
|
|
|
11
11
|
|
|
12
12
|
## Default packages
|
|
13
13
|
|
|
14
|
-
Reviewed on 2026-09-16 against published npm metadata and integrity-verified source archives. All eight top-level packages declare the MIT license. Pins are authoritative in `templates/settings.json`.
|
|
14
|
+
Reviewed on 2026-09-16 against published npm metadata and integrity-verified source archives, and `specpi-jev-guard` 0.4.0 on 2026-09-20. All eight top-level packages declare the MIT license. Pins are authoritative in `templates/settings.json`.
|
|
15
15
|
|
|
16
16
|
SpecPi requests exact npm dependency saves for these pins and checks installed top-level versions before completing installation. The override applies to package acquisition without changing the user's global npm configuration.
|
|
17
17
|
|
|
@@ -24,16 +24,17 @@ SpecPi requests exact npm dependency saves for these pins and checks installed t
|
|
|
24
24
|
| pi-goal-x | 0.31.2 | [tmonk/pi-goal-x](https://github.com/tmonk/pi-goal-x) |
|
|
25
25
|
| @sreetej510/pi-usage | 0.10.0 | [Sreetej510/pi-extensions](https://github.com/Sreetej510/pi-extensions) |
|
|
26
26
|
| @gotgenes/pi-permission-system | 32.0.2 | [gotgenes/pi-packages](https://github.com/gotgenes/pi-packages) |
|
|
27
|
+
| specpi-jev-guard | 0.4.0 | [TannerMidd/specpi-jev-guard](https://github.com/TannerMidd/specpi-jev-guard) |
|
|
27
28
|
|
|
28
29
|
Transitive dependencies and their notices remain in Pi's npm installation tree. Top-level pins do not freeze upstream dependency ranges or constitute a full transitive security audit. Pi invokes npm with its upstream package-management semantics, including dependency lifecycle scripts. Browser QA has no install hook: confirmed SpecPi install/update explicitly invokes its installed Node setup bin, unless acquisition or browser setup is skipped. It downloads package-pinned Chromium without installing OS libraries. BetterWright is optional/manual and retains its own Bun-based setup; SpecPi neither removes Bun nor deletes user-owned tools. Usage reporting and web packages make their own provider/service connections. Consult upstream licenses and security policies before redistributing their components.
|
|
29
30
|
|
|
30
31
|
Pi Lens is no longer a default package. Normal updates retire only unchanged entries added by SpecPi; independent or modified entries and downloaded bytes remain, with their upstream notices. Restart Pi to unload Lens.
|
|
31
32
|
|
|
32
|
-
The combined base is tested with Pi 0.84.4. Pi Goal X declares Pi `>=0.83.0 <0.85.0`; compatibility with newer hosts is not assumed. SpecPi's former custom browser, structural-search, background-task, and command-guard implementations, DonSeTch, and Pi themes have been removed. Its delegation and experiment implementations were not discarded: they now ship as the independent `specpi-delegation` and `specpi-experiments` packages described below. Removal restores owned package settings but does not delete downloaded upstream packages or tools. Retired private runtimes remain in local backups with their notices.
|
|
33
|
+
The combined base is tested with Pi 0.84.4. Pi Goal X declares Pi `>=0.83.0 <0.85.0`; compatibility with newer hosts is not assumed. SpecPi's former custom browser, structural-search, background-task, and native command-guard implementations, DonSeTch, and Pi themes have been removed; command gating is the pinned `specpi-jev-guard` package's again. Its delegation and experiment implementations were not discarded: they now ship as the independent `specpi-delegation` and `specpi-experiments` packages described below. Removal restores owned package settings but does not delete downloaded upstream packages or tools. Retired private runtimes remain in local backups with their notices.
|
|
33
34
|
|
|
34
35
|
## Network services
|
|
35
36
|
|
|
36
|
-
The optional Jev
|
|
37
|
+
The optional Jev advisor and `specpi-jev-guard` both reach TypeSafe's System One endpoint. Neither is a bundled dependency: the advisor is first-party source that uses Node's built-in `fetch` and declares no package, and the guard is a pinned MIT package with no declared runtime dependencies. Both resolve their key the way every other Pi provider credential is resolved, in Pi's own documented order: the `openrouter` entry that `/login openrouter` writes to Pi's `auth.json`, then `OPENROUTER_API_KEY` from the environment (`TYPESAFE_API_KEY` when the direct API is selected — `JEV_BACKEND=typesafe` for the advisor, `/jev-guard backend typesafe` for the guard). Guard 0.3.0 is the release that added the credential-store half; before it the package read the environment alone. The two keys are not interchangeable: the other service rejects the wrong one with a bare 401. SpecPi never provisions or stores a provider credential, and reads one only to place it in the request header of the call the user has switched on.
|
|
37
38
|
|
|
38
39
|
Requests go to `openrouter.ai` by default, or to `api.typesafe.ai` on the direct backend, unless `TYPESAFE_BASE_URL` redirects them, which exists so tests never reach the network and the eval proxy can price advisor traffic. Retention and processing of what is sent are governed by TypeSafe and, on the default route, by OpenRouter as well -- not by SpecPi; the advisor ships off, asks before its first transmission and records a hash of every payload locally. The frozen eval price list carries `jev-1.13.0` at $0.042/MTok input and free output so a measured SpecPi + Jev row never reports a cost that excludes its own advisor.
|
|
39
40
|
|
|
@@ -41,7 +42,7 @@ Worth stating plainly, because it is the part a reader is most likely to assume
|
|
|
41
42
|
|
|
42
43
|
## Standalone delegation and experiments
|
|
43
44
|
|
|
44
|
-
`packages/delegation` and `packages/experiments` are first-party packages extracted from SpecPi's own retired harness code under MIT. Neither bundles third-party runtime code or declares a production dependency; every runtime import is a Node builtin or a Pi-supplied optional peer. Experiments invokes the user's own `git` as an external program through Pi's `exec` seam; Git is not bundled, vendored or version-pinned. Delegation's extraction drops the Command Guard admission path, which
|
|
45
|
+
`packages/delegation` and `packages/experiments` are first-party packages extracted from SpecPi's own retired harness code under MIT. Neither bundles third-party runtime code or declares a production dependency; every runtime import is a Node builtin or a Pi-supplied optional peer. Experiments invokes the user's own `git` as an external program through Pi's `exec` seam; Git is not bundled, vendored or version-pinned. Delegation's extraction drops the Command Guard admission path, which it does not integrate with, and reports its guard posture as `absent`. See their dependency notices and security boundaries: [delegation](packages/delegation/THIRD_PARTY.md) / [boundary](packages/delegation/SECURITY.md), [experiments](packages/experiments/THIRD_PARTY.md) / [boundary](packages/experiments/SECURITY.md).
|
|
45
46
|
|
|
46
47
|
## Standalone browser QA
|
|
47
48
|
|
|
@@ -51,4 +52,4 @@ The separately released `packages/browser-qa` source reuses the retired QA imple
|
|
|
51
52
|
|
|
52
53
|
The website uses the bundled IBM Plex Sans and Plex Mono fonts under the SIL Open Font License 1.1; see [`site/fonts/LICENSE.txt`](https://github.com/TannerMidd/SpecPi/blob/main/site/fonts/LICENSE.txt). The website's scripts and diagrams are first-party code and assets. The README loads public package/license badges from Shields.io and build status from GitHub. Website fonts and media are not included in the npm package.
|
|
53
54
|
|
|
54
|
-
SpecPi Chat 0.
|
|
55
|
+
SpecPi Chat 0.14.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.
|
|
@@ -6,8 +6,8 @@
|
|
|
6
6
|
// inside sanitize.
|
|
7
7
|
//
|
|
8
8
|
// AWAIT ONLY THE SYSTEMS THAT MUTATE WHAT THEY INSPECT. Retention must be awaited, because its
|
|
9
|
-
// answer replaces the tool result it was asked about; so must
|
|
10
|
-
//
|
|
9
|
+
// answer replaces the tool result it was asked about; so must the two tool_call systems, which
|
|
10
|
+
// return a patch or edit `event.input` in place. A system that acts on
|
|
11
11
|
// a later turn must not be awaited: at roughly 300ms a call, a turn-level system firing thirty
|
|
12
12
|
// times would add nine seconds to an attempt that takes a hundred and thirty, to deliver advice
|
|
13
13
|
// that was never going to change the turn it was asked during.
|
|
@@ -22,13 +22,11 @@ import { writeUsage } from "./usage.mjs";
|
|
|
22
22
|
|
|
23
23
|
export const SYSTEM_LABELS = Object.freeze({
|
|
24
24
|
retention: "Tool-result retention",
|
|
25
|
-
compaction: "Compaction guidance",
|
|
26
25
|
gap: "Capability-gap triage",
|
|
27
26
|
sources: "Delegation source ranking",
|
|
28
27
|
progress: "Progress and thrash detection",
|
|
29
28
|
untrusted: "Untrusted-content classification",
|
|
30
29
|
capability: "Turn-zero capability arming",
|
|
31
|
-
guard: "Command guard",
|
|
32
30
|
});
|
|
33
31
|
|
|
34
32
|
export function createBroker(options = {}) {
|