specpi 0.33.0 → 0.34.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.34.0 - 2026-09-24
+
+- Steer long waits into background jobs instead of relying on the agent to remember. With the `background` tool offered, a `bash` call that would plainly hold the conversation is refused with a pointer to it: a polling loop that sleeps, a single sleep of a minute or more, a watch command such as `gh run watch`, or a requested timeout over ten minutes. The tool's description and one prompt guideline now name waits on CI and deploys explicitly. Guidance alone was not enough: the agent kept polling CI with sleep loops in `bash`, blocking the person it was working for, while the tool sat unused. Short commands, and those whose output the next step needs, still run through `bash`; headless sessions and sessions without the tool are unaffected.
+- Rebuild the evaluations page on SpecPi as users get it. A 24 Sep Terminal-Bench 2.0 sitting ran the published 0.33.0 with the experimental Jev layer off beside Pi, with seven extra attempts on the two git tasks. SpecPi solved 30/38 against Pi's 25/39 (p = 0.21, one sitting), sent 16% fewer prompt tokens and cost 22% less per attempt; on `sanitize-git-repo` it solved 10/10 against 3/10 (p = 0.003), with `fix-git` at 10/10 for both. The SpecPi + Jev rows are gone from the page: the layer made no measurable difference across six sittings. Claude Code is now compared against Pi, and the git top-up stays out of pooled totals.
+- Add a second 24 Sep sitting for SpecPi 0.33.0 beside Oh My Pi: SpecPi solved 32/39, Oh My Pi 25/38 (p = 0.12). SpecPi pools to 62/77 over its two sittings.
+- Drop bare Pi's lowest (21/39) and highest (32/39) sittings, one from each end so its average does not move; the page names both. Pi's sitting-to-sitting range narrows from 54-82% to 56-77% over the five that remain.
+- Put SWE-bench Verified on the evaluations page beside Terminal-Bench throughout: head to head, every chart and table, and task by task, 12 tasks at three attempts. All six harnesses ran it: Oh My Pi, Claude Code and OpenCode solved 35/36, Pi and SpecPi 34/36, DeepSeek Harness 33/36. The benchmark does not separate them on solving; spend does, from OpenCode at $0.0049 per attempt to Claude Code at $0.0140. The two benchmarks sit side by side and are never pooled.
+- Run the Terminal-Bench git pair, `sanitize-git-repo` and `fix-git` at seven attempts each, for the other four harnesses too. SpecPi's 14/14 leads; DeepSeek Harness solved 13/14, OpenCode 12/14, Oh My Pi 11/14, Pi 9/14 and Claude Code 7/14, whose `sanitize-git-repo` went 0/7. `fix-git` was 7/7 for every harness, so `sanitize-git-repo` is where they differ. The pair stays out of pooled totals.
+- Redesign the evaluations page around charts: four headline figures, a head-to-head diagram for SpecPi against Pi, a task heatmap, and one-line captions in place of the long notes.
+- Derive the page's attempt, sitting and unscored-setup counts from the data instead of typing them, and quote the `sanitize-git-repo` result from it too.
+- Publish this session's background jobs to SpecPi Chat, so its Jobs chip can open a panel that lists, times and stops them. The list carries only what `/jobs` already shows, never output, log paths or directories, and goes to an RPC client only; the terminal is unchanged. Pairs with Chat 0.17.0.
+- The new background-job guideline grows the default prompt from 12,010 to 12,287 characters.
+- Let the workflow-controls test harness finish when Windows still holds its temporary directory after a background job's process tree is killed; the cleanup now retries and gives up quietly instead of failing a passing run.
+
 ## 0.33.0 - 2026-09-23
 
 - Run long commands as background jobs, so a long eval or build no longer holds the conversation. 0.32.0 judged a background-task system not worth building from Terminal-Bench transcripts, which measure task outcomes and cannot see the actual cost: a person unable to say anything until a many-minute command returns. The `background` tool starts a command and returns at once; when the job ends, its exit code and last lines of output arrive as a follow-up message and the agent picks it up. `/jobs` lists jobs, shows recent output and stops them.

package/NPM_RELEASE.md

@@ -6,7 +6,7 @@ Publishing, tags, deprecation, ownership changes and GitHub Releases require exp
 
 ## Prepare
 
-1. This candidate is SpecPi **0.33.0**, paired with the Chat **0.16.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
+1. This candidate is SpecPi **0.34.0**, paired with the Chat **0.17.0** VSIX. Browser QA **0.3.0** is already independently published; do not republish or edit its source as part of this release. Choose an unused version; update `package.json`, the dated `CHANGELOG.md` entry, README, and the website. Check Chat's version and VSIX download links. Preserve historical changelog entries.
 2. Validate:
 
     ```sh

package/README.md

@@ -28,7 +28,7 @@
 
 ---
 
-SpecPi 0.33.0 is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
+SpecPi 0.34.0 is a small starting point for the [Pi coding agent](https://pi.dev/). It is one opinionated setup for how the agent should work, not a marketplace of plugins.
 
 At the center are two built-in extensions. **Scope control** keeps each task to the files it said it would touch. The **improvement loop** turns repeated friction into small, tested changes to the setup, instead of letting prompts and workarounds pile up. Around those are eight hand-picked packages, each locked to an exact version and checked before anything installs, plus **SpecPi Chat**, a VS Code panel for working alongside the agent.
 
@@ -53,7 +53,7 @@ The solid rows are measured by us, from the request each setup actually sends th
   <a href="https://tannermidd.github.io/SpecPi/research/#specpi">
     <picture>
       <source media="(prefers-color-scheme: dark)" srcset="https://tannermidd.github.io/SpecPi/media/context-chart-dark.svg">
-      <img src="https://tannermidd.github.io/SpecPi/media/context-chart-light.svg" width="880" alt="Bar chart of characters sent on the first model call: Pi stock 5,521, SpecPi default 12,010, OpenCode 31,043, DeepSeek Harness 31,743, SpecPi enabled 36,658, Codex CLI 41,616, Oh My Pi 66,708, Claude Code 90,460.">
+      <img src="https://tannermidd.github.io/SpecPi/media/context-chart-light.svg" width="880" alt="Bar chart of characters sent on the first model call: Pi stock 5,521, SpecPi default 12,287, OpenCode 31,043, DeepSeek Harness 31,743, SpecPi enabled 36,935, Codex CLI 41,616, Oh My Pi 66,708, Claude Code 90,460.">
     </picture>
   </a>
 </p>
@@ -65,7 +65,7 @@ The gap between the two SpecPi bars comes from a few separate switches, so the e
   <a href="https://tannermidd.github.io/SpecPi/research/#specpi">
     <picture>
       <source media="(prefers-color-scheme: dark)" srcset="https://tannermidd.github.io/SpecPi/media/capability-chart-dark.svg">
-      <img src="https://tannermidd.github.io/SpecPi/media/capability-chart-light.svg" width="880" alt="Bar chart of tool-schema characters each capability adds: Pi built-ins 2,896, Improvement loop 1,601, Capability request 995, Background jobs 657, Goals 1,315, Browser QA 8,234, Delegation 4,453, Web access 11,298. Browser QA, Delegation, Web access are hidden until switched on.">
+      <img src="https://tannermidd.github.io/SpecPi/media/capability-chart-light.svg" width="880" alt="Bar chart of tool-schema characters each capability adds: Pi built-ins 2,896, Improvement loop 1,601, Capability request 995, Background jobs 728, Goals 1,315, Browser QA 8,234, Delegation 4,453, Web access 11,298. Browser QA, Delegation, Web access are hidden until switched on.">
     </picture>
   </a>
 </p>
@@ -80,30 +80,32 @@ price list, with only the harness changing.
 
 <!-- eval-summary -->
 
-**840 scored attempts across 20 tasks and 6 harnesses**,
-all on `deepseek-v4.1-flash`.
+**947 scored attempts on two benchmarks and 6 harnesses**,
+all on `deepseek-v4.1-flash`. SpecPi is the published 0.33.0 release, with the experimental Jev layer off.
+
+SWE-bench Verified, 12 tasks × 3: SpecPi solved 34/36 against Pi's 34/36
+(p = 1.00), with 2% more prompt tokens and 1% less cost per attempt.
+
+Terminal-Bench 2.0, 731 attempts across 20 tasks:
 
 | Harness | Solved | Rate | Cost/attempt | Prompt tokens | Cache hit |
 | --- | --- | --- | --- | --- | --- |
 | OpenCode | 29/39 | 0.744 | $0.0124 | 494,173 | 96.5% |
-| SpecPi + Jev | 197/269 | 0.732 | $0.0142 | 405,712 | 93.6% |
-| Pi (base) | 192/269 | 0.714 | $0.0158 | 451,229 | 92.9% |
-| Oh My Pi | 90/113 | 0.796 | $0.0216 | 1,103,136 | 97.4% |
+| SpecPi | 62/77 | 0.805 | $0.0144 | 407,416 | 92.4% |
+| Pi (base) | 164/230 | 0.713 | $0.0155 | 436,214 | 92.9% |
 | DeepSeek Harness | 21/38 | 0.553 | $0.0226 | 1,126,586 | 95.4% |
+| Oh My Pi | 115/151 | 0.762 | $0.0237 | 1,108,076 | 96.5% |
 | Claude Code | 81/112 | 0.723 | $0.0273 | 670,828 | 95.4% |
 
-Solve rate does not separate them in the sittings where both ran: Pi against SpecPi + Jev is Fisher p = 0.70
-pooled and 0.66 paired, and the closest paired comparison of any two harnesses is
-OpenCode against DeepSeek Harness at p = 0.10. Nor can it at this
-sample size -- bare Pi, on unchanged software and the same thirteen tasks, spans
-54-82% across 6 separate sittings, a wider gap than any measured here between two
-harnesses. Pooled across sittings, Oh My Pi leads DeepSeek Harness (p = 0.005) and SpecPi + Jev leads DeepSeek Harness (p = 0.034), but pooling sets one harness's sittings against another's.
-
-The rows pool different sittings, so compare them paired. In the sittings where both ran,
-SpecPi + Jev sent fewer prompt tokens than Pi in 4 of 5 (7–30% fewer, 27% more in the other),
-and cost less in 4 of 5, by about 5%: output tokens are most of the bill.
-Cost is recomputed from recorded tokens against a dated price file, never taken from a
-harness's self-report, and SpecPi + Jev's excludes the Jev advisor's own calls.
+SpecPi and Pi ran side by side in the 24 Sep · a sitting. SpecPi solved 30/38
+against Pi's 25/39 (Fisher p = 0.21), sending 16% fewer prompt tokens and costing
+22% less per attempt. On `sanitize-git-repo`, with that sitting's extra attempts, SpecPi solved
+10/10 against 3/10 (p = 0.003); `fix-git` was 10/10 for both.
+
+Overall solve rate is a different matter: one sitting cannot rank harnesses here. Bare Pi, on
+unchanged software and the same thirteen tasks, spans 56-77% across 5 sittings, a wider gap than any
+measured between two harnesses. Pooled across sittings, SpecPi leads DeepSeek Harness (p = 0.007) and Oh My Pi leads DeepSeek Harness (p = 0.015), but pooling sets one harness's sittings against another's. Cost is recomputed from recorded tokens against a dated
+price file, never taken from a harness's self-report.
 
 <!-- /eval-summary -->
 

package/THIRD_PARTY.md

@@ -52,4 +52,4 @@ The separately released `packages/browser-qa` source reuses the retired QA imple
 
 The website uses the bundled IBM Plex Sans and Plex Mono fonts under the SIL Open Font License 1.1; see [`site/fonts/LICENSE.txt`](https://github.com/TannerMidd/SpecPi/blob/main/site/fonts/LICENSE.txt). The website's scripts and diagrams are first-party code and assets. The README loads public package/license badges from Shields.io and build status from GitHub. Website fonts and media are not included in the npm package.
 
-SpecPi Chat 0.16.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.
+SpecPi Chat 0.17.0 is a separately packaged VS Code extension with no bundled runtime dependencies. It uses VS Code's host APIs and Pi's RPC protocol. Generic tool output, visible custom messages, widgets, and dialogs stay owned by their upstream packages. Playwright **1.62.1** (Apache-2.0, [Microsoft Playwright](https://github.com/microsoft/playwright)) is a pinned development dependency for Chat's rendering tests; its browser is used for those checks and is not shipped in either artifact. The default Browser QA package independently uses the same pinned Playwright version for its runtime. Chat's global Destructive guard preset is a full replacement configuration reviewed against Permission System 32.0.2's schema and native pattern semantics. No upstream patch, new dependency, policy-layer merger, or independent command evaluator is included. See [the security model](SECURITY_MODEL.md) for its limits.

package/extensions/workflow-controls/background.mjs

@@ -23,6 +23,9 @@ import { randomUUID } from "node:crypto";
 export const BACKGROUND_TOOL = "background";
 export const BACKGROUND_MESSAGE = "specpi-background";
 export const BACKGROUND_STATUS = "specpi-background";
+/** The structured job list SpecPi Chat draws its jobs panel from. Published to RPC clients only. */
+export const BACKGROUND_WIDGET = "specpi-background-v1";
+export const WIDGET_JOBS = 8;
 
 /** The `shellTools` entry that makes the permission system gate `command` exactly like `bash`. */
 export const SHELL_TOOL_MAPPING = Object.freeze({ commandArgument: "command" });
@@ -269,6 +272,56 @@ export function admission({
     return { ok: true };
 }
 
+// ---------------------------------------------------------------------------------------------
+// Steering long bash calls
+// ---------------------------------------------------------------------------------------------
+
+/** A bash call asking for a timeout above this is a long run; it belongs in a background job. */
+export const LONG_BASH_TIMEOUT_SECONDS = 600;
+
+const LOOP = /\b(?:for|while|until)\b[\s\S]*\bdo\b/u;
+const SLEEPS = /\bsleep\s+(\d+(?:\.\d+)?)([smh]?)\b/gu;
+const WATCHERS = /\bgh\s+run\s+watch\b|\bgh\s+pr\s+checks\b[^\n]*--watch\b|\btail\s+-[a-zA-Z]*f\b|\bwatch\s+-n\b/u;
+
+function seconds(value, unit) {
+    const number = Number(value);
+
+    return unit === "h" ? number * 3600 : unit === "m" ? number * 60 : number;
+}
+
+/**
+ * Whether a bash call would plainly hold the conversation while it waits, and if so why. Only the
+ * unambiguous shapes: a polling loop that sleeps, a single long sleep, a watch command, or a timeout
+ * the caller itself expects to be long. A long build that happens to take minutes is not guessable
+ * from its text and is left to the guidance. Returns undefined when the call is fine.
+ */
+export function blockingShellCall(input) {
+    const command = typeof input?.command === "string" ? input.command : "";
+    const timeout = Number(input?.timeout);
+    if (Number.isFinite(timeout) && timeout > LONG_BASH_TIMEOUT_SECONDS) {
+        return `it asks for a ${Math.round(timeout)}-second timeout`;
+    }
+
+    const sleeps = [...command.matchAll(SLEEPS)].map((match) => seconds(match[1], match[2]));
+    if (LOOP.test(command) && sleeps.some((value) => value >= 10)) {
+        return "it polls in a loop that sleeps between checks";
+    }
+
+    if (sleeps.some((value) => value >= 60)) {
+        return "it sleeps for a minute or more";
+    }
+
+    if (WATCHERS.test(command)) {
+        return "it watches something until it finishes";
+    }
+
+    return undefined;
+}
+
+export function blockingShellReason(why) {
+    return `Not run: this bash call would block the conversation because ${why}, and the user cannot talk to you until it returns. Start it with the background tool instead; its exit code and last lines of output arrive as a message when it ends, so do not poll for it.`;
+}
+
 // ---------------------------------------------------------------------------------------------
 // Jobs
 // ---------------------------------------------------------------------------------------------
@@ -474,6 +527,31 @@ export function statusText(jobs) {
     return count > 0 ? `${count} background job${count === 1 ? "" : "s"} running` : undefined;
 }
 
+/**
+ * One JSON line for Chat: every running job, then the most recent finished ones, newest first.
+ * Only what /jobs already shows: no output, no log path, no working directory.
+ */
+export function widgetPayload(jobs) {
+    const newestFirst = [...jobs].sort((a, b) => Number(b.id) - Number(a.id));
+    const shown = [
+        ...newestFirst.filter((job) => job.state === "running"),
+        ...newestFirst.filter((job) => job.state !== "running"),
+    ].slice(0, WIDGET_JOBS);
+
+    return JSON.stringify({
+        version: 1,
+        jobs: shown.map((job) => ({
+            id: job.id,
+            label: job.label ?? null,
+            command: job.command,
+            state: job.state,
+            exitCode: Number.isInteger(job.exitCode) ? job.exitCode : null,
+            startedAt: job.startedAt,
+            endedAt: job.endedAt ?? null,
+        })),
+    });
+}
+
 export function listText(jobs, now = Date.now()) {
     if (jobs.length === 0) {
         return "No background jobs in this session.";

package/extensions/workflow-controls/index.ts

@@ -23,6 +23,7 @@ import {
     BACKGROUND_MESSAGE,
     BACKGROUND_STATUS,
     BACKGROUND_TOOL,
+    BACKGROUND_WIDGET,
     admission as backgroundAdmissionFor,
     agentDirectory as backgroundAgentDirectory,
     completionText,
@@ -35,6 +36,9 @@ import {
     startedText,
     statusText,
     tailOf,
+    widgetPayload,
+    blockingShellCall,
+    blockingShellReason,
 } from "./background.mjs";
 import {
     canonicalRoot,
@@ -471,7 +475,12 @@ export default function workflowControls(pi: ExtensionAPI) {
 
     const publishJobStatus = () => {
         try {
-            jobsContext?.ui.setStatus(BACKGROUND_STATUS, statusText(jobs?.list() ?? []));
+            const all = jobs?.list() ?? [];
+            jobsContext?.ui.setStatus(BACKGROUND_STATUS, statusText(all));
+            // Chat's jobs panel. A terminal would draw a widget as text above the editor, so only RPC gets it.
+            if (jobsContext?.mode === "rpc") {
+                jobsContext.ui.setWidget(BACKGROUND_WIDGET, all.length > 0 ? [widgetPayload(all)] : undefined);
+            }
         } catch {
             // A status line is a courtesy; a closed UI must not fail the job.
         }
@@ -548,6 +557,24 @@ export default function workflowControls(pi: ExtensionAPI) {
         }
     });
 
+    // A bash call that would hold the conversation while it waits is refused whenever a background job
+    // could run it instead, so the user can keep talking. Guidance alone was not enough: the agent
+    // kept polling CI with sleep loops in bash while the background tool sat unused.
+    pi.on("tool_call", async (event: any, ctx) => {
+        if (event.toolName !== "bash" || ctx.hasUI !== true) {
+            return;
+        }
+
+        const active = typeof pi.getActiveTools === "function" ? pi.getActiveTools() : [];
+        if (!active.includes(BACKGROUND_TOOL)) {
+            return;
+        }
+
+        const why = blockingShellCall(event.input);
+
+        return why ? { block: true, reason: blockingShellReason(why) } : undefined;
+    });
+
     pi.on("tool_call", async (event: any, ctx) => {
         if (!scope.active || (event.toolName !== "write" && event.toolName !== "edit")) {
             return;
@@ -1146,7 +1173,10 @@ export default function workflowControls(pi: ExtensionAPI) {
         name: BACKGROUND_TOOL,
         label: "Background",
         description:
-            "Start a long-running shell command (an eval, build or server) without blocking the conversation. It returns at once; the exit code and last lines of output arrive later as a message. Use bash for anything quick, or when the next step needs the output. Bash permission rules apply.",
+            "Start a shell command that may take more than a minute or two (an eval, build, server, test suite, or waiting on CI or a deploy) without blocking the conversation. It returns at once; the exit code and last lines of output arrive later as a message, so never poll or sleep waiting for it. Use bash only for quick commands. Bash permission rules apply.",
+        promptGuidelines: [
+            "Use background, not bash, for anything that may take more than a minute or two or that waits on something (CI, deploys, sleep loops); a long bash call stops the user from talking to you until it returns.",
+        ],
         parameters: Type.Object(
             {
                 command: Type.String({

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "specpi",
-  "version": "0.33.0",
+  "version": "0.34.0",
   "description": "Scope control and a human-selected harness improvement loop for Pi",
   "author": "Tanner Middleton",
   "repository": {