pi-background-tasks 0.7.4 → 0.7.6

package/PUBLISHING.md

@@ -1,6 +1,6 @@
 # Publishing pi-background-tasks
 
-Release checklist for npm publishing and standalone git publishing. The current release candidate is 0.7.4; version 0.7.0 introduced the Fusion public surfaces (`/fusion`, `/fusion-models`, `fusion_brainstorm`) in addition to the background-task surfaces. Do not advertise the GitHub install target until the standalone repository has the exact release commit and tag.
+Release checklist for npm publishing and standalone git publishing. The current release candidate is 0.7.6; version 0.7.0 introduced the Fusion public surfaces (`/fusion`, `/fusion-models`, `fusion_brainstorm`) in addition to the background-task surfaces. Do not advertise the GitHub install target until the standalone repository has the exact release commit and tag.
 
 ## Preconditions
 
@@ -37,8 +37,8 @@ npm publish --access public
 Pi install smoke after publish:
 
 ```bash
-PI_CODING_AGENT_DIR=$(mktemp -d) pi -e npm:pi-background-tasks@0.7.4 --offline --no-tools --no-session -p "/jobs"
-pi install npm:pi-background-tasks@0.7.4
+PI_CODING_AGENT_DIR=$(mktemp -d) pi -e npm:pi-background-tasks@0.7.6 --offline --no-tools --no-session -p "/jobs"
+pi install npm:pi-background-tasks@0.7.6
 ```
 
 ## Publish to git
@@ -51,15 +51,15 @@ git status --short --branch
 git log --oneline -3
 git remote -v
 git push origin main
-git tag v0.7.4
-git push origin v0.7.4
+git tag v0.7.6
+git push origin v0.7.6
 ```
 
 Pi install smoke after git tag, using an isolated Pi agent directory so no local checkout or user `~/.pi` state is involved:
 
 ```bash
-PI_CODING_AGENT_DIR=$(mktemp -d) pi -e git:github.com/ismailsaleekh/pi-background-tasks@v0.7.4 --offline --no-tools --no-session -p "/jobs"
-pi install git:github.com/ismailsaleekh/pi-background-tasks@v0.7.4
+PI_CODING_AGENT_DIR=$(mktemp -d) pi -e git:github.com/ismailsaleekh/pi-background-tasks@v0.7.6 --offline --no-tools --no-session -p "/jobs"
+pi install git:github.com/ismailsaleekh/pi-background-tasks@v0.7.6
 ```
 
 ## pi.dev/packages

package/README.md

@@ -9,19 +9,19 @@ This package adds named, tracked background shell jobs with durable output files
 From npm after publish:
 
 ```bash
-pi install npm:pi-background-tasks@0.7.4
+pi install npm:pi-background-tasks@0.7.6
 ```
 
 From git after pushing this package to its standalone repository and tagging:
 
 ```bash
-pi install git:github.com/ismailsaleekh/pi-background-tasks@v0.7.4
+pi install git:github.com/ismailsaleekh/pi-background-tasks@v0.7.6
 ```
 
 For project-local install:
 
 ```bash
-pi install -l npm:pi-background-tasks@0.7.4
+pi install -l npm:pi-background-tasks@0.7.6
 ```
 
 ## Commands
@@ -122,7 +122,7 @@ Progress is surfaced through `fusion` status updates, TUI cancellable loader UI
 
 ### Conversation context policy
 
-Fusion children receive a **versioned conversation projection**, not a raw execution transcript. The canonical input schema is `pi-background-tasks.fusion-input.v2` and every run states exactly what was included and what was omitted.
+Fusion children receive a **versioned conversation projection**, not a raw execution transcript. The canonical input schema is `pi-background-tasks.fusion-input.v3` and every run states exactly what was included and what was omitted.
 
 The projection transform (`visible-conversation-ledger-v1`) is shared by both entry points:
 
@@ -137,7 +137,13 @@ The projection transform (`visible-conversation-ledger-v1`) is shared by both en
 | Tool-result images | excluded; recorded as an omission receipt (never raw bytes) |
 | Active `fusion_brainstorm` call and its sibling calls | scope-excluded from the branch |
 
-Omissions are **explicit, deterministic, and auditable** — never silent. Each omitted event produces a ledger row with its kind, exact byte count, and SHA-256 of the omitted bytes; contiguous omissions collapse into source-ordered `omitted_activity` receipts carrying counts, byte totals, and a run hash. The complete ledger is persisted as `context-omission-ledger.json`. **No head, tail, or preview of an omitted payload is ever forwarded** (`tool_payload_preview_bytes` is `0`), because an arbitrary prefix is usually irrelevant and can leak secrets or carry tool-output prompt injection. Repeated construction is byte-identical, so hashes are stable.
+Omissions are **explicit, deterministic, and auditable** — never silent. Each omitted event produces a ledger row with its kind, exact byte count, and SHA-256 of the omitted bytes. Contiguous omissions collapse into source-ordered `omitted_activity` receipts. Each receipt is deliberately compact, carrying only what a reading model can act on:
+
+```json
+{"at":[1,9],"bytes":29019,"counts":{"tool_calls":5,"tool_result_texts":5},"kind":"omitted_activity"}
+```
+
+`at` is the inclusive source-ordinal span, `bytes` is the total omitted non-image payload for that run, and a zero-valued count is absent rather than serialized. Per-event hashes, ledger indices, and the per-kind byte map live in `context-omission-ledger.json`, not in the prompt: a child cannot verify a hash of payload it does not hold, so forwarding one only consumed context. That ledger also carries a `projection_map` proving every ledger row is represented by exactly one receipt or image marker, and `accounting.omission_receipt_utf8_bytes` records the exact receipt cost. The complete ledger is persisted as `context-omission-ledger.json`. **No head, tail, or preview of an omitted payload is ever forwarded** (`tool_payload_preview_bytes` is `0`), because an arbitrary prefix is usually irrelevant and can leak secrets or carry tool-output prompt injection. Repeated construction is byte-identical, so hashes are stable.
 
 Two entry points share the transform but differ in request authority:
 
@@ -150,16 +156,30 @@ Two entry points share the transform but differ in request authority:
 
 ### Stage budgets
 
-Every prompt-expansion stage — candidate, evaluator, evaluation repair, and merger — is size-checked **before any child process is created**. Safety is based on the **smallest** configured model's context window, never the largest, so adding one small-context slot cannot be masked by large-context siblings.
+Every prompt-expansion stage — candidate, evaluator, evaluation repair, and merger — is size-checked **before any child process is created**, and each stage is checked against **its own configured route**, so a large-context slot cannot hide a small-context sibling and a small slot cannot veto stages it never serves.
 
 The input token bound is `ceil(utf8Bytes / 2)`. This is a ceiling, not an estimate: across 159 real large Fusion prompts the densest observed ratio was 3.552 bytes per input token, so the divisor keeps roughly a 1.7x margin and also bounds dense non-ASCII input.
 
-Downstream growth is guaranteed by two cooperating layers rather than by an assumption:
+Each stage is forecast with its **real prompt builder**, rendered with empty embedded-output slots, plus the enforced output contracts for whatever that stage will embed:
+
+| Stage | Embeds | Mandatory |
+|---|---|---|
+| `candidate` | canonical input only | yes, once per slot |
+| `evaluation` | + 3 candidate answers | yes |
+| `evaluation_repair` | + 3 candidates, invalid evaluator output, diagnostics | conditional, still budgeted |
+| `merge` | + 3 candidates, validated evaluation | yes |
+
+`evaluation_repair` only runs when the evaluator returns schema-invalid JSON, but it is budgeted unconditionally: "uncommon" is not a reliability contract.
+
+Output sizes are guaranteed by **enforced contracts**, not assumptions. Each stage has a maximum response size measured in *JSON-rendered transfer bytes* — the bytes the response actually costs once a later stage embeds it, escaping included (candidate 48 KiB, evaluator 64 KiB, merger 64 KiB, repair diagnostics 8 KiB). Measuring the rendered form removes any escaping guess: quotes, backslashes, and newlines expand 2x and control characters up to 6x. A response over its contract is a loud `child_output_cap` failure; it is preserved in the run artifacts and is never sliced, truncated, or forwarded.
+
+When a workflow cannot fit, the error names the **first failing mandatory stage** in deterministic order (candidate slots, then evaluation, then merge), lists every other blocking stage, and labels conditional ones. It never blames a stage that would have succeeded.
+
+Remediation is derived, not guessed: Fusion re-plans the entire workflow **with the request removed**. If it still fails, the error says plainly that shortening the request cannot help and points at starting a fresh conversation or raising the route's context window. If it then fits, the request is what determines feasibility, and the error states the exact minimum byte reduction and the maximum safe request size.
 
-1. **Enforced output contracts.** Each stage has a maximum response size measured in *JSON-rendered transfer bytes* — the bytes the response actually costs once a later stage embeds it, escaping included (candidate 48 KiB, evaluator 64 KiB, merger 64 KiB, repair diagnostics 8 KiB). Measuring the rendered form removes any escaping guess: quotes, backslashes, and newlines expand 2x and control characters up to 6x, so a raw-byte contract would not bound the embedded size. A response over its contract is a loud `child_output_cap` failure; it is preserved in the run artifacts and is never sliced, truncated, or forwarded.
-2. **An exact reserve.** The canonical input must leave room for the widest stage — the evaluation repair, which embeds three candidate answers, the invalid evaluator output, and diagnostics — plus fixed wrapper overhead. Because the contracts above are already rendered-byte bounds, this is an exact sum, not a raw size inflated by an estimated factor. The reserve is converted to tokens with the *same* `ceil(bytes / 2)` function used to measure prompts; reserving output *tokens* directly would understate the cost of re-embedding those bytes.
+Runs that do fit still emit an advisory utilization warning for any stage at or above 80% of its allowance, recorded in `budget-plan.json` and as a progress event, so a session approaching the ceiling is visible before it hard-fails. The warning never alters behaviour.
 
-Each route additionally reserves its output contract, 4,096 framing, and 4,096 safety tokens. Because every step is uniformly conservative, the policy requires roughly a **168,000-token context window per configured slot**. Smaller routes are rejected at configuration time with an actionable error naming the requirement, rather than being accepted and failing later at the provider. Route capacities and the pre-candidate feasibility decision are persisted as `budget-plan.json`.
+Every configured route must also satisfy a documented minimum capacity; smaller routes are rejected at configuration time with an actionable error naming the requirement, rather than being accepted and failing later at the provider. All route capacities, per-stage forecasts, headroom, utilization, the byte composition of the blocking stage, and the blockers list are persisted as `budget-plan.json`.
 
 If an input still exceeds the safe budget, Fusion fails with a typed `prompt_budget_exceeded` error naming the stage, measured bytes, measured token upper bound, allowed tokens, the limiting configured model and its context window, and concrete remediation. **Zero children are launched** when preflight rejects. Provider context-window failures remain loud child failures; there is no hidden local truncation and no silent fallback anywhere in this path.
 

package/TESTING.md

@@ -86,7 +86,7 @@ npm run test:rpc
 npm run test:agent-loop
 ```
 
-The Fusion SDK/RPC/scripted-provider tests install a deterministic fake child `pi` in a temp `PATH` from `tests/helpers/fusion-fake-pi.ts`. Parent Pi remains the real SDK/RPC runtime; only direct child `pi --mode text` calls with the package-owned private compact metadata extension are intercepted. `PI_CODING_AGENT_DIR` is pointed at the temp agent directory so `fusion-models.json` is never read from the user's real global Pi directory. Fusion context coverage is conversation-projection coverage. `tests/unit/fusion-context-prompts.test.ts` verifies that a synthetic session carrying more than 1 MB of tool arguments/results still yields a small canonical input, that user and assistant text survive verbatim, that thinking and tool payloads never appear (including no head/tail/preview sentinel), that omission counts, byte totals, and hashes are exact and stable, that repeated construction is byte-identical, that the active `fusion_brainstorm` leaf and sibling calls stay scope-excluded, that images remain marker-only or ledger-only with no raw base64 in child prompts, and that every retained source block receives exactly one disposition. `tests/unit/fusion-budget.test.ts` covers stage budgets: the limiting model is the smallest configured route (including when it is the evaluator rather than a candidate), unknown or too-small capacities fail before spawn, boundary prompts pass at exactly the limit and fail one byte past it, the child system prompt counts as input, dense multi-byte UTF-8 cannot bypass byte accounting, and candidate, evaluator, evaluation-repair, and merger expansions are each rejected before their child is spawned with zero partial launches. `tests/package/typebox-compat.test.ts` pins the TypeBox posture and compiles nullable-array schemas. The release-only `npm run test:compat` packs the package, installs exact supported Pi versions, runs `/jobs`, runs `/fusion` through the installed package entrypoint with the fake child Pi, verifies five child invocations, verifies `/fusion-models` rejects non-TUI mode, asserts the resolved `typebox` is Pi's bundled peer rather than a private or nested copy, and scans the installed package bytes for TypeBox APIs removed in the 1.3.x line. It then drives the current host Pi through a real RPC `fusion_brainstorm` parent-agent loop, checks the persisted tool result carries the complete Pi `Usage.cost` object, invokes `get_session_stats` (the same aggregation boundary used by the TUI footer), reopens the durable session, and verifies identical token/cost totals. All parent and child inference remains deterministic and local.
+The Fusion SDK/RPC/scripted-provider tests install a deterministic fake child `pi` in a temp `PATH` from `tests/helpers/fusion-fake-pi.ts`. Parent Pi remains the real SDK/RPC runtime; only direct child `pi --mode text` calls with the package-owned private compact metadata extension are intercepted. `PI_CODING_AGENT_DIR` is pointed at the temp agent directory so `fusion-models.json` is never read from the user's real global Pi directory. Fusion context coverage is conversation-projection coverage. `tests/unit/fusion-context-prompts.test.ts` verifies that a synthetic session carrying more than 1 MB of tool arguments/results still yields a small canonical input, that user and assistant text survive verbatim, that thinking and tool payloads never appear (including no head/tail/preview sentinel), that omission counts, byte totals, and hashes are exact and stable, that repeated construction is byte-identical, that the active `fusion_brainstorm` leaf and sibling calls stay scope-excluded, that images remain marker-only or ledger-only with no raw base64 in child prompts, and that every retained source block receives exactly one disposition. `tests/unit/fusion-high-cardinality.test.ts` covers the receipt-cardinality regression: a session of many short interleaved tool events (rather than a few enormous results) pins per-receipt cost, proves the compact `omitted_activity` fields are the only model-facing ones, reconciles every receipt against the ledger through `projection_map`, and proves the whole workflow fits a real route budget. `tests/unit/fusion-budget.test.ts` covers stage budgets: the limiting model is the smallest configured route (including when it is the evaluator rather than a candidate), unknown or too-small capacities fail before spawn, boundary prompts pass at exactly the limit and fail one byte past it, the child system prompt counts as input, dense multi-byte UTF-8 cannot bypass byte accounting, and candidate, evaluator, evaluation-repair, and merger expansions are each rejected before their child is spawned with zero partial launches. `tests/package/typebox-compat.test.ts` pins the TypeBox posture and compiles nullable-array schemas. The release-only `npm run test:compat` packs the package, installs exact supported Pi versions, runs `/jobs`, runs `/fusion` through the installed package entrypoint with the fake child Pi, verifies five child invocations, verifies `/fusion-models` rejects non-TUI mode, asserts the resolved `typebox` is Pi's bundled peer rather than a private or nested copy, and scans the installed package bytes for TypeBox APIs removed in the 1.3.x line. It then drives the current host Pi through a real RPC `fusion_brainstorm` parent-agent loop, checks the persisted tool result carries the complete Pi `Usage.cost` object, invokes `get_session_stats` (the same aggregation boundary used by the TUI footer), reopens the durable session, and verifies identical token/cost totals. All parent and child inference remains deterministic and local.
 
 ## Coverage summary
 

package/TEST_PLAN.md

@@ -53,8 +53,8 @@ This package follows:
 | Stop task from LLM tool | `bg_kill` |  | yes |  |  |  |  |  | Covers running kill and already-finished loud failure. |
 | Fusion command direct result | `/fusion`, `fusion-result` custom message | yes | yes | yes |  | yes | yes |  | Core unit covers deterministic v2 conversation-projection construction, explicit image omission markers without raw image data, direct final-text child argv/stdin/compact-metadata parsing, exact multi-block answer reconstruction, artifacts, pre-abort handling, and orchestration. SDK runs real `/fusion` against a fake child `pi`, verifies exactly five child text-mode invocations, hidden prompt persistence, visible exact merged custom message, no parent assistant rewrite, renderer output, progress/status behavior, no-argument editor flow, editor cancellation, and malformed config causing zero children. RPC verifies command discovery, `/fusion <prompt>` with U+2028/U+2029 content, custom-message result emission, no parent `agent_start`, no-argument editor protocol, malformed config/child failure visibility, and child isolation flags. PTY verifies `/fusion` renders the exact merged answer directly in a real TUI. Compatibility smoke runs `/fusion` through every supported Pi version with the installed package entrypoint. |
 | Fusion tool result | `fusion_brainstorm({prompt})` | yes | yes |  |  | current-host stats/replay | yes | yes | Tool is registered at load and re-added on `session_start`; no eligibility/quota/routine/justification gates are implemented. BUG-182 unit/SDK coverage pins the exact host `Usage` contract, complete cost-component aggregation, rejection of legacy `costTotal`, v2 child/result/manifest schemas, exact merged text, progress, context exclusion, and final details. The release compatibility gate drives a real current-host RPC agent through the tool, verifies persisted tool usage, invokes the same session-stat path used by the footer, then reopens the durable session and verifies identical stats. Scripted-provider coverage proves normal parent consumption and evaluator schema repair. |
-| Fusion conversation projection | canonical input `fusion-input.v2`, `context-omission-ledger.json` | yes | yes | yes |  |  | yes |  | Unit covers a >1 MB synthetic tool-heavy session staying within budget, verbatim user/assistant text, thinking exclusion, zero tool-payload preview bytes (head/tail sentinels absent), exact and stable omission counts/byte totals/hashes, byte-identical repeated construction, hash change on omitted-payload mutation without exposing the payload, contiguous omission-run collapsing with dense source-ordered ledger indices, active-tool-call-leaf and sibling-call exclusion, marker-only user images plus ledger-only tool-result images with no base64, per-entry-point policy ids and request authority, and the exactly-one-disposition property for every retained block. SDK verifies the exact child stdin carries `conversation_projection` and never `conversation_transcript`. RPC verifies the v2 request object. Package tests assert the ledger artifact ships in the documented artifact set. |
-| Fusion stage budgets | `budget-plan.json`, typed `prompt_budget_exceeded` | yes |  |  |  |  |  |  | Unit covers the conservative bytes-per-token bound, smallest-configured-route limiting selection (candidate or evaluator), rejection of unknown/zero/negative/too-small context windows, the reserved canonical-input share, boundary accept at exactly the limit and reject one byte past, the child system prompt counted as input, multi-byte UTF-8 byte accounting, pre-spawn rejection with zero children for candidate, evaluator, evaluation-repair, and merger expansion, safe prompts completing all five calls, persisted route/plan snapshots including negative slack on rejection, and the reproduced 1 MB failure shape now fitting the smallest configured budget. Errors carry stage, measured size, allowed size, limiting model, and remediation in both structured detail and message text. |
+| Fusion conversation projection | canonical input `fusion-input.v3`, `context-omission-ledger.json` | yes | yes | yes |  |  | yes |  | Unit covers a >1 MB synthetic tool-heavy session staying within budget, verbatim user/assistant text, thinking exclusion, zero tool-payload preview bytes (head/tail sentinels absent), exact and stable omission counts/byte totals/hashes, byte-identical repeated construction, hash change on omitted-payload mutation without exposing the payload, contiguous omission-run collapsing into compact `omitted_activity` receipts carrying only `at`/`bytes`/`counts` (no hashes, ledger indices, or per-kind byte map), receipt-to-ledger reconciliation through `projection_map`, active-tool-call-leaf and sibling-call exclusion, marker-only user images plus ledger-only tool-result images with no base64, per-entry-point policy ids and request authority, and the exactly-one-disposition property for every retained block. SDK verifies the exact child stdin carries `conversation_projection` and never `conversation_transcript`. RPC verifies the versioned request object. A dedicated high-cardinality fixture (340 receipt runs over 1,360 omitted events with incident-matching visible-text volume) pins per-receipt cost and proves the pre-fix verbose format would overflow the merge stage while the compact format fits. Package tests assert the ledger artifact ships in the documented artifact set. |
+| Fusion stage budgets | `budget-plan.json` (v2 per-stage forecasts), typed `prompt_budget_exceeded` | yes |  |  |  |  |  |  | Unit covers the conservative bytes-per-token bound, smallest-configured-route limiting selection (candidate or evaluator), rejection of unknown/zero/negative/too-small context windows, the reserved canonical-input share, boundary accept at exactly the limit and reject one byte past, the child system prompt counted as input, multi-byte UTF-8 byte accounting, per-stage forecasts built from the real prompt builders against each stage's own route, deterministic primary-blocker selection naming the first failing mandatory stage with conditional repair listed separately, the empty-request counterfactual that suppresses request-shortening advice when history dominates and states an exact byte target when it does not, advisory >=80% utilization warnings on runs that fit, pre-spawn rejection with zero children for candidate, evaluator, evaluation-repair, and merger expansion, safe prompts completing all five calls, persisted route/plan snapshots including negative slack on rejection, and the reproduced 1 MB failure shape now fitting the smallest configured budget. Errors carry stage, measured size, allowed size, limiting model, and remediation in both structured detail and message text. |
 | Pi/TypeBox compatibility | `peerDependencies`, packed bytes | yes |  |  |  |  | yes |  | Unit pins typebox as a `"*"` peer that is neither a runtime nor bundled dependency, requires the resolved TypeBox to be Pi 0.83's 1.3.x line, requires all four supported Pi/TUI lines in the peer range, scans all package TypeScript for the seven removed TypeBox APIs, and compiles the shipped tool schema plus nullable-array/nullable-string and optional-field shapes under TypeBox 1.3. The release compat gate repeats the removed-API scan against installed package bytes per Pi version. |
 | Fusion model selector | `/fusion-models`, `fusion-models.json`, `FusionModelSelector` | yes | yes | yes | yes | yes | yes |  | Unit covers strict config parsing, duplicates, slash-containing model IDs, stale model failures, `$current`, atomic save, inter-process lock/revision compare-and-swap, and deterministic concurrent-save conflict. Component covers all five slots, duplicate selection, searchable model list, stale display, reset/save/cancel, persistence errors, and width safety. SDK drives the real command in a synthetic TUI context without `ctx.mode` to cover old Pi compatibility and verifies duplicate `$current`/explicit model persistence; SDK also verifies headless no-UI rejection. RPC verifies non-TUI notification without hanging, PTY verifies the real selector opens in a TUI, and compatibility smoke verifies print-mode rejection for every supported Pi version. |
 | Fusion child isolation/lifecycle | child `pi --mode text`, private compact metadata extension, shutdown cleanup, `.pi/fusion` artifacts | yes | yes | yes |  | current-host stats/replay | yes |  | BUG-180 coverage proves final-text transport, reasoning exclusion, strict response reconstruction, unchanged caps, isolation, process cleanup, failure artifacts, and lifecycle semantics. BUG-182 upgrades the private compact metadata and artifact manifest to v2, preserves all token and cost components, centralizes clone/add/empty operations, rejects missing/legacy/unknown/non-finite cost shapes, and proves successful plus failed/cancelled attempt costs aggregate exactly once. SDK/RPC verify public child isolation; session shutdown tracks initializing and live runs. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-background-tasks",
-  "version": "0.7.4",
+  "version": "0.7.6",
   "description": "Pi extension package for background shell tasks plus five-model fusion brainstorming: bg_run tools, /bg commands, /fusion, fusion_brainstorm, model selector UI, bounded logs, kill/timeout safety, and completion wakeups.",
   "type": "module",
   "license": "ISC",

package/src/core/fusion/artifacts.ts

@@ -14,7 +14,7 @@ import {
   type FusionAttemptArtifactRecord,
   type FusionBudgetPlanV1,
   type FusionCandidateId,
-  type FusionContextOmissionLedgerV1,
+  type FusionContextOmissionLedgerV2,
   type FusionChildRunResult,
   type FusionModelConfigV1,
   type FusionSource,
@@ -288,7 +288,7 @@ export class FusionArtifactStore {
    * a separate artifact so canonical input carries only compact run receipts
    * while the full omission accounting stays locally auditable.
    */
-  async writeContextLedger(ledger: FusionContextOmissionLedgerV1): Promise<void> {
+  async writeContextLedger(ledger: FusionContextOmissionLedgerV2): Promise<void> {
     await this.writeArtifact('context-omission-ledger.json', canonicalJson(ledger));
   }