carthagent 0.0.0-stage → 1.0.7

package/docs/deepseek-jev-benchmark-v4-results.md

@@ -0,0 +1,119 @@
+# V4: all nine task runs evaluated
+
+## Bottom line
+
+All 3 tasks × 3 arms were attempted once in the frozen v4 matrix, with no candidate retries, source repairs, or evaluator feedback to generators. All nine retained outputs were externally evaluated. **Plain pi and Carthagent+Jev each passed 25/25 external categories; Carthagent without Jev passed 16/25.** These are category counts, not exhaustive correctness scores or statistical evidence of Jev superiority.
+
+The strongest observed operational difference was completion: **plain pi delivered final answers for voxel and ledger; no harness-arm run reached final handoff**. Five harness runs hit the frozen request-size ceiling; the sixth hit per-response output length. Plain shop hit the 15-minute deadline. Consequently there is no clean fully completed three-way comparison, even though every cell now has an evaluated artifact outcome.
+
+## Protocol and provenance
+
+- Execution commit `e157563`; source freeze under `/home/baissi/benchmarks/deepseek-jev-v4/freeze.json`. Actual frozen hashes reverified after all nine evaluations. Freeze format version 3; attempt identifier v4.
+- Same installed pi 0.85.1 engine and generator `deepseek/deepseek-v4.1-flash`, high reasoning, all arms. Plain includes the identical benchmark-only budget guard, not delivery planning. Treatments add Carthagent delivery extension/skill and optionally Jev.
+- Separate Jev Decisions calls: returned `typesafe/jev-1.13-20260917`. No generator router.
+- Frozen ceilings: 900 seconds, 120 tool attempts admitted by benchmark guard, 100 paid requests including up to 3 Jev calls, 16,384 output tokens/request, 400KB incoming / 402KB reconstructed request envelope, $2.79/cell. Compaction disabled. See `deepseek-jev-benchmark-v4-protocol.md`.
+- Earlier v1–v3 attempts, defects, unknown charges and partial candidates remain retained and disclosed. V4 is not an independent replicate of a silently discarded experiment.
+
+## Results
+
+| Task | Arm | External categories | Terminal outcome | Time | Tool starts | Known API cost | Candidate-owned tests |
+|---|---|---:|---|---:|---:|---:|---|
+| Voxel | Plain pi | **8/8** | Final answer completed | 284.3s | 28 | $0.04217 | Pass: 15 unit + 8 HTTP |
+| Voxel | Carthagent | **0/8** | Output-token limit | 276.7s | 22 | $0.03834 | Cannot run: no Cargo.toml |
+| Voxel | Carthagent + Jev | **8/8** | Input-byte limit | 332.5s | 36 | $0.05754 | Fail: Rust test code does not compile |
+| Shop | Plain pi | **8/8** | 900s wall-time limit | 900.3s | 41 | $0.05119 + unknown hold | Pass: 14 tests |
+| Shop | Carthagent | **7/8** | Input-byte limit | 264.5s | 31 | $0.11445 | Fail: referenced test files missing |
+| Shop | Carthagent + Jev | **8/8** | Input-byte limit | 255.9s | 53 | $0.12673 | Pass: 13 tests |
+| Ledger | Plain pi | **9/9** | Final answer completed | 129.1s | 15 | $0.01526 | Pass: 22 tests |
+| Ledger | Carthagent | **9/9** | Input-byte limit | 317.6s | 39 | $0.05291 | 29/30 pass; erroneous assertion |
+| Ledger | Carthagent + Jev | **9/9** | Input-byte limit | 321.0s | 32 | $0.05147 | 36/38 pass; erroneous assertions |
+
+Category counts weight unrelated categories equally and must not be treated as percentage of all requirements. Missing runnable voxel/harness accounts for its eight failed categories, not eight separately diagnosed implementation defects.
+
+### Aggregate descriptive numbers (not quality-adjusted rankings)
+
+| Arm | External categories | Final-answer completions | Tool starts | Sum of run time | Known cost |
+|---|---:|---:|---:|---:|---:|
+| Plain pi | 25/25 | 2/3 | 84 | 1,313.7s | $0.10863 + $0.15 unresolved hold |
+| Carthagent | 16/25 | 0/3 | 92 | 858.8s | $0.20570 |
+| Carthagent + Jev | 25/25 | 0/3 | 121 | 909.4s | $0.23574 |
+
+Do **not** interpret shorter harness time as greater speed: those runs terminated at resource ceilings, while plain shop spent much of its time debugging its own browser setup until timeout. Time includes tooling, provider latency, and dependencies; provider endpoints differed. Known cost excludes unresolved charges and includes repeated/cache context.
+
+## Detailed observations
+
+### Voxel
+
+Plain and Jev-assisted implementations passed release compilation, deterministic terrain, movement/gravity/jump checks, block interaction checks, invalid/unreachable action nonmutation, save/corrupt-load behavior, changing Rust PPM output, and real Firefox keyboard interaction. Screenshots/frame artifacts were captured; no human aesthetic rating is claimed.
+
+Carthagent-only completed a captured plan review/approval but exhausted a single model response at `length` before a Cargo project was written. It was not stuck at an unapproved gate. Its planning attempts included malformed IDs/schema repair. No post-stop continuation was supplied.
+
+Jev-assisted runtime passed the external checks, but candidate `cargo test` failed on a type/borrow mistake in `src/http.rs:214`. Release compilation excludes that test-only code. External functional pass does not excuse broken promised tests.
+
+### Shop
+
+Both plain and Jev-assisted implementations passed API pricing/filtering/input rejection, orders/idempotency/restart, concurrent inventory, desktop cart persistence, and mobile checkout through native Firefox clicks.
+
+Carthagent-only passed seven categories but failed the native mobile checkout click: `#status` overlaid the checkout button, causing `element click intercepted`. Its `npm test` referenced missing files. Neither was repaired by the supervisor.
+
+Plain pi spent time trying Playwright/browser dependencies, including a missing `libasound.so.2` and broad host-library searches. The external geckodriver/Firefox evaluator worked and all 14 candidate tests passed afterward. Its final deadline occurred while a model request was in flight; the request was canceled, not replayed. Broad host discovery also demonstrates that prompt-based workspace isolation is not an OS sandbox; it does not prove evaluator or credential confidentiality.
+
+### Exact-money ledger
+
+All three passed the independent BigInt oracle checks, including large magnitudes beyond IEEE-754 cent precision, malformed decimal input, normalized idempotency/conflicts, rejected-key reuse, atomic overflow rejection, restart, corrupted-journal preservation, and seeded transfers.
+
+Carthagent-only's failed candidate assertion expected a valid `open` following failed requests to fail, then expected the newly opened account to exist on the next line. That is a contradictory test expectation, not evidence from this test of an implementation bug.
+
+The Jev candidate's two failing tests also contain wrong expectations: 10.00 + 0.50 is **1050 cents**, not its asserted 1025; `compte_é` violates the public ASCII-only account regex, although its test expected acceptance while supposedly testing a Unicode *journal path*. These are concrete test-quality defects despite the external 9/9 runtime score. Source remains unchanged.
+
+## What Jev actually contributed
+
+**Four successful advisory calls**, known combined Jev cost **$0.00107814**:
+
+| Task/revision | SRP probability | DI probability | Tests probability | Topics at >=0.5 |
+|---|---:|---:|---:|---|
+| Voxel | .33 | .19 | .55 | tests |
+| Shop first capture | .32 | .15 | .43 | none |
+| Shop later capture | .31 | .15 | .46 | none |
+| Ledger | .19 | .13 | .37 | none |
+
+The voxel candidate recorded dispositions about renderer cohesion, the WorldView seam and real-entry-point tests. It incorrectly described the below-threshold SRP/DI probabilities as "flags"; the adapter itself only marked `tests`. Its intended integration tests still contained a compile defect. Other Jev calls produced no above-threshold topics. These probabilities are not calibrated defect likelihood measurements from this benchmark.
+
+The Jev arm's higher external category total than Carthagent-only is **observed, not causally established**. One sampled run per task, different generated plans, endpoint variability, and early stops preclude attributing the difference to four advisory calls. Jev cost was small; generator/context cost and unfinished delivery were the larger operational factors.
+
+## Harness behavior and resource pressure
+
+All six harness runs performed design validation, captured inspection, model-authored review and approval at least once. Schema/reference repairs consumed additional revisions; revisions at termination were voxel 2/3, shop 3/5, ledger 3/3 (without/with Jev). Shop+Jev was re-inspecting its fifth revision when stopped. No harness run called final `delivery_review` or `delivery_finish`; no `verified` delivery is claimed. The harness did not falsely mark these partial outputs complete.
+
+Five harness runs exceeded the local 400KB request body with compaction disabled. This cap counts tool schemas, full plan/review captures, source/tool results, and reasoning replay metadata. It can bind before time, tool, or dollar limits and disproportionately affects verbose planning histories. Therefore this is specifically a **fixed-budget/no-compaction harness comparison**, not an evaluation of a well-tuned compaction-enabled configuration. A follow-up should test bounded planning receipts/context management uniformly and increase neither limits nor memory selectively after seeing results.
+
+## Transparent continuation and raw-status defects
+
+The frozen runner classified input-size refusal as generic `model-error` and stopped after voxel+Jev. A supervisor inspected the receipt and continued **only the six unstarted cells**, with no frozen changes. Shop/plain's wall-time cancellation was classified `infrastructure-error` because its canceled in-flight request retained a hold; after verifying `stopReason=wall-time` and transport `ABORTED`, the supervisor continued **only the three unstarted ledgers**. These are protocol execution adjustments, not retries of failed cells.
+
+All raw result records remain unchanged. Supplemental classification in evaluation/summary files distinguishes `input-byte-limit`, `output-token-limit`, and `wall-time`. For cells skipped by the frozen evaluator's generic model-error rule, supervisor scripts called the **same frozen evaluator functions** on the retained resource-limited outputs. This procedure and its limits are documented at `RESOURCE-STOP-CONTINUATION.md`.
+
+There were **no v4 TLS, malformed-JSON, or upstream non-2xx errors**. The only ambiguous v4 bill is the request interrupted at plain shop's wall-time deadline. The large-body transport fix worked beyond the previously failing size. This is still finite evidence, not a universal network guarantee.
+
+## Billing
+
+- V4 known candidate charges: **$0.5500629984** (includes the four Jev calls).
+- V4 unresolved in-flight reservation: **$0.15**, not a confirmed charge.
+- Shared candidate accounting after all attempts: conservative recorded debit **$2.2395021996** plus **$0.30** current holds = **$2.5395021996**. The debit includes $1.50 of historical unknown holds seeded during earlier repair, so it is not all confirmed provider spend.
+- Known candidate bills across all attempts: approximately **$0.7395021996**. Unknown conservative candidate holds across historical seed/current state: **$1.80**. No account-level/vendor cap guarantee is claimed.
+- Known smoke/diagnostic bills: **$0.0104371476** plus **$0.04** unresolved diagnostic hold. Both shared accounts remain below the original $27/$0.10 ceilings.
+- Provider endpoints were not fixed: AtlasCloud, Relace, Parasail, Sail Research appeared while model ID remained pinned. This confounds timing/cost comparisons; it is not evidence the generator model ID changed.
+
+## Artifacts and limitations
+
+Root `/home/baissi/benchmarks/deepseek-jev-v4`:
+
+- `summary.json`: all nine cells, raw/supplemental outcomes, scores, time/tools/cost/tokens.
+- `matrix.log`, `continuation.log`, `ledger-continuation.log`, `RESOURCE-STOP-CONTINUATION.md` and preserved supervisor scripts.
+- Per-cell `events.jsonl`, `gateway.jsonl`, `result.json`, `invocation.json`, `guard.jsonl`, sessions, workspace, `.harness` reports and `advisor.jsonl` where applicable.
+- Per-cell `evaluation/result.json`, `candidate-tests.json`, server/build/browser logs, screenshots/PPM where checks reached capture.
+- `freeze.json`, `preflight/`, `live-preflight/` preserve readiness and hashes.
+
+Readiness passed 20 benchmark tests and 348 repository tests before execution. The reference/mutant evaluator tests do not make the evaluator exhaustive; voxel regression used a copy of a historical candidate rather than an independently built reference. Screenshots are evidence, not a scored visual review. Acceptance beyond the published assertions can still fail. Same-user shell filesystem access is not sandboxed. Captured reviews remain model-authored, not independent review. No general statistical claim follows from this nine-cell exploratory run.
+
+**Practical conclusion:** plain pi was the most reliable final-delivery baseline in this sample; Jev-assisted Carthagent matched its external functional categories but did not reach handoff and retained test defects. The next harness improvement suggested by these runs is reducing planning/context overhead and finishing verification within bounds—not adding more advisory calls based on this evidence alone.

package/docs/deepseek-jev-http400-diagnosis.md

@@ -0,0 +1,45 @@
+# HTTP 400 diagnosis and regression-tested fix
+
+## Confirmed cause
+
+The v3 HTTP 400 was caused by the benchmark's curl adapter, **not demonstrated to be DeepSeek's tool-history validation or context window**.
+
+1. Reconstructed the failed request using the installed pi serializer, original session messages, tool schemas, system prompt and model compatibility configuration. No tools were executed and the original session/candidate was not changed.
+2. Reconstructed bytes exactly matched the recorded request: **97,377 bytes**, SHA-256 `c7d747793525d4a7019922d59076d481f4a723d3a593d66dd1fbc095c79208b3`. All 11 tool calls had matching results; no orphan results or duplicate IDs were found.
+3. Made the one authorized diagnostic replay, changing **only `max_tokens` from 16384 to 32**. OpenRouter returned HTTP 400, `error.message: "JSON parsing failed"`. This replay was not resumed as a candidate, and no returned tool could execute.
+4. Sent the original reconstructed payload through the old adapter to a real local HTTP receiver. **The receiver obtained zero body bytes**, despite the supervisor's pre-transport hash correctly describing 97,377 bytes.
+5. The adapter put the entire JSON body into a single curl stdin-config `data-binary` line. That path hits curl's config-line length limitation; curl can still issue the request without the intended body. The earlier 60KB readiness inputs did not catch it.
+6. The fixed adapter delivered all **97,377 bytes byte-for-byte**, with the receiver hash matching the original. This offline reproduction directly demonstrates the transport defect and correction.
+
+The earlier v1/v2 TLS failure remains a separate historical issue; this finding does not retroactively establish its cause.
+
+## Fix
+
+- Keep fresh curl/OpenSSL transport, verified HTTPS, HTTP/1.1, no redirects and **no automatic paid POST retry**.
+- Send the request body using `--data-binary @FILE`, outside curl's inline-config parser. The body file is created in a private **0700** temporary directory with **0600** permissions and removed when curl closes, including normal failure/abort paths.
+- Credentials still enter only through the short stdin configuration, never argv or body files. This changes the previous claim that payloads never touch disk: payload files now exist temporarily and abrupt supervisor termination may leave them. It is **not zero-retention or an OS credential sandbox**.
+- Preserve selected response headers (`x-request-id`, `request-id`, `cf-ray`) and parse interim HTTP headers while retaining response/deadline bounds.
+- On HTTP failure, record request hash, status, safe error fields, request ID when supplied, and a bounded sanitized body excerpt. Known key variants and bearer tokens are redacted before display truncation; oversized bodies are omitted and canceled. This is not universal secret detection: error bodies can reflect submitted source/planning data.
+- Keep ambiguous billing holds and stop instead of assuming an HTTP 400 is free.
+
+## Tests / evidence
+
+Focused tests cover certificate verification, real TLS, cancellation/deadline/response bounds, redirects, missing curl, response headers, and **actual server-received request bytes** at 60KB, 90KB, 97KB, 130KB and 399KB plus Unicode/quotes/backslashes/newlines and the reconstructed historical multi-turn request. Error tests cover reflected credentials, oversized/non-JSON error bodies, no paid retry, held unknown cost and exact historical request reconstruction.
+
+Artifacts: `/home/baissi/benchmarks/deepseek-jev-v3/http400-diagnostic/`:
+
+- `payload.json`: exact reconstruction, private mode 0600; contains original candidate source/context, not the supervisor API key.
+- `inspection.json`: hashes and structural history inspection.
+- `replay.json`: one-shot paid diagnostic, bounded sanitized provider error.
+- `wire-inspection.json`: old adapter received-body length 0.
+- `wire-inspection-fixed.json`: fixed adapter exact byte/hash match.
+
+Test logs: `/tmp/benchmark-http400-tests.log`, `/tmp/benchmark-http400-fixed.log`. Full validation passed: **20 benchmark tests** (`/tmp/benchmark-http400-full.log`) and `npm run quality` including **348 repository tests**, lint/typecheck/build (`/tmp/benchmark-http400-quality.log`).
+
+## Billing and next execution
+
+The diagnostic retains a **$0.04 smoke reservation**, since HTTP 400 did not return billing. Existing known smoke spend is approximately **$0.010010970**; known smoke plus this hold is **$0.050010970**, under the original $0.10 ceiling. The prior $0.15 candidate reservation is unchanged. These held sums are not confirmed charges.
+
+Only **one paid replay** was made. No paid post-fix replay, new candidate run, provider pinning, model change or matrix restart was performed in this repair. The specific defect is verified locally using the exact historical bytes; fixed live acceptance remains to be tested under a newly authorized bounded probe.
+
+The v3 freeze is intentionally now stale: do not overwrite it or its readiness receipts. A new protocol/root/freeze is required before another matrix. Old v3 readiness cannot authorize these changed transport/evaluator sources. The previous `benchmark-running.md` description of pipe-only bodies is historical and superseded by this document; its old freeze commands must not be used to overwrite v3.

package/docs/delivery-review-evidence.md

@@ -0,0 +1,53 @@
+# Review the outcome, not just the parts
+
+Delivery review captures tracked changes and the contents of new text source files.
+Dependency, cache and output directories use the fingerprint's shared directory
+exclusions. The receipt lists that scope explicitly. Binary additions carry a
+digest and require an appropriate viewer; they are not visual evidence. Large
+receipts remain available through the full capture path, and excessive new files
+fail explicitly instead of silently losing source. Git-ignored files are outside
+the new-file capture; include relevant ignored sources in an explicit manual review.
+
+For changes connecting components, select a few checks that demonstrate the
+connection the user relies on. A game needs input/camera/movement agreement; a
+form needs submission/persistence agreement. A valid screenshot file or passing
+physics test does not demonstrate those connections. Local changes still need
+only relevant focused checks: this is not an extra mandatory planning loop.
+
+`coverage` accepts optional `status: "demonstrated" | "unverified"`. Use
+`unverified` when assertions do not establish a requirement. Such gaps and all
+submitted findings are saved with stable IDs. A failed review record returns
+`recorded:false` and unresolved findings, invalidates earlier review evidence,
+and persists through both CLI and HTTP entry points. Empty findings, recapture,
+session restore, and plan revision do not erase them.
+
+Resolve a finding with `resolutions:[{id,change,evidence}]` on a current capture
+after checks pass. Explain the repair or concrete evidence that the original
+requirement is met. An environment limitation is not a resolution of untested
+core behavior. If it cannot be demonstrated, finish blocked and describe what
+did pass. The finding and resolution history is retained in the review evidence.
+
+These are traceability safeguards, not a semantic correctness oracle: assertions
+and resolutions remain model-authored. Existing clients may omit coverage status,
+but cannot erase previously recorded findings by omitting them.
+
+## Outcome contracts for larger changes
+
+Plans may opt into experimental top-level `outcomes` with `id`, exact
+`requirement`, real `entrypoint`, externally `observable` result, explicit
+`substitutions`, a plausible `missedFailure`, and linked non-static `checks`.
+No task is automatically enrolled based on component count. Standard
+plans without a declared architecture should still use outcomes for substantial
+changes; automatic inference of complexity is not claimed.
+
+Build and exercise a thin real execution path early. At final review, each outcome
+needs `{id,check,observation,challenge,status}`. The harness binds it to the actual
+fresh passing check receipt. A missing record or unrelated check cannot verify;
+an explicit unverified result creates a persistent finding. Outcome IDs and
+requirement ownership cannot be dropped during plan revision. A successful
+alternate mode is not evidence for the normal entrypoint. External dependencies
+may be isolated, but resulting substitutions must remain visible.
+
+The harness validates linkage and persistence. It does not execute prose,
+understand every assertion, or prove that a model-authored challenge actually
+exercises the claimed behavior. Live evaluation remains necessary.

package/docs/delivery-timeout-repair-plan.md

@@ -0,0 +1,51 @@
+# Delivery timeout and schema-friction repair plan
+
+Status: proposed, author-challenged; no implementation or paid replay in this change.
+
+## Model selection and observed failure
+
+User-selected Jev OpenRouter model ID is `typesafe/jev-1.13`. Both production adapter and benchmark broker already request it through POST https://openrouter.ai/api/alpha/decisions (not chat completions). Keep GLM5.3 Flash as generator; never substitute Jev router/chat generation. No scored GLM Jev cell dispatched advice because neither reached valid inspect.
+
+GLM append/Jev: gateway request at20:45:33.084Z,68,342 bytes, kind generation, request hash c623ecaa4aac41673b4dcddaf5caf939e310c1f37a436a13ee6dc849ac84c804. Failed after180,005ms with ABORTED. Session events identify compaction. The shared180s gateway deadline fired; unknown$0.15 hold retained. This proves no completed response arrived before deadline, NOT absence of any upstream progress or that Jev stalled. Current curl adapter buffers stdout until process close; gateway buffers/parses complete SSE before releasing it. Existing receipts cannot distinguish prefill/queue delay from slow/interrupted streamed output.
+
+GLM vary/Jev: overall900s candidate deadline canceled final generation/compaction request after22,288ms, not its180s provider limit. Time consumed by schema/workflow repairs and discovery. Other900s cells likewise include unfinished tests/review. Do not conflate wall-time, command deadline, provider deadline, byte overflow or advisor failure.
+
+Concrete repair evidence: scenarios-only nested design patch replaced entire design, requiring full resubmission; invalid optional workflow schemas/references; missing revision reason/acceptance; review walkthrough mismatch; wrong progress IDs; candidate t.plan with standalone node:assert; unbounded shell test pipelines. Broader provider root cause unresolved.
+
+## Stage1 — instrument and stop truthfully (transport/runner cohesion)
+
+- Explicit request purpose generation/compaction/advice, deadline owner, wall remaining, request ID, request bytes, output ceiling, connect/header/first-body timing where observable, cumulative bytes and last-progress time. No raw body/key telemetry. Purpose comes from trusted lifecycle/integration metadata, not inference from prompt text; unknown if unavailable. Correlate split compaction requests without concurrent mislabeling.
+- Streaming-aware transport observer parses bounded SSE incrementally and retains known usage/model/id as they arrive, while candidate delivery may remain buffered until billing validation. Keep certificate checks, no retry/redirect, hard byte limit. Partial bill is reconciled only when authoritative complete usage is valid; otherwise keep hold.
+- Distinct first-response/progress/total/wall deadlines with documented precedence; provider progress cannot extend total indefinitely. Initially keep existing total180s until measurements justify change. Tests: delay headers, progress stream, partial usage, malformed SSE, unknown bill, downstream cancel, wall vs provider races and no duplicate dispatch.
+- On stop, persist supervisor blocked/interrupted handoff containing exact pending requirements/evidence status and reason without another paid generation. Do not manufacture model-authored review or verified completion. No repeated failed compaction attempts after gateway closes.
+
+## Stage2 — remove structural rewrite loops (planning API cohesion)
+
+- Add explicitly separate versioned design-section patch operation. Merge only named components/ports/scenarios/risks; arrays replace within section; absent sections retained. Unknown fields fail with exact JSON path/allowed keys. Existing whole-design replacement semantics unchanged for compatibility. Revision/digest change invalidates approval/evidence per existing rules; exact acceptance/outputs/findings/history/user checks remain.
+- Offer compact deterministic reference manifest: normalized step IDs/indices, component/check/scenario IDs and exact requirement references. Structural repair skeleton has empty trace/assertion slots visibly marked incomplete; never auto-fill reviews/approval.
+- Review errors list missing/duplicate/unknown scenario IDs and missing fields; progress errors list valid steps. Top-level schema errors point to current guide and concrete missing key. Update guide to explicitly omit optional authored workflow for simple tasks: automatic mandatory workflow overlay still applies. If supplied, validate full authored graph; do not silently discard it.
+- Tests replay each observed invalid call, prove actionable error followed by minimal valid repair, preserve all original acceptance/outputs/validators/blockers and stale-source locks. Assert partial scenario patch cannot delete components/ports/risks.
+
+## Stage3 — test execution and verification reserve (execution host cohesion)
+
+- Explicit bounded-run policy shared by all arms: default per-shell timeout60s, absolute120s unless protocol authorizes a declared longer check. Supervise process groups and deadline independently of shell pipeline exit status. Timeout remains timeout even if tail/head exits0. Candidate cannot reset session clock or buy a larger check limit through a tool argument.
+- Show remaining time and pending obligations from supervisor monotonic clock. At <=180s recommend no optional expansion, rerun only necessary checks, record review/handoff or explicit blockers; preserve every requirement. Do not initiate another compaction if insufficient configured request/handoff reserve; stop honestly if context cannot proceed safely rather than overflowing or dropping requirements. This is bounded-run policy, not universal production behavior.
+- Local fixtures demonstrate real test-runner APIs (t.assert vs standalone assert counting) and detect hanging servers/cleanup problems. Do not require running arbitrary probes before approval or auto-modify user tests. Actionable hints only, no weakening assertions.
+- Tests: hanging pipeline descendants killed, nonzero original command not laundered, budget cannot reset by revision, failed/stale evidence survives supervisor handoff, enough time remains to finish a successful bounded fixture.
+
+## Stage4 — Jev integration parity and meaningful exercise
+
+- Retain exact `typesafe/jev-1.13` Decisions model, explicit opt-in,15s local advisor bound,3 attempts/run and existing unknown-bill holds. Advice remains non-authoritative.
+- Eliminate drift: broker currently replaces production ADVISOR_QUESTIONS with its own older QUESTIONS strings. Use a shared frozen question contract/hash, include check argv/outputs/assumptions, assert direct and broker payload equivalence. Broker still rejects client model/endpoint overrides.
+- Invalid schema remains deterministic: zero paid calls/attempt reservations until valid capture. First valid inspect triggers automatic semantic advice; identical run/revision/digest/source reuses receipt. Review must visibly address flagged topics but model-authored disposition is not proof. Failure means unavailable, not clean plan.
+- Scripted local end-to-end run must actually reach valid inspect/advice, use updated question content, then implementation/checks/review/finish. Same run repeated with advisor timeout proves ordinary review remains possible without authority leakage. Independently test shared campaign policy: unknown paid billing may intentionally stop benchmark even though ordinary production advisory failure is nonblocking.
+
+## Stage5 — readiness then new evaluation
+
+1. Offline: all above adversarial tests, real installed-engine scripted full workflow, repository quality. Valid/invalid/hanging scenarios and real local streaming HTTP, not only mocked functions. Save plan challenge/author review; disclose independence limits.
+2. Small separate GLM/Jev development pilot before matrix: budget cap agreed in advance, charges incl compaction and unresolved holds. Require actual advice invocation, measured request stages and honest terminal handoff; analyze failures before expanding. No claiming classifier effectiveness from one successful call.
+3. Freeze fresh tasks/root/evaluator/runtime/provider policy. Same model, time/command caps, compaction, docs and isolation all arms. >=3 repetitions/task for primary comparison; fixed provider policy if supported/available, no silently different fallback. No candidate repair/retry after hidden evaluation. Preserve historical artifacts and old bills.
+
+## Author challenge and acceptance boundary
+
+A longer timeout alone trades one stop for lost task time; do not start there. More guidance alone failed to prevent schema loops; implement deterministic ergonomic operations, not probabilistic syntax repair. Model summaries cannot redefine requirements, approvals or findings. Partial semantic advice before a valid captured design adds cost and unclear identity, so not proposed. No silent workflow simplification, check deletion, model switching, shell bypass or automatic unknown-bill replay. Changes to test budget policy must apply to plain pi too. Timing tells where delay happens, not why vendor internals stall. End-to-end scripted success demonstrates mechanics only; effectiveness requires new model runs. Scope excludes general OS sandbox proof and universal schema/model correctness.

package/docs/evaluation-readiness.md

@@ -0,0 +1,31 @@
+# Controlled web-delivery evaluation protocol
+
+## Status and spending authorization
+
+This is an evaluation protocol, not evidence that a model delivers complete applications reliably or economically. No paid trials were run for this hardening series. Before any live requests, obtain an explicit aggregate spending cap, per-run allocation, approved providers, retry policy and billing-monitoring owner. Tool/time/repair budgets are cooperative controls, not dollar caps. Stop admission before the remaining allocation is insufficient for a request; do not assume cancellation refunds an in-flight request.
+
+## Freeze the experiment
+
+Record repository commit, lockfile, runtime and browser versions, OS, task prompt, model configuration, harness configuration, independent acceptance suite and its hash. Use fresh isolated worktrees and least-privilege credentials. Keep the acceptance evaluator outside the agent-writable workspace. Do not replace failed validators with easier checks. Record every task attempt, including infrastructure failures, refusals, exhausted budgets and incomplete results.
+
+Compare baseline and candidate on the same tasks and budgets, with repeated runs and randomized ordering. Declare sample size before inspecting outcomes. Include a small CRUD application, async error/retry UI, authentication/authorization boundaries, persistence, accessibility and an existing-project change. Include Unicode, paths with spaces, malformed input, delayed responses and offline failures. Phones and tablets are outside this platform scope.
+
+## Independent acceptance and repository conformance
+
+For each task enumerate every explicit user requirement and give it an independent observable oracle. Run the target repository's configured lint, formatting checks, typecheck, tests, build and mandatory validators; discover conventions instead of imposing this repository's tooling universally. Report pre-existing failures separately from regressions. Do not perform repository-wide formatting cleanup inside a feature task.
+
+Use real-browser interaction checks for modules and application behavior. Capture startup console errors, exceptions and rejections before navigation; unavailable capture is a failure, not a clean result. Test expected network error states separately from unexpected application errors. Bind executable evidence and review findings to the accepted source snapshot. Screenshots alone are not visual review: record who inspected them, or explicitly mark visual review unperformed.
+
+Review the diff and neighboring architecture for duplicated canonical logic, unnecessary abstractions/dependencies, boundary validation, error handling, cleanup/cancellation, stale async state, unused code and weakened types. Record each finding as severity, location, rationale, resolution and remaining uncertainty. Keep model judgments separate from executable results and human review.
+
+## Metrics and result record
+
+Record task/run ID; all frozen identities; platform; start/end time; admitted tools; repair rounds; provider failures; input/output/cached tokens; actual billed cost; intervention count and minutes; independent acceptance pass/fail; harness success claim; false-success flag; unresolved findings; and links to logs, diff, browser diagnostics and evidence.
+
+Report completion rate, false-success rate (claimed success with independent failure), recovery rate after injected faults, human intervention frequency, maintainability findings by severity, median/tail latency, total cost and cost per independently accepted change. Include spending on failed attempts in total cost. With zero accepted changes, cost per accepted change is undefined, not zero. Publish denominators and uncertainty; do not generalize a single successful task or a token anecdote into model-wide claims.
+
+## Current assurance boundary
+
+The hardening series adds real Firefox error capture and CI, scoped first-party JavaScript lint, actual-engine offline tool/time/repair budget regressions, same-run restoration reconciliation and stale-review rejection. Native Windows/macOS/Linux Node 22/24 CI is distinct from WSL local evidence. WSL Python validation still requires missing local dependencies; native Python CI does not establish WSL Python success.
+
+Outstanding work includes enforced read-only review isolation, stronger validator/runtime identity and fingerprint coverage, and independent controlled model trials. These are not completed by passing the current test suite. Review timeout and source/head checks are described in [review-assurance.md](review-assurance.md); quality scope is in [quality.md](quality.md).

package/docs/evaluation.md

@@ -0,0 +1,70 @@
+# Verification and live Mercury trials
+
+## Runtime migration note (2026-10)
+
+The earlier evidence below was collected with Pi 0.85.1 and its CLI JSONL event protocol. Those runs are preserved as historical records and **must not be pooled, ranked, or treated as directly comparable** with native OpenCode 2 evaluation runs. Native runs record `evidenceSchema`, `runtime: "opencode-2"`, exact OpenCode version, session ID, and `campaignComparability: "not-comparable-to-pi-runtime"`. Freeze v4 rejects the previous Pi-based campaign freeze. Historical reports and raw files are intentionally immutable.
+
+The migrated harness uses the OpenCode 2 HTTP client against a dedicated isolated service. The evaluator starts its own service with a private home; the benchmark rejects arbitrary `CTG_OPENCODE_URL` overrides and starts an authenticated service it owns. Non-fixture provider runs require `OPENROUTER_API_KEY`. Neither path discovers the shared user service or searches `~/.pi` for credentials. Completed runs poll for the exact session's authoritative terminal outcome and use `session.get()` aggregate cost/token usage; missing outcomes or unknown usage fail closed. A native model-request hook checks a session-metadata budget handshake against the active turn/cost guard before primary requests. Productive turns are reconciled from the paginated durable session transcript; the evaluation plugin additionally maintains an ID-keyed persistent ledger for admission across compaction/restart. Missing terminal outcomes trigger a bounded `resume:false` interrupt of only the evaluator-owned session. The adapter also accepts an injected client for offline tests.
+
+Offline checks: `node --test tools/test-native-swe.mjs tools/benchmark/runner.test.mjs`. These validate event accounting and bounded runner setup without provider calls. They are harness tests, not model efficacy evidence.
+
+The previous real-host SWE receipt `/tmp/opencode/ctg-product-eval314-swe.log` is **invalid as native verification evidence**: it reports `productiveTurns: 3` but only one shell call and one edit, and therefore proves no post-edit native shell check. The SWE host acceptance now requires completed native tool results in strict order—red shell, successful edit, then a distinct green shell with explicit exit code 0, one pass and zero failures—before the fixture provider can return a successful final answer. An independent final workspace check remains supplementary, not a substitute for that host-call chain. Earlier documentation or claims that imply the prior receipt verified the repair must not be relied on.
+
+> Context: these trials ran in the development workspace of this package, alongside a voxel-game fixture used as the evaluation task. `scenarios/game/baseline/minecraft.py` in this repository is that fixture: the deliberately defective baseline copied into each trial's isolated work directory. References to "the repository game"/"the improved game" mean an upgraded reference implementation from that workspace, which is not part of this repository; screenshots and raw event streams live in its ignored `artifacts/` and `.harness/` directories.
+
+## Scope of the evidence
+
+The improved game in the repository was implemented and repaired in this coding session. **It was not produced by Mercury in the trials below.** Mercury trials started from `scenarios/game/baseline/minecraft.py` in new work directories. Do not attribute the repository game's passing checks to the smaller model.
+
+Environment: Windows, Python 3.11, Ursina 7.0.0, Panda3D 1.10.16, Node 23.9.0, pi 0.85.1. Model: `openrouter/inception/mercury-2.5-preview` (the available Mercury 2.5 endpoint). This model's catalog advertises text input, not image input.
+
+## Deterministic local checks
+
+- Node library + actual-extension integration tests cover contract/check validation, subprocess exit/error/timeout behavior, stale evidence, undeclared file edits, generated-only scopes, serial check queues, required validators, active-branch restore, compaction context and bounded follow-ups.
+- Python core tests cover deterministic/solid worlds, safe spawn, placement, protected bedrock, chunk-boundary remeshing, face winding, grid traversal, reach, movement/jump/wall/ceiling collisions, stalled-frame substeps, void recovery, atomic saves, malformed saves and import safety.
+- A bounded onscreen check (`scripts/onscreen_smoke.py`) verifies actual keyboard-event dispatch, hotbar selection, pause/resume and mouse capture. It passed on this workstation; this is not extended human playtesting.
+- The real game smoke checks selection/scroll, mining/placing, pause/resume, save/load, jump, movement, framebuffer output and HUD placement. The external Panda3D probe independently verifies real renderer construction and captures its framebuffer.
+- The final default world has 12,644 blocks, 16 terrain entities and 9,456 visible faces. These are measured counters, not FPS claims. A representative combined generation/interaction smoke run took about 1.3–1.7 seconds on this workstation; it excludes some engine startup overhead.
+- The inspected screenshot is `artifacts/game.png`. Screenshot review caught missing top faces, huge offscreen UI scale, weak text contrast and inconsistent overlay depth; those were repaired. Logic tests alone did not catch them.
+- D2 0.7.1 rendered `workflow.d2` and `game-architecture.d2`. The local Go toolchain could not build D2 0.7.1, so an official release binary was downloaded into ignored `.tools/`; no system toolchain was changed.
+
+## Live trials and what changed
+
+Raw evidence remains under `.harness/evaluations/<trial>/`: exact event streams, generated programs, execution logs, summary JSON and delivery reports. All trials were opt-in and bounded. Provider-reported costs are recorded, not assumed to be the actual invoice.
+
+| Trial folder prefix | Condition | Observed result |
+|---|---|---|
+| `1788823697128-cli-7817b8` | Baseline vs initial custom | Both passed 7/7 external CLI checks. Baseline exceeded its 45-turn budget; custom ended in 35 turns. Custom wasted many calls rejecting parallel check siblings. |
+| `1788823943886-game-9e9a8d` | Initial custom game | Candidate smoke exited zero and wrote an image, but did not establish real rendering. It also declared only `artifacts/`, producing an empty source fingerprint. **This was an invalid apparent success**, not a playable-game pass. |
+| `1788824255457-cli-cc97dc` | Baseline vs queued/fingerprinted custom | Baseline passed 7/7 in 24 turns. Custom passed 6/7 in 29 turns: Unicode output crashed under Windows cp1252. Two connection errors recovered. No claim of a custom win. |
+| `1788824548169-game-d4a443` | Baseline vs stricter custom game | Both failed the external framebuffer probe. Baseline exceeded 90 turns; custom ended after repeated connection errors. Neither is a successful game delivery. |
+| `1788825278064-cli-8a4aab` | Custom with required external validators | Completed in 42 turns; 7/7 external behavioral checks passed, including non-Latin Unicode output. Required-validator feedback was available during development. Two connection errors recovered. |
+| `1788825278071-game-bb7c52` | Custom with required renderer validator | Failed graphics verification and exceeded the 90-turn budget. Thirteen connection errors were observed. An inspected real framebuffer showed a gray screen and oversized white line, not a world. The failure remains recorded. |
+| `1788827018813-game-83c5b1` | Custom, first run of the revised harness | Burned the whole 600 s wall clock on an un-timed `find / -name verify_ursina.py` despite the absolute probe path being in the prompt; 2 turns, zero product work. (13 of the previous trial's 91 turns were zero-token connection errors that had also eaten its budget.) Drove the bounded-shell and productive-turn changes below. |
+| `1788827921291-game-6bbbae` | Custom + bash cap + staged prompt + productive-turn accounting | **First verified game delivery:** 23 productive turns, 100 s, $0.022 reported. External probe passed with 106 sampled colors at 1280×720; import-safe launcher, procedural textures, save/load. Honest limitations still listed missing WASD/collision/chunked meshing/gameplay tests — a premature, scope-short completion. |
+| `1788828169501-game-9d5ccd` | Custom + scope-reconciliation guidance | Verified in 41 productive turns, 186 s, $0.050 reported. Probe passed at the 3-color minimum ("3 colored cubes"); O(1) dict world, 15 world tests, versioned atomic saves with corruption handling, pause/selection. Movement/collision still absent and honestly declared in the handoff. |
+
+### Changes driven by those failures
+
+1. **FIFO check queue and `id="all"`:** no retry storm when Mercury batches verification calls.
+2. **Full-project source fingerprint:** undeclared source changes invalidate evidence; output-only scopes are rejected. Changing source during a check also fails it.
+3. **Required external validators:** user-owned checks are injected into the contract; the model cannot omit or replace their IDs to finish. Invalid config fails closed. This project enables core, harness and renderer checks in `.pi/delivery.json`.
+4. **Renderer credibility probe:** a hand-drawn screenshot or logic-only smoke branch does not count as graphics evidence. The probe records actual framebuffer dimensions and sampled colors; its minimal three-color threshold is a coarse blank-frame heuristic, not a visual quality metric. A later re-probe confirmed the failed Mercury framebuffer had only two sampled colors; the tested starter has three and the actual game substantially more.
+5. **Cross-platform integration guidance:** run actual subprocesses in the normal environment, include non-ASCII output, and do not hide encoding errors by fixing only the test's environment.
+6. **Verified Ursina starter and API notes:** provide a runnable slice for offscreen lifecycle, UI lens sizing, correct color APIs, no GraphicsBuffer mouse locking, actual input handling and framebuffer capture.
+7. **Eager recipe context for explicit Ursina/Minecraft/voxel requests:** smaller models sometimes skip skill discovery, so those prompts receive the game skill and starter directly. The 2025-09-07 trials below exercise this path end-to-end.
+8. **Honest completion states and budgets:** failed/stale checks cannot produce verified tool status; a blocked handoff is supported, follow-ups are bounded, and the evaluation runner records time/turn/provider failures instead of treating them as success.
+9. **Productive-turn accounting:** provider connection errors (13 of 91 turns in `1788825278071`) no longer consume the model-turn budget; the runner records `productiveTurns`, `providerErrors` and `endedOnProviderError` per trial (unit-tested in `lib/delivery.mjs`).
+10. **Bounded shell commands (`--delivery-bash-cap`):** pi's bash/powershell tools have no default timeout, so one un-timed `find /` consumed an entire 600 s trial (`1788827018813`). The extension now caps un-timed/oversized shell timeouts at 120 s in trials via the documented `tool_call` input mutation (integration-tested), and the environment forbids filesystem-wide searches and tells the model to use provided absolute paths as-is.
+11. **Renderer-first staging and self-verifying smoke:** the game prompt now requires adapting the verified starter into a probe-passing `--smoke` before gameplay breadth. The starter asserts pixel variety (`assert_nonblank`, unit-tested against flat/black/missing images), so a saved PNG alone is no longer accepted as render evidence; check evidence persists output tails and repair nudges quote the failing check's exit code and tail, denying the "environment limitation" rationalization observed in `1788825278071`.
+12. **Scope-reconciliation guidance:** "verified" with a contract that omits core prompt requirements is now explicitly called a scope failure in the delivery guidance, the finish tool description, the game skill and the trial environment. Two post-fix trials still stopped before movement/collision while honestly declaring it — the remaining gap is model behavior, now visible rather than hidden.
+
+## Interpretation
+
+These trials are engineering regression evidence, **not a statistically powered benchmark**. The implementation changed between trials, task sizes/budgets differ, and the final custom condition receives external validator feedback during development while baseline is scored afterward. Later trials are informed by earlier failures. Do not compare them as a blind, identical-condition model ranking.
+
+The evidence supports better failure detection and a tested verification/recovery mechanism. It **does not establish reliable game generation by Mercury 2.5 in a single run**, nor a general solve-rate improvement. Model capability, endpoint reliability, independent test quality and visual review still matter. The upgraded repository game is a working reference and test fixture, not proof that the smaller model can recreate it unaided.
+
+After the 2025-09-07 hardening, Mercury 2.5 under the improved harness went from zero successful game trials to **verified, external-probe-passing deliveries in 23–41 productive turns** (100–186 s, $0.02–$0.05 reported) where it previously exhausted 91 turns or starved on connection errors and blank framebuffers. Both successful deliveries are still incomplete games: neither implemented movement or collision, and the second cleared the probe's blank-frame bar with only three sampled colors. Gameplay breadth within one delivery run remains a model-capability limit that the harness now surfaces honestly in each handoff's limitations rather than letting it hide behind green checks.
+
+For future evaluation, run multiple fresh trials of both conditions with fixed prompts, budgets, dependency environments and a separate hidden test set. Measure behavioral correctness, final gate status, renderer/visual review, wall time, failed tool calls and reported cost. Keep the failures in the report.