openmerit 0.1.6 → 0.1.7-preview.1

package/docs/testing.md

@@ -1,118 +1,77 @@
-# Testing
+# Testing and validation
 
 ## Standard verification
 
 ```sh
-npm install
-npm test
-npm run check
+npm ci
 npm run verify
 npm pack --dry-run
 ```
 
-Tests cover protocol contracts, lifecycle decisions, post-build target detection, incumbent identity, baseline sufficiency, durable candidate handoffs, candidate/frontier/swap sequencing, conservative Pareto verification, persistence, audit events, safe evidence identifiers, Pi commands, private intent delivery, evidence requirements, and mutation boundaries.
+`verify` runs workspace tests, syntax checks, TypeScript checks, the package build, demo tests, generated-documentation checks, and whitespace validation. It does not publish a package or deploy the website.
 
-The routine `packages/pi/test/live-wakeup.test.ts` test uses a disposable project to verify that a due, insufficient baseline is checked and recorded without starting Pi. A prior opt-in live OpenRouter run proved the former harness-driven assessment could issue an intent and call tools, but Pi did not complete it within the bounded test. Candidate discovery and model changes still require a representative headless end-to-end trial before claiming unattended operation.
+## Coordinator and Pi coverage
 
-Automation tests additionally cover task and elapsed cadence, idempotent target-bound signal delivery, durable pending work, catalogue changes, metric regression thresholds, native wakeup requests, unsupported-scheduler fallbacks, delayed post-swap verification, and Pi dispatch from explicit application-task signals. Pi's own settled-turn telemetry is not a product-observation test path.
+Automated tests cover protocol contracts, task-specific sampling, application-target binding, setup and post-build target detection, incumbent identity, deterministic baseline sufficiency, durable candidate/assessment handoffs, frontier conformance, approval, swap and rollback requests, persistence, evidence requirements, budget accounting, and audit redaction.
 
-## Inference Terminal verification (unreleased)
+Automation coverage includes task and elapsed cadence, signal idempotency, durable pending work, catalogue changes, regression thresholds, delayed post-swap checks, scheduler capability gaps, and persistent pause. Pi's own usage and coding checkpoints are not counted as application measurements.
 
-`npm test --workspace=@openmerit/terminal` tests metadata projection, unknown/redacted values, incremental UTF-8 imports, stable-ID updates, rotation, truncation, project boundaries, source precedence, local runtime contracts, aggregate counter scope, CLI arguments, sample isolation, chart sample gaps and inspection, and the loopback read-only API. Local HTTP fixtures prove which endpoints the collectors call; they do not prove every external runtime version is compatible.
+The disposable-project test in `packages/pi/test/live-wakeup.test.ts` verifies that a due, insufficient baseline is recorded without starting Pi. Scheduler tests validate managed LaunchAgent configuration and authorization boundaries. They do not establish reliable unattended candidate discovery, comparison, or swapping; those require a representative headless product trial.
 
-The source UI was also reviewed in a browser at desktop and 390-pixel widths, including all six views, model search and location filters, time windows, node inspection, the command palette, Escape dismissal, and a real empty project. The full `npm run verify` command includes terminal tests and syntax checks through the workspace scripts. See the [terminal guide](inference-terminal.md) for current collection coverage and limitations.
-
-The enterprise sample tests assert 40 reported agents, 140 tasks, 21 used models, 9,600 unique requests, consistent task-to-agent assignments, all path classifications, and unknown measurements on running requests. Browser checks cover the larger inventories, agent/task lists and nested details, command search, map coverage, and desktop/mobile layouts. This fixture exercises interface density; it does not establish production throughput or hardware performance.
-
-The sidebar and animation revision was additionally checked for persistent chart/DOM identity across polls, CPU/memory selection, exact-sample keyboard inspection, compact navigation, Torph transitions, immediate keyboard updates, and static charts under reduced motion. These are browser checks, not a committed browser automation suite. Browser source lives in `packages/terminal/client/`; `scripts/build-terminal.mjs` bundles it into the ignored `public/app.js` before `npm run dash` and during the package build.
-
-Path-category tests cover explicit metadata precedence, unknown/invalid types, case-insensitive inventory/source matching, conflicting declarations, and mixed paths without double counting. They verify that endpoint location alone never establishes a request path. Sample data exercises direct, local, router, and gateway paths, including one model reached through multiple types.
-
-Deployment and capacity tests cover strict identity matching, duplicate-ID rejection, ordered path projection and entry conflicts, shared-pool deduplication, stale/future/missing/zero readings, positive-limit ratios, safe project-local capacity files, export failure clearing, and highest-series vLLM cache usage. The enterprise fixture checks 27 deployments, 12 pools, 9,479 identified requests, and explicit stale/missing capacity coverage.
-
-## Pi load smoke test
+## Inference Terminal coverage
 
 ```sh
-./node_modules/.bin/pi --no-session --no-extensions -e ./packages/pi/src/index.ts --list-models
+npm test --workspace=@openmerit/terminal
 ```
 
-This proves the real Pi runtime loads the extension and enumerates its available catalogue. It does not prove a provider can complete an end-to-end product intent.
-
-CI also packs `openmerit`, installs the tarball in a temporary directory,
-imports its public core and protocol entry points, and loads the installed Pi
-extension. This checks the published artifact rather than only the workspace.
+Tests cover metadata projection and redaction, source containment, recursive syntax indexing, phased worker progress, incremental reuse, coverage limits, endpoint classification, authorized loopback probes, supported request-log formats, UTF-8 boundaries, rotation/truncation, source reconciliation, historical ranges, unknown versus zero measurements, and the read-only API.
 
-## Evidence levels
+Inventory and UI coverage includes model/service identity, explicit deployment and host links, shared capacity, stale/missing hardware, path classification, map completeness, search and filtering, preference-dependent demo decisions, and preservation of unknown costs. Runtime fixtures establish which endpoints collectors call; they do not prove compatibility with every external runtime version.
 
-- **Unit verified** — deterministic behavior has automated coverage.
-- **Runtime smoke verified** — the actual Pi runtime loads the extension.
-- **Product trial verified** — Pi creates evals and observability in a real product and completes an intent with evidence.
-- **Production verified** — representative repeated runs support reliability and distribution claims.
+The three fictional scenarios are separate:
 
-The current repository is unit- and runtime-smoke-verified. Product and production claims require target-product evidence.
+| Scenario | Agents | Tasks | Models / offers | Requests |
+| --- | ---: | ---: | --- | ---: |
+| Local sample (`--demo` or sample selector) | 40 | 140 | 21 models | 9,600 over 24 hours |
+| Hosted enterprise sample | 200 | 699 | 21 models | 9.6 million over 30 days |
+| Hosted before/after comparison | 280 | 980 | 26 reviewed offers across six services; assignments depend on preference | 9.6 million over 30 days |
 
-## Experimental Pi/OpenMerit sandbox demo
+The comparison tests preserve workload and token volume across all six preferences, enforce service/output compatibility, disclose benchmark/efficiency overrides, and verify modeled costs. Sample tests exercise interface scale and arithmetic, not measured throughput, application quality, or hardware performance.
 
-This repository includes a disposable, live-provider demo for showing the complete experimental loop. It is not the shipping product workflow and its small evaluation set is not evidence for reliability claims.
+Browser review covers the six views, inspectors, search, activity ranges, map and chart interaction, keyboard navigation, reduced motion, and desktop/narrow layouts. The source light/dark change has local browser verification; it is not yet published to npm or the hosted site. Browser checks are distinct from the committed unit suite.
 
-Requirements: Node.js 22.19+, Pi 0.87.x with OpenRouter coding-harness authentication, and an OpenRouter application credential available as `OPENROUTER_API_KEY` or in Pi's local `auth.json`. The coding-harness credential is not passed to the generated application; the demo runner supplies the application credential only while making provider calls.
+## Pi load and package smoke tests
 
 ```sh
-npm run demo
-```
-
-Pi builds a small sentiment-classification app and infers a task/evaluation plan in a temporary directory. The runner validates the plan against available collectors, shows the selected metrics and rationales, smoke-tests the app's configurable route through a live application call, and passes the inferred profile to OpenMerit's coordinator. Task quality is scored only by exact sentiment-label match; rationale plausibility is not graded. The runner collects a baseline, randomly interleaves incumbent/challenger calls over Pi's same three to five representative cases with two repetitions per case, records provider-reported token use and request cost plus measured latency and task outcomes, asks OpenMerit to calculate and verify the Pareto frontier, applies a selected frontier route only in that temporary app, and runs a new post-swap check for every case/repetition. It stops on missing model availability, missing provider usage/cost, invalid output, or a $1 default application-provider spend threshold checked after each response. The saved summary includes its randomization seed.
-
-The default baseline is `openai/gpt-4.1-mini`, with `openai/gpt-4o-mini` as the challenger. Override the selection with `OPENMERIT_DEMO_BASELINE_MODEL` and comma-separated `OPENMERIT_DEMO_CANDIDATE_MODELS`; candidates must be available in the current OpenRouter catalogue. `OPENMERIT_DEMO_PI_MODEL` selects the Pi coding model. `OPENMERIT_DEMO_REPETITIONS` accepts 2–5, `OPENMERIT_DEMO_MAX_SPEND_USD` accepts a cap up to $5, and `OPENMERIT_DEMO_SEED` supplies a repeatable ordering seed.
-
-The temporary directory path is printed at startup and completion. It contains the generated app and inferred plan, `.openmerit/` lifecycle state and redacted event log, per-call `artifacts/application-runs.jsonl`, comparison manifest, and summary. The runner executes application calls serially. Its spend threshold measures application calls only; Pi's coding-model spend is not measured or included. Its tiny task set and sample count are a mechanics demo only: do not present its observed rates, average latency, or frontier as representative reliability, quality, or performance claims.
-
-### Inference Terminal hardware checks (unreleased)
-
-`npm test --workspace=@openmerit/terminal` covers bounded and project-contained hardware exports, duplicate/reserved host IDs, future and stale timestamps, missing versus empty residency, per-host history deduplication/reset, sample placement, and the hashes of locally bundled official favicons. Browser verification covers path-series selection and keyboard bin inspection, enlarged table headers, host and location selection, separate host chart histories, stale/missing hardware, exact-name connection icons, and narrow-screen layouts. These checks exercise passive metadata; they do not run inference or discover remote hardware.
-
-The terminal identity checks additionally cover all 21 sample models across inventory, requests, deployments and residency; official versus arbitrary model namespaces; unknown-name fallbacks; runtime/maker separation; and consistent HTML/SVG asset selection. Browser checks cover model and service marks on all six views, search results, deployment/path/source/model inspectors, and loaded-model rows, plus image loading without external requests, mobile overflow, and nonduplicated accessible names.
-
-
-The terminal naming tests cover source-supplied IDs without display names, absent workload labels, configured-address fallback for unnamed runtimes, full system hostnames, exported host name/hostname/ID precedence, and missing site/rack metadata. Browser checks cover colored all-path stacks and isolated paths, readable activity subtitles, compact Sources rows, name provenance inspectors, and host filters that appear only for reported sites. Sample naming is explicitly fictional.
-
-
-Top-navigation browser checks cover all six tabs, manual arrow/Home/End activation, panel labels, direct hash views, source coverage, environment switching, command search, sticky positioning, responsive tab overflow, and focus/scroll preservation through a passive poll. The old sidebar and collapse controls are absent. Desktop and narrow layouts retain readable content with no page-level horizontal overflow.
-
-### Installed terminal smoke test
-
-The package-smoke CI job installs the npm tarball into a fresh directory, then runs `scripts/terminal-package-smoke.mjs` against its installed executable. It checks the CLI, packaged browser assets, cached passive collection, request-update deduplication, source loss/recovery, sample separation, read-only HTTP boundary, and shutdown. Its default runtime is a deterministic fixture; that test makes no inference calls.
-
-For a separate real-runtime check, supply a disposable Ollama server with `smollm2:135m` already installed and unloaded:
-
-```sh
-node scripts/terminal-package-smoke.mjs /path/to/install/node_modules/.bin/openmerit --ollama-url http://127.0.0.1:11434 --output /tmp/terminal-evidence
+./node_modules/.bin/pi --no-session --no-extensions -e ./packages/pi/src/index.ts --list-models
 ```
 
-This opt-in test application makes four real local generation calls. The terminal itself still reads only inventory endpoints and a temporary metadata export. Two requests are exported with Ollama's actual token and timing measurements. A separate, explicitly synthetic running-to-completed record tests update semantics. A generation without an export must produce no invented request or agent history. The test removes its temporary project and stops its terminal process; the caller owns the disposable runtime's lifecycle. The JSON report excludes prompts and generated text. No cloud credentials are needed.
-
-### Automatic application discovery (unreleased)
+This checks extension loading and catalogue enumeration, not successful provider authentication or a completed product intent. CI also packs the package, installs it in a temporary directory, imports the core/protocol entry points, and loads the installed extension.
 
-Terminal tests cover bounded literal project declarations, credential omission, SDK hints without invented usage, canonical project containment, malformed/oversized inputs, safe loopback endpoints, runtime discovery without configuration, outage handling, and no added polling between discovery passes for never-seen endpoints. Application request logs and explicit exports reconcile without duplicating requests. Application agent identities remain intact. Records explicitly marked as development usage are rejected.
+## Experimental sandbox
 
-A normal CLI fixture starts with application logs, valid legacy Pi history grants, native Pi history, and other coding-tool history directories. It verifies that only application requests contribute to the snapshot and that OpenMerit `product_task_completed` summaries still work when request-level evidence is absent. Retired access flags give an explanatory error; sample JSON bypasses collectors. The installed-package smoke covers runtime/request lifecycle and read-only HTTP boundaries.
+`npm run demo` builds a disposable sentiment-classification application through Pi and runs live provider calls. It requires Pi 0.87 with OpenRouter authentication and an application OpenRouter credential. It is separate from the fictional dashboard comparison and from the recorded website player.
 
-The earlier history-based checks below document a superseded preview. Pi history is no longer a supported terminal source.
+The runner collects a baseline, interleaves incumbent/challenger calls on the prepared cases, records cost/token/latency evidence, submits results to the coordinator, and exercises a change and fresh checks in the temporary app. Missing usage, invalid output, or exhausted allowance stops the run. The default application-spend threshold is $1, checked after responses; Pi's own coding-model spend is excluded. This is not an in-flight provider cutoff.
 
-### Historical VM check with Pi history (superseded)
+The default models are `openai/gpt-4.1-mini` and `openai/gpt-4o-mini`. The runner supports `OPENMERIT_DEMO_BASELINE_MODEL`, comma-separated `OPENMERIT_DEMO_CANDIDATE_MODELS`, `OPENMERIT_DEMO_PI_MODEL`, `OPENMERIT_DEMO_REPETITIONS` (2–5), `OPENMERIT_DEMO_MAX_SPEND_USD` (up to $5), and `OPENMERIT_DEMO_SEED`. The printed temporary directory retains the app, plan, lifecycle state, per-call artifacts, comparison manifest, and summary. Small prepared examples do not establish representative product reliability.
 
-On 8 October 2026, the preview was built and installed in a dedicated Ubuntu 24.04.5 ARM64 VM with four virtual CPUs and 4 GiB of RAM, using Node 22.23.1, Ollama 0.40.1, and Pi 0.87.0. Eighteen actual Pi requests ran across `smollm2:135m`, `qwen2.5:0.5b`, and `smollm2:360m`. After explicit project-scoped history consent, the installed dashboard collected all 18 completed requests and their reported 1,402 input and 587 output tokens directly from native Pi sessions. No activity export supplied those requests.
+## What evidence establishes
 
-This was a controlled local-model workload, not a production-traffic test. Pi's configured local-model rates were zero; displayed cost does not measure hardware or electricity. In that initial build, route names remained unknown because the adapter omitted Pi’s recorded provider connection ID. Deployments, agent/task identities, and request durations were not reported by the history. The VM had no host-directory mounts or forwarded host credentials, and its dashboard was forwarded only to host loopback.
+- **Unit verified:** deterministic behavior has automated coverage.
+- **Runtime smoke verified:** the runtime loads the extension or executable.
+- **Product trial verified:** a representative application completes the workflow with inspected evidence.
+- **Production verified:** repeated representative runs support the claimed operating conditions and reliability.
 
-The real browser check exposed an overview coverage check that omitted native history. A regression test then covered Pi-only usage, connected empty sources, retained partial history, and inventory without request evidence; it now covers application logs instead. The same count check is used by the overview, path summary, and model table.
+Implementation checks, historical demo recordings, and public model benchmarks cannot substitute for target-product validation. See [current limitations](roadmap.md) before relying on unattended changes.
 
 
-### Historical Pi connection identity (superseded, 9 October 2026)
+## Agent-evaluation preview validation — 15 October 2026
 
-That preview adapter preserved Pi's recorded provider connection ID as the route name. Its regression coverage included Ollama, an arbitrarily named local runtime, a declared gateway, unknown connection types, conflicting declarations, and an end-to-end approved-history collector check. Generic activity records still do not acquire a route from their serving-provider name. These checks do not infer physical deployments or reconstruct historical endpoints from mutable credentials or current Pi model configuration.
+The npm preview `0.1.7-preview.1` package adds a separate agent-evaluation loop. `npm run verify` passes 393 tests, plus syntax/type/build/documentation checks. Fifteen evaluation-runner tests cover fresh workspaces, frozen input bytes, scope acknowledgement, final-output parsing, balanced/adaptive repeats, revision compatibility, process timeout, invalid graders, connection errors, cumulative unknown-cost reservations, corrupt ledgers, file collisions, locking and watch coalescing.
 
+An isolated installation of the actual npm tarball completed init, real hosted Scout planning and a disabled-budget check. Twelve simulated agent attempts through that installed command verified two CLI revisions, six attempts each, compatible deltas and persistent accounting. Simulated metrics are packaging tests, not model-performance evidence.
 
-### Application-only VM acceptance (unreleased, 9 October 2026)
+Real macOS smoke runs used Claude Code 2.1.7, Codex CLI 0.162.0-alpha.17.2 and OpenCode 1.18.30. On one tiny JavaScript normalization fixture, with three fresh attempts per route, Claude/Haiku 4.5 and Codex/GPT-6 Luna passed all executable checks. OpenCode/Gemini 2.5 Flash Lite passed one of three. Costs remained unknown for Codex. These different model/harness combinations validate execution and grading paths; they are not an unbiased comparison of agents or a general model ranking. Earlier smoke runs deliberately exposed unsupported authentication/model access, native permission failures, a timeout and invalid JSON. Those cases informed connection-error handling and conservative partial-cost accounting. No model-specific ranking rule was added.
 
-The updated installed CLI was run as `openmerit dash` from the VM application directory with no terminal configuration or custom activity feed. All 240 existing application requests, 40 worker identities, and 120 task identities remained unchanged; the 36 earlier Pi-history requests were excluded. The default server, JSON CLI, next passive poll, and private dashboard agreed. Application log hashes and the legacy grant file were unchanged, and runtime inventory remained available. No new inference was performed. This supersedes the Pi-history acceptance scope above and verifies supported existing application logs, not universal discovery.
+Scout Worker `7ddeacd3-1310-48bd-b546-60fcab4fa4ef` passed live HTTP/MCP checks for pre-ranking model eligibility, empty inventory and duplicate-ID rejection. npm publication is separate from the successful tarball installation; the registry credential returned HTTP 401 during release preparation.

package/docs/troubleshooting.md

@@ -42,11 +42,11 @@ Pending or active intent state is durable. Inspect `state.json` and `events.json
 
 ## Checks are not running while Pi is closed
 
-This is a current adapter limitation. Pi reports session and task signals while running, but does not provision its own background wakeups. A persistent policy reports an external-scheduler requirement. See [automatic checks](automation.md).
+On macOS, check that the confirmed policy requests persistent execution and allows infrastructure changes, that the Mac user is logged in, and that Pi authentication works without shell-only credentials. `/openmerit doctor` reports scheduler diagnostics; `.openmerit/scheduler-status.json` records completed wakeups and failures. An active pause unloads the managed LaunchAgent. Other platforms need an external scheduler. A successful schedule does not establish that model-driven work completed. See [automatic checks](automation.md).
 
-## Status says paused but a check starts
+## Work continues after pausing
 
-This is a known 0.1.5 limitation: the automation-signal dispatch path does not check the pause flag. The flag is also reset by a process restart. Pause does not cancel existing work. See [current limitations](roadmap.md).
+Pause is durable and gates automatic work, but an already-active intent can continue and explicit commands remain available. Inspect the active intent and audit timestamps to distinguish prior work from a new automatic dispatch. `/openmerit cancel` records abandonment; it does not terminate a provider request or harness process. The standalone terminal is independent of Pi pause; stop its CLI with Ctrl+C to stop collection. If a new automatic intent starts while the persisted pause is active, record a sanitized reproduction and your installed version.
 
 ## Report a reproducible problem
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.6",
+  "version": "0.1.7-preview.1",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -19,6 +19,7 @@
     "dist/protocol/",
     "dist/pi/",
     "dist/terminal/",
+    "dist/eval/",
     "docs/",
     "CHANGELOG.md",
     "LICENSE",
@@ -77,10 +78,14 @@
     "check:docs": "npm run build:docs && node --test scripts/docs.test.mjs",
     "check:demo": "node --test demo/test/*.test.mjs",
     "prepare": "npm run build",
-    "verify": "npm test && npm run check && npm run typecheck && npm run build && npm run check:demo && npm run check:docs && git diff --check",
+    "verify": "npm test && npm run check && npm run typecheck && npm run build && npm run check:demo && npm run check:scout && npm run check:docs && npm run build:scout && git diff --check",
     "prepublishOnly": "npm run verify",
     "build:dash-preview": "node scripts/build-terminal-preview.mjs",
-    "deploy:dash-preview": "npx --yes wrangler@4.136.3 deploy --config wrangler.terminal.jsonc"
+    "deploy:dash-preview": "npx --yes wrangler@4.136.3 deploy --config wrangler.terminal.jsonc",
+    "scout": "node scout/dev.mjs",
+    "scout:refresh": "node scout/refresh.mjs",
+    "check:scout": "node --test scout/test/*.test.mjs && node --check scout/worker.mjs && node --check site/scout/scout.js",
+    "build:scout": "node scripts/build-scout.mjs"
   },
   "dependencies": {
     "@earendil-works/pi-ai": "0.87.0",
@@ -104,6 +109,7 @@
     "@xyflow/react": "12.12.0",
     "elkjs": "0.12.0",
     "esbuild": "0.28.2",
+    "fflate": "0.8.2",
     "liveline": "0.0.7",
     "markdown-it": "15.0.2",
     "minisearch": "7.2.0",