openmerit 0.1.12 → 0.2.0

package/docs/testing.md

@@ -1,6 +1,6 @@
 # Testing and validation
 
-**Pi source replacement — unreleased and unpublished:** the new Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. The dated Pi 0.87 lifecycle/scheduler results below remain historical receipts, not proof of the replacement or supported current commands. Package, browser-approval, native Pi and SDK acceptance must be reported for the new build separately; a successful documentation build alone establishes none of them. Core coordinator and standalone evaluation tests remain separate.
+**Pi adapter — npm 0.1.12 and later:** the Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. Dated acceptance records below describe the packages tested at the time; some mention package exports that have since been removed. Standalone evaluation tests remain separate.
 
 ## Stable 0.1.7 acceptance — 24 October 2026
 
@@ -34,7 +34,7 @@ The final production deployment passed **203/203 VM assertions and 8/8 filter/am
 
 The published `0.1.7-preview.4` package passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Registry integrity matches the reviewed tarball, the `preview` tag resolves to this version, and stable `latest` remains `0.1.6`. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
 
-The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy.
+The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures were recorded separately at the time. These checks do not establish universal semantic reliability or improved task accuracy.
 
 Fresh registry installation in the existing Ubuntu ARM64 Lima VM verified all 101 packaged file hashes, registry integrity and public core/protocol/Pi exports. Seven workloads were sent through both HTTP and the official MCP SDK. Six of those fourteen requests failed their functional expectations: books/posters and invoice HTTP requests returned unavailable responses, while SQL generation and safety classification with rewriting returned empty shortlists through both interfaces. The matching MCP books/posters and invoice requests succeeded; bird identification, label-only classification and unsupported audio behaved as expected. The first harness stopped when it assumed SQL had returned an endpoint. A separate continuation tested exact endpoint scope using the successful bird result, request/protocol errors and installed CLI planning, preserving all original failures. Together the runs recorded 187 passing checks and nine failed assertions, including duplicate agreement failures caused by unavailable responses and an installed-CLI candidate check. Both CLI email and SQL plans were empty. Planning created no execution or spend state, and the existing global VM installation was retained. This is not a clean acceptance pass. The seven first HTTP workload requests had a median wall time of 4.924 seconds and a maximum of 6.229 seconds, including unavailable responses; repeated cached requests were faster.
 
@@ -60,14 +60,6 @@ npm pack --dry-run
 
 `verify` runs workspace tests, syntax checks, strict TypeScript checks and the TypeScript migration ratchet, the package build, demo tests, generated-documentation checks, and whitespace validation. It does not publish a package or deploy the website.
 
-## Coordinator and Pi coverage
-
-Automated tests cover protocol contracts, task-specific sampling, application-target binding, setup and post-build target detection, incumbent identity, deterministic baseline sufficiency, durable candidate/assessment handoffs, frontier conformance, approval, swap and rollback requests, persistence, evidence requirements, budget accounting, and audit redaction.
-
-Automation coverage includes task and elapsed cadence, signal idempotency, durable pending work, catalogue changes, regression thresholds, delayed post-swap checks, scheduler capability gaps, and persistent pause. Pi's own usage and coding checkpoints are not counted as application measurements.
-
-The disposable-project test in `packages/pi/test/live-wakeup.test.ts` verifies that a due, insufficient baseline is recorded without starting Pi. Scheduler tests validate managed LaunchAgent configuration and authorization boundaries. They do not establish reliable unattended candidate discovery, comparison, or swapping; those require a representative headless product trial.
-
 ## Inference Terminal coverage
 
 ```sh
@@ -76,35 +68,15 @@ npm test --workspace=@openmerit/terminal
 
 Tests cover metadata projection and redaction, source containment, recursive syntax indexing, phased worker progress, incremental reuse, coverage limits, endpoint classification, authorized loopback probes, supported request-log formats, UTF-8 boundaries, rotation/truncation, source reconciliation, historical ranges, unknown versus zero measurements, and the read-only API.
 
-Inventory and UI coverage includes model/service identity, explicit deployment and host links, shared capacity, stale/missing hardware, path classification, map completeness, search and filtering, preference-dependent demo decisions, and preservation of unknown costs. Runtime fixtures establish which endpoints collectors call; they do not prove compatibility with every external runtime version.
+Inventory and UI coverage includes model/service identity, explicit deployment and host links, shared capacity, stale/missing hardware, path classification, map completeness, search and filtering, and preservation of unknown costs. Runtime fixtures establish which endpoints collectors call; they do not prove compatibility with every external runtime version.
 
-The three fictional scenarios are separate:
+The local fictional sample (`--demo` or the sample selector) has 40 agents, 140 tasks, 21 models and 9,600 requests over 24 hours. Sample tests exercise interface scale and arithmetic, not measured throughput, application quality, or hardware performance.
 
-| Scenario | Agents | Tasks | Models / offers | Requests |
-| --- | ---: | ---: | --- | ---: |
-| Local sample (`--demo` or sample selector) | 40 | 140 | 21 models | 9,600 over 24 hours |
-| Hosted enterprise sample | 200 | 699 | 21 models | 9.6 million over 30 days |
-| Hosted before/after comparison | 280 | 980 | 26 reviewed offers across six services; assignments depend on preference | 9.6 million over 30 days |
-
-The comparison tests preserve workload and token volume across all six preferences, enforce service/output compatibility, disclose benchmark/efficiency overrides, and verify modeled costs. Sample tests exercise interface scale and arithmetic, not measured throughput, application quality, or hardware performance.
-
-Browser review covers the six views, inspectors, search, activity ranges, map and chart interaction, keyboard navigation, reduced motion, and desktop/narrow layouts. The source light/dark change has local browser verification; it is not yet published to npm or the hosted site. Browser checks are distinct from the committed unit suite.
+Browser review covers the six views, inspectors, search, activity ranges, map and chart interaction, keyboard navigation, reduced motion, and desktop/narrow layouts. Browser checks are distinct from the committed unit suite.
 
 ## Pi load and package smoke tests
 
-```sh
-./node_modules/.bin/pi --no-session --no-extensions -e ./packages/pi/src/index.ts --list-models
-```
-
-This checks extension loading and catalogue enumeration, not successful provider authentication or a completed product intent. CI also packs the package, installs it in a temporary directory, imports the core/protocol entry points, and loads the installed extension.
-
-## Experimental sandbox
-
-`npm run demo` builds a disposable sentiment-classification application through Pi and runs live provider calls. It requires Pi 0.87 with OpenRouter authentication and an application OpenRouter credential. It is separate from the fictional dashboard comparison and from the recorded website player.
-
-The runner collects a baseline, interleaves incumbent/challenger calls on the prepared cases, records cost/token/latency evidence, submits results to the coordinator, and exercises a change and fresh checks in the temporary app. Missing usage, invalid output, or exhausted allowance stops the run. The default application-spend threshold is $1, checked after responses; Pi's own coding-model spend is excluded. This is not an in-flight provider cutoff.
-
-The default models are `openai/gpt-4.1-mini` and `openai/gpt-4o-mini`. The runner supports `OPENMERIT_DEMO_BASELINE_MODEL`, comma-separated `OPENMERIT_DEMO_CANDIDATE_MODELS`, `OPENMERIT_DEMO_PI_MODEL`, `OPENMERIT_DEMO_REPETITIONS` (2–5), `OPENMERIT_DEMO_MAX_SPEND_USD` (up to $5), and `OPENMERIT_DEMO_SEED`. The printed temporary directory retains the app, plan, lifecycle state, per-call artifacts, comparison manifest, and summary. Small prepared examples do not establish representative product reliability.
+CI packs the package, installs it in a temporary directory, imports `openmerit/pi`, smoke-tests the installed terminal command, and loads the installed extension in Pi with `--list-models`. These checks establish packaging and loading, not successful provider authentication or Scout recommendations.
 
 ## What evidence establishes
 
@@ -113,7 +85,7 @@ The default models are `openai/gpt-4.1-mini` and `openai/gpt-4o-mini`. The runne
 - **Product trial verified:** a representative application completes the workflow with inspected evidence.
 - **Production verified:** repeated representative runs support the claimed operating conditions and reliability.
 
-Implementation checks, historical demo recordings, and public model benchmarks cannot substitute for target-product validation. See [current limitations](roadmap.md) before relying on unattended changes.
+Implementation checks and public model benchmarks cannot substitute for target-product validation. See [current limitations](roadmap.md).
 
 
 ## Agent-evaluation preview validation — 15 October 2026

package/docs/troubleshooting.md

@@ -33,7 +33,7 @@ That is expected. Pi has no OpenMerit auto-update hooks. Explicitly rerun `insta
 
 ## `/openmerit` or evaluation tools are missing
 
-They were removed, not renamed. The Pi adapter now supplies only native Scout MCP and shared guidance. It does not set up evaluations, recover intents, schedule work, swap models, or install a LaunchAgent. Existing legacy background jobs must be removed separately; installing the new adapter is not evidence that they stopped. Preserve old evaluation state and audit files if needed. For new evaluations use [the standalone CLI](agent-evaluations.md); custom harnesses may still use [the core coordinator](adapter-guide.md).
+They were removed, not renamed. The Pi adapter now supplies only native Scout MCP and shared guidance. It does not set up evaluations, recover intents, schedule work, swap models, or install a LaunchAgent. Existing legacy background jobs must be removed separately; installing the new adapter is not evidence that they stopped. Preserve old evaluation state and audit files if needed. For new evaluations use [the standalone CLI](agent-evaluations.md).
 
 ## Scout was not consulted or its evidence is insufficient
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.12",
+  "version": "0.2.0",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -11,12 +11,9 @@
     "pi-package",
     "pi-extension",
     "model-routing",
-    "evaluation",
-    "pareto-frontier"
+    "evaluation"
   ],
   "files": [
-    "dist/core/",
-    "dist/protocol/",
     "dist/pi/",
     "dist/terminal/",
     "dist/eval/",
@@ -26,18 +23,6 @@
     "README.md"
   ],
   "exports": {
-    ".": {
-      "types": "./dist/core/src/index.d.ts",
-      "default": "./dist/core/src/index.js"
-    },
-    "./core": {
-      "types": "./dist/core/src/index.d.ts",
-      "default": "./dist/core/src/index.js"
-    },
-    "./protocol": {
-      "types": "./dist/protocol/src/index.d.ts",
-      "default": "./dist/protocol/src/index.js"
-    },
     "./pi": {
       "types": "./dist/pi/src/index.d.ts",
       "default": "./dist/pi/src/index.js"
@@ -77,7 +62,6 @@
     "deploy": "node scripts/deploy.ts",
     "deploy:status": "node scripts/deploy.ts status",
     "build": "node scripts/build.mjs",
-    "demo": "node scripts/demo.mjs",
     "predash": "node scripts/build-terminal.mjs",
     "dash": "node packages/terminal/src/cli.mjs dash",
     "build:docs": "node scripts/build-docs.mjs",
@@ -87,8 +71,6 @@
     "prepare": "npm run build",
     "verify": "npm test && npm run check && npm run typecheck && npm run check:typescript && npm run build && npm run check:demo && npm run check:scout && npm run check:docs && npm run build:scout && git diff --check",
     "prepublishOnly": "npm run verify",
-    "build:dash-preview": "node scripts/build-terminal-preview.mjs",
-    "deploy:dash-preview": "node scripts/deploy.ts terminal-preview",
     "scout": "node scout/dev.mjs",
     "scout:refresh": "node scout/refresh.mjs",
     "check:scout": "node --test scout/test/*.test.mjs scout/test/*.test.ts && node --check scout/worker.mjs",
@@ -98,8 +80,6 @@
     "@iarna/toml": "2.2.5",
     "@lezer/javascript": "1.5.6",
     "@lezer/python": "1.1.19",
-    "hyparquet": "1.31.5",
-    "typebox": "1.3.27",
     "yaml": "2.9.1"
   },
   "peerDependencies": {
@@ -117,10 +97,9 @@
     "@resvg/resvg-js": "2.6.2",
     "@types/markdown-it": "14.1.2",
     "@types/node": "^22.18.6",
-    "@xyflow/react": "12.12.0",
-    "elkjs": "0.12.0",
     "esbuild": "0.28.2",
     "fflate": "0.8.2",
+    "hyparquet": "1.31.5",
     "liveline": "0.0.7",
     "markdown-it": "15.0.2",
     "miniflare": "5.20260921.0-alpha",
@@ -129,6 +108,7 @@
     "react": "19.3.0",
     "react-dom": "19.3.0",
     "torph": "0.1.3",
+    "typebox": "1.3.27",
     "typescript": "^5.9.3",
     "web-vitals": "6.2.3",
     "wrangler": "4.136.3"

package/dist/core/src/index.d.ts

@@ -1,101 +0,0 @@
-import type { AnyIntentResult, AutomationPlan, AutomationSignal, CandidateAssessment, FrontierSnapshot, HarnessCapability, HarnessDescriptor, HarnessDispatchReceipt, HarnessWakeupRequest, TaskProfile, ImprovementCycleState, IntentLifecycleUpdate, IntentKind, OpenMeritIntent, OpenMeritProjectConfig, OpenMeritProjectState, ObservabilityCoverage, JsonValue, SupervisedGraduationPolicy } from "../../protocol/src/index.ts";
-import type { OpenMeritStore } from "./store.ts";
-export { OPENMERIT_DIRECTORY, ProjectStore, type AuditEventExporter, type OpenMeritStore, type ProjectStoreOptions, } from "./store.ts";
-export interface HarnessAdapter {
-    readonly descriptor: HarnessDescriptor;
-    dispatch(intent: OpenMeritIntent): Promise<HarnessDispatchReceipt>;
-    scheduleWakeup?(request: HarnessWakeupRequest): Promise<HarnessDispatchReceipt>;
-}
-export interface IntentFactoryOptions {
-    readonly taskProfileId?: string | undefined;
-    readonly authorizationPolicyId?: string;
-    readonly policy?: SupervisedGraduationPolicy | undefined;
-    readonly id?: string;
-    readonly requestedAt?: string;
-    readonly evaluationSpend?: number | undefined;
-    readonly targetId?: string | undefined;
-    readonly additionalConstraints?: Readonly<Record<string, JsonValue>>;
-}
-export declare function createOpenMeritIntent(kind: IntentKind, options?: IntentFactoryOptions): OpenMeritIntent;
-export declare function missingHarnessCapabilities(descriptor: HarnessDescriptor, kind: IntentKind): readonly HarnessCapability[];
-export declare function validateHarnessDescriptor(descriptor: HarnessDescriptor): VerificationResult;
-export declare function dispatchToHarness(adapter: HarnessAdapter, intent: OpenMeritIntent): Promise<HarnessDispatchReceipt>;
-export declare function startedStage(kind: IntentKind): ImprovementCycleState["stage"];
-export declare function completedStage(kind: IntentKind, succeeded: boolean): ImprovementCycleState["stage"];
-export interface VerificationResult {
-    readonly valid: boolean;
-    readonly errors: readonly string[];
-}
-export interface ResultInterpretation extends VerificationResult {
-    readonly nextStage?: ImprovementCycleState["stage"];
-    readonly config?: OpenMeritProjectConfig | undefined;
-    readonly cyclePatch?: Partial<ImprovementCycleState> | undefined;
-}
-export declare function validateObservabilityCoverage(profile: TaskProfile, coverage: ObservabilityCoverage, requireReady: boolean): readonly string[];
-export declare function interpretIntentResult(intent: OpenMeritIntent, result: AnyIntentResult, cycle: ImprovementCycleState, config: OpenMeritProjectConfig): ResultInterpretation;
-export type OrchestrationDecision = {
-    readonly action: "issue_intent";
-    readonly intentKind: IntentKind;
-    readonly reason: string;
-} | {
-    readonly action: "await_user";
-    readonly reason: string;
-} | {
-    readonly action: "observe";
-    readonly reason: string;
-};
-export declare function nextImprovementAction(state: ImprovementCycleState): OrchestrationDecision;
-export declare function verifyFrontierSnapshot(profile: TaskProfile, assessments: readonly CandidateAssessment[], snapshot: FrontierSnapshot): VerificationResult;
-export interface IssuedIntent {
-    readonly intent: OpenMeritIntent;
-    readonly state: OpenMeritProjectState;
-    readonly receipt: HarnessDispatchReceipt;
-}
-export interface CoordinatorIssueOptions {
-    readonly additionalConstraints?: Readonly<Record<string, JsonValue>> | undefined;
-}
-export interface AcceptedIntentResult {
-    readonly intentKind: IntentKind;
-    readonly state: OpenMeritProjectState;
-    readonly interpretation: ResultInterpretation;
-    readonly automation: {
-        readonly plan: AutomationPlan;
-        readonly receipt?: HarnessDispatchReceipt;
-    };
-}
-export interface ObservationResult {
-    readonly state: OpenMeritProjectState;
-    readonly assessmentDue: boolean;
-}
-export interface CatalogChangeResult {
-    readonly state: OpenMeritProjectState;
-    readonly changed: boolean;
-    readonly reassessmentDue: boolean;
-}
-export interface AutomationSignalResult {
-    readonly state: OpenMeritProjectState;
-    readonly duplicate: boolean;
-    readonly decision: OrchestrationDecision;
-}
-export declare function buildAutomationPlan(state: OpenMeritProjectState, descriptor: HarnessDescriptor): AutomationPlan;
-/** Harness-neutral durable coordinator. Adapters transport jobs; this class owns lifecycle state. */
-export declare class OpenMeritCoordinator {
-    private readonly store;
-    private readonly adapter;
-    constructor(store: OpenMeritStore, adapter: HarnessAdapter);
-    issue(kind: IntentKind, options?: CoordinatorIssueOptions): Promise<IssuedIntent>;
-    reconcileAutomation(now?: string): Promise<{
-        readonly plan: AutomationPlan;
-        readonly receipt?: HarnessDispatchReceipt;
-    }>;
-    acceptResult(result: AnyIntentResult): Promise<AcceptedIntentResult>;
-    recordLifecycle(update: IntentLifecycleUpdate): Promise<void>;
-    recordTaskObservation(details?: Readonly<Record<string, JsonValue>>): Promise<ObservationResult>;
-    assessBaseline(now?: string): Promise<{
-        readonly state: OpenMeritProjectState;
-        readonly errors: readonly string[];
-        readonly decision: OrchestrationDecision;
-    }>;
-    recordAutomationSignal(signal: AutomationSignal): Promise<AutomationSignalResult>;
-    recordModelCatalogFingerprint(fingerprint: string): Promise<CatalogChangeResult>;
-}