openmerit 0.1.4 → 0.1.6-preview.0

package/docs/adapter-guide.md

@@ -0,0 +1,189 @@
+# Building a harness adapter
+
+OpenMerit adapters are transport bridges. They do not reproduce OpenMerit's lifecycle, validation, persistence, or Pareto rules, and OpenMerit does not reproduce the harness's intelligence.
+
+## Required components
+
+An adapter provides a `HarnessAdapter` with:
+
+- a stable harness descriptor;
+- the OpenMerit protocol versions it supports;
+- declared capabilities;
+- supported execution modes;
+- a `dispatch` method that accepts a canonical `OpenMeritIntent` and returns an acceptance receipt.
+
+```ts
+import {
+  OpenMeritCoordinator,
+  ProjectStore,
+  type HarnessAdapter,
+} from "openmerit/core";
+import { PROTOCOL_VERSION, OPENMERIT_SCHEMA_DIALECT } from "openmerit/protocol";
+
+const adapter: HarnessAdapter = {
+  descriptor: {
+    id: "my-harness",
+    name: "My coding harness",
+    version: "1.0.0",
+    protocolVersions: [PROTOCOL_VERSION],
+    capabilities: [
+      "user_confirmation",
+      "evaluation_authoring",
+      "observability_instrumentation",
+      "production_observation",
+      "candidate_discovery",
+      "challenger_execution",
+      "frontier_calculation",
+      "model_mutation",
+      "post_swap_verification",
+      "rollback",
+    ],
+    executionModes: ["interactive"],
+    structuredOutput: {
+      schemaDialect: OPENMERIT_SCHEMA_DIALECT,
+      presentation: "adapter_validation",
+      enforcement: "adapter",
+    },
+    automation: {
+      supportedSignals: ["product_task_completed", "metric_window_available", "scheduled_tick", "model_catalog_changed", "verification_window_completed"],
+      persistentScheduling: false,
+      backgroundExecution: false,
+      wakeupProvisioning: "none",
+    },
+  },
+  async dispatch(intent) {
+    await myHarness.enqueue(intent);
+    return { accepted: true, harnessJobId: intent.id };
+  },
+};
+
+const coordinator = new OpenMeritCoordinator(
+  new ProjectStore(process.cwd()),
+  adapter,
+);
+```
+
+`ProjectStore` is the included local-filesystem backend. A remote or hosted integration can implement the `OpenMeritStore` interface instead.
+
+The confirmed task profile identifies the application LLM call under evaluation. Every metric, assessment, candidate, frontier artifact, and automation signal must carry that target ID; the harness's own model usage is orchestration data and must not be reported as product evidence. Application-provider cost, token, and latency telemetry belongs on the application signal or metric record.
+
+In the unreleased source checkout, setup may return `preliminaryModelLeads`: model IDs with sourced price estimates and reputation summaries. The field is optional under protocol 0.1 for older adapters. Core stores the leads and includes both the confirmed task profile and these leads in later discovery intent constraints, so a headless harness run does not depend on prior chat context. Leads are research priors, never measured candidate assessments.
+
+## External telemetry export
+
+OpenMerit always writes its canonical, redacted audit stream locally. A host integration may pass `eventExporters` to `ProjectStore` to mirror those same safe events to OpenTelemetry, Langfuse, LangSmith, or another observability backend:
+
+```ts
+const store = new ProjectStore(projectRoot, {
+  eventExporters: [{
+    id: "my-otel-bridge",
+    async export(event) {
+      await telemetry.emit("openmerit.audit", event);
+    },
+  }],
+});
+```
+
+Exporters are not the durable source of truth. They are best effort, and failures are recorded locally without blocking OpenMerit decisions.
+
+## Dispatching work
+
+```ts
+const issued = await coordinator.issue("establish_evals");
+```
+
+The coordinator builds the canonical intent, checks the descriptor's protocol and capabilities, creates durable state, records audit events, and calls the adapter. The harness decides how to execute the job.
+
+If a required capability is absent, dispatch is rejected before harness work starts.
+
+## Reporting progress
+
+A harness may send durable lifecycle progress through `recordLifecycle`:
+
+```ts
+await coordinator.recordLifecycle({
+  protocolVersion: PROTOCOL_VERSION,
+  intentId: issued.intent.id,
+  status: "running",
+  occurredAt: new Date().toISOString(),
+  message: "Generating representative evaluation cases",
+});
+```
+
+Use `waiting_for_user` when an interactive confirmation is required but cannot be completed in the current execution mode.
+
+## Completing work
+
+Each intent kind has a typed output in `IntentOutputMap`. Return an `IntentResult<K>` with durable evidence references, then submit it to the coordinator:
+
+```ts
+await coordinator.acceptResult(result);
+```
+
+The coordinator validates the active intent, protocol version, evidence references, and intent-specific output, including applicable frontier and candidate-identity checks, before updating lifecycle state. Adapters must not duplicate this logic. The harness remains responsible for performing only the authorized work; result validation does not sandbox its tools or authenticate external evidence.
+
+### Present the canonical result schema
+
+Import `completionToolSchemaFor(intent.kind)` from `openmerit/protocol` and present it through the harness's strongest structured-output mechanism. Do not recreate result schemas inside an adapter. A harness that supports changing tool definitions should specialize the completion tool whenever an intent becomes active; other harnesses may expose intent-specific tools or perform adapter-side validation.
+
+Declare the behavior in `HarnessDescriptor.structuredOutput`:
+
+- `schemaDialect` identifies the canonical OpenMerit JSON Schema dialect;
+- `presentation` states whether the adapter uses a dynamic tool, a static union, or adapter validation;
+- `enforcement` states whether validation occurs at the provider and harness, harness only, or adapter only.
+
+Schema validation catches malformed structure. The coordinator still owns semantic checks such as authorized candidate identity, metric readiness for frontier comparisons, lifecycle consistency, and Pareto conformance.
+
+## Reporting normal work and catalogue changes
+
+After a real product task completes:
+
+```ts
+await coordinator.recordTaskObservation({
+  modelId: "vendor/model",
+});
+
+// In the unreleased source checkout, core has already checked baseline
+// sufficiency when the confirmed cadence became due. Inspect durable state
+// and dispatch only a pending downstream intent, if any.
+```
+
+`recordTaskObservation` resolves the confirmed application target from project state. New adapters should prefer `recordAutomationSignal` so the task signal can carry the explicit `targetId`, durable evidence, and application-provider telemetry.
+
+When the harness's available model catalogue changes, compute a stable fingerprint and call `recordModelCatalogFingerprint`. The returned `reassessmentDue` flag tells the adapter whether the lifecycle justifies candidate discovery.
+
+New adapters should use `recordAutomationSignal` for product tasks, metric windows, scheduled ticks, catalogue observations, and verification windows. Signal IDs must remain stable across delivery retries. If the returned decision issues an intent, dispatch it through the coordinator. `recordTaskObservation` and `recordModelCatalogFingerprint` remain compatibility helpers.
+
+Call `reconcileAutomation` after configuration changes or when restoring a project. It returns an `AutomationPlan`. Implement `scheduleWakeup` only when the harness can genuinely arrange the requested wakeup. Declare persistent scheduling, background execution, supported signal sources, and provisioning requirements accurately in `HarnessDescriptor.automation`.
+
+If persistent scheduling is unavailable, expose the plan's active-session or external-scheduler fallback. Do not report a seven-day schedule as configured merely because the adapter checks policy when an interactive session starts.
+
+## Capability mapping
+
+| Intent | Required capability |
+| --- | --- |
+| `establish_evals` | confirmation, evaluation authoring, observability instrumentation |
+| `instrument_observability` | observability instrumentation |
+| `run_assessment` | production observation |
+| `discover_candidates` | candidate discovery |
+| `run_challenger_trials` | challenger execution |
+| `calculate_frontier` | frontier calculation |
+| `investigate_regression` | production observation |
+| `apply_model_swap` | model mutation |
+| `verify_model_swap` | post-swap verification |
+| `rollback_model_swap` | rollback |
+
+## Conformance checklist
+
+An adapter is ready when it:
+
+1. advertises only capabilities it can actually execute;
+2. preserves intent IDs and protocol versions;
+3. never reports success without durable evidence;
+4. returns the typed output for every supported intent;
+5. performs model mutation only when the intent explicitly authorizes it;
+6. reports ordinary product tasks without counting OpenMerit orchestration turns;
+7. passes a durable issue-and-complete cycle using `OpenMeritCoordinator`;
+8. proves the requested side effect, not merely a generated description.
+
+The fake non-Pi adapter in `packages/core/test/adapter-conformance.test.ts` is the executable reference.

package/docs/architecture.md

@@ -0,0 +1,59 @@
+# Architecture
+
+## Design principle
+
+OpenMerit is an orchestrator, not an intelligence layer. The product application LLM call is the system under test; the coding harness is the evaluator and execution environment.
+
+```text
+User or lifecycle event
+        |
+        v
+OpenMerit deterministic orchestration
+        |
+        | typed intent: outcome, constraints, evidence
+        v
+Coding harness such as Pi
+        |
+        | intelligent work, tools, evals, calculation
+        v
+Typed result and durable evidence references
+        |
+        v
+OpenMerit validation, policy, persistence, next nudge
+```
+
+An automatic OpenMerit action means automatic issuance of an approved intent. OpenMerit does not perform candidate research, evaluate quality, calculate a frontier, or mutate model configuration.
+
+## Packages
+
+### `openmerit/protocol`
+
+Defines the JSON-safe boundary: durable intents, evidence references, metrics, task profiles, candidate assessments, frontier snapshots, budgets, policies, state, and audit events.
+
+It also owns the versioned runtime schema registry for all intent results. Harness adapters consume these schemas rather than defining their own result shapes, report their structured-output capabilities, and may present the active schema using harness-native tools or adapter validation. A confirmed application LLM target is carried through profiles, metrics, candidates, frontier results, automation signals, and swaps; the harness model is not a valid substitute target.
+
+### `openmerit/core`
+
+Contains deterministic behavior: choosing the next lifecycle intent according to confirmation and rollback policy, persisting state, and checking harness-produced frontiers against the normative definition. The verifier rejects invalid harness results; it is not a model-selection engine.
+
+`OpenMeritCoordinator` is the reusable adapter boundary. It constructs canonical intents, negotiates capabilities, records lifecycle progress and observations, validates results, and advances durable state. `OpenMeritStore` makes persistence replaceable; `ProjectStore` is the bundled filesystem backend.
+
+The coordinator also accepts idempotent application-target automation signals and evaluates the user-confirmed `AutomationCheckPolicy`. `buildAutomationPlan` tells an adapter which signals and wakeups are required and reports unsupported persistent scheduling honestly. OpenMerit never installs a scheduler or creates a paid cloud worker itself.
+
+### `openmerit/pi`
+
+Exposes the command surface, supplies private typed intent context, triggers Pi, accepts one structured completion for the active intent, records evidence, advances the lifecycle, and detects model-catalog changes.
+
+The Pi package implements `HarnessAdapter`. It does not define canonical intents, per-intent result contracts, stage transitions, or validation rules.
+
+### Inference Terminal (unreleased)
+
+`packages/terminal` supplies a standalone `openmerit dash` executable, passive collectors, a loopback read-only HTTP API, and bundled dark-only browser assets. It does not invoke the coordinator or harness. One collector cycle reads existing metadata and runtime inventory; browser clients receive the same cached snapshot. Imported activity, runtime counters, and sampled whole-machine resources retain their separate scopes. Missing measurements remain unknown, and fictional sample data is isolated from the real snapshot. See the [terminal guide](inference-terminal.md) for the supported sources and JSON contract.
+
+## Intelligence resources
+
+Models, web research, tools, evaluators, and fast decision systems such as Jev System One are harness resources. They are configured and invoked by Pi. This keeps OpenMerit provider-neutral and lets future adapters choose their own implementation.
+
+## Trust boundary
+
+The harness is trusted to perform the work and return honest evidence. OpenMerit reduces unsupported claims through structured outputs, evidence references, compatible profile revisions, complete comparisons, and conservative uncertainty handling. It cannot prove that an arbitrary external evidence URI contains truthful data.

package/docs/automation.md

@@ -0,0 +1,74 @@
+# Harness-neutral automation
+
+The user defines cadence, thresholds, budget, scheduling mode, and automation permission. OpenMerit stores that policy, evaluates durable signals, and decides whether a bounded intent is due. A harness adapter arranges native wakeups and reports events. The coding harness performs the resulting intelligent work.
+
+```text
+User-confirmed policy
+        ↓
+OpenMerit due-check engine
+        ↓
+Harness-native wakeup or lifecycle hook
+        ↓
+Standard automation signal
+        ↓
+No work due, or one typed OpenMerit intent
+```
+
+## Check policy
+
+`AutomationCheckPolicy` independently configures:
+
+- baseline assessment cadence;
+- frontier reassessment cadence and catalogue-change behavior;
+- regression-check cadence and per-metric relative deterioration thresholds;
+- post-swap verification windows;
+- a cooldown between mature-system checks;
+- active-session or persistent execution;
+- whether wakeup provisioning may make infrastructure changes.
+
+Task-count and elapsed-time triggers use OR semantics. A policy with `afterCompletedTasks: 100` and `afterElapsedSeconds: 604800` becomes due at the first threshold reached. A scheduled wakeup does not directly launch expensive work: it submits `scheduled_tick`, and OpenMerit evaluates policy and durable state before issuing an intent.
+
+In the unreleased source checkout, the baseline elapsed-time clock starts when setup or required instrumentation finishes. Ordinary task and metric signals do not restart that clock. OpenMerit retains the latest reported aggregate window for each application metric, including sample count and evidence references. It never sums possibly overlapping windows. At a due baseline check, core requires ready instrumentation and a finite measured value, enough samples, and an evidence reference for every required objective. An insufficient check records the gaps and waits for more data; sufficient evidence advances to candidate discovery.
+
+## Signals
+
+Adapters may report:
+
+- `product_task_completed`;
+- `metric_window_available`;
+- `scheduled_tick`;
+- `model_catalog_changed`;
+- `verification_window_completed`.
+
+Signal IDs are idempotency keys. OpenMerit persists a bounded history, does not count duplicate task completions twice, and keeps due work pending until an adapter successfully dispatches it.
+
+## Capability negotiation
+
+Every `HarnessDescriptor` declares supported signals, persistent scheduling, background execution, and whether wakeup provisioning is native, unavailable, or requires an infrastructure change. OpenMerit produces an `AutomationPlan` containing required signals, the next wakeup, scheduling strategy, and explicit gaps.
+
+If persistent scheduling is requested but unsupported, the plan says `external_scheduler_required`. The adapter must not claim that a schedule exists. Active-session adapters can submit `scheduled_tick` when they start. An external scheduler may also wake a non-interactive harness and submit the same signal.
+
+OpenMerit calls `scheduleWakeup` only when the confirmed policy requests persistent execution, the adapter declares persistent scheduling and background execution, and any infrastructure-changing provisioning has been authorized. Paid agents, system scheduler installation, and cloud infrastructure remain harness implementation details and authorization boundaries.
+
+## Intent mapping
+
+OpenMerit owns deterministic mapping from due conditions to intents:
+
+| Due condition | Intent |
+| --- | --- |
+| Baseline cadence | Core checks stored baseline-window sufficiency; if ready, `discover_candidates` |
+| Mature frontier cadence or changed catalogue | `discover_candidates` |
+| Regression cadence or crossed metric threshold | `investigate_regression` |
+| Completed post-swap window | `verify_model_swap` |
+
+Subsequent lifecycle transitions continue through candidate trials, frontier calculation, supervised swap, verification, or rollback. The harness executes those intents; OpenMerit validates their structured result and evidence.
+
+## Pi behavior
+
+Pi reports catalogue and scheduled-tick signals on session start. Application instrumentation submits product-task, metric-window, and post-swap verification signals through its typed `openmerit_report_signal` tool; Pi's own settled-turn telemetry is not treated as product evidence. Published `openmerit@0.1.5` cannot provision persistent background wakeups and reports an external-scheduler gap.
+
+In the unreleased source checkout on macOS, a confirmed policy with `execution.mode: "persistent"` and `infrastructureChangesAllowed: true` allows the Pi adapter to install a per-project user LaunchAgent. It uses the model selected in the interactive Pi session and checks that Pi authentication works without shell-only credentials. The agent checks local state every five minutes without calling a model. When a baseline check is due, the runner submits `scheduled_tick` directly to core: an insufficient result exits without starting Pi; a sufficient result starts Pi for candidate discovery. The Mac must be logged in; this is not a cloud service, an exact-time alarm, or a source of product metrics. Task-count and metric-window checks still depend on application signals.
+
+This background path is experimental in the source checkout. A local runner test verifies that an insufficient baseline check needs no Pi process. An earlier live OpenRouter/Pi test reached the former assessment intent and made tool calls, but did not complete it within the bounded window. Candidate discovery, comparison, and swapping from a background Pi run are not yet product-validated. A background Pi run is capped at 15 minutes and writes its exit code and any still-active intent ID to `.openmerit/scheduler-status.json`; an unfinished intent requires inspection and explicit recovery in Pi.
+
+`/openmerit pause` in this source checkout persists the pause, unloads the project's LaunchAgent, and prevents automatic due decisions while still recording incoming signals. `/openmerit resume` clears the pause and restores an authorized schedule. Neither command cancels already-active work. A declined or failed scheduler installation is reported as a gap, not success. On other platforms, persistent policies still require an external scheduler.

package/docs/budgets.md

@@ -0,0 +1,37 @@
+# Budgets and permissions
+
+Setup asks you to confirm the limits and permissions for evaluation work. OpenMerit stores the policy and passes its constraints to the harness.
+
+## Evaluation costs
+
+OpenMerit has no subscription fee. Model-provider charges still apply to setup, assessments, challenger trials, and verification. Paid tools, telemetry, or infrastructure used by your harness may add costs. Application-provider cost for ordinary product tasks can be reported separately as production evidence; Pi's own orchestration usage is not application cost and is not charged to the challenger-evaluation budget.
+
+## Budget fields
+
+| Field | Meaning |
+| --- | --- |
+| `currency` | Currency for the evaluation budget. |
+| `maximumSpend` | Spending limit the harness is instructed to respect. |
+| `maximumCandidateCount` | Optional limit on candidate count. |
+| `maximumRunsPerCandidate` | Optional limit on runs per candidate. |
+| `expiresAt` | Optional expiration for the evaluation budget. |
+
+In the unreleased development version, OpenMerit keeps a separate ledger for controlled challenger trials. Each challenger intent receives the amount already spent and the remaining allowance. Challenger results that exceed the approved limit are rejected, and another challenger intent cannot start after the limit is exhausted. Provider-side spending controls remain the strongest protection against a request that exceeds its estimate while it is running.
+
+Setup, baseline product work, assessments, and swap verification remain visible costs, but do not consume `evaluationBudget.maximumSpend`. This separation prevents normal product usage from silently exhausting the allowance reserved for comparing challengers.
+
+## Model changes
+
+OpenMerit normally waits for `/openmerit approve` before issuing a swap request. It can request an automatic swap when `automaticSwapsEnabled` is true and `confirmationRequiredUntilVerifiedSwaps` has been reached. That threshold may be zero.
+
+Post-swap evaluation remains part of the workflow. The harness performs the change and reports whether it passed evaluation. `rollbackOnRegression` determines whether a reported post-swap regression can trigger a rollback request.
+
+## Infrastructure changes
+
+Persistent scheduling is a separate permission. An adapter that provisions infrastructure must declare that capability, and the confirmed check policy must allow infrastructure changes before OpenMerit asks it to provision a wakeup.
+
+The current Pi adapter does not provide persistent scheduling or background execution. An external scheduler is needed to start Pi for unattended checks. See [automatic checks](automation.md).
+
+## What a policy can establish
+
+OpenMerit uses policy and lifecycle state to decide which requests to issue, then checks the results returned by the harness. It is not a sandbox around the harness’s tools or provider access. See [security](security.md) for the responsibility boundary.

package/docs/commands.md

@@ -0,0 +1,85 @@
+# Commands
+
+Run these slash commands inside Pi with the OpenMerit extension loaded. Shell commands such as `pi install` run in your terminal instead.
+
+## Inference Terminal (preview)
+
+The `openmerit@preview` npm package adds a shell command, `openmerit dash [path]`, for a passive local view of models, connections, usage, and compute. From this checkout, use `npm run dash -- [path]`. It is independent of Pi setup and does not run model comparisons or inference. Published `openmerit@0.1.5` does not include it.
+
+See the [Inference Terminal guide](inference-terminal.md) for local installation, automatic discovery, `--no-discovery`, `--json`, `--demo`, collection intervals, and data limitations. Discovery is part of `dash` and is scoped to application evidence. Coding-harness histories are not collected.
+
+## Inspect status
+
+```text
+/openmerit
+/openmerit status
+```
+
+Both show the task profile, metric readiness, evaluation status, active work, next assessment, lifecycle stage, incumbent candidate, candidate and assessment counts, challenger spend, proposed candidate, and pause flag. A readiness label is based on the harness’s structured reports; inspect the evidence for the underlying measurements.
+
+## Start or recover setup
+
+```text
+/openmerit setup
+```
+
+Requests setup explicitly or reopens setup to correct stale target/incumbent state. Accepted replacement setup clears prior candidate, assessment, frontier, and baseline handoffs while retaining the audit log.
+
+Request setup explicitly only when recovering, rerunning, or configuring a custom integration. In the unreleased source checkout, an unconfigured project starts setup when bounded source detection finds an application LLM call at session start or after a settled Pi build turn. OpenMerit refuses another intent while work is active.
+
+## Cancel interrupted work
+
+```text
+/openmerit cancel
+```
+
+Record the active intent as cancelled after an interrupted or abandoned harness run. This clears durable active-work state without claiming success and releases any interactive request that was queued behind setup.
+
+## Request an assessment
+
+```text
+/openmerit assess
+```
+
+In the unreleased source checkout, ask OpenMerit core to check stored baseline metric windows now. It reports missing or immature required metrics without starting Pi, or starts candidate discovery when they pass. Published `openmerit@0.1.5` still asks Pi to assess. Neither path creates missing measurements or overrides sample requirements.
+
+## Request a frontier
+
+```text
+/openmerit frontier
+```
+
+Ask the harness to calculate a model comparison now. OpenMerit validates the returned frontier against the confirmed profile and supplied assessments. The command does not itself authorize a model change.
+
+## Approve a proposal
+
+```text
+/openmerit approve
+```
+
+Approve the current proposal after its frontier result has passed validation. If there is no proposal in the expected lifecycle stage, the command reports that no verified proposal is awaiting approval. See [comparisons and swaps](lifecycle.md) for automatic-swap policy.
+
+## Pause and resume
+
+```text
+/openmerit pause
+/openmerit resume
+```
+
+In 0.1.5, pause suppresses some lifecycle follow-ups, but task, measurement, and session-start signals can still start due checks. The flag resets on process restart and does not cancel active work. Explicit commands still work. Do not treat “paused” as a guarantee that all OpenMerit work has stopped.
+
+## Locate logs and evidence
+
+```text
+/openmerit logs
+```
+
+Show the local audit, exporter-error, evidence-manifest, and state paths. See [project files and logs](operations.md) for their contents.
+
+## Check the installation
+
+```text
+/openmerit doctor
+```
+
+Show adapter diagnostics, UI availability, and scheduling support. This checks extension wiring; it does not prove provider authentication, successful evaluations, or a completed model swap.

package/docs/demo-backfill.md

@@ -0,0 +1,29 @@
+# Backfill recovery
+
+The [agent workflow](https://openmerit.site/demo?task=backfill) prepares a recovery manifest for an interrupted data backfill. Its goal is to resume recoverable partitions without duplicating committed data, quarantine malformed input, and stop for investigation when the failure is unknown or involves a checksum conflict. It never executes the manifest or writes to a database.
+
+## Workflow structure
+
+Three dependent model turns perform one goal: request the incident through `inspect_job`, request its authoritative ledger through `read_checkpoint`, then combine those results with the recovery policy into a manifest. The tools are read-only functions over fixed synthetic records. Their requests are JSON commands dispatched by the prepared adapter, not arbitrary code. The runner fixes the three-task sequence; the model supplies the job identifiers and final plan. Invalid requests become scored failures and are not silently repaired.
+
+The fixed agent uses GPT-4.1 mini for all three tasks. Its challenger is one prepared mixed-model route:
+
+| Task | Without OpenMerit | With the proposed route |
+| --- | --- | --- |
+| Inspect incident | GPT-4.1 mini | GPT-4.1 nano |
+| Read checkpoint | GPT-4.1 mini | GPT-4.1 mini |
+| Plan recovery | GPT-4.1 mini | Gemini 2.5 Flash Lite |
+
+Each model receives the accumulated context and tool results from preceding tasks. OpenMerit evaluates the **whole route** and can change the application's three-step model configuration after approval. The baseline remains fixed. The route is a supplied candidate, not an unrestricted search over all model combinations or evidence that each individual assignment is optimal. The workflow itself remains prepared in advance.
+
+## Evaluation policy
+
+Eight synthetic recovery goals cover stale runbooks, non-contiguous commits, malformed rows, completed jobs, checksum conflicts, unknown failures, and instructions embedded in incident notes. Eligibility requires all eight complete manifests to match, correct tool use and output structure on every goal, no committed or nonexistent partition scheduled for replay/quarantine, and all required evidence citations. Mean whole-goal latency must be at most eight seconds and mean whole-goal cost at most $0.003. These are constraints on the observed suite, not per-request timeouts or guarantees about other jobs. The lowest-cost eligible frontier candidate must also cost less than the incumbent. Six disjoint goals verify an approved change.
+
+Cost adds every model turn in each goal. Latency measures the complete sequential workflow, including tool dispatch. The `total_task_cost` assessment retains actual spend over the sample window, including every agent call. The per-goal budget is multiplied by the window size for comparison and by the six-goal verification size after a swap; the display divides actual spend by goal count. Provider-level call records, tool requests/results, checks, and manifests are retained in the run evidence. The recording includes the sample's actual tool trace. The evaluation records are `BF-4101` through `BF-4108`; their incidents and ledgers are fixed.
+
+## Recorded outcome
+
+In the recorded run, the fixed agent completed 6/8 goals and the mixed-model route completed 8/8. Mean whole-goal costs were $0.0007742 and $0.0004018625; latencies were 4.12s and 3.80s. This is about 48% lower cost and 8% lower latency in that run. All six post-swap goals passed, and a custom request returned the expected quarantine plan through both paths. The displayed sample preserves a baseline mistake: it scheduled the malformed partition for both replay and quarantine; the mixed route excluded it from replay. The recording's step models come from the actual provider calls and tool traces, not labels added to the earlier single-model recording. These are one run's observations, not promised outcomes.
+
+See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-fieldkit.md

@@ -0,0 +1,47 @@
+# Field kit · Six tasks
+
+[Watch the six-task field-kit recording](https://openmerit.site/demo/?task=fieldkit). The captured run compares two heterogeneous model routes using real OpenRouter calls through Pi and OpenMerit. Each route uses six distinct models; five assignments change and the feasibility model is retained.
+
+The field-kit workflow prepares an offline recording handoff from a Spanish technician brief, a synthetic equipment-label image, an HTML operating manual, and a Python capture script. Both paths already use **six different models**. The approved route pairs narrow specialists with broader coding, reasoning, and writing models:
+
+## Workflow structure
+
+| Task | Original mix | Approved mix | Capability being exercised |
+| --- | --- | --- | --- |
+| Translate brief | [Qwen3.6 Plus](https://openrouter.ai/qwen/qwen3.6-plus) | [Hy-MT2 1.8B](https://openrouter.ai/tencent/hy-mt2-1.8b) | Spanish → English translation; a compact translation specialist |
+| Read label | [Qwen3.6 Flash](https://openrouter.ai/qwen/qwen3.6-flash) | [Perceptron Mk1.5](https://openrouter.ai/perceptron/perceptron-mk1.5) | Visual perception from an actual PNG input |
+| Extract limits | [Gemini 3.1 Flash Lite](https://openrouter.ai/google/gemini-3.1-flash-lite) | [Schematron V2 Turbo](https://openrouter.ai/inference-net/schematron-v2-turbo) | Active-profile extraction from HTML; a 3B extraction model |
+| Inspect code | [MiMo-V2.5 Pro](https://openrouter.ai/xiaomi/mimo-v2.5-pro) | [Laguna XS 2.1](https://openrouter.ai/poolside/laguna-xs-2.1) | Static Python analysis; a coding model with 33B total / 3B active parameters |
+| Check feasibility | [Qwen3.7 Plus](https://openrouter.ai/qwen/qwen3.7-plus) | Qwen3.7 Plus (retained) | Reconcile the translated brief with equipment, operating limits, and code |
+| Write handoff | [Gemma 4 31B](https://openrouter.ai/google/gemma-4-31b-it) | [Gemma 4 26B A4B](https://openrouter.ai/google/gemma-4-26b-a4b-it) | Copy the decision into the final schema with ordered evidence |
+
+These are the supplied model assignments for this development workflow. Each route uses six distinct models; five assignments change and the reasoning model is retained. They describe the capabilities being exercised; the small suite does not establish broad capability rankings.
+
+## Handoff rules
+
+Six constraints govern the handoff: no network uploads, a compatible capture sample rate, enough battery energy, enough storage, completion by the same-day deadline, and a matching connector. Equality at a limit passes. The result must list every failing constraint in policy order, or return `ready` when none fail. No hardware is changed or dispatched, scripts are never executed, and no recording is uploaded.
+
+The runner fixes the six-stage sequence and dispatches read-only source adapters. Translation feeds feasibility; label and manual extraction provide equipment limits; code analysis reports the final sample rate and upload flag; feasibility combines those actual outputs; writing copies that decision and its evidence. Intermediate errors are retained and scored even if a later model happens to produce a correct final answer. Raw scripts and source documents are data, not instructions to change the workflow. The vision models receive the same image pixels, without a text transcription of the label. The HTML extractor receives its extraction instructions in the JSON Schema. Translation uses plain text; code analysis requests JSON without enforced response formatting because Laguna does not support that API parameter. Other structured tasks use JSON Schema. Both routes receive the same task inputs, encodings, and output budgets: 180 tokens for each of the first four tasks, then 2,048 for the feasibility model (including a low reasoning effort) and 350 for the handoff. Optional reasoning is disabled for the other tasks where supported. No failed answer is repaired or substituted.
+
+## Evaluation policy
+
+Four evaluation goals cover ready and blocked kits across all six constraints. Every structured intermediate output, the final manifest, and the evidence list must match. The translation is checked through the resulting duration, deadline, and offline decision rather than a verbatim wording match. The source-adapter completion check is reported as `tool_use_performance`; it does not imply free-form tool selection by the models. Mean whole-goal latency must be ≤45 seconds and inference cost ≤$0.006. Four disjoint goals verify the approved route, including exact resource and deadline boundaries. This smaller suite keeps six-turn goals within the existing 120-request, ten-minute, and $1.25 session bounds. The evaluation records are `KIT-8101` through `KIT-8104`.
+
+## Captured result
+
+The supervised local run was verified on September 28, 2026.
+
+| Path | Complete evaluation goals | Mean model cost per goal | Mean whole-goal latency |
+| --- | --- | --- | --- |
+| Original six-model mix | 4/4 | $0.001867339 | 29.49s |
+| Approved six-model mix | 4/4 | $0.001449088 | 25.63s |
+
+The approved route had **22% lower measured cost** and **13% lower mean latency**, with all intermediate and final checks passing. After operator approval, the application configuration changed and all 4/4 disjoint verification goals passed. The displayed sample is fieldkit-01; both paths return the same correct handoff. This demonstrates an observed efficiency improvement on the prepared suite, not an accuracy gain or a production reliability estimate.
+
+The recording is a compressed replay of local real-provider calls, not a fresh Cloudflare run. All application models were released within March 28–September 28, 2026; dated OpenRouter sources are included in Run details and the downloaded evidence. Build: `demo-20260928-fieldkit-local-v5`. Source export SHA-256: `9f0986917f41354bb8ce80421bb9a413825fb457eb37c78af119747880a1514b`. Costs exclude Pi orchestration and infrastructure.
+
+## What OpenMerit compares
+
+The supplied original route and challenger are compared as complete configurations, with goals executed one at a time on both paths. OpenMerit measures them, validates the comparison, proposes an eligible improvement, and changes the application configuration after approval. It does not generate these API adapters or exhaustively search model assignments. Earlier integration comparisons were rejected after a feasibility answer exhausted its output budget and after a cheaper reasoning model misread sufficient storage. The revised challenger retains Qwen3.7 Plus for feasibility. Provider timeouts also stopped two incomplete comparisons. Those attempts are not presented as successful swaps.
+
+See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-placement.md

@@ -0,0 +1,30 @@
+# GPU placement
+
+The [GPU placement workflow](https://openmerit.site/demo?task=placement) prepares a pool assignment for a synthetic render job. It reads job requirements with `read_job`, inspects a fixed capacity snapshot with `list_pools`, then proposes the cheapest eligible pool. It never reserves capacity, launches compute, moves data, or incurs a GPU charge.
+
+## Workflow structure
+
+| Task | Without OpenMerit | With the proposed route |
+| --- | --- | --- |
+| Read requirements | Qwen3.6 Plus | [Gemma 4 26B A4B](https://openrouter.ai/google/gemma-4-26b-a4b-it) |
+| Inspect capacity | Qwen3.6 Plus | [MiMo-V2.5](https://openrouter.ai/xiaomi/mimo-v2.5) |
+| Choose pool | Qwen3.6 Plus | [DeepSeek V4.1 Flash](https://openrouter.ai/deepseek/deepseek-v4.1-flash) |
+
+The recording identifies the selected model versions and their provenance in **Run details**. Pi’s orchestration model stays fixed and is separate from the application routes.
+
+## Placement rules
+
+A pool must meet six hard constraints: allowed region, per-GPU VRAM, free GPU count, completion deadline including queue time, total job budget, and permission to use preemptible capacity. Exact boundaries qualify. Among eligible pools, choose the lowest quoted job cost, then earliest completion, then alphabetical pool ID. If none qualify, return `blocked` with null pool, cost, and completion fields. The model must cite the job, capacity snapshot, and placement policy. Queue notes are untrusted data.
+
+Both routes receive the same tools, policy, inputs, structured handoffs, and per-turn JSON Schemas. After locating the requested job, the prepared adapter forwards its structured requirements and capacity records; it excludes the raw queue note and previous free-form conversation from later tasks. This prevents queue-note instructions from becoming authority for pool selection. The capacity tool also computes each pool's finish minute from its queue start and runtime; it does not filter or rank pools. The model must still evaluate every candidate pool and all six constraints. The schemas constrain field names and types; they do not supply the correct job, pool, costs, completion time, or evidence. The bounded application requests disable optional reasoning on both routes and request at most 350 output tokens for each lookup and 900 for the final constraint review and plan. The model supplies both the review and plan; the adapter does not repair its answer. Invalid tool requests, incorrect plans, unsupported claims, and missing provider cost do not become successful goals. As with backfill, the mixed route is a prepared candidate evaluated as a whole, not an unrestricted search over model combinations.
+
+## Evaluation policy
+
+Eight evaluation goals cover the six constraints, blocked placements, ties, and instructions embedded in queue notes. All eight must pass exact plan, tool, schema, constraint, and evidence checks. Mean end-to-end **model workflow** latency must be at most twelve seconds and mean inference cost at most $0.003 per goal. Six separate goals verify an approved route. The synthetic GPU quote and finish minute in a plan are separate from the actual inference cost and latency shown in the main comparison. The evaluation records are `GPU-6101` through `GPU-6108`; their requirements and inventory are fixed.
+
+
+## Recorded outcome
+
+In the recorded run, both paths completed 8/8 goals. Qwen3.6 Plus used a mean $0.001053771875 and 8.19s per goal; the mixed route used $0.000409093125 and 7.09s—about 61% lower cost and 13% lower latency. All six post-swap goals passed, with a 10.96s mean verification latency below the twelve-second requirement. A custom budget-constrained request correctly returned `blocked` through both paths. These are real local OpenRouter calls from one prepared run, not promised savings or production reliability estimates.
+
+See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-spam.md

@@ -0,0 +1,15 @@
+# Spam filter · Jev
+
+The [Jev task](https://openmerit.site/demo?task=spam) classifies a community comment as `publish` or `hold` under a narrow promotional-spam policy. Ordinary discussion, criticism, and reports of spam are allowed. Unsolicited promotional solicitations are held. The fixtures contain twelve examples of each class, with six separate verification comments.
+
+## Models and integration
+
+The challenger is [TypeSafe's Jev 1.13](https://openrouter.ai/docs/guides/community/jev), a model that returns typed decisions through OpenRouter's Decisions API. The baseline generates a JSON decision; Jev answers a prepared Choice question. Both receive the same comment and decision criteria, and the application normalizes both results to the same one-field schema. The API adapter is prepared in advance. This demo shows OpenMerit evaluating and switching the application configuration; it does not demonstrate automatically rewriting a chat application to use Jev. Pi's own harness model stays fixed.
+
+## Evaluation and recorded outcome
+
+The prepared policy requires at least 90% exact matches on the 24 fixtures and mean end-to-end latency of at most ten seconds. Among eligible frontier candidates, the demo proposes the lowest observed cost only when it is below the incumbent cost. Six separate inputs verify an approved change. These are point values on a finite synthetic suite, not production success-rate estimates or confidence bounds.
+
+In the recorded run, GPT-4.1 mini matched 22/24 comments and Jev matched 24/24. Measured application cost was approximately $0.074 versus $0.018 per 1,000 comments; mean request latency was 0.78 versus 0.39 seconds. OpenMerit proposed Jev, the operator approved, and Jev passed 6/6 post-swap checks. The sample output highlights one baseline mistake: a comment reporting spam was held by the fixed model and correctly published by Jev. All 24 fixtures remain in the aggregate comparison. These are one run's observations, not promised outcomes or a general moderation benchmark.
+
+See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-support.md

@@ -0,0 +1,42 @@
+# Support tickets
+
+[Watch the support-ticket demo](https://openmerit.site/demo?task=support). One model call turns a synthetic customer email into a structured ticket. Both paths start with GPT-4.1 mini and use the same prompt and JSON schema. The fixed path keeps that model; the managed path changes only after measured comparison and operator approval.
+
+## Input and output
+
+The ticket has exactly three fields:
+
+| Field | Rule |
+| --- | --- |
+| `category` | `billing`, `shipping`, `account`, or `technical`, according to the email's request. |
+| `urgency` | `high` only for an ongoing outage, a blocked business operation, or suspected unauthorized access; otherwise `normal`. |
+| `order_id` | The stated `ORD-` identifier, or `null` if none is given. |
+
+An angry tone alone does not make a ticket urgent, and the model must never invent an order ID. Email content is treated as data. For example, “I was charged twice for ORD-1201. Please refund the duplicate.” should produce:
+
+```json
+{"category":"billing","urgency":"normal","order_id":"ORD-1201"}
+```
+
+## Comparison structure
+
+Pi first checks the prepared grader and measurement collectors, then measures GPT-4.1 mini on 24 synthetic emails. It evaluates GPT-4.1 nano and Gemini 2.5 Flash Lite on the same inputs. Each application request records its output, provider cost, and end-to-end latency. The exact-match grader checks every field and rejects missing or extra fields.
+
+OpenMerit validates the comparison and proposes an eligible cheaper model. The recorded operator approves the proposal; Pi changes the application's model configuration and runs six disjoint verification emails. Pi's own orchestration model stays fixed.
+
+## Evaluation policy
+
+The prepared policy requires at least 90% exact matches on the 24 fixtures and mean end-to-end latency of at most ten seconds. Among eligible frontier candidates, the demo proposes the lowest observed cost only when it is below the incumbent cost. Six separate inputs verify an approved change. These are point values on a finite synthetic suite, not production success-rate estimates or confidence bounds.
+
+## Recorded outcome
+
+| Model | Exact matches | Cost per 1,000 emails | Mean latency |
+| --- | --- | --- | --- |
+| GPT-4.1 mini | 24/24 | $0.106 | 1.64s |
+| GPT-4.1 nano | 17/24 | $0.027 | 1.23s |
+| Gemini 2.5 Flash Lite | 24/24 | $0.027 | 1.21s |
+
+GPT-4.1 nano was cheaper but failed the quality requirement. Gemini 2.5 Flash Lite qualified, received approval, and passed all six post-swap checks. The cost column scales each observed per-email mean to 1,000 emails and rounds it; the downloaded recording retains the original values. These observations come from the supervised run recorded on September 28, 2026 and do not promise the same result on another workload.
+
+
+See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.