openmerit 0.1.11 → 0.2.0

package/docs/automation.md

@@ -1,74 +0,0 @@
-# Harness-neutral automation
-
-The user defines cadence, thresholds, budget, scheduling mode, and automation permission. OpenMerit stores that policy, evaluates durable signals, and decides whether a bounded intent is due. A harness adapter arranges native wakeups and reports events. The coding harness performs the resulting intelligent work.
-
-```text
-User-confirmed policy
-        ↓
-OpenMerit due-check engine
-        ↓
-Harness-native wakeup or lifecycle hook
-        ↓
-Standard automation signal
-        ↓
-No work due, or one typed OpenMerit intent
-```
-
-## Check policy
-
-`AutomationCheckPolicy` independently configures:
-
-- baseline assessment cadence;
-- frontier reassessment cadence and catalogue-change behavior;
-- regression-check cadence and per-metric relative deterioration thresholds;
-- post-swap verification windows;
-- a cooldown between mature-system checks;
-- active-session or persistent execution;
-- whether wakeup provisioning may make infrastructure changes.
-
-Task-count and elapsed-time triggers use OR semantics. A policy with `afterCompletedTasks: 100` and `afterElapsedSeconds: 604800` becomes due at the first threshold reached. A scheduled wakeup does not directly launch expensive work: it submits `scheduled_tick`, and OpenMerit evaluates policy and durable state before issuing an intent.
-
-The baseline elapsed-time clock starts when setup or required instrumentation finishes. Ordinary task and metric signals do not restart that clock. OpenMerit retains the latest reported aggregate window for each application metric, including sample count and evidence references. It never sums possibly overlapping windows. At a due baseline check, core requires ready instrumentation and a finite measured value, enough samples, and an evidence reference for every required objective. An insufficient check records the gaps and waits for more data; sufficient evidence advances to candidate discovery.
-
-## Signals
-
-Adapters may report:
-
-- `product_task_completed`;
-- `metric_window_available`;
-- `scheduled_tick`;
-- `model_catalog_changed`;
-- `verification_window_completed`.
-
-Signal IDs are idempotency keys. OpenMerit persists a bounded history, does not count duplicate task completions twice, and keeps due work pending until an adapter successfully dispatches it.
-
-## Capability negotiation
-
-Every `HarnessDescriptor` declares supported signals, persistent scheduling, background execution, and whether wakeup provisioning is native, unavailable, or requires an infrastructure change. OpenMerit produces an `AutomationPlan` containing required signals, the next wakeup, scheduling strategy, and explicit gaps.
-
-If persistent scheduling is requested but unsupported, the plan says `external_scheduler_required`. The adapter must not claim that a schedule exists. Active-session adapters can submit `scheduled_tick` when they start. An external scheduler may also wake a non-interactive harness and submit the same signal.
-
-OpenMerit calls `scheduleWakeup` only when the confirmed policy requests persistent execution, the adapter declares persistent scheduling and background execution, and any infrastructure-changing provisioning has been authorized. Paid agents, system scheduler installation, and cloud infrastructure remain harness implementation details and authorization boundaries.
-
-## Intent mapping
-
-OpenMerit owns deterministic mapping from due conditions to intents:
-
-| Due condition | Intent |
-| --- | --- |
-| Baseline cadence | Core checks stored baseline-window sufficiency; if ready, `discover_candidates` |
-| Mature frontier cadence or changed catalogue | `discover_candidates` |
-| Regression cadence or crossed metric threshold | `investigate_regression` |
-| Completed post-swap window | `verify_model_swap` |
-
-Subsequent lifecycle transitions continue through candidate trials, frontier calculation, supervised swap, verification, or rollback. The harness executes those intents; OpenMerit validates their structured result and evidence.
-
-## Pi behavior
-
-Pi reports catalogue and scheduled-tick signals on session start. Application instrumentation submits product-task, metric-window, and post-swap verification signals through its typed `openmerit_report_signal` tool; Pi's own settled-turn telemetry is not treated as product evidence.
-
-On macOS, a confirmed policy with `execution.mode: "persistent"` and `infrastructureChangesAllowed: true` allows the Pi adapter to install a per-project user LaunchAgent. It uses the model selected in the interactive Pi session and checks that Pi authentication works without shell-only credentials. The agent checks local state every five minutes without calling a model. When a baseline check is due, the runner submits `scheduled_tick` directly to core: an insufficient result exits without starting Pi; a sufficient result starts Pi for candidate discovery. The Mac must be logged in; this is not a cloud service, an exact-time alarm, or a source of product metrics. Task-count and metric-window checks still depend on application signals.
-
-Background candidate work remains experimental. A local runner test verifies that an insufficient baseline check needs no Pi process. Candidate discovery, comparison, and swapping from a background Pi run are not yet product-validated. A background Pi run is capped at 15 minutes and writes its exit code and any still-active intent ID to `.openmerit/scheduler-status.json`; an unfinished intent requires inspection and explicit recovery in Pi.
-
-`/openmerit pause` persists the pause, unloads the project's LaunchAgent, and prevents automatic due decisions while still recording incoming signals. `/openmerit resume` clears the pause and restores an authorized schedule. Neither command cancels already-active work. A declined or failed scheduler installation is reported as a gap, not success. On other platforms, persistent policies still require an external scheduler.

package/docs/budgets.md

@@ -1,37 +0,0 @@
-# Budgets and permissions
-
-Setup asks you to confirm the limits and permissions for evaluation work. OpenMerit stores the policy and passes its constraints to the harness.
-
-## Evaluation costs
-
-OpenMerit has no subscription fee. Model-provider charges still apply to setup, assessments, challenger trials, and verification. Paid tools, telemetry, or infrastructure used by your harness may add costs. Application-provider cost for ordinary product tasks can be reported separately as production evidence; Pi's own orchestration usage is not application cost and is not charged to the challenger-evaluation budget.
-
-## Budget fields
-
-| Field | Meaning |
-| --- | --- |
-| `currency` | Currency for the evaluation budget. |
-| `maximumSpend` | Spending limit the harness is instructed to respect. |
-| `maximumCandidateCount` | Optional limit on candidate count. |
-| `maximumRunsPerCandidate` | Optional limit on runs per candidate. |
-| `expiresAt` | Optional expiration for the evaluation budget. |
-
-OpenMerit keeps a separate ledger for controlled challenger trials. Each challenger intent receives the amount already spent and the remaining allowance. Challenger results that exceed the approved limit are rejected, and another challenger intent cannot start after the limit is exhausted. Provider-side spending controls remain the strongest protection against a request that exceeds its estimate while it is running.
-
-Setup, baseline product work, assessments, and swap verification remain visible costs, but do not consume `evaluationBudget.maximumSpend`. This separation prevents normal product usage from silently exhausting the allowance reserved for comparing challengers.
-
-## Model changes
-
-OpenMerit normally waits for `/openmerit approve` before issuing a swap request. It can request an automatic swap when `automaticSwapsEnabled` is true and `confirmationRequiredUntilVerifiedSwaps` has been reached. That threshold may be zero.
-
-Post-swap evaluation remains part of the workflow. The harness performs the change and reports whether it passed evaluation. `rollbackOnRegression` determines whether a reported post-swap regression can trigger a rollback request.
-
-## Infrastructure changes
-
-Persistent scheduling is a separate permission. An adapter that provisions infrastructure must declare that capability, and the confirmed check policy must allow infrastructure changes before OpenMerit asks it to provision a wakeup.
-
-On macOS, the Pi adapter can install a per-project user LaunchAgent when persistent execution and infrastructure changes are both authorized. The user must remain logged in and Pi authentication must work in the background environment. Other platforms require an external scheduler. Background candidate work still needs representative product validation. See [automatic checks](automation.md).
-
-## What a policy can establish
-
-OpenMerit uses policy and lifecycle state to decide which requests to issue, then checks the results returned by the harness. It is not a sandbox around the harness’s tools or provider access. See [security](security.md) for the responsibility boundary.

package/docs/demo-backfill.md

@@ -1,31 +0,0 @@
-# Backfill recovery
-
-> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
-
-The agent workflow in the checkout’s recorded player (`site/demo/`) prepares a recovery manifest for an interrupted data backfill. Its goal is to resume recoverable partitions without duplicating committed data, quarantine malformed input, and stop for investigation when the failure is unknown or involves a checksum conflict. It never executes the manifest or writes to a database.
-
-## Workflow structure
-
-Three dependent model turns perform one goal: request the incident through `inspect_job`, request its authoritative ledger through `read_checkpoint`, then combine those results with the recovery policy into a manifest. The tools are read-only functions over fixed synthetic records. Their requests are JSON commands dispatched by the prepared adapter, not arbitrary code. The runner fixes the three-task sequence; the model supplies the job identifiers and final plan. Invalid requests become scored failures and are not silently repaired.
-
-The fixed agent uses GPT-4.1 mini for all three tasks. Its challenger is one prepared mixed-model route:
-
-| Task | Without OpenMerit | With the proposed route |
-| --- | --- | --- |
-| Inspect incident | GPT-4.1 mini | GPT-4.1 nano |
-| Read checkpoint | GPT-4.1 mini | GPT-4.1 mini |
-| Plan recovery | GPT-4.1 mini | Gemini 2.5 Flash Lite |
-
-Each model receives the accumulated context and tool results from preceding tasks. OpenMerit evaluates the **whole route** and can change the application's three-step model configuration after approval. The baseline remains fixed. The route is a supplied candidate, not an unrestricted search over all model combinations or evidence that each individual assignment is optimal. The workflow itself remains prepared in advance.
-
-## Evaluation policy
-
-Eight synthetic recovery goals cover stale runbooks, non-contiguous commits, malformed rows, completed jobs, checksum conflicts, unknown failures, and instructions embedded in incident notes. Eligibility requires all eight complete manifests to match, correct tool use and output structure on every goal, no committed or nonexistent partition scheduled for replay/quarantine, and all required evidence citations. Mean whole-goal latency must be at most eight seconds and mean whole-goal cost at most $0.003. These are constraints on the observed suite, not per-request timeouts or guarantees about other jobs. The lowest-cost eligible frontier candidate must also cost less than the incumbent. Six disjoint goals verify an approved change.
-
-Cost adds every model turn in each goal. Latency measures the complete sequential workflow, including tool dispatch. The `total_task_cost` assessment retains actual spend over the sample window, including every agent call. The per-goal budget is multiplied by the window size for comparison and by the six-goal verification size after a swap; the display divides actual spend by goal count. Provider-level call records, tool requests/results, checks, and manifests are retained in the run evidence. The recording includes the sample's actual tool trace. The evaluation records are `BF-4101` through `BF-4108`; their incidents and ledgers are fixed.
-
-## Recorded outcome
-
-In the recorded run, the fixed agent completed 6/8 goals and the mixed-model route completed 8/8. Mean whole-goal costs were $0.0007742 and $0.0004018625; latencies were 4.12s and 3.80s. This is about 48% lower cost and 8% lower latency in that run. All six post-swap goals passed, and a custom request returned the expected quarantine plan through both paths. The displayed sample preserves a baseline mistake: it scheduled the malformed partition for both replay and quarantine; the mixed route excluded it from replay. The recording's step models come from the actual provider calls and tool traces, not labels added to the earlier single-model recording. These are one run's observations, not promised outcomes.
-
-See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-fieldkit.md

@@ -1,49 +0,0 @@
-# Field kit · Six tasks
-
-> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
-
-Watch the six-task field-kit recording in the checkout’s recorded player (`site/demo/`). The captured run compares two heterogeneous model routes using real OpenRouter calls through Pi and OpenMerit. Each route uses six distinct models; five assignments change and the feasibility model is retained.
-
-The field-kit workflow prepares an offline recording handoff from a Spanish technician brief, a synthetic equipment-label image, an HTML operating manual, and a Python capture script. Both paths already use **six different models**. The approved route pairs narrow specialists with broader coding, reasoning, and writing models:
-
-## Workflow structure
-
-| Task | Original mix | Approved mix | Capability being exercised |
-| --- | --- | --- | --- |
-| Translate brief | [Qwen3.6 Plus](https://openrouter.ai/qwen/qwen3.6-plus) | [Hy-MT2 1.8B](https://openrouter.ai/tencent/hy-mt2-1.8b) | Spanish → English translation; a compact translation specialist |
-| Read label | [Qwen3.6 Flash](https://openrouter.ai/qwen/qwen3.6-flash) | [Perceptron Mk1.5](https://openrouter.ai/perceptron/perceptron-mk1.5) | Visual perception from an actual PNG input |
-| Extract limits | [Gemini 3.1 Flash Lite](https://openrouter.ai/google/gemini-3.1-flash-lite) | [Schematron V2 Turbo](https://openrouter.ai/inference-net/schematron-v2-turbo) | Active-profile extraction from HTML; a 3B extraction model |
-| Inspect code | [MiMo-V2.5 Pro](https://openrouter.ai/xiaomi/mimo-v2.5-pro) | [Laguna XS 2.1](https://openrouter.ai/poolside/laguna-xs-2.1) | Static Python analysis; a coding model with 33B total / 3B active parameters |
-| Check feasibility | [Qwen3.7 Plus](https://openrouter.ai/qwen/qwen3.7-plus) | Qwen3.7 Plus (retained) | Reconcile the translated brief with equipment, operating limits, and code |
-| Write handoff | [Gemma 4 31B](https://openrouter.ai/google/gemma-4-31b-it) | [Gemma 4 26B A4B](https://openrouter.ai/google/gemma-4-26b-a4b-it) | Copy the decision into the final schema with ordered evidence |
-
-These are the supplied model assignments for this development workflow. Each route uses six distinct models; five assignments change and the reasoning model is retained. They describe the capabilities being exercised; the small suite does not establish broad capability rankings.
-
-## Handoff rules
-
-Six constraints govern the handoff: no network uploads, a compatible capture sample rate, enough battery energy, enough storage, completion by the same-day deadline, and a matching connector. Equality at a limit passes. The result must list every failing constraint in policy order, or return `ready` when none fail. No hardware is changed or dispatched, scripts are never executed, and no recording is uploaded.
-
-The runner fixes the six-stage sequence and dispatches read-only source adapters. Translation feeds feasibility; label and manual extraction provide equipment limits; code analysis reports the final sample rate and upload flag; feasibility combines those actual outputs; writing copies that decision and its evidence. Intermediate errors are retained and scored even if a later model happens to produce a correct final answer. Raw scripts and source documents are data, not instructions to change the workflow. The vision models receive the same image pixels, without a text transcription of the label. The HTML extractor receives its extraction instructions in the JSON Schema. Translation uses plain text; code analysis requests JSON without enforced response formatting because Laguna does not support that API parameter. Other structured tasks use JSON Schema. Both routes receive the same task inputs, encodings, and output budgets: 180 tokens for each of the first four tasks, then 2,048 for the feasibility model (including a low reasoning effort) and 350 for the handoff. Optional reasoning is disabled for the other tasks where supported. No failed answer is repaired or substituted.
-
-## Evaluation policy
-
-Four evaluation goals cover ready and blocked kits across all six constraints. Every structured intermediate output, the final manifest, and the evidence list must match. The translation is checked through the resulting duration, deadline, and offline decision rather than a verbatim wording match. The source-adapter completion check is reported as `tool_use_performance`; it does not imply free-form tool selection by the models. Mean whole-goal latency must be ≤45 seconds and inference cost ≤$0.006. Four disjoint goals verify the approved route, including exact resource and deadline boundaries. This smaller suite keeps six-turn goals within the existing 120-request, ten-minute, and $1.25 session bounds. The evaluation records are `KIT-8101` through `KIT-8104`.
-
-## Captured result
-
-The supervised local run was verified on September 28, 2026.
-
-| Path | Complete evaluation goals | Mean model cost per goal | Mean whole-goal latency |
-| --- | --- | --- | --- |
-| Original six-model mix | 4/4 | $0.001867339 | 29.49s |
-| Approved six-model mix | 4/4 | $0.001449088 | 25.63s |
-
-The approved route had **22% lower measured cost** and **13% lower mean latency**, with all intermediate and final checks passing. After operator approval, the application configuration changed and all 4/4 disjoint verification goals passed. The displayed sample is fieldkit-01; both paths return the same correct handoff. This demonstrates an observed efficiency improvement on the prepared suite, not an accuracy gain or a production reliability estimate.
-
-The recording is a compressed replay of local real-provider calls, not a fresh Cloudflare run. All application models were released within March 28–September 28, 2026; dated OpenRouter sources are included in Run details and the downloaded evidence. Build: `demo-20260928-fieldkit-local-v5`. Source export SHA-256: `9f0986917f41354bb8ce80421bb9a413825fb457eb37c78af119747880a1514b`. Costs exclude Pi orchestration and infrastructure.
-
-## What OpenMerit compares
-
-The supplied original route and challenger are compared as complete configurations, with goals executed one at a time on both paths. OpenMerit measures them, validates the comparison, proposes an eligible improvement, and changes the application configuration after approval. It does not generate these API adapters or exhaustively search model assignments. Earlier integration comparisons were rejected after a feasibility answer exhausted its output budget and after a cheaper reasoning model misread sufficient storage. The revised challenger retains Qwen3.7 Plus for feasibility. Provider timeouts also stopped two incomplete comparisons. Those attempts are not presented as successful swaps.
-
-See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-placement.md

@@ -1,32 +0,0 @@
-# GPU placement
-
-> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
-
-The GPU placement workflow in the checkout’s recorded player (`site/demo/`) prepares a pool assignment for a synthetic render job. It reads job requirements with `read_job`, inspects a fixed capacity snapshot with `list_pools`, then proposes the cheapest eligible pool. It never reserves capacity, launches compute, moves data, or incurs a GPU charge.
-
-## Workflow structure
-
-| Task | Without OpenMerit | With the proposed route |
-| --- | --- | --- |
-| Read requirements | Qwen3.6 Plus | [Gemma 4 26B A4B](https://openrouter.ai/google/gemma-4-26b-a4b-it) |
-| Inspect capacity | Qwen3.6 Plus | [MiMo-V2.5](https://openrouter.ai/xiaomi/mimo-v2.5) |
-| Choose pool | Qwen3.6 Plus | [DeepSeek V4.1 Flash](https://openrouter.ai/deepseek/deepseek-v4.1-flash) |
-
-The recording identifies the selected model versions and their provenance in **Run details**. Pi’s orchestration model stays fixed and is separate from the application routes.
-
-## Placement rules
-
-A pool must meet six hard constraints: allowed region, per-GPU VRAM, free GPU count, completion deadline including queue time, total job budget, and permission to use preemptible capacity. Exact boundaries qualify. Among eligible pools, choose the lowest quoted job cost, then earliest completion, then alphabetical pool ID. If none qualify, return `blocked` with null pool, cost, and completion fields. The model must cite the job, capacity snapshot, and placement policy. Queue notes are untrusted data.
-
-Both routes receive the same tools, policy, inputs, structured handoffs, and per-turn JSON Schemas. After locating the requested job, the prepared adapter forwards its structured requirements and capacity records; it excludes the raw queue note and previous free-form conversation from later tasks. This prevents queue-note instructions from becoming authority for pool selection. The capacity tool also computes each pool's finish minute from its queue start and runtime; it does not filter or rank pools. The model must still evaluate every candidate pool and all six constraints. The schemas constrain field names and types; they do not supply the correct job, pool, costs, completion time, or evidence. The bounded application requests disable optional reasoning on both routes and request at most 350 output tokens for each lookup and 900 for the final constraint review and plan. The model supplies both the review and plan; the adapter does not repair its answer. Invalid tool requests, incorrect plans, unsupported claims, and missing provider cost do not become successful goals. As with backfill, the mixed route is a prepared candidate evaluated as a whole, not an unrestricted search over model combinations.
-
-## Evaluation policy
-
-Eight evaluation goals cover the six constraints, blocked placements, ties, and instructions embedded in queue notes. All eight must pass exact plan, tool, schema, constraint, and evidence checks. Mean end-to-end **model workflow** latency must be at most twelve seconds and mean inference cost at most $0.003 per goal. Six separate goals verify an approved route. The synthetic GPU quote and finish minute in a plan are separate from the actual inference cost and latency shown in the main comparison. The evaluation records are `GPU-6101` through `GPU-6108`; their requirements and inventory are fixed.
-
-
-## Recorded outcome
-
-In the recorded run, both paths completed 8/8 goals. Qwen3.6 Plus used a mean $0.001053771875 and 8.19s per goal; the mixed route used $0.000409093125 and 7.09s—about 61% lower cost and 13% lower latency. All six post-swap goals passed, with a 10.96s mean verification latency below the twelve-second requirement. A custom budget-constrained request correctly returned `blocked` through both paths. These are real local OpenRouter calls from one prepared run, not promised savings or production reliability estimates.
-
-See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-spam.md

@@ -1,17 +0,0 @@
-# Spam filter · Jev
-
-> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
-
-The Jev task in the checkout’s recorded player (`site/demo/`) classifies a community comment as `publish` or `hold` under a narrow promotional-spam policy. Ordinary discussion, criticism, and reports of spam are allowed. Unsolicited promotional solicitations are held. The fixtures contain twelve examples of each class, with six separate verification comments.
-
-## Models and integration
-
-The challenger is [TypeSafe's Jev 1.13](https://openrouter.ai/docs/guides/community/jev), a model that returns typed decisions through OpenRouter's Decisions API. The baseline generates a JSON decision; Jev answers a prepared Choice question. Both receive the same comment and decision criteria, and the application normalizes both results to the same one-field schema. The API adapter is prepared in advance. This demo shows OpenMerit evaluating and switching the application configuration; it does not demonstrate automatically rewriting a chat application to use Jev. Pi's own harness model stays fixed.
-
-## Evaluation and recorded outcome
-
-The prepared policy requires at least 90% exact matches on the 24 fixtures and mean end-to-end latency of at most ten seconds. Among eligible frontier candidates, the demo proposes the lowest observed cost only when it is below the incumbent cost. Six separate inputs verify an approved change. These are point values on a finite synthetic suite, not production success-rate estimates or confidence bounds.
-
-In the recorded run, GPT-4.1 mini matched 22/24 comments and Jev matched 24/24. Measured application cost was approximately $0.074 versus $0.018 per 1,000 comments; mean request latency was 0.78 versus 0.39 seconds. OpenMerit proposed Jev, the operator approved, and Jev passed 6/6 post-swap checks. The sample output highlights one baseline mistake: a comment reporting spam was held by the fixed model and correctly published by Jev. All 24 fixtures remain in the aggregate comparison. These are one run's observations, not promised outcomes or a general moderation benchmark.
-
-See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo-support.md

@@ -1,44 +0,0 @@
-# Support tickets
-
-> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
-
-Watch the support-ticket demo in the checkout’s recorded player (`site/demo/`). One model call turns a synthetic customer email into a structured ticket. Both paths start with GPT-4.1 mini and use the same prompt and JSON schema. The fixed path keeps that model; the managed path changes only after measured comparison and operator approval.
-
-## Input and output
-
-The ticket has exactly three fields:
-
-| Field | Rule |
-| --- | --- |
-| `category` | `billing`, `shipping`, `account`, or `technical`, according to the email's request. |
-| `urgency` | `high` only for an ongoing outage, a blocked business operation, or suspected unauthorized access; otherwise `normal`. |
-| `order_id` | The stated `ORD-` identifier, or `null` if none is given. |
-
-An angry tone alone does not make a ticket urgent, and the model must never invent an order ID. Email content is treated as data. For example, “I was charged twice for ORD-1201. Please refund the duplicate.” should produce:
-
-```json
-{"category":"billing","urgency":"normal","order_id":"ORD-1201"}
-```
-
-## Comparison structure
-
-Pi first checks the prepared grader and measurement collectors, then measures GPT-4.1 mini on 24 synthetic emails. It evaluates GPT-4.1 nano and Gemini 2.5 Flash Lite on the same inputs. Each application request records its output, provider cost, and end-to-end latency. The exact-match grader checks every field and rejects missing or extra fields.
-
-OpenMerit validates the comparison and proposes an eligible cheaper model. The recorded operator approves the proposal; Pi changes the application's model configuration and runs six disjoint verification emails. Pi's own orchestration model stays fixed.
-
-## Evaluation policy
-
-The prepared policy requires at least 90% exact matches on the 24 fixtures and mean end-to-end latency of at most ten seconds. Among eligible frontier candidates, the demo proposes the lowest observed cost only when it is below the incumbent cost. Six separate inputs verify an approved change. These are point values on a finite synthetic suite, not production success-rate estimates or confidence bounds.
-
-## Recorded outcome
-
-| Model | Exact matches | Cost per 1,000 emails | Mean latency |
-| --- | --- | --- | --- |
-| GPT-4.1 mini | 24/24 | $0.106 | 1.64s |
-| GPT-4.1 nano | 17/24 | $0.027 | 1.23s |
-| Gemini 2.5 Flash Lite | 24/24 | $0.027 | 1.21s |
-
-GPT-4.1 nano was cheaper but failed the quality requirement. Gemini 2.5 Flash Lite qualified, received approval, and passed all six post-swap checks. The cost column scales each observed per-email mean to 1,000 emails and rounds it; the downloaded recording retains the original values. These observations come from the supervised run recorded on September 28, 2026 and do not promise the same result on another workload.
-
-
-See [the demo overview](demo.md) for playback controls, the shared comparison lifecycle, and how to read the measurements.

package/docs/demo.md

@@ -1,74 +0,0 @@
-# Demos and samples
-
-## Explore the terminal
-
-- [Hosted enterprise sample](https://dash.openmerit.ai/): 200 fictional agents, 699 task IDs, 21 models, and 9.6 million requests across 30 days. It demonstrates inventory, paths, activity, shared compute, and source coverage without reading your machine.
-- [Before/after comparison](https://dash.openmerit.ai/demo): fourteen fictional insurance workflows and six preference choices across 26 model/service offers. Workloads and token volume stay fixed; costs are modeled from the dated price catalog. Benchmark and efficiency exceptions remain visible as preference overrides.
-
-Neither hosted terminal example makes inference calls, applies model changes, or establishes application quality or latency. See [the comparison methodology](inference-terminal.md#hosted-enterprise-sample-and-comparison) for the exact selection rules, costs, and evidence limits.
-
-## Interactive website experiment
-
-The separately deployed [delivery-resolution experiment](https://openmerit.ai/demo/) uses fictional customer cases and bounded live provider calls. Its [guide](interactive-demo.md) explains the current scenarios, approval and verification steps, and session limits. This is distinct from the passive dashboard samples above.
-
-## Run your own supervised trial
-
-Use [Pi setup](getting-started.md#set-up-model-comparisons-with-pi) and [your first trial](first-trial.md) for real application measurements. The repository's separate `npm run demo` sandbox makes live provider calls; its requirements and spend boundaries are in [testing](testing.md#experimental-sandbox).
-
-## Recorded player
-
-Main also retains a recorded player in `site/demo/`, served at `/demo` by this checkout's static preview. Its September 28, 2026 recordings and the walkthroughs below document specific earlier experiments. They are not a description of the separately deployed interactive website demo, current provider availability, or a new validation of main. Existing recording links remain available for inspecting their evidence.
-
-## Recorded examples in this checkout
-
-| Scope | Demo | How it is structured |
-| --- | --- | --- |
-| Tasks | [Support tickets](demo-support.md) | One model call turns an email into a ticket; three models face the same exact-match checks. |
-| Tasks | [Spam filter · Jev](demo-spam.md) | A chat model and a typed-decision specialist classify comments under the same promotional-spam policy. |
-| Agents | [Backfill recovery](demo-backfill.md) | Three dependent turns inspect an incident, read a checkpoint, and prepare a recovery manifest. |
-| Agents | [GPU placement](demo-placement.md) | Three dependent turns read requirements, inspect capacity, and choose a pool under six constraints. |
-| Agents | [Field kit · Six tasks](demo-fieldkit.md) | A prepared six-model workflow translates, reads an image, extracts HTML, inspects code, checks feasibility, and writes a handoff. Both paths use six distinct models. |
-
-## How a comparison works
-
-Every recorded comparison uses a prepared application, synthetic fixtures, a grading rubric, instrumentation, API adapters, and a bounded model shortlist. Both application paths receive the same inputs and task rules. The **without OpenMerit** path keeps its original configuration. The managed path follows this sequence:
-
-1. **Measure a baseline.** Pi runs the incumbent on the fixed evaluation suite and saves outputs, costs, latency, and grading results.
-2. **Compare challengers.** The same suite measures alternative models or complete model routes. OpenMerit checks the returned evidence references, result contracts, and frontier conformance. A cheap candidate that fails a requirement cannot qualify.
-3. **Request approval.** The prepared demo policy proposes the lowest observed cost among eligible frontier candidates only when it is below the incumbent cost. Keeping the incumbent is a valid outcome.
-4. **Apply and verify.** After operator approval, Pi changes the application's configuration and checks it on separate inputs. The confirmed policy requests restoration of the incumbent if verification fails.
-
-The agent examples compare supplied route configurations as whole workflows. They do not search all possible per-task model assignments. Pi's orchestration model stays fixed. The adapters, fixtures, and helpers are prepared in advance; OpenMerit does not generate those integrations in these demos.
-
-Pi invokes bounded helpers and submits saved artifacts by name through a demo-specific transport bridge. The bridge expands those references into the adapter's canonical results and signals. The actual OpenMerit lifecycle then validates and advances the comparison. Missing provider cost stops a comparison instead of being treated as zero; a partial run is not a successful outcome.
-
-The repository runner supplies explicit sampling plans, metric aggregations, and an incumbent descriptor to match the current protocol. Candidate limits count the incumbent and challengers together. Its comparison manifest records the fixed fixtures, rubric, model routes, and actual run order; the incumbent's baseline is reused in the comparison and included in evaluation-spend accounting. Runs are not randomized. The recordings retain their original execution evidence; the current runner does not establish a new validation result without a new run.
-
-## Playback controls
-
-The player in `site/demo/` is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
-
-- **Pause / Play** stops or resumes playback. Seeking on the timeline pauses at the selected position.
-- **Replay** starts the recording again from the beginning.
-- **Reset** returns to the beginning paused and closes Run details.
-- **Run details** shows the evaluation policy, sample input and outputs, provenance, and **Download evidence**, which saves the selected recording as JSON.
-
-Switching task or scope starts the selected recording from the beginning and closes Run details. Returning to a scope restores its last selected example. Left/Right arrows and Home/End move within each tab row. Task-specific URLs open the matching scope and example. Playback stops at the end and does not advance while the browser tab is hidden.
-
-The available recordings were captured on September 28, 2026. Support and spam ran in Cloudflare; backfill, GPU placement, and field kit ran locally with real OpenRouter calls, as labeled in their provenance. The 36-second presentation compresses the original timing. Operator approval, outputs, and measurements are captured events, not new work performed while watching.
-
-## Reading the results
-
-Each example has a short description below its tabs explaining the task and intended result. It remains visible throughout playback, including when paused, completed, or reset.
-
-The main view shows the decision, both application paths, and the run trace. A single-task demo shows one model per path; an agent recording shows each task's model. The managed assignments stay on the baseline until the recorded change is applied. Resetting or seeking backward restores the corresponding earlier state.
-
-Quality is the number of matching fixtures or completed goals in the finite suite. Cost is the observed mean application cost multiplied by 1,000 and rounded to $0.001; it is not a separate 1,000-input test. Latency is mean end-to-end application request time, including the complete sequential workflow for agents, shown to two decimal places in seconds. Percentage changes use the original unrounded measurements, and slower outcomes are labeled as higher latency.
-
-Application costs exclude Pi orchestration and infrastructure. GPU quotes inside a placement plan are synthetic job data, separate from measured inference costs. Evidence downloads retain original unrounded measurements and provenance; the player presents a curated recording rather than the full temporary runner filesystem.
-
-## Scope and limitations
-
-These examples use small synthetic suites, supplied prompts, fixed helpers, and preselected candidates. Their observed results are not production success-rate estimates, confidence bounds, or promised savings. Recovery manifests are never executed, GPU capacity is never reserved, and field-kit scripts are never run. Watching downloads static assets and recorded synthetic evidence; it sends no inputs to model providers.
-
-For your own workload, follow [your first trial](first-trial.md). The [architecture guide](architecture.md) explains the boundary between OpenMerit and the coding harness.

package/docs/first-trial.md

@@ -1,60 +0,0 @@
-# Your first supervised trial
-
-Use one representative product task to learn whether OpenMerit and your harness can collect useful evidence. This guide describes a trial to perform; it does not claim that a trial has already succeeded for your product.
-
-## Choose a bounded task
-
-Pick a task with outcomes you can inspect, such as extracting fields from a document, answering questions over a known collection, or completing a repeatable coding task. Identify the current model configuration so it can be restored if a challenger regresses.
-
-Choose examples that resemble actual use, including difficult cases. A single successful example cannot establish a success rate or reliability claim.
-
-## Confirm the setup
-
-Complete [installation](getting-started.md) and open the product repository in an interactive Pi session. During setup, confirm:
-
-- the required quality, cost, latency, and other metrics;
-- the sample requirements and unacceptable outcomes;
-- the evaluation budget and any candidate or run limits;
-- when reassessments and post-swap checks should happen;
-- whether model changes or rollback may happen automatically.
-
-For a supervised first trial, leave automatic swaps disabled. A zero graduation threshold can allow an immediate automatic swap if automation is enabled. Review [budgets and permissions](budgets.md) before confirming.
-
-## Inspect the evaluations
-
-Ask Pi to show the evaluation artifacts and instrumentation it created, explain what they measure, and demonstrate a representative run. Check that all required metrics have a source and that evaluation data is appropriate to send to the configured providers.
-
-```text
-/openmerit status
-/openmerit logs
-```
-
-These commands help locate readiness and evidence. They do not independently certify that an evaluation is correct. Inspect the artifacts and measurements themselves.
-
-## Collect and compare
-
-Use the product normally while its evaluations collect representative measurements. OpenMerit also records completed-work checkpoints; these checkpoints alone do not establish product quality.
-
-OpenMerit core checks baseline sufficiency from reported metric windows; when they pass, it asks Pi to discover candidates, run challenger trials, and calculate a comparison. You can request a check explicitly:
-
-```text
-/openmerit assess
-```
-
-Review the candidate results, sample counts, costs, and unresolved measurements. A missing metric is a reason to gather evidence, not to assume a model passed or failed.
-
-## Approve and verify a change
-
-If a proposal passes the comparison checks and you want to try it, approve the specific proposal:
-
-```text
-/openmerit approve
-```
-
-The harness applies the change. Allow the configured post-swap evaluation window to complete and inspect its results. If a regression is reported, the workflow can request rollback when your policy permits it. Ask Pi to demonstrate that the previous configuration can be restored.
-
-## Decide what to trust next
-
-Keep the trial’s artifacts and audit trail. Before enabling automatic changes, confirm that your evaluations detect meaningful failures, provider costs are visible, and post-swap verification and rollback work for this task.
-
-Persistent checks require an authorized macOS LaunchAgent or an external scheduler on other platforms. Pause survives restarts and stops automatic due decisions, but does not cancel active work or disable explicit commands. Review [automatic checks](automation.md) and keep the trial supervised; background candidate work is not yet product-validated.

package/docs/interactive-demo.md

@@ -1,118 +0,0 @@
-# Interactive delivery-resolution experiment
-
-This guide describes the separately deployed website experiment, build `guided-20261002-v9`. Its application source is preserved on `codex/main-before-terminal-2026-10-12`; main’s `site/demo/` remains the historical recorded player. The hosted experiment and the terminal’s fictional [before/after sample](https://dash.openmerit.ai/demo) are separate products.
-
-Revision `guided-20261002-v9` of the [interactive demo](/demo/) presents five business scenarios: a delivery policy update, privacy update, an in-house deployment preference, peak-time response needs and an incomplete customer request. The agent has one prepared goal: resolve a damaged delivery before a customer's event. Four sequential model calls read the store policy, assess the affected order and delivery options, choose a resolution, and draft a customer reply. The fourth call is a real model response. No refund, shipment or customer message is executed.
-
-## Run one request
-
-Select **Run request**. John is a member with a $240 order. One $90 desk lamp arrived damaged two days ago; the other items are fine. He needs a replacement before an event in three days. Stock is available, but standard delivery takes five days. Express delivery takes two days and costs the store $12.
-
-The agent must use the affected item's price rather than the order total, apply damage coverage despite the final-sale label, check stock and the delivery deadline, apply the member express allowance, and respect the human-approval limit. Express coverage starts off. John’s standard delivery would miss his event, so the initial correct offer is a $90 item-only refund. Enabling express coverage changes the correct offer to a free two-day replacement, with the store covering the $12 shipping cost. Other prepared cases require an item-only refund, review or a decline.
-
-The starting assignment is Gemini 3.1 Flash Lite through OpenRouter for all four tasks. This is a fixed starting example, not an OpenMerit-selected or approved setup. Each output is checked. It runs once per session; its four calls and cost count toward the session allowance. A passing opening request does not consume a comparison cycle. Since revision v8, a failed business check, response-time check or provider call automatically starts the existing full comparison under the same rules. That recovery consumes one fresh comparison and uses the existing call/spend allowance, preserving verification resources.
-
-The task blocks light up blue as calls run, green for passed checks, and red for failures. A completed passing request shows its proposed resolution and the actual fourth model's **Draft reply · not sent**. A failed opening request remains in the evidence. Since revision v8 the demo immediately compares alternatives and keeps scenario controls locked; it does not ask the visitor to diagnose the failure or relax a business rule. A qualifying result is a proposal, still requiring **Use this setup** and fresh verification. If the bounded search fails, no setup is applied. Drafts and request metrics are hidden during a new run, after a rule change, or when the draft is not valid for the displayed session state.
-
-## Decision instructions and recovery
-
-The resolution prompt separates damage claims from ordinary returns. Damage claims explicitly ignore the final-sale label and ordinary membership window, then apply damage expiry, human approval, stock and on-time shipping in order. The model must return the reason belonging to the selected branch. The application still passes real upstream output to later calls and compares the result against an independent expected outcome; no answer repair or relaxed grading is introduced.
-
-This addresses the observed v7 error where a covered damaged item was declined as final sale, or refunded with an inconsistent `final_sale` reason. Live decision-only canaries test eight cases on each of the starting OpenRouter model and direct OpenAI model, including an undamaged final-sale counterexample; these are separate from whole-agent qualification. Passing one request or one task is not proof that a complete setup qualifies. In the subsequent local v8 browser validation, a mixed setup with three hosted open-weight assignments and a direct OpenAI resolution assignment passed all four comparison cases, both fresh verification cases and John’s rerun. Earlier failed candidates remained in the evidence. This is a small prepared-suite result, not a guarantee for other requests or future runs.
-
-## Explore business scenarios
-
-This revision has five yes/no scenarios under **What changed?**, all off initially. Run the opening request, then select a scenario. Each selection immediately starts reassessment; it does not directly choose a model.
-
-| Scenario | What changes in the experiment |
-| --- | --- |
-| **We now cover urgent replacements** — Policy update | Members receive express delivery costing the store at most $15 when it can meet their deadline and standard shipping cannot. The policy extraction, expected resolution and reply checks change. John's confirmed damage claim changes from a $90 item-only refund to a free two-day replacement. |
-| **Customer data needs tighter protection** — Privacy update | The fictional company allows customer-data tasks only through its approved endpoint, direct OpenAI. Public-policy calls can still use OpenRouter. This is endpoint eligibility, not a claim that a provider guarantees privacy; hosted open weights are subject to the same restriction. The business-facing card does not name or prescribe a model. |
-| **We want to run this ourselves later** — In-house option | Prefer compatible reviewed open-weight assignments that pass the complete suite, then measured cost or response time. These calls still use OpenRouter in the cloud. This does not measure local deployment feasibility. |
-| **The support queue is filling up** — Peak time | Prioritize the lowest measured worst-case response time, with a five-second limit per complete test instead of twenty seconds. This represents the business need for faster answers; the demo does not generate a traffic burst, measure throughput or establish a production capacity guarantee. |
-| **The customer gives us less to go on** — Incomplete request | John's message no longer establishes whether the lamp is damaged, opened or unused. The facts call must return `condition: unknown`, the resolution must be `clarify / missing_context`, and the reply must ask one relevant question without offering a refund or replacement. The displayed customer message changes too. |
-
-The incomplete-request scenario changes both comparison and fresh verification inputs. Other cases still contain explicit damage, so a setup that always asks a question cannot pass. Missing condition takes precedence over a remedy or approval decision: the agent must establish what happened first. Enabling a delivery benefit cannot authorize a replacement when condition is unknown. Inputs supplied to the models never include the hidden expected condition or repaired outputs. Every proposed four-call setup must pass the entire suite; a stronger model is not assigned by name.
-
-There is no scenario-to-model mapping. OpenMerit selects from setups that satisfy endpoint eligibility, business checks and the response limit using measured tradeoffs. A scenario can change the correct customer outcome without requiring a different model. Models that still pass and remain the best measured choice stay in place; no swap is staged for the demo.
-
-Human approval above $150 per affected item remains a fixed store rule in the UI, rather than a scenario toggle. Exactly $150 does not trigger it, and the whole order total is not substituted for the affected item's price. Damage claims retain a 14-day inclusive window and override final-sale labels. A missing receipt requires review once condition is known. Prefer an in-stock replacement that meets the deadline: standard shipping first, then covered member express shipping when enabled. Otherwise offer an item-only refund. There are no sliders or model-picking controls. The backend retains bounded inputs for prepared experiments, including older approval/model restrictions. Older builds require a fresh session.
-
-All five switches and **Reset scenarios** are disabled immediately while a request starts, comparisons run, approval is verified, or John's request is rerun. They visibly dim and explain when they unlock. A lost connection keeps them locked until the session status is recovered. Approval stays near the result, above the scenarios.
-
-Changing delivery policy or customer context requires fresh comparison evidence. Measurements can be reused only for exactly the same business policy and prepared cases in the same build. Privacy changes recheck endpoint eligibility; the deployment preference and response target are reevaluated against applicable measurements. Reuse is labelled in the explanation and trace. The one-case opening request is never substituted for four-case comparison evidence. Changing a scenario pauses the previous setup until approval and verification.
-
-Changed task assignments show the previous and proposed model. **Why this choice?** reports whether the earlier setup failed a task/business check, missed the deadline, or remained a qualifying but less preferred measured tradeoff. An individual step's passing output is not proof that the entire setup qualifies. A comparison with no fitting initial setup automatically starts a broader search under unchanged business rules.
-
-With the in-house preference off, the first comparison tests four complete configurations: OpenRouter AutoRouter, Gemini through OpenRouter, direct OpenAI models, and a mixed assignment. Later policy comparisons include a previously activated expanded assignment in place of the original Gemini starting setup. Other prepared assignments, cached applicable results, new task combinations and catalogue models are available to broader search. Keeping the initial batch to four leaves room for successive business-rule changes within the existing session limits; every fresh batch also reserves verification resources. AutoRouter is an inference option, not an opponent. Its selection is restricted to the disclosed shortlist; served models are retained in the trace and downloaded evidence. No configuration is forced to win.
-
-## Hosted open-weight scenario
-
-**We want to run this ourselves later** uses a reviewed shortlist: [Qwen 3.5 9B](https://huggingface.co/Qwen/Qwen3.5-9B), [Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), [Gemma 3 12B](https://huggingface.co/google/gemma-3-12b-it), and [Mistral Small 3.2 24B](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506). This is a curated candidate set, not a popularity ranking or a hardware recommendation. The live OpenRouter catalogue must still list each candidate with compatible structured outputs, context size and prices. A release with an announced catalogue expiration is skipped. Unreviewed models are not inferred to have open weights from their vendor name.
-
-Task cards use the shorter **Open weights** badge. A note below the scenarios states that all calls run in the cloud. The read-only inspector, model-call trace and downloaded evidence retain **Open weights · Hosted on OpenRouter** and the actual provider. The demo measures hosted API charges and wall-clock latency. It does not measure local hardware, memory, throughput, deployment feasibility or local data privacy. Hosting on OpenRouter still sends inputs to cloud providers, and endpoint restrictions continue to apply to every private step.
-
-Enabling the scenario tests compatible open-weight assignments first, using permitted direct models on any tasks whose endpoint restrictions exclude OpenRouter. All four business cases and the response-time limit must still pass. A derived `preference_fit` score records the fraction of the four assignments using reviewed open-weight releases; it is metadata-based preference evidence, not a quality or privacy score. The coordinator includes it in the verified frontier. Selection prefers the highest fraction, then the existing cost or live-chat timing priority. A more expensive qualifying open-weight setup can therefore win without inventing a saving.
-
-When the initial candidates fail, broader search can retain successful open-weight tasks and test another model on the weakest task, then use the existing bounded fallback search. Every combination faces the complete suite. A mixed setup or a setup with no open-weight assignments may be proposed when it is the qualifying result found within the allowance; the explanation reports the actual count. If catalogue discovery fails, other permitted options remain available. No switch forces a specific model to pass or relaxes a business check. Turning the preference off restores the usual cost/timing selection and can reuse evidence for unchanged business cases. Approval and fresh verification remain required.
-
-The pool stays bounded at eight catalogue candidates, including compatible reviewed open-weight releases and direct OpenAI candidates. Existing call, spend and verification reservations are unchanged. In the 2 October v7 local live check, all four reviewed families were called successfully, but none of the tested complete or mixed setups passed every default-policy case. The best mixed setups passed three of four cases. No setup was approved; successful dispatch is not evidence of a qualifying agent.
-
-## Automatic broader search
-
-A comparison with no qualifying initial configuration automatically enters **Searching other models**. Search can also investigate a better open-weight fit when an existing qualifying setup uses fewer open-weight tasks than the endpoint rules permit; the UI distinguishes this preference search from a failed business check. The response-time limit, express policy, approval requirement, business fixtures and grading remain unchanged during the search. No additional switch or approval is needed for evaluation within the already authorized session allowance.
-
-The harness reads OpenRouter's model catalogue and the configured OpenAI account's model list. OpenRouter candidates must support this text/structured-output request format, have a sufficient context window and stay within the existing $1 input/$5 output per million token dispatch ceiling with no request fee. Catalogue latency ordering is a research lead, not a measurement of this workflow. Direct OpenAI discovery uses account availability intersected with adapters that have checked request semantics and token rates: GPT-5.4 nano and GPT-4o mini supplement the original GPT-4.1 options. Other OpenAI models are not yet supported by this demo. Catalogue failures are recorded independently; one unavailable source does not hide the other source's candidates.
-
-The bounded pool contains up to eight additional compatible models. Task-level checks and timings can suggest a new combination of previously tried models; untested prepared assignments and newly discovered models can also be tried. Every proposed combination is run through all four comparison cases with real sequential calls. It cannot qualify on estimated latency, catalogue rankings or successful individual steps alone. Each round uses prior task results to suggest a combination, then considers remaining prepared assignments and compatible catalogue models. No assignment is automatically promoted just because it is new.
-
-The search stops at the first qualifying additional setup (or first improvement in open-weight coverage during a preference search), after at most six extra setups per comparison, when its compatible options are tested, or when session/call/spend limits prevent another complete test. Twelve calls and a conservative spend allowance are preserved for approval verification and the original request rerun. The comparison and its expansion count as one fresh comparison. Failed configurations and catalogue sources remain in the evidence download. A failed fresh verification or rerun also disqualifies that assignment on subsequent comparisons wherever the recorded failure still violates the current requirements; earlier passing comparison evidence cannot silently override it. Unchanged workload results can be reused on subsequent rule changes; their current restrictions and time limit are always rechecked.
-
-While search runs, controls stay locked and the read-only IDE and trace follow the new model assignments. If no setup qualifies, the UI identifies the stopping reason and, when available, the fastest correct setup's slowest measured case against the deadline. Earlier task blocks are labelled **Earlier scenario**, not **Passed**. Search exhaustion does not prove that no model outside this bounded search could qualify. The demo never silently relaxes the requirement or applies a failed configuration.
-
-## Read and approve the choice
-
-The compact page shows one customer request and four task blocks. During a comparison, a caption identifies the prepared case currently being tested, so the blocks are not presented as calls for John alone. Amber marks a proposed changed assignment. **Why this choice?** contains the explanation, measurements, other candidate results and session usage. Labels accompany every semantic color.
-
-The UI pairs the demo on the left with a read-only **Behind the scenes** view on the right; the two panels are centered together on desktop and stack on smaller screens. Taller workflows scroll normally. The inspector uses a muted gray surface, softer borders and restrained syntax/status colors so the interactive panel remains the focus. Text, live highlights and errors remain readable. One code view shows the current constraints, prepared steps, model assignments and checks. The active call is highlighted and brought into view automatically. Changed rules and proposed model assignments are marked in amber. These are views of configuration and session data, not editable source files. The run trace stays visible underneath as a compact monospace console, with timestamps, source labels and colored RUN/OK/WARN/ERROR/INFO markers. The original event messages remain intact. All actions, including **Download run evidence** under **Why this choice?**, remain on the left. The right side has no tabs, buttons or editing controls and starts no model calls.
-
-The view distinguishes the starting assignment, a proposal awaiting approval, verification in progress, and a verified setup. Earlier rules, stopped sessions and lost connections are labelled explicitly. A comparison can run two setups concurrently; the code follows the latest reported task while the trace retains events from both. Returned output is shown only when the displayed request has completed and the rules still match. Scrolling back preserves the reader’s position; a new action on the left resumes automatic code following. The trace follows new events while the reader is at its bottom.
-
-Each configuration faces four synthetic cases: John's event deadline, an out-of-stock item requiring a refund, a high-value item that tests the approval switch, and a missing receipt with instructions to falsely claim the entire order was refunded. Two routes may run concurrently; calls and cases within each route are sequential. Policy, extracted facts, resolution and reply commitment fields are checked exactly against independent expected values under the current rules. The draft also receives bounded wording checks for required outcome wording, amounts and common false action claims. These checks do not constitute a full semantic evaluation of its prose.
-
-An upstream mistake is retained and passed to subsequent calls; it is not repaired with an expected answer. The application supplies arithmetic comparisons calculated from the actual extracted policy and facts, including delivery timing and express coverage. The third model applies rule precedence and chooses the resolution; the fourth model receives that actual resolution and writes the draft. These helpers and inventory/delivery snapshots are prepared application code and fictional inputs, not live commerce integrations.
-
-The prepared harness calculates the comparison; the actual OpenMerit coordinator validates the typed evidence and frontier and records its lifecycle. Model and harness activity are labelled separately from OpenMerit events. This demo does not run Pi as an autonomous investigator, invent integrations, or exhaustively search all model assignments. Its broader search uses the bounded strategy described above.
-
-The frontier compares task success, observed maximum whole-request time and API cost on the same cases. Local v7 also includes derived open-weight preference fit when that scenario is enabled. After the enabled open-weight preference is applied, live chat recommends the lowest observed maximum time; otherwise it recommends the lowest measured API cost within the time limit. Ties use the other metric, then a stable configuration ID. The core validates the frontier and the evidence-backed preference; the prepared harness calculates the recommendation. Reported average latency is separate. Business rules can increase cost or latency, and no saving is guaranteed. **Use this setup** records the exact approval, applies the selected configuration, and makes fresh calls on two separate boundary cases: exactly day 14, exactly the review threshold and exactly $15 express cost; and expired damage coverage on day 15. Every completed case makes four calls. The guided flow then reruns John’s original damaged-delivery request with fresh calls. It does not display a verification fixture as John’s request. The core’s reference is a measured candidate within the comparison; the separate one-case opening example is not treated as that four-case baseline. If the comparison reference remains best, the core records its use and fresh checks without claiming a core model-swap transition. Failed verification stops the application; it does not restore a setup that may violate new requirements.
-
-**Run John’s request again**, under the explanation, reruns the same known fictional case with fresh model calls. It is not an unseen validation set. Changing a requirement disables this action until reassessment. A failed live example pauses further work.
-
-## Measurements and data
-
-Times are observed wall-clock durations, including all four sequential calls and small harness overhead. The suite is too small to estimate production reliability or latency distributions. The requirement uses the observed maximum, not a promised service level.
-
-OpenRouter cost is the provider-reported charge. Direct OpenAI cost is an estimate calculated from actual input, cached-input, and output token counts using checked published rates for each supported direct model (GPT-4.1 mini/nano, GPT-4o mini and GPT-5.4 nano). It is not an invoice. The cost and latency cards beneath the result cover John’s displayed request, labelled as a comparison case when proposed. Values stay neutral because a rule change may increase or decrease either metric. They appear after a completed measurement and are hidden while a run is in progress or the rules have changed. Expanded comparison measurements report the four-case average and maximum separately. Changes in John’s request cost and time are shown under the explanation; different business rules can increase or decrease either. The session ledger also includes the opening request, other candidates, verification and live examples. Container infrastructure and engineering are excluded. Missing cost evidence is not treated as zero.
-
-All requests are constructed from fictional fixtures. The browser never receives API keys or container authorization tokens. Model output, synthetic inputs, measurements, and an event trace are available through **Download run**. The full experiment and core audit artifacts are also written inside the temporary container. The demo is cloud-only; it does not claim local inference or measure local hardware.
-
-## Session limits
-
-A session lasts up to 30 minutes and permits at most three fresh comparisons, 12 comparison/review cycles, 240 model calls, and $2 of conservative inference reservations. There are at most two concurrent sessions and 12 admitted sessions per UTC day shared across the site. There is no separate three-session limit for people on the same network. A session that closes or expires without reserving any model calls returns its admission slot; this also applies when a container fails to start or starts with an outdated runtime. Startup attempts remain limited to 36 per UTC day. Once a model call is reserved, that session counts toward the daily allowance even if the call fails. Reservations are retained conservatively. Temporary API keys and service availability can make live inference unavailable.
-
-If the shared allowance is exhausted, new sessions open at 00:00 UTC (4 a.m. Dubai). Existing unexpired sessions can continue within their own limits.
-
-If the connection drops while starting, the page checks whether a session was created. If none exists, **Run request** becomes available again; no model call is implied by a startup or reconnecting message. Expired sessions and sessions from an earlier demo version offer **Start a new session**. Completed evidence belongs to its original session.
-
-**End session** closes admission and destroys the container. A session cannot resume its live work after a process restart. It keeps its stopped status rather than presenting stale approval as active. These limits and the small prepared suite make this an experiment, not production readiness evidence. The earlier recorded demo and its guides remain archived.
-
-## Provider references
-
-- [OpenRouter AutoRouter](https://openrouter.ai/docs/guides/routing/routers/auto-router)
-- [OpenAI structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
-- [GPT-4.1 mini rates](https://developers.openai.com/api/docs/models/gpt-4.1-mini)
-- [GPT-4.1 nano rates](https://developers.openai.com/api/docs/models/gpt-4.1-nano)
-- [OpenRouter model catalogue](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties)
-- [OpenAI available-model list](https://developers.openai.com/api/reference/resources/models/methods/list)
-- [GPT-5.4 nano request support and rates](https://developers.openai.com/api/docs/models/gpt-5.4-nano)
-- [GPT-4o mini request support and rates](https://developers.openai.com/api/docs/models/gpt-4o-mini)