openmerit 0.1.4 → 0.1.6-preview.0

package/docs/demo.md

@@ -0,0 +1,57 @@
+# Demo
+
+The [demo player](https://openmerit.site/demo) shows recorded, supervised OpenMerit comparisons. The guides in this section explain each example's inputs, model assignments, evaluation policy, and outcome. These are early development demonstrations from this checkout; they do not establish general product reliability or behavior in the published npm release.
+
+## Choose a walkthrough
+
+| Scope | Demo | How it is structured |
+| --- | --- | --- |
+| Tasks | [Support tickets](demo-support.md) | One model call turns an email into a ticket; three models face the same exact-match checks. |
+| Tasks | [Spam filter · Jev](demo-spam.md) | A chat model and a typed-decision specialist classify comments under the same promotional-spam policy. |
+| Agents | [Backfill recovery](demo-backfill.md) | Three dependent turns inspect an incident, read a checkpoint, and prepare a recovery manifest. |
+| Agents | [GPU placement](demo-placement.md) | Three dependent turns read requirements, inspect capacity, and choose a pool under six constraints. |
+| Agents | [Field kit · Six tasks](demo-fieldkit.md) | A prepared six-model workflow translates, reads an image, extracts HTML, inspects code, checks feasibility, and writes a handoff. Both paths use six distinct models. |
+
+## How a comparison works
+
+Every recorded comparison uses a prepared application, synthetic fixtures, a grading rubric, instrumentation, API adapters, and a bounded model shortlist. Both application paths receive the same inputs and task rules. The **without OpenMerit** path keeps its original configuration. The managed path follows this sequence:
+
+1. **Measure a baseline.** Pi runs the incumbent on the fixed evaluation suite and saves outputs, costs, latency, and grading results.
+2. **Compare challengers.** The same suite measures alternative models or complete model routes. OpenMerit checks the returned evidence references, result contracts, and frontier conformance. A cheap candidate that fails a requirement cannot qualify.
+3. **Request approval.** The prepared demo policy proposes the lowest observed cost among eligible frontier candidates only when it is below the incumbent cost. Keeping the incumbent is a valid outcome.
+4. **Apply and verify.** After operator approval, Pi changes the application's configuration and checks it on separate inputs. The confirmed policy requests restoration of the incumbent if verification fails.
+
+The agent examples compare supplied route configurations as whole workflows. They do not search all possible per-task model assignments. Pi's orchestration model stays fixed. The adapters, fixtures, and helpers are prepared in advance; OpenMerit does not generate those integrations in these demos.
+
+Pi invokes bounded helpers and submits saved artifacts by name through a demo-specific transport bridge. The bridge expands those references into the adapter's canonical results and signals. The actual OpenMerit lifecycle then validates and advances the comparison. Missing provider cost stops a comparison instead of being treated as zero; a partial run is not a successful outcome.
+
+The unreleased source runner now supplies explicit sampling plans, metric aggregations, and an incumbent descriptor to match the current protocol. Candidate limits count the incumbent and challengers together. Its comparison manifest records the fixed fixtures, rubric, model routes, and actual run order; the incumbent's baseline is reused in the comparison and included in evaluation-spend accounting. Runs are not randomized. These source changes do not alter the existing recordings or establish a new live validation result.
+
+## Playback controls
+
+The player is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
+
+- **Pause / Play** stops or resumes playback. Seeking on the timeline pauses at the selected position.
+- **Replay** starts the recording again from the beginning.
+- **Reset** returns to the beginning paused and closes Run details.
+- **Run details** shows the evaluation policy, sample input and outputs, provenance, and **Download evidence**, which saves the selected recording as JSON.
+
+Switching task or scope starts the selected recording from the beginning and closes Run details. Returning to a scope restores its last selected example. Left/Right arrows and Home/End move within each tab row. Task-specific URLs open the matching scope and example. Playback stops at the end and does not advance while the browser tab is hidden.
+
+The available recordings were captured on September 28, 2026. Support and spam ran in Cloudflare; backfill, GPU placement, and field kit ran locally with real OpenRouter calls, as labeled in their provenance. The 36-second presentation compresses the original timing. Operator approval, outputs, and measurements are captured events, not new work performed while watching.
+
+## Reading the results
+
+Each example has a short description below its tabs explaining the task and intended result. It remains visible throughout playback, including when paused, completed, or reset.
+
+The main view shows the decision, both application paths, and the run trace. A single-task demo shows one model per path; an agent recording shows each task's model. The managed assignments stay on the baseline until the recorded change is applied. Resetting or seeking backward restores the corresponding earlier state.
+
+Quality is the number of matching fixtures or completed goals in the finite suite. Cost is the observed mean application cost multiplied by 1,000 and rounded to $0.001; it is not a separate 1,000-input test. Latency is mean end-to-end application request time, including the complete sequential workflow for agents, shown to two decimal places in seconds. Percentage changes use the original unrounded measurements, and slower outcomes are labeled as higher latency.
+
+Application costs exclude Pi orchestration and infrastructure. GPU quotes inside a placement plan are synthetic job data, separate from measured inference costs. Evidence downloads retain original unrounded measurements and provenance; the player presents a curated recording rather than the full temporary runner filesystem.
+
+## Scope and limitations
+
+These examples use small synthetic suites, supplied prompts, fixed helpers, and preselected candidates. Their observed results are not production success-rate estimates, confidence bounds, or promised savings. Recovery manifests are never executed, GPU capacity is never reserved, and field-kit scripts are never run. Watching downloads static assets and recorded synthetic evidence; it sends no inputs to model providers.
+
+For your own workload, follow [your first trial](first-trial.md). The [architecture guide](architecture.md) explains the boundary between OpenMerit and the coding harness.

package/docs/first-trial.md

@@ -0,0 +1,60 @@
+# Your first supervised trial
+
+Use one representative product task to learn whether OpenMerit and your harness can collect useful evidence. This guide describes a trial to perform; it does not claim that a trial has already succeeded for your product.
+
+## Choose a bounded task
+
+Pick a task with outcomes you can inspect, such as extracting fields from a document, answering questions over a known collection, or completing a repeatable coding task. Identify the current model configuration so it can be restored if a challenger regresses.
+
+Choose examples that resemble actual use, including difficult cases. A single successful example cannot establish a success rate or reliability claim.
+
+## Confirm the setup
+
+Complete [installation](getting-started.md) and open the product repository in an interactive Pi session. During setup, confirm:
+
+- the required quality, cost, latency, and other metrics;
+- the sample requirements and unacceptable outcomes;
+- the evaluation budget and any candidate or run limits;
+- when reassessments and post-swap checks should happen;
+- whether model changes or rollback may happen automatically.
+
+For a supervised first trial, leave automatic swaps disabled. A zero graduation threshold can allow an immediate automatic swap if automation is enabled. Review [budgets and permissions](budgets.md) before confirming.
+
+## Inspect the evaluations
+
+Ask Pi to show the evaluation artifacts and instrumentation it created, explain what they measure, and demonstrate a representative run. Check that all required metrics have a source and that evaluation data is appropriate to send to the configured providers.
+
+```text
+/openmerit status
+/openmerit logs
+```
+
+These commands help locate readiness and evidence. They do not independently certify that an evaluation is correct. Inspect the artifacts and measurements themselves.
+
+## Collect and compare
+
+Use the product normally while its evaluations collect representative measurements. OpenMerit also records completed-work checkpoints; these checkpoints alone do not establish product quality.
+
+In the unreleased source checkout, OpenMerit core checks baseline sufficiency from reported metric windows; when they pass, it asks Pi to discover candidates, run challenger trials, and calculate a comparison. Published `openmerit@0.1.5` still asks Pi to assess the baseline. You can request a check explicitly:
+
+```text
+/openmerit assess
+```
+
+Review the candidate results, sample counts, costs, and unresolved measurements. A missing metric is a reason to gather evidence, not to assume a model passed or failed.
+
+## Approve and verify a change
+
+If a proposal passes the comparison checks and you want to try it, approve the specific proposal:
+
+```text
+/openmerit approve
+```
+
+The harness applies the change. Allow the configured post-swap evaluation window to complete and inspect its results. If a regression is reported, the workflow can request rollback when your policy permits it. Ask Pi to demonstrate that the previous configuration can be restored.
+
+## Decide what to trust next
+
+Keep the trial’s artifacts and audit trail. Before enabling automatic changes, confirm that your evaluations detect meaningful failures, provider costs are visible, and post-swap verification and rollback work for this task.
+
+Pi does not wake itself while closed. Version 0.1.5 also has a [partial pause limitation](roadmap.md#known-limitations). Plan supervision around those behaviors.

package/docs/getting-started.md

@@ -0,0 +1,65 @@
+# Getting started
+
+## Prerequisites
+
+- Node.js 22.19 or newer.
+- npm.
+- Pi 0.87-compatible provider authentication.
+- A product repository in which Pi can create and verify evaluation and observability artifacts.
+
+OpenMerit does not require or accept an LLM provider credential. Provider access belongs to the coding harness.
+
+## Install from npm
+
+```sh
+pi install npm:openmerit
+pi list
+```
+
+Run `/openmerit doctor` inside Pi to confirm that the extension is available.
+
+## Install from source
+
+```sh
+git clone https://github.com/laz-aslam/openmerit.git
+cd openmerit
+npm ci
+```
+
+Start Pi with the extension:
+
+```sh
+./node_modules/.bin/pi --no-extensions -e ./packages/pi/src/index.ts
+```
+
+`--no-extensions` prevents a globally installed npm copy from conflicting with the source extension. Run `/openmerit doctor` inside Pi to confirm that the adapter and intent executor are available.
+
+## Establish a product profile
+
+Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. In the unreleased source checkout it also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
+
+OpenMerit sends Pi a private, typed `establish_evals` intent. Pi must identify the application LLM call or shared route under evaluation and its exact incumbent application model. It inspects the application's outputs, tools, and failure modes to infer task-relevant metrics instead of asking you to invent them. Before running setup, Pi shows the proposed metrics, why each matters, the representative case and repetition counts, aggregation, any actual performance threshold, and estimated evaluation cost. You confirm or edit that plan together with the stable target ID, call sites, route key, budget, check cadence, scheduling mode, and automation permissions. Pi's own model is never an application target.
+
+For an active-session demo, Pi defaults the baseline and post-swap verification cadence to one completed evaluation task unless you change it. OpenMerit rejects empty baseline or post-swap cadences.
+
+In this source checkout (not yet published in npm `openmerit@0.1.5`), Pi is also asked to research preliminary model leads during setup. Each reported lead has a current price estimate and reputation summary with sources and check times. OpenMerit stores these leads with the project profile. The field is optional for compatibility with existing protocol-0.1 harnesses, and an empty list means no credible leads were found. It does not change the application model or count as a measurement.
+
+OpenMerit rejects successful setup without a confirmed task profile, at least one valid objective with an explicit sampling plan, a supervised-graduation policy, non-empty baseline and post-swap cadences, a budget, and evidence.
+
+The source checkout's setup instruction asks Pi to run the application path with working provider authentication, execute each required collector and grader, inspect the resulting metric values, and retain verification artifacts. A metric that cannot be produced must be marked missing. This smoke run verifies the pipeline; it is not counted as a production baseline sample. OpenMerit still checks the reported coverage and evidence references, not the authenticity of a grader run itself.
+
+## Collect the baseline
+
+After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. In the unreleased source checkout, OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient. Published `openmerit@0.1.5` still nudges Pi to assess.
+
+## Find the frontier
+
+When the baseline meets every required metric threshold, OpenMerit issues the next approved nudges: discover candidates, run controlled challenger trials, and calculate the Pareto frontier. Pi performs that work. OpenMerit checks the returned result against the [Pareto specification](pareto-spec.md).
+
+Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application. In npm `openmerit@0.1.5`, the shortlist comes after the application's baseline has enough evidence. The unreleased source checkout collects preliminary leads at setup, then refreshes them, probes availability, and freezes the incumbent plus challengers. Pi must use the same cases, application revision, prompt, tools, grader, and run count, with a recorded shuffle seed. It preserves one experiment manifest and reports measured quality, cost, latency, tokens, and other required metrics. OpenMerit durably carries the accepted candidate set and assessments into later intents rather than relying on Pi conversation history. A model proposal must come from the verified frontier of those measured candidate results, not from price or reputation alone.
+
+## Approve an early swap
+
+With automatic swaps disabled, a model proposal that passes frontier validation waits for `/openmerit approve`. That authorizes Pi to apply only the proposed change. OpenMerit then requires post-swap evaluation; the approved policy determines rollback behavior. Automatic swaps may bypass per-proposal approval when enabled and the configured verified-swap threshold is reached, including when that threshold is zero.
+
+Continue with [your first supervised trial](first-trial.md) for a practical walkthrough.

package/docs/index.md

@@ -0,0 +1,40 @@
+# OpenMerit documentation
+
+OpenMerit helps your coding harness compare models against the work your product needs to do. You define what matters; the harness gathers measurements and tests candidates; OpenMerit coordinates the process and checks the returned results.
+
+The current release is intended for supervised trials with Pi. Start with a representative task and working evaluations before relying on automatic model changes.
+
+## Start with your product
+
+Install the extension in Pi, then open your product repository:
+
+```sh
+pi install npm:openmerit
+```
+
+Interactive setup asks Pi to establish your task requirements, evaluation budget, automation permissions, and measurements. Follow the [getting started guide](getting-started.md), then work through a [first supervised trial](first-trial.md).
+
+## Explore the demos
+
+The [Demo section](demo.md) explains how each prepared example works, from a single ticket-classification call to a multi-model agent workflow. Watch the recorded comparisons and inspect their inputs, grading rules, model routes, and evidence.
+
+## Understand the decisions
+
+OpenMerit compares candidates against the objectives you confirm, such as quality, cost, and latency. Several models may offer useful tradeoffs. A cheaper model is not automatically a better fit.
+
+- [Metrics and evidence](metrics-and-evidence.md) explains which measurements and samples a comparison needs.
+- [Comparisons and swaps](lifecycle.md) follows a baseline through challenger trials, approval, and post-swap checks.
+- [Budgets and permissions](budgets.md) describes the limits you give the harness and where enforcement depends on it.
+- [Automatic checks](automation.md) explains when work becomes due and what needs an external scheduler.
+
+## Operate with visibility
+
+Use [commands](commands.md) to inspect status and request work. [Project files and logs](operations.md) explains what is stored, and [troubleshooting](troubleshooting.md) covers common setup and evidence problems.
+
+OpenMerit checks structured results and model comparisons. It relies on the harness to execute the work and provide honest evidence. Read the [security notes](security.md) and [current limitations](roadmap.md) before enabling automation.
+
+## Build an integration
+
+Pi 0.87 is the only implemented adapter today. The core and protocol can support other harnesses through an adapter.
+
+Start with the [architecture](architecture.md), then the [adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines the comparison rules, and [testing and validation](testing.md) explains the evidence behind implementation claims.