openmerit 0.1.6 → 0.1.7-preview.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/CHANGELOG.md +4 -4
  2. package/README.md +61 -40
  3. package/dist/eval/src/adapters.mjs +73 -0
  4. package/dist/eval/src/cli.mjs +103 -0
  5. package/dist/eval/src/config.mjs +102 -0
  6. package/dist/eval/src/process.mjs +44 -0
  7. package/dist/eval/src/report.mjs +51 -0
  8. package/dist/eval/src/runner.mjs +106 -0
  9. package/dist/eval/src/scout.mjs +37 -0
  10. package/dist/terminal/public/app.js +36 -36
  11. package/dist/terminal/public/flow.css +1 -1
  12. package/dist/terminal/public/flow.js +26 -26
  13. package/dist/terminal/public/index.html +3 -2
  14. package/dist/terminal/public/styles.css +320 -91
  15. package/dist/terminal/src/cli.mjs +2 -0
  16. package/dist/terminal/src/demo-policy.mjs +205 -0
  17. package/dist/terminal/src/enterprise-sample.mjs +151 -0
  18. package/docs/adapter-guide.md +2 -2
  19. package/docs/agent-evaluations.md +101 -0
  20. package/docs/architecture.md +3 -3
  21. package/docs/automation.md +5 -5
  22. package/docs/budgets.md +2 -2
  23. package/docs/commands.md +7 -5
  24. package/docs/demo-backfill.md +3 -1
  25. package/docs/demo-fieldkit.md +3 -1
  26. package/docs/demo-placement.md +3 -1
  27. package/docs/demo-spam.md +3 -1
  28. package/docs/demo-support.md +3 -1
  29. package/docs/demo.md +22 -5
  30. package/docs/first-trial.md +2 -2
  31. package/docs/getting-started.md +23 -12
  32. package/docs/index.md +18 -29
  33. package/docs/inference-terminal.md +90 -54
  34. package/docs/interactive-demo.md +118 -0
  35. package/docs/lifecycle.md +4 -4
  36. package/docs/metrics-and-evidence.md +3 -3
  37. package/docs/operations.md +3 -5
  38. package/docs/pi-extension.md +6 -6
  39. package/docs/roadmap.md +17 -17
  40. package/docs/scout.md +206 -0
  41. package/docs/security.md +6 -4
  42. package/docs/site-demo.md +4 -0
  43. package/docs/site-design.md +7 -1
  44. package/docs/site-documentation.md +9 -3
  45. package/docs/site-domain-migration.md +57 -0
  46. package/docs/site-dynamic-og.md +2 -2
  47. package/docs/site-faq-maintenance.md +121 -6
  48. package/docs/site-hero-resolution.md +2 -0
  49. package/docs/site-illustration-sequences.md +3 -1
  50. package/docs/site-illustration-transparency.md +80 -0
  51. package/docs/site-memo.md +6 -0
  52. package/docs/site-og-image.md +1 -1
  53. package/docs/site-publication-2026-10-14.md +29 -0
  54. package/docs/site-scout-decisions-experiment.md +87 -0
  55. package/docs/site-scout.md +254 -0
  56. package/docs/site-terminal-preview.md +96 -3
  57. package/docs/site-themes.md +30 -0
  58. package/docs/testing.md +37 -78
  59. package/docs/troubleshooting.md +3 -3
  60. package/package.json +9 -3
package/docs/commands.md CHANGED
@@ -1,10 +1,12 @@
1
1
  # Commands
2
2
 
3
- Run these slash commands inside Pi with the OpenMerit extension loaded. Shell commands such as `pi install` run in your terminal instead.
3
+ The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
4
+
5
+ The npm `0.1.7-preview.1` release (`openmerit@preview`) also provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks, budgets and release status. These are shell commands, separate from Pi commands.
4
6
 
5
7
  ## Inference Terminal
6
8
 
7
- Install `openmerit@0.1.6` with `npm install -g openmerit` to use the shell command `openmerit dash [path]`, for a passive local view of models, connections, usage, and compute. From this checkout, use `npm run dash -- [path]`. It is independent of Pi setup and does not run model comparisons or inference. Earlier `openmerit@0.1.5` does not include it.
9
+ Install with `npm install -g openmerit` to use `openmerit dash [path]` for a passive local view of models, connections, usage, and compute. From this checkout, use `npm run dash -- [path]`. It is independent of Pi setup and does not run model comparisons or inference.
8
10
 
9
11
  See the [Inference Terminal guide](inference-terminal.md) for local installation, automatic discovery, `--no-discovery`, `--json`, `--demo`, collection intervals, and data limitations. Discovery is part of `dash` and is scoped to application evidence. Coding-harness histories are not collected.
10
12
 
@@ -25,7 +27,7 @@ Both show the task profile, metric readiness, evaluation status, active work, ne
25
27
 
26
28
  Requests setup explicitly or reopens setup to correct stale target/incumbent state. Accepted replacement setup clears prior candidate, assessment, frontier, and baseline handoffs while retaining the audit log.
27
29
 
28
- Request setup explicitly only when recovering, rerunning, or configuring a custom integration. In the unreleased source checkout, an unconfigured project starts setup when bounded source detection finds an application LLM call at session start or after a settled Pi build turn. OpenMerit refuses another intent while work is active.
30
+ Request setup explicitly only when recovering, rerunning, or configuring a custom integration. An unconfigured project starts setup when bounded source detection finds an application LLM call at session start or after a settled Pi build turn. OpenMerit refuses another intent while work is active.
29
31
 
30
32
  ## Cancel interrupted work
31
33
 
@@ -41,7 +43,7 @@ Record the active intent as cancelled after an interrupted or abandoned harness
41
43
  /openmerit assess
42
44
  ```
43
45
 
44
- In the unreleased source checkout, ask OpenMerit core to check stored baseline metric windows now. It reports missing or immature required metrics without starting Pi, or starts candidate discovery when they pass. Published `openmerit@0.1.5` still asks Pi to assess. Neither path creates missing measurements or overrides sample requirements.
46
+ Ask OpenMerit core to check stored baseline metric windows now. It reports missing or immature required metrics without starting Pi, or starts candidate discovery when they pass. It does not create missing measurements or override sample requirements.
45
47
 
46
48
  ## Request a frontier
47
49
 
@@ -66,7 +68,7 @@ Approve the current proposal after its frontier result has passed validation. If
66
68
  /openmerit resume
67
69
  ```
68
70
 
69
- In 0.1.5, pause suppresses some lifecycle follow-ups, but task, measurement, and session-start signals can still start due checks. The flag resets on process restart and does not cancel active work. Explicit commands still work. Do not treat “paused” as a guarantee that all OpenMerit work has stopped.
71
+ Pause is persisted in `.openmerit/state.json`, blocks automatic due decisions and lifecycle follow-ups, and unloads the project's managed macOS LaunchAgent. Incoming signals can still be recorded. Resume clears pause and restores scheduling only when the confirmed policy authorizes it. Neither command cancels an already-active intent; explicit commands remain available. The standalone Inference Terminal is independent and keeps collecting until its CLI process exits.
70
72
 
71
73
  ## Locate logs and evidence
72
74
 
@@ -1,6 +1,8 @@
1
1
  # Backfill recovery
2
2
 
3
- The [agent workflow](https://openmerit.site/demo?task=backfill) prepares a recovery manifest for an interrupted data backfill. Its goal is to resume recoverable partitions without duplicating committed data, quarantine malformed input, and stop for investigation when the failure is unknown or involves a checksum conflict. It never executes the manifest or writes to a database.
3
+ > Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
4
+
5
+ The agent workflow in the checkout’s recorded player (`site/demo/`) prepares a recovery manifest for an interrupted data backfill. Its goal is to resume recoverable partitions without duplicating committed data, quarantine malformed input, and stop for investigation when the failure is unknown or involves a checksum conflict. It never executes the manifest or writes to a database.
4
6
 
5
7
  ## Workflow structure
6
8
 
@@ -1,6 +1,8 @@
1
1
  # Field kit · Six tasks
2
2
 
3
- [Watch the six-task field-kit recording](https://openmerit.site/demo/?task=fieldkit). The captured run compares two heterogeneous model routes using real OpenRouter calls through Pi and OpenMerit. Each route uses six distinct models; five assignments change and the feasibility model is retained.
3
+ > Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
4
+
5
+ Watch the six-task field-kit recording in the checkout’s recorded player (`site/demo/`). The captured run compares two heterogeneous model routes using real OpenRouter calls through Pi and OpenMerit. Each route uses six distinct models; five assignments change and the feasibility model is retained.
4
6
 
5
7
  The field-kit workflow prepares an offline recording handoff from a Spanish technician brief, a synthetic equipment-label image, an HTML operating manual, and a Python capture script. Both paths already use **six different models**. The approved route pairs narrow specialists with broader coding, reasoning, and writing models:
6
8
 
@@ -1,6 +1,8 @@
1
1
  # GPU placement
2
2
 
3
- The [GPU placement workflow](https://openmerit.site/demo?task=placement) prepares a pool assignment for a synthetic render job. It reads job requirements with `read_job`, inspects a fixed capacity snapshot with `list_pools`, then proposes the cheapest eligible pool. It never reserves capacity, launches compute, moves data, or incurs a GPU charge.
3
+ > Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
4
+
5
+ The GPU placement workflow in the checkout’s recorded player (`site/demo/`) prepares a pool assignment for a synthetic render job. It reads job requirements with `read_job`, inspects a fixed capacity snapshot with `list_pools`, then proposes the cheapest eligible pool. It never reserves capacity, launches compute, moves data, or incurs a GPU charge.
4
6
 
5
7
  ## Workflow structure
6
8
 
package/docs/demo-spam.md CHANGED
@@ -1,6 +1,8 @@
1
1
  # Spam filter · Jev
2
2
 
3
- The [Jev task](https://openmerit.site/demo?task=spam) classifies a community comment as `publish` or `hold` under a narrow promotional-spam policy. Ordinary discussion, criticism, and reports of spam are allowed. Unsolicited promotional solicitations are held. The fixtures contain twelve examples of each class, with six separate verification comments.
3
+ > Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
4
+
5
+ The Jev task in the checkout’s recorded player (`site/demo/`) classifies a community comment as `publish` or `hold` under a narrow promotional-spam policy. Ordinary discussion, criticism, and reports of spam are allowed. Unsolicited promotional solicitations are held. The fixtures contain twelve examples of each class, with six separate verification comments.
4
6
 
5
7
  ## Models and integration
6
8
 
@@ -1,6 +1,8 @@
1
1
  # Support tickets
2
2
 
3
- [Watch the support-ticket demo](https://openmerit.site/demo?task=support). One model call turns a synthetic customer email into a structured ticket. Both paths start with GPT-4.1 mini and use the same prompt and JSON schema. The fixed path keeps that model; the managed path changes only after measured comparison and operator approval.
3
+ > Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
4
+
5
+ Watch the support-ticket demo in the checkout’s recorded player (`site/demo/`). One model call turns a synthetic customer email into a structured ticket. Both paths start with GPT-4.1 mini and use the same prompt and JSON schema. The fixed path keeps that model; the managed path changes only after measured comparison and operator approval.
4
6
 
5
7
  ## Input and output
6
8
 
package/docs/demo.md CHANGED
@@ -1,8 +1,25 @@
1
- # Demo
1
+ # Demos and samples
2
2
 
3
- The [demo player](https://openmerit.site/demo) shows recorded, supervised OpenMerit comparisons. The guides in this section explain each example's inputs, model assignments, evaluation policy, and outcome. These are early development demonstrations from this checkout; they do not establish general product reliability or behavior in the published npm release.
3
+ ## Explore the terminal
4
4
 
5
- ## Choose a walkthrough
5
+ - [Hosted enterprise sample](https://dash.openmerit.ai/): 200 fictional agents, 699 task IDs, 21 models, and 9.6 million requests across 30 days. It demonstrates inventory, paths, activity, shared compute, and source coverage without reading your machine.
6
+ - [Before/after comparison](https://dash.openmerit.ai/demo): fourteen fictional insurance workflows and six preference choices across 26 model/service offers. Workloads and token volume stay fixed; costs are modeled from the dated price catalog. Benchmark and efficiency exceptions remain visible as preference overrides.
7
+
8
+ Neither hosted terminal example makes inference calls, applies model changes, or establishes application quality or latency. See [the comparison methodology](inference-terminal.md#hosted-enterprise-sample-and-comparison) for the exact selection rules, costs, and evidence limits.
9
+
10
+ ## Interactive website experiment
11
+
12
+ The separately deployed [delivery-resolution experiment](https://openmerit.ai/demo/) uses fictional customer cases and bounded live provider calls. Its [guide](interactive-demo.md) explains the current scenarios, approval and verification steps, and session limits. This is distinct from the passive dashboard samples above.
13
+
14
+ ## Run your own supervised trial
15
+
16
+ Use [Pi setup](getting-started.md#set-up-model-comparisons-with-pi) and [your first trial](first-trial.md) for real application measurements. The repository's separate `npm run demo` sandbox makes live provider calls; its requirements and spend boundaries are in [testing](testing.md#experimental-sandbox).
17
+
18
+ ## Recorded player
19
+
20
+ Main also retains a recorded player in `site/demo/`, served at `/demo` by this checkout's static preview. Its September 28, 2026 recordings and the walkthroughs below document specific earlier experiments. They are not a description of the separately deployed interactive website demo, current provider availability, or a new validation of main. Existing recording links remain available for inspecting their evidence.
21
+
22
+ ## Recorded examples in this checkout
6
23
 
7
24
  | Scope | Demo | How it is structured |
8
25
  | --- | --- | --- |
@@ -25,11 +42,11 @@ The agent examples compare supplied route configurations as whole workflows. The
25
42
 
26
43
  Pi invokes bounded helpers and submits saved artifacts by name through a demo-specific transport bridge. The bridge expands those references into the adapter's canonical results and signals. The actual OpenMerit lifecycle then validates and advances the comparison. Missing provider cost stops a comparison instead of being treated as zero; a partial run is not a successful outcome.
27
44
 
28
- The unreleased source runner now supplies explicit sampling plans, metric aggregations, and an incumbent descriptor to match the current protocol. Candidate limits count the incumbent and challengers together. Its comparison manifest records the fixed fixtures, rubric, model routes, and actual run order; the incumbent's baseline is reused in the comparison and included in evaluation-spend accounting. Runs are not randomized. These source changes do not alter the existing recordings or establish a new live validation result.
45
+ The repository runner supplies explicit sampling plans, metric aggregations, and an incumbent descriptor to match the current protocol. Candidate limits count the incumbent and challengers together. Its comparison manifest records the fixed fixtures, rubric, model routes, and actual run order; the incumbent's baseline is reused in the comparison and included in evaluation-spend accounting. Runs are not randomized. The recordings retain their original execution evidence; the current runner does not establish a new validation result without a new run.
29
46
 
30
47
  ## Playback controls
31
48
 
32
- The player is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
49
+ The player in `site/demo/` is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
33
50
 
34
51
  - **Pause / Play** stops or resumes playback. Seeking on the timeline pauses at the selected position.
35
52
  - **Replay** starts the recording again from the beginning.
@@ -35,7 +35,7 @@ These commands help locate readiness and evidence. They do not independently cer
35
35
 
36
36
  Use the product normally while its evaluations collect representative measurements. OpenMerit also records completed-work checkpoints; these checkpoints alone do not establish product quality.
37
37
 
38
- In the unreleased source checkout, OpenMerit core checks baseline sufficiency from reported metric windows; when they pass, it asks Pi to discover candidates, run challenger trials, and calculate a comparison. Published `openmerit@0.1.5` still asks Pi to assess the baseline. You can request a check explicitly:
38
+ OpenMerit core checks baseline sufficiency from reported metric windows; when they pass, it asks Pi to discover candidates, run challenger trials, and calculate a comparison. You can request a check explicitly:
39
39
 
40
40
  ```text
41
41
  /openmerit assess
@@ -57,4 +57,4 @@ The harness applies the change. Allow the configured post-swap evaluation window
57
57
 
58
58
  Keep the trial’s artifacts and audit trail. Before enabling automatic changes, confirm that your evaluations detect meaningful failures, provider costs are visible, and post-swap verification and rollback work for this task.
59
59
 
60
- Pi does not wake itself while closed. Version 0.1.5 also has a [partial pause limitation](roadmap.md#known-limitations). Plan supervision around those behaviors.
60
+ Persistent checks require an authorized macOS LaunchAgent or an external scheduler on other platforms. Pause survives restarts and stops automatic due decisions, but does not cancel active work or disable explicit commands. Review [automatic checks](automation.md) and keep the trial supervised; background candidate work is not yet product-validated.
@@ -1,15 +1,26 @@
1
1
  # Getting started
2
2
 
3
- ## Prerequisites
3
+ The [npm agent-evaluation preview](agent-evaluations.md) adds a separate CLI-change loop for Claude Code, Codex and OpenCode. It is not part of stable npm 0.1.6 or the Pi setup below.
4
4
 
5
- - Node.js 22.19 or newer.
6
- - npm.
7
- - Pi 0.87-compatible provider authentication.
8
- - A product repository in which Pi can create and verify evaluation and observability artifacts.
5
+ ## Open your inference environment
9
6
 
10
- OpenMerit does not require or accept an LLM provider credential. Provider access belongs to the coding harness.
7
+ With Node.js 22.19 or newer and npm installed, run:
11
8
 
12
- ## Install from npm
9
+ ```sh
10
+ npm install -g openmerit
11
+ cd /path/to/your/application
12
+ openmerit dash
13
+ ```
14
+
15
+ The Inference Terminal is included in stable `openmerit@0.1.6`. It opens a local, read-only view of the selected application's services, models, recorded requests, and linked compute. It does not require Pi or a provider API key, make inference calls, or change model assignments. Supported code/configuration declarations identify inventory; request history requires existing supported logs or exports. See [the terminal guide](inference-terminal.md) for collection, privacy, configuration, and diagnostics.
16
+
17
+ To inspect fictional data without installation, open the [hosted dashboard](https://dash.openmerit.ai/) or its [before/after comparison](https://dash.openmerit.ai/demo). Neither connects to your machine.
18
+
19
+ ## Set up model comparisons with Pi
20
+
21
+ The Pi integration is a separate workflow for evaluations and authorized model changes. It requires Pi 0.87 with a model provider configured, and a product repository where Pi can create and verify evaluation and observability artifacts. Provider credentials belong to Pi or its environment; OpenMerit does not accept them.
22
+
23
+ Install the extension in Pi:
13
24
 
14
25
  ```sh
15
26
  pi install npm:openmerit
@@ -36,27 +47,27 @@ Start Pi with the extension:
36
47
 
37
48
  ## Establish a product profile
38
49
 
39
- Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. In the unreleased source checkout it also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
50
+ Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. It also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
40
51
 
41
52
  OpenMerit sends Pi a private, typed `establish_evals` intent. Pi must identify the application LLM call or shared route under evaluation and its exact incumbent application model. It inspects the application's outputs, tools, and failure modes to infer task-relevant metrics instead of asking you to invent them. Before running setup, Pi shows the proposed metrics, why each matters, the representative case and repetition counts, aggregation, any actual performance threshold, and estimated evaluation cost. You confirm or edit that plan together with the stable target ID, call sites, route key, budget, check cadence, scheduling mode, and automation permissions. Pi's own model is never an application target.
42
53
 
43
54
  For an active-session demo, Pi defaults the baseline and post-swap verification cadence to one completed evaluation task unless you change it. OpenMerit rejects empty baseline or post-swap cadences.
44
55
 
45
- In this source checkout (not yet published in npm `openmerit@0.1.5`), Pi is also asked to research preliminary model leads during setup. Each reported lead has a current price estimate and reputation summary with sources and check times. OpenMerit stores these leads with the project profile. The field is optional for compatibility with existing protocol-0.1 harnesses, and an empty list means no credible leads were found. It does not change the application model or count as a measurement.
56
+ Pi is also asked to research preliminary model leads during setup. Each reported lead has a current price estimate and reputation summary with sources and check times. OpenMerit stores these leads with the project profile. The field is optional for compatibility with existing protocol-0.1 harnesses, and an empty list means no credible leads were found. It does not change the application model or count as a measurement.
46
57
 
47
58
  OpenMerit rejects successful setup without a confirmed task profile, at least one valid objective with an explicit sampling plan, a supervised-graduation policy, non-empty baseline and post-swap cadences, a budget, and evidence.
48
59
 
49
- The source checkout's setup instruction asks Pi to run the application path with working provider authentication, execute each required collector and grader, inspect the resulting metric values, and retain verification artifacts. A metric that cannot be produced must be marked missing. This smoke run verifies the pipeline; it is not counted as a production baseline sample. OpenMerit still checks the reported coverage and evidence references, not the authenticity of a grader run itself.
60
+ The setup instruction asks Pi to run the application path with working provider authentication, execute each required collector and grader, inspect the resulting metric values, and retain verification artifacts. A metric that cannot be produced must be marked missing. This smoke run verifies the pipeline; it is not counted as a production baseline sample. OpenMerit still checks the reported coverage and evidence references, not the authenticity of a grader run itself.
50
61
 
51
62
  ## Collect the baseline
52
63
 
53
- After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. In the unreleased source checkout, OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient. Published `openmerit@0.1.5` still nudges Pi to assess.
64
+ After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient.
54
65
 
55
66
  ## Find the frontier
56
67
 
57
68
  When the baseline meets every required metric threshold, OpenMerit issues the next approved nudges: discover candidates, run controlled challenger trials, and calculate the Pareto frontier. Pi performs that work. OpenMerit checks the returned result against the [Pareto specification](pareto-spec.md).
58
69
 
59
- Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application. In npm `openmerit@0.1.5`, the shortlist comes after the application's baseline has enough evidence. The unreleased source checkout collects preliminary leads at setup, then refreshes them, probes availability, and freezes the incumbent plus challengers. Pi must use the same cases, application revision, prompt, tools, grader, and run count, with a recorded shuffle seed. It preserves one experiment manifest and reports measured quality, cost, latency, tokens, and other required metrics. OpenMerit durably carries the accepted candidate set and assessments into later intents rather than relying on Pi conversation history. A model proposal must come from the verified frontier of those measured candidate results, not from price or reputation alone.
70
+ Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application. OpenMerit collects preliminary leads at setup, then refreshes them, probes availability, and freezes the incumbent plus challengers. Pi must use the same cases, application revision, prompt, tools, grader, and run count, with a recorded shuffle seed. It preserves one experiment manifest and reports measured quality, cost, latency, tokens, and other required metrics. OpenMerit durably carries the accepted candidate set and assessments into later intents rather than relying on Pi conversation history. A model proposal must come from the verified frontier of those measured candidate results, not from price or reputation alone.
60
71
 
61
72
  ## Approve an early swap
62
73
 
package/docs/index.md CHANGED
@@ -1,51 +1,40 @@
1
1
  # OpenMerit documentation
2
2
 
3
- OpenMerit helps your coding harness compare models against the work your product needs to do. You define what matters; the harness gathers measurements and tests candidates; OpenMerit coordinates the process and checks the returned results.
3
+ These guides describe the implementation on **main**. The standalone Inference Terminal is included in stable npm `openmerit@0.1.6`; interface changes available in source and the hosted dashboard but absent from that npm release are identified separately. The website, hosted samples, and npm package are deployed independently.
4
4
 
5
- The optional orchestration integration is intended for supervised trials with Pi. Start with a representative task and working evaluations before relying on automatic model changes.
5
+ ## See your inference environment
6
6
 
7
- ## View your application inference
8
-
9
- Version `0.1.6` includes the local Inference Terminal as an early starting point for OpenMerit. Requires Node.js 22.19 or newer:
7
+ With Node.js 22.19 or newer:
10
8
 
11
9
  ```sh
12
10
  npm install -g openmerit
11
+ cd /path/to/your/application
13
12
  openmerit dash
14
13
  ```
15
14
 
16
- Run it from your application's directory. See the [Inference Terminal guide](inference-terminal.md) for supported sources, recorded activity, background indexing, and data limits. Pi setup is separate and optional for terminal users.
17
-
18
- ## Start with your product
15
+ The terminal shows application services, model declarations, recorded requests, and linked compute. It reads supported local evidence, needs no Pi installation or provider API key, and makes no inference calls or model changes. Configuration can identify declared services; actual activity requires existing supported logs or exports.
19
16
 
20
- Install the extension in Pi, then open your product repository:
17
+ Start with [installation and setup](getting-started.md) or the [Inference Terminal guide](inference-terminal.md). Explore the [hosted sample](https://dash.openmerit.ai/) and [before/after dashboard demo](https://dash.openmerit.ai/demo) without connecting your machine. The [demo guide](demo.md) explains their fictional data and comparison limits.
21
18
 
22
- ```sh
23
- pi install npm:openmerit
24
- ```
19
+ ## Discover model candidates
25
20
 
26
- Interactive setup asks Pi to establish your task requirements, evaluation budget, automation permissions, and measurements. Follow the [getting started guide](getting-started.md), then work through a [first supervised trial](first-trial.md).
21
+ For a shortlist before building an integration, the separate [Scout beta](scout.md) accepts a task description and returns up to five evidence-backed model candidates. It is available at [openmerit.ai/scout/](https://openmerit.ai/scout/), with a prompt-only JSON API and MCP tool for coding agents.
27
22
 
28
- ## Explore the demos
23
+ ## Evaluate model changes with Pi
29
24
 
30
- The [Demo section](demo.md) explains how each prepared example works, from a single ticket-classification call to a multi-model agent workflow. Watch the recorded comparisons and inspect their inputs, grading rules, model routes, and evidence.
25
+ The separate Pi 0.87 integration asks a coding harness to establish evaluations, gather application measurements, and test candidates. OpenMerit coordinates the approved lifecycle, records evidence references, validates structured results and frontier conformance, and requests authorized changes. The harness executes the work; the terminal does not perform this workflow.
31
26
 
32
- ## Understand the decisions
27
+ Follow [Pi setup](getting-started.md#set-up-model-comparisons-with-pi), then [your first supervised trial](first-trial.md).
33
28
 
34
- OpenMerit compares candidates against the objectives you confirm, such as quality, cost, and latency. Several models may offer useful tradeoffs. A cheaper model is not automatically a better fit.
29
+ - [Metrics and evidence](metrics-and-evidence.md): measurements, sample requirements, and readiness.
30
+ - [Comparisons and swaps](lifecycle.md): baseline, challenger trials, approval, verification, and rollback.
31
+ - [Budgets and permissions](budgets.md): challenger accounting and provider-spend limits.
32
+ - [Automatic checks](automation.md): durable signals, cadence, and authorized macOS scheduling.
35
33
 
36
- - [Metrics and evidence](metrics-and-evidence.md) explains which measurements and samples a comparison needs.
37
- - [Comparisons and swaps](lifecycle.md) follows a baseline through challenger trials, approval, and post-swap checks.
38
- - [Budgets and permissions](budgets.md) describes the limits you give the harness and where enforcement depends on it.
39
- - [Automatic checks](automation.md) explains when work becomes due and what needs an external scheduler.
34
+ ## Operate and troubleshoot
40
35
 
41
- ## Operate with visibility
42
-
43
- Use [commands](commands.md) to inspect status and request work. [Project files and logs](operations.md) explains what is stored, and [troubleshooting](troubleshooting.md) covers common setup and evidence problems.
44
-
45
- OpenMerit checks structured results and model comparisons. It relies on the harness to execute the work and provide honest evidence. Read the [security notes](security.md) and [current limitations](roadmap.md) before enabling automation.
36
+ [Commands](commands.md) and [project files and logs](operations.md) explain status, pause, diagnostics, and recovery. [Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) describe the boundaries to check before relying on automation.
46
37
 
47
38
  ## Build an integration
48
39
 
49
- Pi 0.87 is the only implemented adapter today. The core and protocol can support other harnesses through an adapter.
50
-
51
- Start with the [architecture](architecture.md), then the [adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines the comparison rules, and [testing and validation](testing.md) explains the evidence behind implementation claims.
40
+ Pi is the implemented coding-harness adapter. Other harnesses can implement the protocol and coordinator boundary described in [architecture](architecture.md) and [the adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines comparison rules; [testing and validation](testing.md) distinguishes automated coverage from product evidence.