openmerit 0.1.6 → 0.1.7-preview.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +4 -4
- package/README.md +61 -40
- package/dist/eval/src/adapters.mjs +73 -0
- package/dist/eval/src/cli.mjs +103 -0
- package/dist/eval/src/config.mjs +102 -0
- package/dist/eval/src/process.mjs +44 -0
- package/dist/eval/src/report.mjs +51 -0
- package/dist/eval/src/runner.mjs +106 -0
- package/dist/eval/src/scout.mjs +37 -0
- package/dist/terminal/public/app.js +36 -36
- package/dist/terminal/public/flow.css +1 -1
- package/dist/terminal/public/flow.js +26 -26
- package/dist/terminal/public/index.html +3 -2
- package/dist/terminal/public/styles.css +320 -91
- package/dist/terminal/src/cli.mjs +2 -0
- package/dist/terminal/src/demo-policy.mjs +205 -0
- package/dist/terminal/src/enterprise-sample.mjs +151 -0
- package/docs/adapter-guide.md +2 -2
- package/docs/agent-evaluations.md +101 -0
- package/docs/architecture.md +3 -3
- package/docs/automation.md +5 -5
- package/docs/budgets.md +2 -2
- package/docs/commands.md +7 -5
- package/docs/demo-backfill.md +3 -1
- package/docs/demo-fieldkit.md +3 -1
- package/docs/demo-placement.md +3 -1
- package/docs/demo-spam.md +3 -1
- package/docs/demo-support.md +3 -1
- package/docs/demo.md +22 -5
- package/docs/first-trial.md +2 -2
- package/docs/getting-started.md +23 -12
- package/docs/index.md +18 -29
- package/docs/inference-terminal.md +90 -54
- package/docs/interactive-demo.md +118 -0
- package/docs/lifecycle.md +4 -4
- package/docs/metrics-and-evidence.md +3 -3
- package/docs/operations.md +3 -5
- package/docs/pi-extension.md +6 -6
- package/docs/roadmap.md +17 -17
- package/docs/scout.md +206 -0
- package/docs/security.md +6 -4
- package/docs/site-demo.md +4 -0
- package/docs/site-design.md +7 -1
- package/docs/site-documentation.md +9 -3
- package/docs/site-domain-migration.md +57 -0
- package/docs/site-dynamic-og.md +2 -2
- package/docs/site-faq-maintenance.md +121 -6
- package/docs/site-hero-resolution.md +2 -0
- package/docs/site-illustration-sequences.md +3 -1
- package/docs/site-illustration-transparency.md +80 -0
- package/docs/site-memo.md +6 -0
- package/docs/site-og-image.md +1 -1
- package/docs/site-publication-2026-10-14.md +29 -0
- package/docs/site-scout-decisions-experiment.md +87 -0
- package/docs/site-scout.md +254 -0
- package/docs/site-terminal-preview.md +96 -3
- package/docs/site-themes.md +30 -0
- package/docs/testing.md +37 -78
- package/docs/troubleshooting.md +3 -3
- package/package.json +9 -3
package/docs/commands.md
CHANGED
|
@@ -1,10 +1,12 @@
|
|
|
1
1
|
# Commands
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
|
|
4
|
+
|
|
5
|
+
The npm `0.1.7-preview.1` release (`openmerit@preview`) also provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks, budgets and release status. These are shell commands, separate from Pi commands.
|
|
4
6
|
|
|
5
7
|
## Inference Terminal
|
|
6
8
|
|
|
7
|
-
Install
|
|
9
|
+
Install with `npm install -g openmerit` to use `openmerit dash [path]` for a passive local view of models, connections, usage, and compute. From this checkout, use `npm run dash -- [path]`. It is independent of Pi setup and does not run model comparisons or inference.
|
|
8
10
|
|
|
9
11
|
See the [Inference Terminal guide](inference-terminal.md) for local installation, automatic discovery, `--no-discovery`, `--json`, `--demo`, collection intervals, and data limitations. Discovery is part of `dash` and is scoped to application evidence. Coding-harness histories are not collected.
|
|
10
12
|
|
|
@@ -25,7 +27,7 @@ Both show the task profile, metric readiness, evaluation status, active work, ne
|
|
|
25
27
|
|
|
26
28
|
Requests setup explicitly or reopens setup to correct stale target/incumbent state. Accepted replacement setup clears prior candidate, assessment, frontier, and baseline handoffs while retaining the audit log.
|
|
27
29
|
|
|
28
|
-
Request setup explicitly only when recovering, rerunning, or configuring a custom integration.
|
|
30
|
+
Request setup explicitly only when recovering, rerunning, or configuring a custom integration. An unconfigured project starts setup when bounded source detection finds an application LLM call at session start or after a settled Pi build turn. OpenMerit refuses another intent while work is active.
|
|
29
31
|
|
|
30
32
|
## Cancel interrupted work
|
|
31
33
|
|
|
@@ -41,7 +43,7 @@ Record the active intent as cancelled after an interrupted or abandoned harness
|
|
|
41
43
|
/openmerit assess
|
|
42
44
|
```
|
|
43
45
|
|
|
44
|
-
|
|
46
|
+
Ask OpenMerit core to check stored baseline metric windows now. It reports missing or immature required metrics without starting Pi, or starts candidate discovery when they pass. It does not create missing measurements or override sample requirements.
|
|
45
47
|
|
|
46
48
|
## Request a frontier
|
|
47
49
|
|
|
@@ -66,7 +68,7 @@ Approve the current proposal after its frontier result has passed validation. If
|
|
|
66
68
|
/openmerit resume
|
|
67
69
|
```
|
|
68
70
|
|
|
69
|
-
|
|
71
|
+
Pause is persisted in `.openmerit/state.json`, blocks automatic due decisions and lifecycle follow-ups, and unloads the project's managed macOS LaunchAgent. Incoming signals can still be recorded. Resume clears pause and restores scheduling only when the confirmed policy authorizes it. Neither command cancels an already-active intent; explicit commands remain available. The standalone Inference Terminal is independent and keeps collecting until its CLI process exits.
|
|
70
72
|
|
|
71
73
|
## Locate logs and evidence
|
|
72
74
|
|
package/docs/demo-backfill.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Backfill recovery
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
|
|
4
|
+
|
|
5
|
+
The agent workflow in the checkout’s recorded player (`site/demo/`) prepares a recovery manifest for an interrupted data backfill. Its goal is to resume recoverable partitions without duplicating committed data, quarantine malformed input, and stop for investigation when the failure is unknown or involves a checksum conflict. It never executes the manifest or writes to a database.
|
|
4
6
|
|
|
5
7
|
## Workflow structure
|
|
6
8
|
|
package/docs/demo-fieldkit.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Field kit · Six tasks
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
|
|
4
|
+
|
|
5
|
+
Watch the six-task field-kit recording in the checkout’s recorded player (`site/demo/`). The captured run compares two heterogeneous model routes using real OpenRouter calls through Pi and OpenMerit. Each route uses six distinct models; five assignments change and the feasibility model is retained.
|
|
4
6
|
|
|
5
7
|
The field-kit workflow prepares an offline recording handoff from a Spanish technician brief, a synthetic equipment-label image, an HTML operating manual, and a Python capture script. Both paths already use **six different models**. The approved route pairs narrow specialists with broader coding, reasoning, and writing models:
|
|
6
8
|
|
package/docs/demo-placement.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# GPU placement
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
|
|
4
|
+
|
|
5
|
+
The GPU placement workflow in the checkout’s recorded player (`site/demo/`) prepares a pool assignment for a synthetic render job. It reads job requirements with `read_job`, inspects a fixed capacity snapshot with `list_pools`, then proposes the cheapest eligible pool. It never reserves capacity, launches compute, moves data, or incurs a GPU charge.
|
|
4
6
|
|
|
5
7
|
## Workflow structure
|
|
6
8
|
|
package/docs/demo-spam.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Spam filter · Jev
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
|
|
4
|
+
|
|
5
|
+
The Jev task in the checkout’s recorded player (`site/demo/`) classifies a community comment as `publish` or `hold` under a narrow promotional-spam policy. Ordinary discussion, criticism, and reports of spam are allowed. Unsolicited promotional solicitations are held. The fixtures contain twelve examples of each class, with six separate verification comments.
|
|
4
6
|
|
|
5
7
|
## Models and integration
|
|
6
8
|
|
package/docs/demo-support.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Support tickets
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> Recorded example · September 28, 2026. This guide describes the recording retained in main, not a current product benchmark or the live interactive demo. For current samples, see [Demos and samples](demo.md).
|
|
4
|
+
|
|
5
|
+
Watch the support-ticket demo in the checkout’s recorded player (`site/demo/`). One model call turns a synthetic customer email into a structured ticket. Both paths start with GPT-4.1 mini and use the same prompt and JSON schema. The fixed path keeps that model; the managed path changes only after measured comparison and operator approval.
|
|
4
6
|
|
|
5
7
|
## Input and output
|
|
6
8
|
|
package/docs/demo.md
CHANGED
|
@@ -1,8 +1,25 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Demos and samples
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
## Explore the terminal
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
- [Hosted enterprise sample](https://dash.openmerit.ai/): 200 fictional agents, 699 task IDs, 21 models, and 9.6 million requests across 30 days. It demonstrates inventory, paths, activity, shared compute, and source coverage without reading your machine.
|
|
6
|
+
- [Before/after comparison](https://dash.openmerit.ai/demo): fourteen fictional insurance workflows and six preference choices across 26 model/service offers. Workloads and token volume stay fixed; costs are modeled from the dated price catalog. Benchmark and efficiency exceptions remain visible as preference overrides.
|
|
7
|
+
|
|
8
|
+
Neither hosted terminal example makes inference calls, applies model changes, or establishes application quality or latency. See [the comparison methodology](inference-terminal.md#hosted-enterprise-sample-and-comparison) for the exact selection rules, costs, and evidence limits.
|
|
9
|
+
|
|
10
|
+
## Interactive website experiment
|
|
11
|
+
|
|
12
|
+
The separately deployed [delivery-resolution experiment](https://openmerit.ai/demo/) uses fictional customer cases and bounded live provider calls. Its [guide](interactive-demo.md) explains the current scenarios, approval and verification steps, and session limits. This is distinct from the passive dashboard samples above.
|
|
13
|
+
|
|
14
|
+
## Run your own supervised trial
|
|
15
|
+
|
|
16
|
+
Use [Pi setup](getting-started.md#set-up-model-comparisons-with-pi) and [your first trial](first-trial.md) for real application measurements. The repository's separate `npm run demo` sandbox makes live provider calls; its requirements and spend boundaries are in [testing](testing.md#experimental-sandbox).
|
|
17
|
+
|
|
18
|
+
## Recorded player
|
|
19
|
+
|
|
20
|
+
Main also retains a recorded player in `site/demo/`, served at `/demo` by this checkout's static preview. Its September 28, 2026 recordings and the walkthroughs below document specific earlier experiments. They are not a description of the separately deployed interactive website demo, current provider availability, or a new validation of main. Existing recording links remain available for inspecting their evidence.
|
|
21
|
+
|
|
22
|
+
## Recorded examples in this checkout
|
|
6
23
|
|
|
7
24
|
| Scope | Demo | How it is structured |
|
|
8
25
|
| --- | --- | --- |
|
|
@@ -25,11 +42,11 @@ The agent examples compare supplied route configurations as whole workflows. The
|
|
|
25
42
|
|
|
26
43
|
Pi invokes bounded helpers and submits saved artifacts by name through a demo-specific transport bridge. The bridge expands those references into the adapter's canonical results and signals. The actual OpenMerit lifecycle then validates and advances the comparison. Missing provider cost stops a comparison instead of being treated as zero; a partial run is not a successful outcome.
|
|
27
44
|
|
|
28
|
-
The
|
|
45
|
+
The repository runner supplies explicit sampling plans, metric aggregations, and an incumbent descriptor to match the current protocol. Candidate limits count the incumbent and challengers together. Its comparison manifest records the fixed fixtures, rubric, model routes, and actual run order; the incumbent's baseline is reused in the comparison and included in evaluation-spend accounting. Runs are not randomized. The recordings retain their original execution evidence; the current runner does not establish a new validation result without a new run.
|
|
29
46
|
|
|
30
47
|
## Playback controls
|
|
31
48
|
|
|
32
|
-
The player is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
|
|
49
|
+
The player in `site/demo/` is for recorded playback. It makes no model calls, starts no sandbox, and accepts no custom inputs. The **How it works** link opens the guide for the selected example.
|
|
33
50
|
|
|
34
51
|
- **Pause / Play** stops or resumes playback. Seeking on the timeline pauses at the selected position.
|
|
35
52
|
- **Replay** starts the recording again from the beginning.
|
package/docs/first-trial.md
CHANGED
|
@@ -35,7 +35,7 @@ These commands help locate readiness and evidence. They do not independently cer
|
|
|
35
35
|
|
|
36
36
|
Use the product normally while its evaluations collect representative measurements. OpenMerit also records completed-work checkpoints; these checkpoints alone do not establish product quality.
|
|
37
37
|
|
|
38
|
-
|
|
38
|
+
OpenMerit core checks baseline sufficiency from reported metric windows; when they pass, it asks Pi to discover candidates, run challenger trials, and calculate a comparison. You can request a check explicitly:
|
|
39
39
|
|
|
40
40
|
```text
|
|
41
41
|
/openmerit assess
|
|
@@ -57,4 +57,4 @@ The harness applies the change. Allow the configured post-swap evaluation window
|
|
|
57
57
|
|
|
58
58
|
Keep the trial’s artifacts and audit trail. Before enabling automatic changes, confirm that your evaluations detect meaningful failures, provider costs are visible, and post-swap verification and rollback work for this task.
|
|
59
59
|
|
|
60
|
-
|
|
60
|
+
Persistent checks require an authorized macOS LaunchAgent or an external scheduler on other platforms. Pause survives restarts and stops automatic due decisions, but does not cancel active work or disable explicit commands. Review [automatic checks](automation.md) and keep the trial supervised; background candidate work is not yet product-validated.
|
package/docs/getting-started.md
CHANGED
|
@@ -1,15 +1,26 @@
|
|
|
1
1
|
# Getting started
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The [npm agent-evaluation preview](agent-evaluations.md) adds a separate CLI-change loop for Claude Code, Codex and OpenCode. It is not part of stable npm 0.1.6 or the Pi setup below.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
- npm.
|
|
7
|
-
- Pi 0.87-compatible provider authentication.
|
|
8
|
-
- A product repository in which Pi can create and verify evaluation and observability artifacts.
|
|
5
|
+
## Open your inference environment
|
|
9
6
|
|
|
10
|
-
|
|
7
|
+
With Node.js 22.19 or newer and npm installed, run:
|
|
11
8
|
|
|
12
|
-
|
|
9
|
+
```sh
|
|
10
|
+
npm install -g openmerit
|
|
11
|
+
cd /path/to/your/application
|
|
12
|
+
openmerit dash
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The Inference Terminal is included in stable `openmerit@0.1.6`. It opens a local, read-only view of the selected application's services, models, recorded requests, and linked compute. It does not require Pi or a provider API key, make inference calls, or change model assignments. Supported code/configuration declarations identify inventory; request history requires existing supported logs or exports. See [the terminal guide](inference-terminal.md) for collection, privacy, configuration, and diagnostics.
|
|
16
|
+
|
|
17
|
+
To inspect fictional data without installation, open the [hosted dashboard](https://dash.openmerit.ai/) or its [before/after comparison](https://dash.openmerit.ai/demo). Neither connects to your machine.
|
|
18
|
+
|
|
19
|
+
## Set up model comparisons with Pi
|
|
20
|
+
|
|
21
|
+
The Pi integration is a separate workflow for evaluations and authorized model changes. It requires Pi 0.87 with a model provider configured, and a product repository where Pi can create and verify evaluation and observability artifacts. Provider credentials belong to Pi or its environment; OpenMerit does not accept them.
|
|
22
|
+
|
|
23
|
+
Install the extension in Pi:
|
|
13
24
|
|
|
14
25
|
```sh
|
|
15
26
|
pi install npm:openmerit
|
|
@@ -36,27 +47,27 @@ Start Pi with the extension:
|
|
|
36
47
|
|
|
37
48
|
## Establish a product profile
|
|
38
49
|
|
|
39
|
-
Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight.
|
|
50
|
+
Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. It also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
|
|
40
51
|
|
|
41
52
|
OpenMerit sends Pi a private, typed `establish_evals` intent. Pi must identify the application LLM call or shared route under evaluation and its exact incumbent application model. It inspects the application's outputs, tools, and failure modes to infer task-relevant metrics instead of asking you to invent them. Before running setup, Pi shows the proposed metrics, why each matters, the representative case and repetition counts, aggregation, any actual performance threshold, and estimated evaluation cost. You confirm or edit that plan together with the stable target ID, call sites, route key, budget, check cadence, scheduling mode, and automation permissions. Pi's own model is never an application target.
|
|
42
53
|
|
|
43
54
|
For an active-session demo, Pi defaults the baseline and post-swap verification cadence to one completed evaluation task unless you change it. OpenMerit rejects empty baseline or post-swap cadences.
|
|
44
55
|
|
|
45
|
-
|
|
56
|
+
Pi is also asked to research preliminary model leads during setup. Each reported lead has a current price estimate and reputation summary with sources and check times. OpenMerit stores these leads with the project profile. The field is optional for compatibility with existing protocol-0.1 harnesses, and an empty list means no credible leads were found. It does not change the application model or count as a measurement.
|
|
46
57
|
|
|
47
58
|
OpenMerit rejects successful setup without a confirmed task profile, at least one valid objective with an explicit sampling plan, a supervised-graduation policy, non-empty baseline and post-swap cadences, a budget, and evidence.
|
|
48
59
|
|
|
49
|
-
The
|
|
60
|
+
The setup instruction asks Pi to run the application path with working provider authentication, execute each required collector and grader, inspect the resulting metric values, and retain verification artifacts. A metric that cannot be produced must be marked missing. This smoke run verifies the pipeline; it is not counted as a production baseline sample. OpenMerit still checks the reported coverage and evidence references, not the authenticity of a grader run itself.
|
|
50
61
|
|
|
51
62
|
## Collect the baseline
|
|
52
63
|
|
|
53
|
-
After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics.
|
|
64
|
+
After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient.
|
|
54
65
|
|
|
55
66
|
## Find the frontier
|
|
56
67
|
|
|
57
68
|
When the baseline meets every required metric threshold, OpenMerit issues the next approved nudges: discover candidates, run controlled challenger trials, and calculate the Pareto frontier. Pi performs that work. OpenMerit checks the returned result against the [Pareto specification](pareto-spec.md).
|
|
58
69
|
|
|
59
|
-
Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application.
|
|
70
|
+
Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application. OpenMerit collects preliminary leads at setup, then refreshes them, probes availability, and freezes the incumbent plus challengers. Pi must use the same cases, application revision, prompt, tools, grader, and run count, with a recorded shuffle seed. It preserves one experiment manifest and reports measured quality, cost, latency, tokens, and other required metrics. OpenMerit durably carries the accepted candidate set and assessments into later intents rather than relying on Pi conversation history. A model proposal must come from the verified frontier of those measured candidate results, not from price or reputation alone.
|
|
60
71
|
|
|
61
72
|
## Approve an early swap
|
|
62
73
|
|
package/docs/index.md
CHANGED
|
@@ -1,51 +1,40 @@
|
|
|
1
1
|
# OpenMerit documentation
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
These guides describe the implementation on **main**. The standalone Inference Terminal is included in stable npm `openmerit@0.1.6`; interface changes available in source and the hosted dashboard but absent from that npm release are identified separately. The website, hosted samples, and npm package are deployed independently.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## See your inference environment
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
Version `0.1.6` includes the local Inference Terminal as an early starting point for OpenMerit. Requires Node.js 22.19 or newer:
|
|
7
|
+
With Node.js 22.19 or newer:
|
|
10
8
|
|
|
11
9
|
```sh
|
|
12
10
|
npm install -g openmerit
|
|
11
|
+
cd /path/to/your/application
|
|
13
12
|
openmerit dash
|
|
14
13
|
```
|
|
15
14
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
## Start with your product
|
|
15
|
+
The terminal shows application services, model declarations, recorded requests, and linked compute. It reads supported local evidence, needs no Pi installation or provider API key, and makes no inference calls or model changes. Configuration can identify declared services; actual activity requires existing supported logs or exports.
|
|
19
16
|
|
|
20
|
-
|
|
17
|
+
Start with [installation and setup](getting-started.md) or the [Inference Terminal guide](inference-terminal.md). Explore the [hosted sample](https://dash.openmerit.ai/) and [before/after dashboard demo](https://dash.openmerit.ai/demo) without connecting your machine. The [demo guide](demo.md) explains their fictional data and comparison limits.
|
|
21
18
|
|
|
22
|
-
|
|
23
|
-
pi install npm:openmerit
|
|
24
|
-
```
|
|
19
|
+
## Discover model candidates
|
|
25
20
|
|
|
26
|
-
|
|
21
|
+
For a shortlist before building an integration, the separate [Scout beta](scout.md) accepts a task description and returns up to five evidence-backed model candidates. It is available at [openmerit.ai/scout/](https://openmerit.ai/scout/), with a prompt-only JSON API and MCP tool for coding agents.
|
|
27
22
|
|
|
28
|
-
##
|
|
23
|
+
## Evaluate model changes with Pi
|
|
29
24
|
|
|
30
|
-
The
|
|
25
|
+
The separate Pi 0.87 integration asks a coding harness to establish evaluations, gather application measurements, and test candidates. OpenMerit coordinates the approved lifecycle, records evidence references, validates structured results and frontier conformance, and requests authorized changes. The harness executes the work; the terminal does not perform this workflow.
|
|
31
26
|
|
|
32
|
-
|
|
27
|
+
Follow [Pi setup](getting-started.md#set-up-model-comparisons-with-pi), then [your first supervised trial](first-trial.md).
|
|
33
28
|
|
|
34
|
-
|
|
29
|
+
- [Metrics and evidence](metrics-and-evidence.md): measurements, sample requirements, and readiness.
|
|
30
|
+
- [Comparisons and swaps](lifecycle.md): baseline, challenger trials, approval, verification, and rollback.
|
|
31
|
+
- [Budgets and permissions](budgets.md): challenger accounting and provider-spend limits.
|
|
32
|
+
- [Automatic checks](automation.md): durable signals, cadence, and authorized macOS scheduling.
|
|
35
33
|
|
|
36
|
-
|
|
37
|
-
- [Comparisons and swaps](lifecycle.md) follows a baseline through challenger trials, approval, and post-swap checks.
|
|
38
|
-
- [Budgets and permissions](budgets.md) describes the limits you give the harness and where enforcement depends on it.
|
|
39
|
-
- [Automatic checks](automation.md) explains when work becomes due and what needs an external scheduler.
|
|
34
|
+
## Operate and troubleshoot
|
|
40
35
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
Use [commands](commands.md) to inspect status and request work. [Project files and logs](operations.md) explains what is stored, and [troubleshooting](troubleshooting.md) covers common setup and evidence problems.
|
|
44
|
-
|
|
45
|
-
OpenMerit checks structured results and model comparisons. It relies on the harness to execute the work and provide honest evidence. Read the [security notes](security.md) and [current limitations](roadmap.md) before enabling automation.
|
|
36
|
+
[Commands](commands.md) and [project files and logs](operations.md) explain status, pause, diagnostics, and recovery. [Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) describe the boundaries to check before relying on automation.
|
|
46
37
|
|
|
47
38
|
## Build an integration
|
|
48
39
|
|
|
49
|
-
Pi
|
|
50
|
-
|
|
51
|
-
Start with the [architecture](architecture.md), then the [adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines the comparison rules, and [testing and validation](testing.md) explains the evidence behind implementation claims.
|
|
40
|
+
Pi is the implemented coding-harness adapter. Other harnesses can implement the protocol and coordinator boundary described in [architecture](architecture.md) and [the adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines comparison rules; [testing and validation](testing.md) distinguishes automated coverage from product evidence.
|