openmerit 0.1.11 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +20 -53
- package/dist/pi/scout-extension.mjs +177 -0
- package/dist/pi/src/index.d.ts +6 -32
- package/dist/pi/src/index.js +49 -787
- package/dist/terminal/scout-update.json +1 -1
- package/dist/terminal/src/install.mjs +102 -166
- package/dist/terminal/src/scout-credentials.d.ts +12 -0
- package/dist/terminal/src/scout-credentials.js +41 -0
- package/dist/terminal/src/scout-credentials.ts +44 -0
- package/dist/terminal/src/scout-instructions.d.ts +5 -0
- package/dist/terminal/src/scout-instructions.js +103 -0
- package/dist/terminal/src/scout-instructions.ts +106 -0
- package/dist/terminal/src/scout-pi.d.ts +44 -0
- package/dist/terminal/src/scout-pi.js +84 -0
- package/dist/terminal/src/scout-pi.ts +76 -0
- package/dist/terminal/src/telemetry.mjs +2 -2
- package/docs/adapter-guide.md +2 -0
- package/docs/architecture.md +7 -5
- package/docs/automation.md +2 -0
- package/docs/budgets.md +2 -0
- package/docs/commands.md +3 -1
- package/docs/demo.md +1 -1
- package/docs/first-trial.md +2 -0
- package/docs/getting-started.md +19 -44
- package/docs/index.md +10 -6
- package/docs/inference-terminal.md +1 -1
- package/docs/lifecycle.md +2 -0
- package/docs/operations.md +2 -0
- package/docs/pi-extension.md +32 -28
- package/docs/roadmap.md +6 -5
- package/docs/scout.md +12 -7
- package/docs/security.md +7 -1
- package/docs/testing.md +2 -0
- package/docs/troubleshooting.md +23 -28
- package/docs/ux-reference.md +1 -1
- package/package.json +3 -4
- package/dist/pi/src/scheduler.d.ts +0 -11
- package/dist/pi/src/scheduler.js +0 -137
- package/dist/pi/src/wakeup.d.ts +0 -2
- package/dist/pi/src/wakeup.js +0 -108
- package/docs/saga/a-steadier-compass.md +0 -25
- package/docs/saga/defining-a-task-for-ai.md +0 -154
- package/docs/saga/the-benchmark-is-not-the-job.md +0 -33
package/docs/getting-started.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Getting started
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
OpenMerit provides a passive Inference Terminal, Scout model-selection guidance, and a separate [agent-evaluation CLI](agent-evaluations.md). Pi Scout setup requires OpenMerit **0.1.12** or newer and Pi **1.5.0**. Package publication and website deployment are independent releases.
|
|
4
4
|
|
|
5
5
|
## Open your inference environment
|
|
6
6
|
|
|
@@ -12,65 +12,40 @@ cd /path/to/your/application
|
|
|
12
12
|
openmerit dash
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
The Inference Terminal
|
|
15
|
+
The Inference Terminal opens a local, read-only view of the selected application's services, models, recorded requests, and linked compute. It does not require Pi or a provider API key, make inference calls, or change model assignments. Supported code/configuration declarations identify inventory; request history requires existing supported logs or exports. See [the terminal guide](inference-terminal.md) for collection, privacy, configuration, and diagnostics.
|
|
16
16
|
|
|
17
17
|
To inspect fictional data without installation, open the [hosted dashboard](https://dash.openmerit.ai/) or its [before/after comparison](https://dash.openmerit.ai/demo). Neither connects to your machine.
|
|
18
18
|
|
|
19
|
-
##
|
|
19
|
+
## Connect Scout to a coding agent
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
Install the extension in Pi:
|
|
24
|
-
|
|
25
|
-
```sh
|
|
26
|
-
pi install npm:openmerit
|
|
27
|
-
pi list
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
Run `/openmerit doctor` inside Pi to confirm that the extension is available.
|
|
31
|
-
|
|
32
|
-
## Install from source
|
|
21
|
+
Claude Code setup is described in [the Scout guide](scout.md#connect-a-coding-agent). For the new Pi integration, use Pi **1.5.0**, Node.js 22.19 or newer, and npm. Run the intended command in your shell:
|
|
33
22
|
|
|
34
23
|
```sh
|
|
35
|
-
|
|
36
|
-
cd openmerit
|
|
37
|
-
npm ci
|
|
24
|
+
npx -y openmerit@latest install --client pi --scope user
|
|
38
25
|
```
|
|
39
26
|
|
|
40
|
-
|
|
27
|
+
Approve the matching device code in the browser, then start a **new Pi session**. Only approve a request you initiated. User scope is the default and applies across projects. For one project, run this from that project's root instead:
|
|
41
28
|
|
|
42
29
|
```sh
|
|
43
|
-
|
|
30
|
+
npx -y openmerit@latest install --client pi --scope project
|
|
44
31
|
```
|
|
45
32
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
## Establish a product profile
|
|
49
|
-
|
|
50
|
-
Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. It also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
|
|
33
|
+
Each client/scope installation has its own private Scout key; project keys are stored outside the repository. Do not install overlapping Pi user/project adapters or a duplicate `openmerit` MCP entry. The installer reports conflicts rather than overwriting edited or unowned files.
|
|
51
34
|
|
|
52
|
-
|
|
35
|
+
The adapter connects Scout through Pi's native MCP and supplies the shared Scout rule and skill. Pi owns tool discovery, tool execution, model choice, permissions, and sessions. Scout recommends application-model candidates; it does not execute those candidates or change Pi's own model. There are no `/openmerit` lifecycle commands, automatic evaluation setup, scheduler, or LaunchAgent in this adapter.
|
|
53
36
|
|
|
54
|
-
|
|
37
|
+
## Check, update, or remove Pi setup
|
|
55
38
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
## Collect the baseline
|
|
63
|
-
|
|
64
|
-
After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient.
|
|
65
|
-
|
|
66
|
-
## Find the frontier
|
|
67
|
-
|
|
68
|
-
When the baseline meets every required metric threshold, OpenMerit issues the next approved nudges: discover candidates, run controlled challenger trials, and calculate the Pareto frontier. Pi performs that work. OpenMerit checks the returned result against the [Pareto specification](pareto-spec.md).
|
|
39
|
+
```sh
|
|
40
|
+
npx -y openmerit@latest status --client pi --scope user --check
|
|
41
|
+
npx -y openmerit@latest doctor --client pi --scope user
|
|
42
|
+
npx -y openmerit@latest remove --client pi --scope user
|
|
43
|
+
```
|
|
69
44
|
|
|
70
|
-
|
|
45
|
+
Use `--scope project` from the original project root for a project installation. `login` and `logout` accept the same `--client` and `--scope` flags. These commands target that installation, not another client's connection. Updating Pi setup requires explicitly rerunning `install` and starting a new session; no Pi auto-update hooks are installed. Installing or updating the npm package alone does not reconfigure Pi.
|
|
71
46
|
|
|
72
|
-
|
|
47
|
+
See [the Pi extension guide](pi-extension.md) for installed paths, migration from Pi 0.87, and SDK embedding. Use [troubleshooting](troubleshooting.md) for setup failures.
|
|
73
48
|
|
|
74
|
-
|
|
49
|
+
## Evaluate application changes
|
|
75
50
|
|
|
76
|
-
|
|
51
|
+
The standalone `openmerit eval` CLI is unchanged by the new Pi adapter. It evaluates Claude Code, Codex and OpenCode through explicit task, scope, budget and grader configuration; see [agent evaluations](agent-evaluations.md). The core evaluation coordinator remains available to custom harness integrations, but the removed Pi 0.87 lifecycle adapter is not the way to invoke it.
|
package/docs/index.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# OpenMerit documentation
|
|
2
2
|
|
|
3
|
-
These guides
|
|
3
|
+
These guides distinguish current source work from published behavior. OpenMerit **0.1.12** adds native Scout support for Pi **1.5.0**, replacing the old Pi 0.87 evaluation extension. The Inference Terminal and standalone agent-evaluation CLI are separate products and are unchanged by that replacement. The website, hosted samples, Scout service and npm package are deployed independently; dated receipts describe the versions actually tested.
|
|
4
4
|
|
|
5
5
|
## See your inference environment
|
|
6
6
|
|
|
@@ -20,11 +20,15 @@ Start with [installation and setup](getting-started.md) or the [Inference Termin
|
|
|
20
20
|
|
|
21
21
|
For a shortlist before building an integration, the separate [Scout beta](scout.md) accepts a task description and returns up to five evidence-backed model candidates. It is available at [openmerit.ai/scout/](https://openmerit.ai/scout/), with a prompt-only JSON API and MCP tool for coding agents.
|
|
22
22
|
|
|
23
|
-
##
|
|
23
|
+
## Connect Scout to Pi
|
|
24
24
|
|
|
25
|
-
The
|
|
25
|
+
The thin Pi 1.5.0 adapter uses native MCP and shared Scout guidance. Pi owns tools, models, permissions and sessions; the adapter has no `/openmerit` lifecycle commands, evaluation scheduling, LaunchAgent or automatic updates.
|
|
26
26
|
|
|
27
|
-
Follow [
|
|
27
|
+
Follow [setup](getting-started.md#connect-scout-to-a-coding-agent) and [the Pi guide](pi-extension.md), including browser approval, per-client/scope credentials, explicit reinstall updates and SDK requirements. Registry setup requires OpenMerit 0.1.12 or newer.
|
|
28
|
+
|
|
29
|
+
## Evaluate application changes
|
|
30
|
+
|
|
31
|
+
Use [the standalone evaluation CLI](agent-evaluations.md) for Claude Code, Codex and OpenCode. Custom harnesses may use the core coordinator. The following guides retain core concepts and historical Pi 0.87 workflows; they do not describe the new Scout adapter:
|
|
28
32
|
|
|
29
33
|
- [Metrics and evidence](metrics-and-evidence.md): measurements, sample requirements, and readiness.
|
|
30
34
|
- [Comparisons and swaps](lifecycle.md): baseline, challenger trials, approval, verification, and rollback.
|
|
@@ -33,8 +37,8 @@ Follow [Pi setup](getting-started.md#set-up-model-comparisons-with-pi), then [yo
|
|
|
33
37
|
|
|
34
38
|
## Operate and troubleshoot
|
|
35
39
|
|
|
36
|
-
[
|
|
40
|
+
[Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) explain current Pi setup and its boundaries. [Commands](commands.md) and [project files and logs](operations.md) also retain explicitly historical Pi lifecycle references alongside standalone/core information; their old pause and recovery commands are not part of the new adapter.
|
|
37
41
|
|
|
38
42
|
## Build an integration
|
|
39
43
|
|
|
40
|
-
Pi
|
|
44
|
+
Pi now supplies a native Scout connection, not an evaluation `HarnessAdapter`. Custom evaluation harnesses can implement the protocol and coordinator boundary described in [architecture](architecture.md) and [the adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines comparison rules; [testing and validation](testing.md) distinguishes automated coverage from product evidence.
|
|
@@ -79,7 +79,7 @@ openmerit dash --no-open
|
|
|
79
79
|
| `--demo` | Use only a fictional local sample; no local collectors run |
|
|
80
80
|
| `--help` | Show usage |
|
|
81
81
|
|
|
82
|
-
Ctrl+C stops collection. The terminal has no persistent daemon and is independent of Pi
|
|
82
|
+
Ctrl+C stops collection. The terminal has no persistent daemon and is independent of core evaluation pause/scheduling policy. The unreleased Pi 1.5.0 Scout adapter exposes no pause or scheduling commands.
|
|
83
83
|
|
|
84
84
|
## Repository indexing
|
|
85
85
|
|
package/docs/lifecycle.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Improvement lifecycle
|
|
2
2
|
|
|
3
|
+
**Core reference / historical Pi workflow:** this lifecycle remains available through the coordinator for custom harnesses. Pi-specific actions below describe the removed Pi 0.87 adapter, not the unreleased Pi 1.5.0 native Scout adapter. The replacement has no lifecycle commands, evaluation setup, swaps, or scheduling. See [Pi setup](pi-extension.md) or the separate, unchanged [evaluation CLI](agent-evaluations.md).
|
|
4
|
+
|
|
3
5
|
## Stages
|
|
4
6
|
|
|
5
7
|
1. **Unconfigured** — no confirmed profile exists.
|
package/docs/operations.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Project state and operations
|
|
2
2
|
|
|
3
|
+
**Scope — unreleased Pi replacement:** the evaluation lifecycle files and `/openmerit` commands below describe the core coordinator and historical Pi 0.87 adapter. The Pi 1.5.0 Scout adapter does not create or advance that lifecycle, manage scheduling, or expose those commands. See [current Pi paths and targeted shell commands](pi-extension.md). Terminal files remain current; standalone evaluation files are documented in [agent evaluations](agent-evaluations.md). Preserve older state and separately remove any legacy background job when migrating.
|
|
4
|
+
|
|
3
5
|
## State location
|
|
4
6
|
|
|
5
7
|
Operational data lives under `.openmerit/` in the target product repository. It should be ignored by version control because it can contain local lifecycle state and evidence locations.
|
package/docs/pi-extension.md
CHANGED
|
@@ -2,53 +2,57 @@
|
|
|
2
2
|
|
|
3
3
|
## Supported version
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
OpenMerit **0.1.12** adds a Scout adapter for Pi **1.5.0**. It replaces, rather than upgrades in place, the Pi 0.87 evaluation adapter. The `@latest` commands below require OpenMerit 0.1.12 or newer. Package publication and website deployment remain independent releases.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Installation
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
from a checkout, load the source extension with:
|
|
9
|
+
With Node.js 22.19 or newer, npm, and Pi 1.5.0 installed:
|
|
11
10
|
|
|
12
11
|
```sh
|
|
13
|
-
|
|
12
|
+
npx -y openmerit@latest install --client pi --scope user
|
|
14
13
|
```
|
|
15
14
|
|
|
16
|
-
|
|
15
|
+
Approve the matching browser device code and start a new Pi session. Scope defaults to `user`; use `--scope project` from the project's root for project-local setup. Each client and scope gets an independent key and approval; project keys are also separated by project. The npm package alone does not configure Pi.
|
|
17
16
|
|
|
18
|
-
|
|
17
|
+
The website's **Add to Pi** action is copy-only and sits below the Claude action, with the Pi logo before the label. It copies the installer command; it does not open a Pi deep link or prove installation. The button requires the corresponding Scout website deployment.
|
|
19
18
|
|
|
20
|
-
|
|
19
|
+
## Installed files and credentials
|
|
21
20
|
|
|
22
|
-
|
|
21
|
+
| File | User scope | Project scope |
|
|
22
|
+
| --- | --- | --- |
|
|
23
|
+
| Native extension bundle | `~/.pi/agent/extensions/openmerit-scout.ts` | `.pi/extensions/openmerit-scout.ts` |
|
|
24
|
+
| Credential-path reference | `~/.pi/agent/extensions/openmerit-scout.json` | `.pi/extensions/openmerit-scout.json` |
|
|
25
|
+
| Shared Scout skill | `~/.pi/agent/skills/openmerit-scout/SKILL.md` | `.pi/skills/openmerit-scout/SKILL.md` |
|
|
26
|
+
| Installer ownership record | `~/.pi/agent/openmerit-install.json` | `.pi/openmerit-install.json` |
|
|
23
27
|
|
|
24
|
-
|
|
28
|
+
`PI_CODING_AGENT_DIR` overrides the user agent directory. The connection JSON contains a `credentialsPath`, not the key. The default private key paths are `~/.config/openmerit/pi/user/credentials.json` for user scope and `~/.config/openmerit/pi/<project-id>/credentials.json` for project scope. The project ID derives from the resolved project root; run targeted commands from that root. `OPENMERIT_CONFIG_DIR` or `XDG_CONFIG_HOME` can override the private configuration base. Do not copy credentials into the project or model instructions.
|
|
25
29
|
|
|
26
|
-
|
|
30
|
+
The extension reads credentials at session startup and registers the `openmerit` HTTP MCP server with direct tool exposure. Authorization is an HTTP header, not prompt content. Missing or invalid configuration produces a sanitized setup diagnostic, not an automatic login. The adapter performs no per-turn HTTP connection checks or credential reads.
|
|
27
31
|
|
|
28
|
-
|
|
32
|
+
## Native ownership and shared guidance
|
|
29
33
|
|
|
30
|
-
|
|
34
|
+
Pi supplies the MCP connection, tool discovery and execution, permissions, and session management. The thin adapter adds the same Scout rule and skill used by the shared installer as structured system-prompt sections. It leaves the host's model, tool allow/exclude lists, and other prompt sections intact. This is guidance to consult Scout before application-model choices, not enforcement of tool invocation or permission to download, spend, deploy, or change models. Scout's [data-handling policy](scout.md#refresh-and-data-handling) applies to submitted tasks.
|
|
31
35
|
|
|
32
|
-
|
|
36
|
+
## Updates and targeted lifecycle commands
|
|
33
37
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
```sh
|
|
39
|
+
npx -y openmerit@latest status --client pi --scope user --check
|
|
40
|
+
npx -y openmerit@latest doctor --client pi --scope user
|
|
41
|
+
npx -y openmerit@latest login --client pi --scope user
|
|
42
|
+
npx -y openmerit@latest logout --client pi --scope user
|
|
43
|
+
npx -y openmerit@latest remove --client pi --scope user
|
|
44
|
+
```
|
|
41
45
|
|
|
42
|
-
|
|
46
|
+
Replace `user` with `project` for a project installation. Removal and logout target that client's scope/key, not a Claude Code installation or another project. Edited or unowned files are not silently overwritten or deleted. Resolve duplicate user/project adapters or existing `openmerit` MCP entries before installing.
|
|
43
47
|
|
|
44
|
-
|
|
48
|
+
Updates are **explicit reinstall only**: rerun `install --client pi` with the same scope and start a new session. There are no Pi auto-update hooks; Claude Code update-hook settings do not apply to Pi.
|
|
45
49
|
|
|
46
|
-
|
|
50
|
+
## SDK embedding
|
|
47
51
|
|
|
48
|
-
|
|
52
|
+
`openmerit/pi` exports `createScoutPiExtension({ credentialsPath })` and a default extension created by that factory. SDK hosts must explicitly load Pi's own **`createMcpExtension()`** alongside the Scout extension; the Scout factory does not create an MCP runtime. The Pi CLI already supplies native MCP. The explicit credential path takes precedence over the installed adjacent connection JSON, with the shared Pi user credential path as the fallback when that JSON is absent. The host supplies its own model, tools, session and permission configuration.
|
|
49
53
|
|
|
50
|
-
|
|
54
|
+
## Migration from Pi 0.87
|
|
51
55
|
|
|
52
|
-
|
|
56
|
+
Remove or disable the old `npm:openmerit` Pi package registration before loading this adapter, and explicitly remove any previously authorized background job through its existing management path. New setup does not promise to uninstall a legacy LaunchAgent. Preserve `.openmerit/` evaluation state and evidence if needed; do not delete them as a setup repair.
|
|
53
57
|
|
|
54
|
-
|
|
58
|
+
The replacement has no `/openmerit` commands, `openmerit_complete_intent` or `openmerit_report_signal` tools, source preflight, evaluation lifecycle, intent recovery, scheduling, automatic swaps, or LaunchAgent. The [core coordinator](architecture.md) remains available for custom harnesses. The [standalone evaluation CLI](agent-evaluations.md) is unchanged. Older lifecycle guides are retained as historical adapter/core references, not instructions for this Scout adapter.
|
package/docs/roadmap.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## Current status
|
|
4
4
|
|
|
5
|
-
The Inference Terminal provides a read-only view of existing application evidence. The Pi
|
|
5
|
+
The Inference Terminal provides a read-only view of existing application evidence. The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. The standalone evaluation CLI and reusable core coordinator remain separate. Neither a populated dashboard nor passing implementation tests establishes that an application has better models, trustworthy evaluations, or reliable unattended automation.
|
|
6
6
|
|
|
7
7
|
## Known limitations
|
|
8
8
|
|
|
@@ -10,10 +10,11 @@ The Inference Terminal provides a read-only view of existing application evidenc
|
|
|
10
10
|
- History and inventory can be incomplete because of source availability and read/parser limits. “All time” means retained supported evidence, not a complete lifetime archive. Cloud account usage is not fetched.
|
|
11
11
|
- Machine and runtime readings describe shared resources; they are not per-request resource measurements. Static endpoints do not prove deployment, availability, or physical location.
|
|
12
12
|
- Hosted samples and the before/after dashboard demo are fictional. Their cost projections and public benchmark screens do not establish application quality or latency.
|
|
13
|
-
- Pi 0.87 is
|
|
14
|
-
-
|
|
15
|
-
-
|
|
16
|
-
-
|
|
13
|
+
- The Pi 0.87 evaluation adapter is removed in source. Pi 1.5.0 setup needs explicit browser approval, a new session and an explicit installer rerun for updates. It has no lifecycle commands, scheduler, LaunchAgent or auto-update hooks. The host owns tools, model, permissions and sessions; SDK hosts must load Pi's native MCP extension explicitly.
|
|
14
|
+
- Scout rule/skill availability does not guarantee invocation or measured application quality. Missing project policy requires asking; unavailable or insufficient Scout evidence requires pausing the model choice.
|
|
15
|
+
- Custom core evaluation harnesses must establish task-specific evaluations and observability. Generic coding checkpoints cannot replace product telemetry, and structured-result validation cannot prove external evidence is truthful.
|
|
16
|
+
- Background provisioning and unattended validation belong to custom evaluation integrations, not the new Pi adapter. Existing legacy Pi jobs require separate cleanup during migration.
|
|
17
|
+
- Core pause policy does not cancel active intents or stop explicit commands or the standalone terminal. It is not a command exposed by the new Pi adapter.
|
|
17
18
|
- Challenger accounting depends on reported costs. It cannot independently meter provider billing or stop an in-flight request. Use provider spending controls as well as the confirmed evaluation policy.
|
|
18
19
|
- The source interface and hosted dashboard follow the device's light/dark preference. That UI change is not included in npm 0.1.6.
|
|
19
20
|
|
package/docs/scout.md
CHANGED
|
@@ -2,16 +2,16 @@
|
|
|
2
2
|
|
|
3
3
|
Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
|
|
4
4
|
|
|
5
|
-
Scout is a separate hosted beta product. Npm 0.1.7
|
|
5
|
+
Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Code setup for it. OpenMerit 0.1.12 adds native Scout access for Pi 1.5.0, replacing the old Pi evaluation adapter without changing the core coordinator, standalone evaluation CLI or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
|
|
6
6
|
|
|
7
7
|
**Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.
|
|
8
8
|
|
|
9
|
-
**Small-screen handoff — source only, not deployed:** below 768 CSS pixels, the Scout page replaces its search interface with a note from Lazim asking visitors to use a larger screen. **Share link** attempts to copy the public Scout URL and opens the device’s native share sheet where supported. If clipboard access fails, the visible URL can be copied manually. At 768 CSS pixels and above, including full-width iPad layouts, the search interface remains available. This does not change JSON API or MCP access.
|
|
10
|
-
|
|
11
9
|
**Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.
|
|
12
10
|
|
|
13
11
|
**Session restoration — unreleased source:** onboarding stays hidden while sign-in is checked, so returning signed-in visitors do not see the introduction flash on reload or reopening. Chat remains locked until an active session is confirmed. Signed-out visitors and failed checks reveal the introduction with sign-in still available; signing out restores it. No saved tour-completion flag grants access. This is not yet deployed and changes no homepage FAQ claim.
|
|
14
12
|
|
|
13
|
+
**Session tokens during agent turns — unreleased source:** each progress poll asks the sign-in client for a current session token instead of reusing the token captured at the start of the turn. This lets an active session renew its short-lived token while Scout works, including when polling retrieves an answer after the initial connection fails. It does not extend an expired or revoked session, resubmit the task or bypass server authentication. A browser-platform regression fixture covers token renewal and answer recovery; the fix is not yet deployed. No homepage FAQ claim changes.
|
|
14
|
+
|
|
15
15
|
**Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.
|
|
16
16
|
|
|
17
17
|
## Five-step introduction modal — deployed, 28 October 2026
|
|
@@ -359,7 +359,7 @@ Scout separates semantic interpretation from exact metadata checks and numerical
|
|
|
359
359
|
| --- | --- |
|
|
360
360
|
| Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
|
|
361
361
|
| Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
|
|
362
|
-
| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and
|
|
362
|
+
| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and Vectorize benchmark embeddings, stored once per content and listed per version. The active pointer changes only after the complete generation is ready. |
|
|
363
363
|
| Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
|
|
364
364
|
| Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
|
|
365
365
|
| Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
|
|
@@ -520,7 +520,11 @@ The comparison table keeps recommendations in Scout rank order. Benchmark mode s
|
|
|
520
520
|
|
|
521
521
|
## Connect a coding agent
|
|
522
522
|
|
|
523
|
-
**
|
|
523
|
+
**Pi native Scout adapter — npm 0.1.12:** Pi **1.5.0** joins Claude Code through the shared browser-approved installer. Run `npx -y openmerit@latest install --client pi [--scope user|project]`. The **Add to Pi** action below Claude copies setup text only; there is no Pi deep link. Approve the matching browser code, then start a new Pi session. Pi owns native MCP, tools, model, permissions and sessions; the adapter adds only the authenticated connection and shared Scout rule/skill. No `/openmerit` lifecycle commands, scheduler, LaunchAgent or automatic-update hooks are carried forward. Explicitly reinstall to update. See [Pi paths, migration and SDK embedding](pi-extension.md).
|
|
524
|
+
|
|
525
|
+
Each new client/scope installation has independent credentials and approval; project keys are also separated by project and stored outside the repository. Shell `login`, `logout`, `status`, `remove` and `doctor` accept `--client pi|claude-code --scope user|project` to target that installation. Pi connection JSON references a private credential path rather than containing the key. The legacy Claude user credential path remains supported. Package publication and website deployment are independent; the versioned Claude setup below describes earlier releases.
|
|
526
|
+
|
|
527
|
+
**Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the Claude commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code was the supported client in that release; Pi support requires 0.1.12 or newer. The older `0.1.7-preview.5` package does not contain these commands.
|
|
524
528
|
|
|
525
529
|
Setup takes four steps, and you act in each one only once:
|
|
526
530
|
|
|
@@ -583,9 +587,9 @@ The agent compares model and endpoint, evidence, reference cost/time and unknown
|
|
|
583
587
|
|
|
584
588
|
### Automatic instruction updates — version 0.1.10
|
|
585
589
|
|
|
586
|
-
|
|
590
|
+
**Claude Code only.** These hooks and `update` commands do not apply to Pi. Pi updates require an explicit installer rerun and a new session.
|
|
587
591
|
|
|
588
|
-
Version 0.1.
|
|
592
|
+
Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.
|
|
589
593
|
|
|
590
594
|
Setup discloses automatic updates and installs asynchronous `SessionStart` and `Stop` command hooks. They check npm's stable `latest` release at most once per 24 hours per installation. Offline or failed checks retain the installed version and retry at the next eligible hook after 15 minutes. `Stop` means Claude finished responding, not that the user's task succeeded. No background service watches connectivity while Claude Code is closed, and checks never ask Claude to keep working. A one-shot `claude -p` process can exit before an asynchronous Stop hook finishes; startup checks and later sessions provide the retry path. Managed policies or disabled hooks can prevent automatic checks; an explicit manual update remains available.
|
|
591
595
|
|
|
@@ -674,6 +678,7 @@ The twice-daily updater fetches eight independent public sources, including Mode
|
|
|
674
678
|
| [HELM Arabic](https://crfm.stanford.edu/helm/) (Stanford CRFM) | Mean score across HELM's Arabic scenarios from the newest release, mostly multiple choice. Models missing a scenario are left out; no per-model dates. |
|
|
675
679
|
| [BRIDGE Medical](https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard) (YLab) | Zero-shot average across clinical-text tasks in several languages, mostly open-weight models. Average only; no per-model dates. |
|
|
676
680
|
| [SuperCLUE](https://www.superclueai.com) | Chinese general-board category scores: math and science reasoning, hallucination control, precise instruction following, agentic coding and planning, for about 20 models. Boards older than six months are not read. |
|
|
681
|
+
| [Fireworks Specialized Intelligence Index](https://fireworks.ai/specialized-intelligence-index/) | Domain benchmarks contributed by practitioner companies (legal, finance and tax, healthcare and life sciences, customer support, software and SRE, web search, marketing, hardware and more), from the index's published JSON. Each benchmark is its own `· Fireworks` series scored by its headline metric; only 0–100 headlines are read, so Elo, US-dollar and relative-score boards and the cost and duration columns are not. Unranked rows and entries Fireworks footnotes as run inside another product or with rejected tasks are left out. Most results are run by the contributor, some by Fireworks, which is itself an inference provider. No per-model dates. |
|
|
677
682
|
| [LLM Stats](https://llm-stats.com/benchmarks) (vendor-reported) | Scores model developers report in announcements, papers and model cards, compiled by LLM Stats and reused with attribution under its terms. Scout reads LLM Stats' 100 most-reported benchmarks and keeps only results it marks self-reported, on 0–1 or 0–100 scales, as separate `· vendor-reported` series that never merge with third-party measurements of the same benchmark. Vendor-reported results count toward ranking. They are not independently verified, and settings such as tools, attempts and reasoning effort follow each vendor. Base (pretrained) checkpoints are never matched to instruct models. A benchmark that leaves the 100 keeps its last results. |
|
|
678
683
|
| [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
|
|
679
684
|
| [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
|
package/docs/security.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Security
|
|
2
2
|
|
|
3
|
+
**Pi replacement — unreleased, unpublished source:** Pi 1.5.0 uses native MCP plus shared Scout rule/skill sections, not the former evaluation adapter. It never initiates login, changes the host model/tools/session, or provisions background jobs. Browser-approved credentials are independent per client/scope/project, stored privately outside the repository, and passed as MCP HTTP headers, never instructions. Pi's installed connection JSON contains only the private credential path. See [Pi paths and targeted removal](pi-extension.md). The standalone evaluation CLI and core coordinator remain separate.
|
|
4
|
+
|
|
3
5
|
The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.
|
|
4
6
|
|
|
5
7
|
The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.
|
|
@@ -12,6 +14,8 @@ Never place secrets in profiles, policies, intent constraints, summaries, eviden
|
|
|
12
14
|
|
|
13
15
|
## Mutation authorization
|
|
14
16
|
|
|
17
|
+
This section describes core/custom evaluation harness policy. Pi-specific lifecycle and background-runner behavior below is **historical Pi 0.87 functionality**, removed from the new Scout adapter. Installing the replacement does not establish that an old background job was removed; manage that job separately.
|
|
18
|
+
|
|
15
19
|
Most intents carry `modelMutationAllowed: false`. Only application-target apply-swap and rollback intents can authorize a mutation, limited to the relevant proposal, prior configuration, and confirmed target ID. Pi's model is explicitly excluded. Automatic swaps require explicit policy opt-in and meeting the configured verified-swap threshold, which can be zero. Otherwise a proposal requires approval. Post-swap evaluation depends on the harness performing the checks and reporting their results. OpenMerit is not a sandbox around the harness's tools.
|
|
16
20
|
|
|
17
21
|
Persistent wakeup provisioning is a separate authorization boundary. An adapter that needs to install a system scheduler, create a paid cloud agent, or change infrastructure must declare `wakeupProvisioning: "infrastructure_change"`; OpenMerit will not request that provisioning unless the confirmed check policy sets `infrastructureChangesAllowed: true`.
|
|
@@ -24,7 +28,7 @@ The background runner imposes a 15-minute wall-clock limit and records a nonzero
|
|
|
24
28
|
|
|
25
29
|
`.openmerit/` may reveal task names, candidate identifiers, evidence locations, budgets, or model choices. Treat it as sensitive project metadata even though it must not contain credentials.
|
|
26
30
|
|
|
27
|
-
|
|
31
|
+
The historical evaluation adapter also recorded OpenMerit requests and results in Pi history. With native Scout, Pi manages its own session/tool records and provider context; the adapter adds no separate evaluation-history store. Task descriptions sent to hosted Scout follow its data policy. Local OpenMerit storage does not imply that all processing stays on your machine.
|
|
28
32
|
|
|
29
33
|
## Evidence integrity
|
|
30
34
|
|
|
@@ -55,6 +59,8 @@ The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses host
|
|
|
55
59
|
|
|
56
60
|
**Automatic instruction updates — version 0.1.10:** updater-enabled setup defaults on with an opt-out, installing asynchronous Claude Code startup/response-end hooks. Checks use npm's public stable registry without credentials, task content or installation IDs. SHA-512 verification checks package bytes against registry metadata; it does not defend against a compromised publisher/registry. The updater reads only a bounded instruction bundle, never executes downloaded code or package scripts, and leaves MCP configuration, keys, privacy settings and project policy unchanged. A supported release contract permits instruction updates; maintainers must change it when permissions or data handling change. This is a publisher assertion, not semantic validation of instructions. Edited files are retained, concurrent writes serialize, and journaled recovery stops on intervening edits. New sessions reliably load new instructions. See [automatic updates](scout.md#automatic-instruction-updates-version-0110) for cadence, offline retries, manual commands and the one-time bootstrap required for older installations. Availability follows npm publication independently of hosted releases.
|
|
57
61
|
|
|
62
|
+
These update hooks are Claude Code only. The new Pi adapter installs none; updates require explicit targeted reinstall. Its `login`, `logout`, `status`, `remove` and `doctor` shell commands target the selected client/scope without revoking another installation's independent key.
|
|
63
|
+
|
|
58
64
|
### OpenMerit coding-agent reset — 23 October 2026
|
|
59
65
|
|
|
60
66
|
**Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
|
package/docs/testing.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# Testing and validation
|
|
2
2
|
|
|
3
|
+
**Pi source replacement — unreleased and unpublished:** the new Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. The dated Pi 0.87 lifecycle/scheduler results below remain historical receipts, not proof of the replacement or supported current commands. Package, browser-approval, native Pi and SDK acceptance must be reported for the new build separately; a successful documentation build alone establishes none of them. Core coordinator and standalone evaluation tests remain separate.
|
|
4
|
+
|
|
3
5
|
## Stable 0.1.7 acceptance — 24 October 2026
|
|
4
6
|
|
|
5
7
|
Published `openmerit@0.1.7` to npm's `latest` channel after full repository verification: 757 passing tests, one existing skip, syntax/type checks, builds and documentation checks. The `preview` tag remains `0.1.7-preview.5`. A fresh registry installation on macOS ARM64 with Node.js 22.23.1 matched the tested 104-file tarball byte-for-byte. Packed core/protocol/Pi exports and the installed terminal runtime smoke checks passed, including passive collection, source removal, runtime outage/recovery and shutdown.
|
package/docs/troubleshooting.md
CHANGED
|
@@ -1,55 +1,50 @@
|
|
|
1
1
|
# Troubleshooting
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The Pi Scout adapter requires OpenMerit **0.1.12** or newer and targets Pi **1.5.0**. Check the resolved package version when using `@latest`. The removed Pi 0.87 `/openmerit` workflow is not a diagnostic path for the replacement.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
/openmerit status
|
|
7
|
-
/openmerit doctor
|
|
8
|
-
/openmerit logs
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
## The command is missing
|
|
5
|
+
## Scout is missing in Pi
|
|
12
6
|
|
|
13
|
-
|
|
7
|
+
Run targeted checks in your shell:
|
|
14
8
|
|
|
15
9
|
```sh
|
|
16
|
-
pi
|
|
10
|
+
npx -y openmerit@latest status --client pi --scope user --check
|
|
11
|
+
npx -y openmerit@latest doctor --client pi --scope user
|
|
17
12
|
```
|
|
18
13
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
## Setup did not begin
|
|
14
|
+
Use `--scope project` from the original project root for project setup. Check Pi and Node.js versions against [setup](getting-started.md), complete browser device approval, then start a **new session**. Installing the npm package or reconnecting an old session does not load new extension instructions. For SDK embedding, load Pi's `createMcpExtension()` explicitly alongside `createScoutPiExtension`; the Scout factory alone is not an MCP runtime.
|
|
22
15
|
|
|
23
|
-
|
|
16
|
+
## The connection configuration is invalid
|
|
24
17
|
|
|
25
|
-
|
|
18
|
+
The installed extension reads `openmerit-scout.json` beside its bundle; that file points to a private credential file outside the repository. Do not put the key in the extension or prompt. Check the scope and configuration-directory overrides against [installed paths](pi-extension.md#installed-files-and-credentials). Reinstall explicitly to repair missing owned files:
|
|
26
19
|
|
|
27
|
-
|
|
20
|
+
```sh
|
|
21
|
+
npx -y openmerit@latest install --client pi --scope user
|
|
22
|
+
```
|
|
28
23
|
|
|
29
|
-
|
|
24
|
+
Expired or revoked credentials require targeted `login` followed by reinstall. The adapter reports a sanitized setup diagnostic; it never starts sign-in on its own. Approve only a browser code you initiated.
|
|
30
25
|
|
|
31
|
-
|
|
26
|
+
## Installation reports a conflict
|
|
32
27
|
|
|
33
|
-
|
|
28
|
+
The installer does not overwrite edited or unowned files. Inspect and preserve your changes before resolving the conflict. Also check for overlapping Pi user/project installations or an existing `openmerit` MCP entry. Remove the unwanted owned installation using its own scope; do not revoke a different client's key as a repair. Disable the old `npm:openmerit` Pi registration when migrating from 0.87.
|
|
34
29
|
|
|
35
|
-
|
|
30
|
+
## The adapter did not update automatically
|
|
36
31
|
|
|
37
|
-
|
|
32
|
+
That is expected. Pi has no OpenMerit auto-update hooks. Explicitly rerun `install --client pi` with the original scope, then start a new session. Claude Code's update settings do not configure Pi.
|
|
38
33
|
|
|
39
|
-
##
|
|
34
|
+
## `/openmerit` or evaluation tools are missing
|
|
40
35
|
|
|
41
|
-
|
|
36
|
+
They were removed, not renamed. The Pi adapter now supplies only native Scout MCP and shared guidance. It does not set up evaluations, recover intents, schedule work, swap models, or install a LaunchAgent. Existing legacy background jobs must be removed separately; installing the new adapter is not evidence that they stopped. Preserve old evaluation state and audit files if needed. For new evaluations use [the standalone CLI](agent-evaluations.md); custom harnesses may still use [the core coordinator](adapter-guide.md).
|
|
42
37
|
|
|
43
|
-
##
|
|
38
|
+
## Scout was not consulted or its evidence is insufficient
|
|
44
39
|
|
|
45
|
-
|
|
40
|
+
The rule and skill are model guidance, not a guaranteed tool gate. Confirm they loaded in a new session and that the host permits the MCP tools. When project model policy is missing, ask the user; when Scout is unavailable or lacks suitable evidence, pause the model choice rather than invent a result. Host model/tool/session restrictions remain authoritative. Scout candidates are not measurements of your application's quality.
|
|
46
41
|
|
|
47
|
-
##
|
|
42
|
+
## Evaluation or terminal problems
|
|
48
43
|
|
|
49
|
-
|
|
44
|
+
The standalone products are separate from Pi setup. Use [agent-evaluation diagnostics](agent-evaluations.md) for tasks, graders, budgets and execution blocks, or [terminal diagnostics](inference-terminal.md) for missing inventory and request evidence. A populated dashboard is not proof of successful evaluations. Never mark missing or rejected evidence successful merely to advance a workflow.
|
|
50
45
|
|
|
51
46
|
## Report a reproducible problem
|
|
52
47
|
|
|
53
|
-
Include
|
|
48
|
+
Include OpenMerit, Pi and Node.js versions, the client/scope, the triggering command, and a sanitized error. Remove credentials, private paths, prompts and proprietary data before sharing. Do not attach credential or connection files containing private configuration.
|
|
54
49
|
|
|
55
50
|
[Open an issue on GitHub](https://github.com/laz-aslam/openmerit/issues)
|
package/docs/ux-reference.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Pi UX behavioral reference
|
|
2
2
|
|
|
3
|
-
This document
|
|
3
|
+
This document is a **historical evaluation-adapter reference**, not the current Pi interface specification. The unreleased Pi 1.5.0 native Scout replacement does not carry forward the `/openmerit` commands, lifecycle, pause controls, or scheduling described here. See [the current Pi guide](pi-extension.md). The list below preserves earlier UX ideas for custom evaluation harnesses only.
|
|
4
4
|
|
|
5
5
|
## Behaviors worth preserving
|
|
6
6
|
|