openmerit 0.1.11 → 0.1.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/CHANGELOG.md +7 -0
  2. package/README.md +20 -53
  3. package/dist/pi/scout-extension.mjs +177 -0
  4. package/dist/pi/src/index.d.ts +6 -32
  5. package/dist/pi/src/index.js +49 -787
  6. package/dist/terminal/scout-update.json +1 -1
  7. package/dist/terminal/src/install.mjs +102 -166
  8. package/dist/terminal/src/scout-credentials.d.ts +12 -0
  9. package/dist/terminal/src/scout-credentials.js +41 -0
  10. package/dist/terminal/src/scout-credentials.ts +44 -0
  11. package/dist/terminal/src/scout-instructions.d.ts +5 -0
  12. package/dist/terminal/src/scout-instructions.js +103 -0
  13. package/dist/terminal/src/scout-instructions.ts +106 -0
  14. package/dist/terminal/src/scout-pi.d.ts +44 -0
  15. package/dist/terminal/src/scout-pi.js +84 -0
  16. package/dist/terminal/src/scout-pi.ts +76 -0
  17. package/dist/terminal/src/telemetry.mjs +2 -2
  18. package/docs/adapter-guide.md +2 -0
  19. package/docs/architecture.md +7 -5
  20. package/docs/automation.md +2 -0
  21. package/docs/budgets.md +2 -0
  22. package/docs/commands.md +3 -1
  23. package/docs/demo.md +1 -1
  24. package/docs/first-trial.md +2 -0
  25. package/docs/getting-started.md +19 -44
  26. package/docs/index.md +10 -6
  27. package/docs/inference-terminal.md +1 -1
  28. package/docs/lifecycle.md +2 -0
  29. package/docs/operations.md +2 -0
  30. package/docs/pi-extension.md +32 -28
  31. package/docs/roadmap.md +6 -5
  32. package/docs/scout.md +12 -7
  33. package/docs/security.md +7 -1
  34. package/docs/testing.md +2 -0
  35. package/docs/troubleshooting.md +23 -28
  36. package/docs/ux-reference.md +1 -1
  37. package/package.json +3 -4
  38. package/dist/pi/src/scheduler.d.ts +0 -11
  39. package/dist/pi/src/scheduler.js +0 -137
  40. package/dist/pi/src/wakeup.d.ts +0 -2
  41. package/dist/pi/src/wakeup.js +0 -108
  42. package/docs/saga/a-steadier-compass.md +0 -25
  43. package/docs/saga/defining-a-task-for-ai.md +0 -154
  44. package/docs/saga/the-benchmark-is-not-the-job.md +0 -33
@@ -1,6 +1,6 @@
1
1
  # Getting started
2
2
 
3
- Stable npm 0.1.7 includes a separate [agent-evaluation CLI-change loop](agent-evaluations.md) for Claude Code, Codex and OpenCode, plus [Scout device setup for Claude Code](scout.md#connect-a-coding-agent). Neither is part of the Pi setup below.
3
+ OpenMerit provides a passive Inference Terminal, Scout model-selection guidance, and a separate [agent-evaluation CLI](agent-evaluations.md). Pi Scout setup requires OpenMerit **0.1.12** or newer and Pi **1.5.0**. Package publication and website deployment are independent releases.
4
4
 
5
5
  ## Open your inference environment
6
6
 
@@ -12,65 +12,40 @@ cd /path/to/your/application
12
12
  openmerit dash
13
13
  ```
14
14
 
15
- The Inference Terminal is included in stable `openmerit@0.1.7`. It opens a local, read-only view of the selected application's services, models, recorded requests, and linked compute. It does not require Pi or a provider API key, make inference calls, or change model assignments. Supported code/configuration declarations identify inventory; request history requires existing supported logs or exports. See [the terminal guide](inference-terminal.md) for collection, privacy, configuration, and diagnostics.
15
+ The Inference Terminal opens a local, read-only view of the selected application's services, models, recorded requests, and linked compute. It does not require Pi or a provider API key, make inference calls, or change model assignments. Supported code/configuration declarations identify inventory; request history requires existing supported logs or exports. See [the terminal guide](inference-terminal.md) for collection, privacy, configuration, and diagnostics.
16
16
 
17
17
  To inspect fictional data without installation, open the [hosted dashboard](https://dash.openmerit.ai/) or its [before/after comparison](https://dash.openmerit.ai/demo). Neither connects to your machine.
18
18
 
19
- ## Set up model comparisons with Pi
19
+ ## Connect Scout to a coding agent
20
20
 
21
- The Pi integration is a separate workflow for evaluations and authorized model changes. It requires Pi 0.87 with a model provider configured, and a product repository where Pi can create and verify evaluation and observability artifacts. Provider credentials belong to Pi or its environment; OpenMerit does not accept them.
22
-
23
- Install the extension in Pi:
24
-
25
- ```sh
26
- pi install npm:openmerit
27
- pi list
28
- ```
29
-
30
- Run `/openmerit doctor` inside Pi to confirm that the extension is available.
31
-
32
- ## Install from source
21
+ Claude Code setup is described in [the Scout guide](scout.md#connect-a-coding-agent). For the new Pi integration, use Pi **1.5.0**, Node.js 22.19 or newer, and npm. Run the intended command in your shell:
33
22
 
34
23
  ```sh
35
- git clone https://github.com/laz-aslam/openmerit.git
36
- cd openmerit
37
- npm ci
24
+ npx -y openmerit@latest install --client pi --scope user
38
25
  ```
39
26
 
40
- Start Pi with the extension:
27
+ Approve the matching device code in the browser, then start a **new Pi session**. Only approve a request you initiated. User scope is the default and applies across projects. For one project, run this from that project's root instead:
41
28
 
42
29
  ```sh
43
- ./node_modules/.bin/pi --no-extensions -e ./packages/pi/src/index.ts
30
+ npx -y openmerit@latest install --client pi --scope project
44
31
  ```
45
32
 
46
- `--no-extensions` prevents a globally installed npm copy from conflicting with the source extension. Run `/openmerit doctor` inside Pi to confirm that the adapter and intent executor are available.
47
-
48
- ## Establish a product profile
49
-
50
- Open the product repository in interactive Pi with the extension loaded. If the project has no OpenMerit state, the Pi adapter performs a bounded source preflight. It also scans after a settled Pi build turn, so an application created from an initially empty directory can enter setup without a slash command. A repository without such a target remains idle; use `/openmerit setup` only to recover or configure a custom integration explicitly.
33
+ Each client/scope installation has its own private Scout key; project keys are stored outside the repository. Do not install overlapping Pi user/project adapters or a duplicate `openmerit` MCP entry. The installer reports conflicts rather than overwriting edited or unowned files.
51
34
 
52
- OpenMerit sends Pi a private, typed `establish_evals` intent. Pi must identify the application LLM call or shared route under evaluation and its exact incumbent application model. It inspects the application's outputs, tools, and failure modes to infer task-relevant metrics instead of asking you to invent them. Before running setup, Pi shows the proposed metrics, why each matters, the representative case and repetition counts, aggregation, any actual performance threshold, and estimated evaluation cost. You confirm or edit that plan together with the stable target ID, call sites, route key, budget, check cadence, scheduling mode, and automation permissions. Pi's own model is never an application target.
35
+ The adapter connects Scout through Pi's native MCP and supplies the shared Scout rule and skill. Pi owns tool discovery, tool execution, model choice, permissions, and sessions. Scout recommends application-model candidates; it does not execute those candidates or change Pi's own model. There are no `/openmerit` lifecycle commands, automatic evaluation setup, scheduler, or LaunchAgent in this adapter.
53
36
 
54
- For an active-session demo, Pi defaults the baseline and post-swap verification cadence to one completed evaluation task unless you change it. OpenMerit rejects empty baseline or post-swap cadences.
37
+ ## Check, update, or remove Pi setup
55
38
 
56
- Pi is also asked to research preliminary model leads during setup. Each reported lead has a current price estimate and reputation summary with sources and check times. OpenMerit stores these leads with the project profile. The field is optional for compatibility with existing protocol-0.1 harnesses, and an empty list means no credible leads were found. It does not change the application model or count as a measurement.
57
-
58
- OpenMerit rejects successful setup without a confirmed task profile, at least one valid objective with an explicit sampling plan, a supervised-graduation policy, non-empty baseline and post-swap cadences, a budget, and evidence.
59
-
60
- The setup instruction asks Pi to run the application path with working provider authentication, execute each required collector and grader, inspect the resulting metric values, and retain verification artifacts. A metric that cannot be produced must be marked missing. This smoke run verifies the pipeline; it is not counted as a production baseline sample. OpenMerit still checks the reported coverage and evidence references, not the authenticity of a grader run itself.
61
-
62
- ## Collect the baseline
63
-
64
- After setup, instrument the confirmed application target and use the product normally. Report application-task completions and cumulative metric windows through Pi's typed signal tool, including the application model where applicable, sample counts, and durable evidence. Pi's own model, tokens, cost, and latency are orchestration telemetry and do not satisfy product metrics. OpenMerit core checks the stored baseline windows at the approved cadence without calling Pi; missing evidence stays insufficient.
65
-
66
- ## Find the frontier
67
-
68
- When the baseline meets every required metric threshold, OpenMerit issues the next approved nudges: discover candidates, run controlled challenger trials, and calculate the Pareto frontier. Pi performs that work. OpenMerit checks the returned result against the [Pareto specification](pareto-spec.md).
39
+ ```sh
40
+ npx -y openmerit@latest status --client pi --scope user --check
41
+ npx -y openmerit@latest doctor --client pi --scope user
42
+ npx -y openmerit@latest remove --client pi --scope user
43
+ ```
69
44
 
70
- Candidate discovery uses current price, capability, availability, reputation, and benchmark signals to shortlist models worth testing. These are starting clues, not evidence that a model performs well on your application. OpenMerit collects preliminary leads at setup, then refreshes them, probes availability, and freezes the incumbent plus challengers. Pi must use the same cases, application revision, prompt, tools, grader, and run count, with a recorded shuffle seed. It preserves one experiment manifest and reports measured quality, cost, latency, tokens, and other required metrics. OpenMerit durably carries the accepted candidate set and assessments into later intents rather than relying on Pi conversation history. A model proposal must come from the verified frontier of those measured candidate results, not from price or reputation alone.
45
+ Use `--scope project` from the original project root for a project installation. `login` and `logout` accept the same `--client` and `--scope` flags. These commands target that installation, not another client's connection. Updating Pi setup requires explicitly rerunning `install` and starting a new session; no Pi auto-update hooks are installed. Installing or updating the npm package alone does not reconfigure Pi.
71
46
 
72
- ## Approve an early swap
47
+ See [the Pi extension guide](pi-extension.md) for installed paths, migration from Pi 0.87, and SDK embedding. Use [troubleshooting](troubleshooting.md) for setup failures.
73
48
 
74
- With automatic swaps disabled, a model proposal that passes frontier validation waits for `/openmerit approve`. That authorizes Pi to apply only the proposed change. OpenMerit then requires post-swap evaluation; the approved policy determines rollback behavior. Automatic swaps may bypass per-proposal approval when enabled and the configured verified-swap threshold is reached, including when that threshold is zero.
49
+ ## Evaluate application changes
75
50
 
76
- Continue with [your first supervised trial](first-trial.md) for a practical walkthrough.
51
+ The standalone `openmerit eval` CLI is unchanged by the new Pi adapter. It evaluates Claude Code, Codex and OpenCode through explicit task, scope, budget and grader configuration; see [agent evaluations](agent-evaluations.md). The core evaluation coordinator remains available to custom harness integrations, but the removed Pi 0.87 lifecycle adapter is not the way to invoke it.
package/docs/index.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # OpenMerit documentation
2
2
 
3
- These guides describe the implementation on **main**. Stable npm `openmerit@0.1.7` includes the Inference Terminal, agent-evaluation CLI and Scout device setup for Claude Code. The website, hosted samples, Scout service and npm package are deployed independently; dated receipts describe the versions actually tested.
3
+ These guides distinguish current source work from published behavior. OpenMerit **0.1.12** adds native Scout support for Pi **1.5.0**, replacing the old Pi 0.87 evaluation extension. The Inference Terminal and standalone agent-evaluation CLI are separate products and are unchanged by that replacement. The website, hosted samples, Scout service and npm package are deployed independently; dated receipts describe the versions actually tested.
4
4
 
5
5
  ## See your inference environment
6
6
 
@@ -20,11 +20,15 @@ Start with [installation and setup](getting-started.md) or the [Inference Termin
20
20
 
21
21
  For a shortlist before building an integration, the separate [Scout beta](scout.md) accepts a task description and returns up to five evidence-backed model candidates. It is available at [openmerit.ai/scout/](https://openmerit.ai/scout/), with a prompt-only JSON API and MCP tool for coding agents.
22
22
 
23
- ## Evaluate model changes with Pi
23
+ ## Connect Scout to Pi
24
24
 
25
- The separate Pi 0.87 integration asks a coding harness to establish evaluations, gather application measurements, and test candidates. OpenMerit coordinates the approved lifecycle, records evidence references, validates structured results and frontier conformance, and requests authorized changes. The harness executes the work; the terminal does not perform this workflow.
25
+ The thin Pi 1.5.0 adapter uses native MCP and shared Scout guidance. Pi owns tools, models, permissions and sessions; the adapter has no `/openmerit` lifecycle commands, evaluation scheduling, LaunchAgent or automatic updates.
26
26
 
27
- Follow [Pi setup](getting-started.md#set-up-model-comparisons-with-pi), then [your first supervised trial](first-trial.md).
27
+ Follow [setup](getting-started.md#connect-scout-to-a-coding-agent) and [the Pi guide](pi-extension.md), including browser approval, per-client/scope credentials, explicit reinstall updates and SDK requirements. Registry setup requires OpenMerit 0.1.12 or newer.
28
+
29
+ ## Evaluate application changes
30
+
31
+ Use [the standalone evaluation CLI](agent-evaluations.md) for Claude Code, Codex and OpenCode. Custom harnesses may use the core coordinator. The following guides retain core concepts and historical Pi 0.87 workflows; they do not describe the new Scout adapter:
28
32
 
29
33
  - [Metrics and evidence](metrics-and-evidence.md): measurements, sample requirements, and readiness.
30
34
  - [Comparisons and swaps](lifecycle.md): baseline, challenger trials, approval, verification, and rollback.
@@ -33,8 +37,8 @@ Follow [Pi setup](getting-started.md#set-up-model-comparisons-with-pi), then [yo
33
37
 
34
38
  ## Operate and troubleshoot
35
39
 
36
- [Commands](commands.md) and [project files and logs](operations.md) explain status, pause, diagnostics, and recovery. [Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) describe the boundaries to check before relying on automation.
40
+ [Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) explain current Pi setup and its boundaries. [Commands](commands.md) and [project files and logs](operations.md) also retain explicitly historical Pi lifecycle references alongside standalone/core information; their old pause and recovery commands are not part of the new adapter.
37
41
 
38
42
  ## Build an integration
39
43
 
40
- Pi is the implemented coding-harness adapter. Other harnesses can implement the protocol and coordinator boundary described in [architecture](architecture.md) and [the adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines comparison rules; [testing and validation](testing.md) distinguishes automated coverage from product evidence.
44
+ Pi now supplies a native Scout connection, not an evaluation `HarnessAdapter`. Custom evaluation harnesses can implement the protocol and coordinator boundary described in [architecture](architecture.md) and [the adapter guide](adapter-guide.md). The [Pareto specification](pareto-spec.md) defines comparison rules; [testing and validation](testing.md) distinguishes automated coverage from product evidence.
@@ -79,7 +79,7 @@ openmerit dash --no-open
79
79
  | `--demo` | Use only a fictional local sample; no local collectors run |
80
80
  | `--help` | Show usage |
81
81
 
82
- Ctrl+C stops collection. The terminal has no persistent daemon and is independent of Pi's pause and scheduling commands.
82
+ Ctrl+C stops collection. The terminal has no persistent daemon and is independent of core evaluation pause/scheduling policy. The unreleased Pi 1.5.0 Scout adapter exposes no pause or scheduling commands.
83
83
 
84
84
  ## Repository indexing
85
85
 
package/docs/lifecycle.md CHANGED
@@ -1,5 +1,7 @@
1
1
  # Improvement lifecycle
2
2
 
3
+ **Core reference / historical Pi workflow:** this lifecycle remains available through the coordinator for custom harnesses. Pi-specific actions below describe the removed Pi 0.87 adapter, not the unreleased Pi 1.5.0 native Scout adapter. The replacement has no lifecycle commands, evaluation setup, swaps, or scheduling. See [Pi setup](pi-extension.md) or the separate, unchanged [evaluation CLI](agent-evaluations.md).
4
+
3
5
  ## Stages
4
6
 
5
7
  1. **Unconfigured** — no confirmed profile exists.
@@ -1,5 +1,7 @@
1
1
  # Project state and operations
2
2
 
3
+ **Scope — unreleased Pi replacement:** the evaluation lifecycle files and `/openmerit` commands below describe the core coordinator and historical Pi 0.87 adapter. The Pi 1.5.0 Scout adapter does not create or advance that lifecycle, manage scheduling, or expose those commands. See [current Pi paths and targeted shell commands](pi-extension.md). Terminal files remain current; standalone evaluation files are documented in [agent evaluations](agent-evaluations.md). Preserve older state and separately remove any legacy background job when migrating.
4
+
3
5
  ## State location
4
6
 
5
7
  Operational data lives under `.openmerit/` in the target product repository. It should be ignored by version control because it can contain local lifecycle state and evidence locations.
@@ -2,53 +2,57 @@
2
2
 
3
3
  ## Supported version
4
4
 
5
- The adapter targets Pi 0.87 and declares `>=0.87.0 <0.88.0`. Development uses 0.87.0 without modifying a globally installed Pi.
5
+ OpenMerit **0.1.12** adds a Scout adapter for Pi **1.5.0**. It replaces, rather than upgrades in place, the Pi 0.87 evaluation adapter. The `@latest` commands below require OpenMerit 0.1.12 or newer. Package publication and website deployment remain independent releases.
6
6
 
7
- ## Loading
7
+ ## Installation
8
8
 
9
- Install the published package with `pi install npm:openmerit`. For development
10
- from a checkout, load the source extension with:
9
+ With Node.js 22.19 or newer, npm, and Pi 1.5.0 installed:
11
10
 
12
11
  ```sh
13
- ./node_modules/.bin/pi --no-extensions -e ./packages/pi/src/index.ts
12
+ npx -y openmerit@latest install --client pi --scope user
14
13
  ```
15
14
 
16
- The isolation flag avoids duplicate `openmerit_complete_intent` and `openmerit_report_signal` registrations when npm `openmerit` is also installed in Pi's user settings.
15
+ Approve the matching browser device code and start a new Pi session. Scope defaults to `user`; use `--scope project` from the project's root for project-local setup. Each client and scope gets an independent key and approval; project keys are also separated by project. The npm package alone does not configure Pi.
17
16
 
18
- Use `/openmerit doctor` to confirm that the adapter and intent executor are available.
17
+ The website's **Add to Pi** action is copy-only and sits below the Claude action, with the Pi logo before the label. It copies the installer command; it does not open a Pi deep link or prove installation. The button requires the corresponding Scout website deployment.
19
18
 
20
- Use `/openmerit logs` to locate the project-local audit stream, exporter failures, evidence manifests, and state while testing an end-to-end run.
19
+ ## Installed files and credentials
21
20
 
22
- On `session_start`, an interactive unconfigured project receives a bounded source preflight. The adapter also repeats that bounded detection after a settled Pi turn. This lets Pi build an application from an empty directory and then receive an automatic `establish_evals` intent without requiring `/openmerit setup`. Durable active-intent state prevents duplicate setup nudges. Non-interactive runs do not attempt a confirmation flow.
21
+ | File | User scope | Project scope |
22
+ | --- | --- | --- |
23
+ | Native extension bundle | `~/.pi/agent/extensions/openmerit-scout.ts` | `.pi/extensions/openmerit-scout.ts` |
24
+ | Credential-path reference | `~/.pi/agent/extensions/openmerit-scout.json` | `.pi/extensions/openmerit-scout.json` |
25
+ | Shared Scout skill | `~/.pi/agent/skills/openmerit-scout/SKILL.md` | `.pi/skills/openmerit-scout/SKILL.md` |
26
+ | Installer ownership record | `~/.pi/agent/openmerit-install.json` | `.pi/openmerit-install.json` |
23
27
 
24
- If Pi exits while an intent is active, the adapter restores its completion tool and resends that same persisted intent on the next interactive session so the harness can continue it.
28
+ `PI_CODING_AGENT_DIR` overrides the user agent directory. The connection JSON contains a `credentialsPath`, not the key. The default private key paths are `~/.config/openmerit/pi/user/credentials.json` for user scope and `~/.config/openmerit/pi/<project-id>/credentials.json` for project scope. The project ID derives from the resolved project root; run targeted commands from that root. `OPENMERIT_CONFIG_DIR` or `XDG_CONFIG_HOME` can override the private configuration base. Do not copy credentials into the project or model instructions.
25
29
 
26
- For configured projects with a confirmed application target, session start reports the available model catalogue and a scheduled tick. Application instrumentation reports idempotent completed-task and metric-window signals. OpenMerit evaluates the confirmed policy before issuing work. On macOS, Pi can install a per-project user LaunchAgent after you confirm persistent execution and authorize infrastructure changes in the policy. It uses the selected authenticated Pi model, checks for due work locally every five minutes, and starts a headless Pi only when the due core check finds downstream harness work. If installation or background authentication fails, the adapter reports a scheduling gap. Other platforms still need an external scheduler.
30
+ The extension reads credentials at session startup and registers the `openmerit` HTTP MCP server with direct tool exposure. Authorization is an HTTP header, not prompt content. Missing or invalid configuration produces a sanitized setup diagnostic, not an automatic login. The adapter performs no per-turn HTTP connection checks or credential reads.
27
31
 
28
- The macOS background runner is not yet product-validated for candidate work. A local runner test verifies that an insufficient baseline is checked and recorded without starting Pi. A model-driven runner is capped at 15 minutes and records a nonzero exit and active intent ID on failure. Do not infer that unattended candidate discovery or swapping has passed end-to-end validation.
32
+ ## Native ownership and shared guidance
29
33
 
30
- The adapter does not convert Pi's own provider usage into application metrics. Application-task signals must carry the application model, target ID, and durable evidence; application telemetry may include provider-reported USD cost, token counts, and wall-clock latency. If automatic setup is active when interactive input arrives, the input is queued and released after setup completes instead of racing the setup turn.
34
+ Pi supplies the MCP connection, tool discovery and execution, permissions, and session management. The thin adapter adds the same Scout rule and skill used by the shared installer as structured system-prompt sections. It leaves the host's model, tool allow/exclude lists, and other prompt sections intact. This is guidance to consult Scout before application-model choices, not enforcement of tool invocation or permission to download, spend, deploy, or change models. Scout's [data-handling policy](scout.md#refresh-and-data-handling) applies to submitted tasks.
31
35
 
32
- Pi also exposes `openmerit_report_signal` for verified `product_task_completed`, `metric_window_available`, and `verification_window_completed` events. Evals or observability work performed by Pi can submit measured application-target metric records through this typed ingress using stable signal IDs. For baseline checks, send cumulative metric windows with sample counts and evidence references. The background wakeup only checks whether a time-based action is due; it does not collect application results by itself. `/openmerit assess` checks those stored windows locally and starts candidate discovery only when all required metrics are ready.
36
+ ## Updates and targeted lifecycle commands
33
37
 
34
- ## Intent delivery
35
-
36
- The extension persists each intent as a custom Pi session entry, sends private extension-authored context with `display: false`, and triggers a follow-up Pi turn. The intent describes the outcome, constraints, evidence, task profile, and authorization boundary without prescribing Pi's method.
37
-
38
- Pi must finish by calling `openmerit_complete_intent` exactly once. Success without evidence is rejected. Setup, assessment, and frontier intents receive additional structured-output checks. During setup, Pi first presents its inferred metrics and sampling plan to the user. The accepted task profile records how many representative cases and repetitions each metric needs and how results are aggregated; sample counts are kept separate from optional performance thresholds.
39
-
40
- If Pi exits while an intent is active, the next session reports the recovered intent. Use `/openmerit cancel` to record an explicit cancellation before retrying; OpenMerit does not silently reinterpret interrupted work as success.
38
+ ```sh
39
+ npx -y openmerit@latest status --client pi --scope user --check
40
+ npx -y openmerit@latest doctor --client pi --scope user
41
+ npx -y openmerit@latest login --client pi --scope user
42
+ npx -y openmerit@latest logout --client pi --scope user
43
+ npx -y openmerit@latest remove --client pi --scope user
44
+ ```
41
45
 
42
- The extension replaces the completion tool definition whenever an intent becomes active. Its `outputs` property therefore exposes the canonical schema for that exact intent instead of untyped JSON. Pi validates the arguments before execution and requests provider-side JSON Schema constrained sampling when supported. OpenMerit validates the canonical schema and semantic rules again when accepting the result. OpenMerit records the exact incumbent during setup, freezes accepted candidates after discovery, and carries accepted assessments and the experiment manifest into frontier calculation so these transitions do not rely on conversational memory.
46
+ Replace `user` with `project` for a project installation. Removal and logout target that client's scope/key, not a Claude Code installation or another project. Edited or unowned files are not silently overwritten or deleted. Resolve duplicate user/project adapters or existing `openmerit` MCP entries before installing.
43
47
 
44
- ## Model changes
48
+ Updates are **explicit reinstall only**: rerun `install --client pi` with the same scope and start a new session. There are no Pi auto-update hooks; Claude Code update-hook settings do not apply to Pi.
45
49
 
46
- The extension never treats Pi's model as the application target. After policy authorization, it sends a bounded, target-bound `apply_model_swap` intent. Pi performs the application-route change and returns configuration proof; the target ID must match the confirmed application LLM target, and Pi's own model must remain unchanged.
50
+ ## SDK embedding
47
51
 
48
- ## Jev System One
52
+ `openmerit/pi` exports `createScoutPiExtension({ credentialsPath })` and a default extension created by that factory. SDK hosts must explicitly load Pi's own **`createMcpExtension()`** alongside the Scout extension; the Scout factory does not create an MCP runtime. The Pi CLI already supplies native MCP. The explicit credential path takes precedence over the installed adjacent connection JSON, with the shared Pi user credential path as the fallback when that JSON is absent. The host supplies its own model, tools, session and permission configuration.
49
53
 
50
- Jev is not integrated into OpenMerit. If configured in Pi, the intent asks Pi to use harness-configured fast decision resources aggressively for bounded ranking, triage, and judgment. Credentials remain outside OpenMerit.
54
+ ## Migration from Pi 0.87
51
55
 
52
- ## Non-interactive modes
56
+ Remove or disable the old `npm:openmerit` Pi package registration before loading this adapter, and explicitly remove any previously authorized background job through its existing management path. New setup does not promise to uninstall a legacy LaunchAgent. Preserve `.openmerit/` evaluation state and evidence if needed; do not delete them as a setup repair.
53
57
 
54
- A harness must not attempt interactive confirmation in a mode without dialog-capable UI. It should return a waiting-for-user or failed result with an actionable reason.
58
+ The replacement has no `/openmerit` commands, `openmerit_complete_intent` or `openmerit_report_signal` tools, source preflight, evaluation lifecycle, intent recovery, scheduling, automatic swaps, or LaunchAgent. The [core coordinator](architecture.md) remains available for custom harnesses. The [standalone evaluation CLI](agent-evaluations.md) is unchanged. Older lifecycle guides are retained as historical adapter/core references, not instructions for this Scout adapter.
package/docs/roadmap.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  ## Current status
4
4
 
5
- The Inference Terminal provides a read-only view of existing application evidence. The Pi integration coordinates supervised model-comparison trials. Neither a populated dashboard nor passing implementation tests establishes that an application has better models, trustworthy evaluations, or reliable unattended automation.
5
+ The Inference Terminal provides a read-only view of existing application evidence. The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. The standalone evaluation CLI and reusable core coordinator remain separate. Neither a populated dashboard nor passing implementation tests establishes that an application has better models, trustworthy evaluations, or reliable unattended automation.
6
6
 
7
7
  ## Known limitations
8
8
 
@@ -10,10 +10,11 @@ The Inference Terminal provides a read-only view of existing application evidenc
10
10
  - History and inventory can be incomplete because of source availability and read/parser limits. “All time” means retained supported evidence, not a complete lifetime archive. Cloud account usage is not fetched.
11
11
  - Machine and runtime readings describe shared resources; they are not per-request resource measurements. Static endpoints do not prove deployment, availability, or physical location.
12
12
  - Hosted samples and the before/after dashboard demo are fictional. Their cost projections and public benchmark screens do not establish application quality or latency.
13
- - Pi 0.87 is the only implemented coding-harness adapter. Jev must be configured and invoked by the harness; OpenMerit provides no Jev client.
14
- - Setup asks the harness to establish task-specific evaluations and observability. Generic coding checkpoints cannot replace product telemetry, and structured-result validation cannot prove external evidence is truthful.
15
- - Background wakeups require a logged-in macOS user, working headless Pi authentication, and explicit persistent-execution/infrastructure permission. Other platforms require an external scheduler. Deterministic baseline checks have automated coverage; unattended candidate discovery and model changes still need representative product validation.
16
- - Pause persists and gates automatic work, but does not cancel active intents, stop explicit commands, or stop the standalone terminal.
13
+ - The Pi 0.87 evaluation adapter is removed in source. Pi 1.5.0 setup needs explicit browser approval, a new session and an explicit installer rerun for updates. It has no lifecycle commands, scheduler, LaunchAgent or auto-update hooks. The host owns tools, model, permissions and sessions; SDK hosts must load Pi's native MCP extension explicitly.
14
+ - Scout rule/skill availability does not guarantee invocation or measured application quality. Missing project policy requires asking; unavailable or insufficient Scout evidence requires pausing the model choice.
15
+ - Custom core evaluation harnesses must establish task-specific evaluations and observability. Generic coding checkpoints cannot replace product telemetry, and structured-result validation cannot prove external evidence is truthful.
16
+ - Background provisioning and unattended validation belong to custom evaluation integrations, not the new Pi adapter. Existing legacy Pi jobs require separate cleanup during migration.
17
+ - Core pause policy does not cancel active intents or stop explicit commands or the standalone terminal. It is not a command exposed by the new Pi adapter.
17
18
  - Challenger accounting depends on reported costs. It cannot independently meter provider billing or stop an in-flight request. Use provider spending controls as well as the confirmed evaluation policy.
18
19
  - The source interface and hosted dashboard follow the device's light/dark preference. That UI change is not included in npm 0.1.6.
19
20
 
package/docs/scout.md CHANGED
@@ -2,16 +2,16 @@
2
2
 
3
3
  Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
4
4
 
5
- Scout is a separate hosted beta product. Npm 0.1.7 includes optional Claude Code setup for it; the hosted service does not change the Pi coordinator or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
5
+ Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Code setup for it. OpenMerit 0.1.12 adds native Scout access for Pi 1.5.0, replacing the old Pi evaluation adapter without changing the core coordinator, standalone evaluation CLI or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
6
6
 
7
7
  **Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.
8
8
 
9
- **Small-screen handoff — source only, not deployed:** below 768 CSS pixels, the Scout page replaces its search interface with a note from Lazim asking visitors to use a larger screen. **Share link** attempts to copy the public Scout URL and opens the device’s native share sheet where supported. If clipboard access fails, the visible URL can be copied manually. At 768 CSS pixels and above, including full-width iPad layouts, the search interface remains available. This does not change JSON API or MCP access.
10
-
11
9
  **Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.
12
10
 
13
11
  **Session restoration — unreleased source:** onboarding stays hidden while sign-in is checked, so returning signed-in visitors do not see the introduction flash on reload or reopening. Chat remains locked until an active session is confirmed. Signed-out visitors and failed checks reveal the introduction with sign-in still available; signing out restores it. No saved tour-completion flag grants access. This is not yet deployed and changes no homepage FAQ claim.
14
12
 
13
+ **Session tokens during agent turns — unreleased source:** each progress poll asks the sign-in client for a current session token instead of reusing the token captured at the start of the turn. This lets an active session renew its short-lived token while Scout works, including when polling retrieves an answer after the initial connection fails. It does not extend an expired or revoked session, resubmit the task or bypass server authentication. A browser-platform regression fixture covers token renewal and answer recovery; the fix is not yet deployed. No homepage FAQ claim changes.
14
+
15
15
  **Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.
16
16
 
17
17
  ## Five-step introduction modal — deployed, 28 October 2026
@@ -359,7 +359,7 @@ Scout separates semantic interpretation from exact metadata checks and numerical
359
359
  | --- | --- |
360
360
  | Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
361
361
  | Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
362
- | Hourly publication | Private R2 records, an indexed D1 fact/observation projection and versioned Vectorize benchmark embeddings. The active pointer changes only after the complete generation is ready. |
362
+ | Hourly publication | Private R2 records, an indexed D1 fact/observation projection and Vectorize benchmark embeddings, stored once per content and listed per version. The active pointer changes only after the complete generation is ready. |
363
363
  | Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
364
364
  | Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
365
365
  | Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
@@ -520,7 +520,11 @@ The comparison table keeps recommendations in Scout rank order. Benchmark mode s
520
520
 
521
521
  ## Connect a coding agent
522
522
 
523
- **Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code is the only supported client; other agents are not yet supported. The older `0.1.7-preview.5` package does not contain these commands.
523
+ **Pi native Scout adapter — npm 0.1.12:** Pi **1.5.0** joins Claude Code through the shared browser-approved installer. Run `npx -y openmerit@latest install --client pi [--scope user|project]`. The **Add to Pi** action below Claude copies setup text only; there is no Pi deep link. Approve the matching browser code, then start a new Pi session. Pi owns native MCP, tools, model, permissions and sessions; the adapter adds only the authenticated connection and shared Scout rule/skill. No `/openmerit` lifecycle commands, scheduler, LaunchAgent or automatic-update hooks are carried forward. Explicitly reinstall to update. See [Pi paths, migration and SDK embedding](pi-extension.md).
524
+
525
+ Each new client/scope installation has independent credentials and approval; project keys are also separated by project and stored outside the repository. Shell `login`, `logout`, `status`, `remove` and `doctor` accept `--client pi|claude-code --scope user|project` to target that installation. Pi connection JSON references a private credential path rather than containing the key. The legacy Claude user credential path remains supported. Package publication and website deployment are independent; the versioned Claude setup below describes earlier releases.
526
+
527
+ **Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the Claude commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code was the supported client in that release; Pi support requires 0.1.12 or newer. The older `0.1.7-preview.5` package does not contain these commands.
524
528
 
525
529
  Setup takes four steps, and you act in each one only once:
526
530
 
@@ -583,9 +587,9 @@ The agent compares model and endpoint, evidence, reference cost/time and unknown
583
587
 
584
588
  ### Automatic instruction updates — version 0.1.10
585
589
 
586
- Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.
590
+ **Claude Code only.** These hooks and `update` commands do not apply to Pi. Pi updates require an explicit installer rerun and a new session.
587
591
 
588
- Version 0.1.11 also recognizes updater entry points reached through symbolic links. Existing installations need an installer rerun to acquire this updater-code fix; automatic instruction updates do not replace the installed updater itself.
592
+ Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.
589
593
 
590
594
  Setup discloses automatic updates and installs asynchronous `SessionStart` and `Stop` command hooks. They check npm's stable `latest` release at most once per 24 hours per installation. Offline or failed checks retain the installed version and retry at the next eligible hook after 15 minutes. `Stop` means Claude finished responding, not that the user's task succeeded. No background service watches connectivity while Claude Code is closed, and checks never ask Claude to keep working. A one-shot `claude -p` process can exit before an asynchronous Stop hook finishes; startup checks and later sessions provide the retry path. Managed policies or disabled hooks can prevent automatic checks; an explicit manual update remains available.
591
595
 
@@ -674,6 +678,7 @@ The twice-daily updater fetches eight independent public sources, including Mode
674
678
  | [HELM Arabic](https://crfm.stanford.edu/helm/) (Stanford CRFM) | Mean score across HELM's Arabic scenarios from the newest release, mostly multiple choice. Models missing a scenario are left out; no per-model dates. |
675
679
  | [BRIDGE Medical](https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard) (YLab) | Zero-shot average across clinical-text tasks in several languages, mostly open-weight models. Average only; no per-model dates. |
676
680
  | [SuperCLUE](https://www.superclueai.com) | Chinese general-board category scores: math and science reasoning, hallucination control, precise instruction following, agentic coding and planning, for about 20 models. Boards older than six months are not read. |
681
+ | [Fireworks Specialized Intelligence Index](https://fireworks.ai/specialized-intelligence-index/) | Domain benchmarks contributed by practitioner companies (legal, finance and tax, healthcare and life sciences, customer support, software and SRE, web search, marketing, hardware and more), from the index's published JSON. Each benchmark is its own `· Fireworks` series scored by its headline metric; only 0–100 headlines are read, so Elo, US-dollar and relative-score boards and the cost and duration columns are not. Unranked rows and entries Fireworks footnotes as run inside another product or with rejected tasks are left out. Most results are run by the contributor, some by Fireworks, which is itself an inference provider. No per-model dates. |
677
682
  | [LLM Stats](https://llm-stats.com/benchmarks) (vendor-reported) | Scores model developers report in announcements, papers and model cards, compiled by LLM Stats and reused with attribution under its terms. Scout reads LLM Stats' 100 most-reported benchmarks and keeps only results it marks self-reported, on 0–1 or 0–100 scales, as separate `· vendor-reported` series that never merge with third-party measurements of the same benchmark. Vendor-reported results count toward ranking. They are not independently verified, and settings such as tools, attempts and reasoning effort follow each vendor. Base (pretrained) checkpoints are never matched to instruct models. A benchmark that leaves the 100 keeps its last results. |
678
683
  | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
679
684
  | [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
package/docs/security.md CHANGED
@@ -1,5 +1,7 @@
1
1
  # Security
2
2
 
3
+ **Pi replacement — unreleased, unpublished source:** Pi 1.5.0 uses native MCP plus shared Scout rule/skill sections, not the former evaluation adapter. It never initiates login, changes the host model/tools/session, or provisions background jobs. Browser-approved credentials are independent per client/scope/project, stored privately outside the repository, and passed as MCP HTTP headers, never instructions. Pi's installed connection JSON contains only the private credential path. See [Pi paths and targeted removal](pi-extension.md). The standalone evaluation CLI and core coordinator remain separate.
4
+
3
5
  The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.
4
6
 
5
7
  The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.
@@ -12,6 +14,8 @@ Never place secrets in profiles, policies, intent constraints, summaries, eviden
12
14
 
13
15
  ## Mutation authorization
14
16
 
17
+ This section describes core/custom evaluation harness policy. Pi-specific lifecycle and background-runner behavior below is **historical Pi 0.87 functionality**, removed from the new Scout adapter. Installing the replacement does not establish that an old background job was removed; manage that job separately.
18
+
15
19
  Most intents carry `modelMutationAllowed: false`. Only application-target apply-swap and rollback intents can authorize a mutation, limited to the relevant proposal, prior configuration, and confirmed target ID. Pi's model is explicitly excluded. Automatic swaps require explicit policy opt-in and meeting the configured verified-swap threshold, which can be zero. Otherwise a proposal requires approval. Post-swap evaluation depends on the harness performing the checks and reporting their results. OpenMerit is not a sandbox around the harness's tools.
16
20
 
17
21
  Persistent wakeup provisioning is a separate authorization boundary. An adapter that needs to install a system scheduler, create a paid cloud agent, or change infrastructure must declare `wakeupProvisioning: "infrastructure_change"`; OpenMerit will not request that provisioning unless the confirmed check policy sets `infrastructureChangesAllowed: true`.
@@ -24,7 +28,7 @@ The background runner imposes a 15-minute wall-clock limit and records a nonzero
24
28
 
25
29
  `.openmerit/` may reveal task names, candidate identifiers, evidence locations, budgets, or model choices. Treat it as sensitive project metadata even though it must not contain credentials.
26
30
 
27
- Pi also records OpenMerit requests and results in its session history. Project context and evaluation data may be sent to the model providers, tools, and evidence systems configured in the harness. Local OpenMerit storage does not imply that all processing stays on your machine.
31
+ The historical evaluation adapter also recorded OpenMerit requests and results in Pi history. With native Scout, Pi manages its own session/tool records and provider context; the adapter adds no separate evaluation-history store. Task descriptions sent to hosted Scout follow its data policy. Local OpenMerit storage does not imply that all processing stays on your machine.
28
32
 
29
33
  ## Evidence integrity
30
34
 
@@ -55,6 +59,8 @@ The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses host
55
59
 
56
60
  **Automatic instruction updates — version 0.1.10:** updater-enabled setup defaults on with an opt-out, installing asynchronous Claude Code startup/response-end hooks. Checks use npm's public stable registry without credentials, task content or installation IDs. SHA-512 verification checks package bytes against registry metadata; it does not defend against a compromised publisher/registry. The updater reads only a bounded instruction bundle, never executes downloaded code or package scripts, and leaves MCP configuration, keys, privacy settings and project policy unchanged. A supported release contract permits instruction updates; maintainers must change it when permissions or data handling change. This is a publisher assertion, not semantic validation of instructions. Edited files are retained, concurrent writes serialize, and journaled recovery stops on intervening edits. New sessions reliably load new instructions. See [automatic updates](scout.md#automatic-instruction-updates-version-0110) for cadence, offline retries, manual commands and the one-time bootstrap required for older installations. Availability follows npm publication independently of hosted releases.
57
61
 
62
+ These update hooks are Claude Code only. The new Pi adapter installs none; updates require explicit targeted reinstall. Its `login`, `logout`, `status`, `remove` and `doctor` shell commands target the selected client/scope without revoking another installation's independent key.
63
+
58
64
  ### OpenMerit coding-agent reset — 23 October 2026
59
65
 
60
66
  **Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
package/docs/testing.md CHANGED
@@ -1,5 +1,7 @@
1
1
  # Testing and validation
2
2
 
3
+ **Pi source replacement — unreleased and unpublished:** the new Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. The dated Pi 0.87 lifecycle/scheduler results below remain historical receipts, not proof of the replacement or supported current commands. Package, browser-approval, native Pi and SDK acceptance must be reported for the new build separately; a successful documentation build alone establishes none of them. Core coordinator and standalone evaluation tests remain separate.
4
+
3
5
  ## Stable 0.1.7 acceptance — 24 October 2026
4
6
 
5
7
  Published `openmerit@0.1.7` to npm's `latest` channel after full repository verification: 757 passing tests, one existing skip, syntax/type checks, builds and documentation checks. The `preview` tag remains `0.1.7-preview.5`. A fresh registry installation on macOS ARM64 with Node.js 22.23.1 matched the tested 104-file tarball byte-for-byte. Packed core/protocol/Pi exports and the installed terminal runtime smoke checks passed, including passive collection, source removal, runtime outage/recovery and shutdown.
@@ -1,55 +1,50 @@
1
1
  # Troubleshooting
2
2
 
3
- Start with the current status and the local audit trail. These show what OpenMerit requested, what the harness reported, and which evidence references were recorded.
3
+ The Pi Scout adapter requires OpenMerit **0.1.12** or newer and targets Pi **1.5.0**. Check the resolved package version when using `@latest`. The removed Pi 0.87 `/openmerit` workflow is not a diagnostic path for the replacement.
4
4
 
5
- ```text
6
- /openmerit status
7
- /openmerit doctor
8
- /openmerit logs
9
- ```
10
-
11
- ## The command is missing
5
+ ## Scout is missing in Pi
12
6
 
13
- Check that OpenMerit is installed and enabled in Pi:
7
+ Run targeted checks in your shell:
14
8
 
15
9
  ```sh
16
- pi list
10
+ npx -y openmerit@latest status --client pi --scope user --check
11
+ npx -y openmerit@latest doctor --client pi --scope user
17
12
  ```
18
13
 
19
- The published extension targets Pi 0.87 and Node.js 22.19 or newer. Follow [getting started](getting-started.md) to install it, then start Pi in your product repository.
20
-
21
- ## Setup did not begin
14
+ Use `--scope project` from the original project root for project setup. Check Pi and Node.js versions against [setup](getting-started.md), complete browser device approval, then start a **new session**. Installing the npm package or reconnecting an old session does not load new extension instructions. For SDK embedding, load Pi's `createMcpExtension()` explicitly alongside `createScoutPiExtension`; the Scout factory alone is not an MCP runtime.
22
15
 
23
- Automatic setup requires an unconfigured project, an interactive Pi UI, and a bounded source-detection match for an application LLM call. Non-interactive runs do not start the confirmation flow, and a project without a detected target remains idle until `/openmerit setup` is requested. Existing project state or active work may also prevent a fresh setup request.
16
+ ## The connection configuration is invalid
24
17
 
25
- Inspect status first. If no intent is active, use `/openmerit setup` to request setup explicitly; this is also the supported correction path when source detection or a prior setup recorded the wrong incumbent. A new setup preserves audit history but replaces the active task profile, candidate/assessment handoffs, frontier, and baseline state after the new result is accepted. Do not delete `.openmerit/` as a routine reset: it contains your policy, lifecycle state, evidence references, and audit history.
18
+ The installed extension reads `openmerit-scout.json` beside its bundle; that file points to a private credential file outside the repository. Do not put the key in the extension or prompt. Check the scope and configuration-directory overrides against [installed paths](pi-extension.md#installed-files-and-credentials). Reinstall explicitly to repair missing owned files:
26
19
 
27
- ## Evidence is missing or incomplete
20
+ ```sh
21
+ npx -y openmerit@latest install --client pi --scope user
22
+ ```
28
23
 
29
- A task profile is not the same as working telemetry. Ask Pi which required metrics are uncovered, where their measurements should come from, and which evaluation artifacts have been run. Normal coding checkpoints do not replace those measurements.
24
+ Expired or revoked credentials require targeted `login` followed by reinstall. The adapter reports a sanitized setup diagnostic; it never starts sign-in on its own. Approve only a browser code you initiated.
30
25
 
31
- The [metrics guide](metrics-and-evidence.md) explains missing and insufficient evidence. More samples or instrumentation may be needed before a candidate can be compared.
26
+ ## Installation reports a conflict
32
27
 
33
- ## A result was rejected
28
+ The installer does not overwrite edited or unowned files. Inspect and preserve your changes before resolving the conflict. Also check for overlapping Pi user/project installations or an existing `openmerit` MCP entry. Remove the unwanted owned installation using its own scope; do not revoke a different client's key as a repair. Disable the old `npm:openmerit` Pi registration when migrating from 0.87.
34
29
 
35
- Read the validation error and audit event. Successful work needs evidence references and the output structure required by that intent. Frontier results must match the task and product revisions, identify their assessments, and satisfy the comparison rules.
30
+ ## The adapter did not update automatically
36
31
 
37
- Have the harness correct the measurement or output problem. Do not mark an intent successful merely to advance the workflow.
32
+ That is expected. Pi has no OpenMerit auto-update hooks. Explicitly rerun `install --client pi` with the original scope, then start a new session. Claude Code's update settings do not configure Pi.
38
33
 
39
- ## Work appears stuck after a restart
34
+ ## `/openmerit` or evaluation tools are missing
40
35
 
41
- Pending or active intent state is durable. Inspect `state.json` and `events.jsonl` using the paths from `/openmerit logs`, and ask the harness to reconcile that work with the current session before requesting another intent. The current adapter may need supervised recovery; automatic completion after a restart is not guaranteed.
36
+ They were removed, not renamed. The Pi adapter now supplies only native Scout MCP and shared guidance. It does not set up evaluations, recover intents, schedule work, swap models, or install a LaunchAgent. Existing legacy background jobs must be removed separately; installing the new adapter is not evidence that they stopped. Preserve old evaluation state and audit files if needed. For new evaluations use [the standalone CLI](agent-evaluations.md); custom harnesses may still use [the core coordinator](adapter-guide.md).
42
37
 
43
- ## Checks are not running while Pi is closed
38
+ ## Scout was not consulted or its evidence is insufficient
44
39
 
45
- On macOS, check that the confirmed policy requests persistent execution and allows infrastructure changes, that the Mac user is logged in, and that Pi authentication works without shell-only credentials. `/openmerit doctor` reports scheduler diagnostics; `.openmerit/scheduler-status.json` records completed wakeups and failures. An active pause unloads the managed LaunchAgent. Other platforms need an external scheduler. A successful schedule does not establish that model-driven work completed. See [automatic checks](automation.md).
40
+ The rule and skill are model guidance, not a guaranteed tool gate. Confirm they loaded in a new session and that the host permits the MCP tools. When project model policy is missing, ask the user; when Scout is unavailable or lacks suitable evidence, pause the model choice rather than invent a result. Host model/tool/session restrictions remain authoritative. Scout candidates are not measurements of your application's quality.
46
41
 
47
- ## Work continues after pausing
42
+ ## Evaluation or terminal problems
48
43
 
49
- Pause is durable and gates automatic work, but an already-active intent can continue and explicit commands remain available. Inspect the active intent and audit timestamps to distinguish prior work from a new automatic dispatch. `/openmerit cancel` records abandonment; it does not terminate a provider request or harness process. The standalone terminal is independent of Pi pause; stop its CLI with Ctrl+C to stop collection. If a new automatic intent starts while the persisted pause is active, record a sanitized reproduction and your installed version.
44
+ The standalone products are separate from Pi setup. Use [agent-evaluation diagnostics](agent-evaluations.md) for tasks, graders, budgets and execution blocks, or [terminal diagnostics](inference-terminal.md) for missing inventory and request evidence. A populated dashboard is not proof of successful evaluations. Never mark missing or rejected evidence successful merely to advance a workflow.
50
45
 
51
46
  ## Report a reproducible problem
52
47
 
53
- Include the OpenMerit, Pi, and Node.js versions, the triggering command or signal, the lifecycle stage, and a sanitized error. Remove credentials, private traces, and proprietary evaluation data before sharing anything in a public issue.
48
+ Include OpenMerit, Pi and Node.js versions, the client/scope, the triggering command, and a sanitized error. Remove credentials, private paths, prompts and proprietary data before sharing. Do not attach credential or connection files containing private configuration.
54
49
 
55
50
  [Open an issue on GitHub](https://github.com/laz-aslam/openmerit/issues)
@@ -1,6 +1,6 @@
1
1
  # Pi UX behavioral reference
2
2
 
3
- This document records user-visible behaviors from the earlier OpenMerit implementation. It is not an implementation specification; the current implementation carries forward only the behaviors listed below, rather than its source code, storage schema, or internal contracts.
3
+ This document is a **historical evaluation-adapter reference**, not the current Pi interface specification. The unreleased Pi 1.5.0 native Scout replacement does not carry forward the `/openmerit` commands, lifecycle, pause controls, or scheduling described here. See [the current Pi guide](pi-extension.md). The list below preserves earlier UX ideas for custom evaluation harnesses only.
4
4
 
5
5
  ## Behaviors worth preserving
6
6