openmerit 0.1.6 → 0.1.7-preview.2

package/docs/inference-terminal.md

@@ -1,72 +1,69 @@
 # Inference Terminal
 
-The Inference Terminal is included in the default `openmerit@0.1.6` npm release. It is an early part of OpenMerit, not the complete product vision. It gives people and agents a local overview of an application’s models, providers, connections, inference activity, and compute. The selected repository or application path defines its scope. A coding agent can launch or read the terminal, but the coding harness’s own requests, tokens, and costs are outside that scope. Earlier `openmerit@0.1.5` does not include this command.
+The terminal gives people and agents a local overview of an application's models, services, connections, inference activity, and linked compute. The selected repository or application path defines its scope. Coding-harness requests, tokens, and costs are outside that scope.
 
-The terminal reads existing metadata independently of inference. It does not call a model, change a route, run evaluations, load or unload models, or store prompts and responses. Closing it does not stop your workloads. Collection consumes a small amount of local CPU, memory, and runtime API capacity; it is not zero-overhead monitoring.
-
-## Share the sample
-
-The [hosted sample](https://dash.openmerit.site/) lets you explore a fictional environment in a browser without installing the terminal. It is deployed separately from this npm package, so its sample data and interface can differ. All identities, requests, costs, and hardware readings in the sample are illustrative.
-
-The hosted page does not connect to your machine, accept project uploads, or run local collectors. Use the npm installation below to view your own application.
+It reads existing evidence independently of inference. It does not call a model, change a route, run evaluations, load or unload models, or retain prompts and responses. Closing it does not stop your workloads. Collection uses local CPU, memory, and authorized runtime API capacity.
 
 ## Install
 
-Requires **Node.js 22.19 or newer**, with npm available in your terminal. From your application's directory:
+Requires **Node.js 22.19 or newer** and npm. Stable `openmerit@0.1.6` includes the terminal:
 
 ```sh
-cd /path/to/your/project
 npm install -g openmerit
+cd /path/to/your/application
 openmerit dash
 ```
 
-Version `0.1.6` brings the code from `0.1.6-preview.5` into npm's default `latest` channel. It includes historical activity ranges, project-evidence corrections, expanded endpoint discovery, code/configuration indexing, counts-only diagnostics, and phased background indexing. References to earlier preview versions below explain when behavior changed; all of those changes are included in `0.1.6`.
+No Git checkout, build step, Pi installation, provider API key, or separate audit command is needed. The global install does not add dependencies or configuration to your application. Run the same install command to update, or `npm uninstall -g openmerit` to remove it. `npm ls -g openmerit --depth=0` shows the installed version; `npm view openmerit dist-tags` reports published channels.
+
+The CLI starts a server bound to `127.0.0.1` and opens the browser before initial collection finishes. Port 4310 is preferred; if occupied, the operating system chooses a free local port. The actual URL is printed. macOS, Windows, and desktop Linux use the default URL handler. SSH and headless sessions print the URL; use a private SSH port forward to view a remote instance. Browser-launch failure does not stop collection.
 
-The terminal is bundled alongside the coordinator, protocol, and Pi adapter. The installation downloads prebuilt assets; no Git checkout, build step, API key, Pi installation, or separate audit command is needed to run the terminal. Installation is global for your Node environment: it does not add a dependency, lockfile, or configuration to the application directory. It replaces any globally installed `openmerit` in the same npm prefix.
+If npm reports `EACCES`, use a user-owned Node installation/npm prefix rather than `sudo`. If the command is not found after installation, reopen the shell and check that npm's global executable directory is on `PATH`.
 
-`openmerit dash` reads the current directory, starts a server bound to `127.0.0.1`, and opens its URL before collection finishes. Port 4310 is preferred; when occupied, the operating system chooses a free local port. The actual URL is always printed. macOS, Windows, and Linux desktop opening use the platform's default URL handler. SSH and headless Linux sessions print the URL instead; use a private SSH port forward to view a remote instance. A browser-launch failure does not stop collection.
+The **source interface and hosted dashboard** follow the device's light/dark preference, including charts and inspectors, and update when that preference changes. There is no theme control or saved override. This appearance change is not included in npm 0.1.6.
 
-Run `npm install -g openmerit` to update, including from an earlier `@preview` or hosted `-dash.` build. Stop the old terminal process and launch `openmerit dash` again after updating. Pin this release with `npm install -g openmerit@0.1.6`; remove it with `npm uninstall -g openmerit`. `npm ls -g openmerit --depth=0` shows the installed version, and `npm view openmerit dist-tags` shows the registry channels. The separate `preview` tag can still point to an older prerelease; use the default install for `0.1.6`.
+Keep the CLI running while using the dashboard. Files unreadable by your user are reported as unavailable. Configuration can identify services but cannot supply missing request history.
 
-If npm reports `EACCES`, use a user-owned Node installation/npm prefix rather than installing with `sudo`. If `openmerit` is not found after a successful install, reopen the terminal and check that your npm global executable directory is on `PATH`.
+## Hosted samples
 
-The page uses dark mode only. Keep the terminal command running while using the dashboard. No extra operating-system permission prompt is expected for readable project files and loopback runtime APIs. Files that your user cannot read are reported as unavailable. Supported request logs populate activity automatically; a repository with only model configuration can show its services but cannot supply missing request history.
+[dash.openmerit.ai](https://dash.openmerit.ai/) shows 200 fictional agents, 699 task IDs, 21 models, and 9.6 million requests over 30 days. The [before/after comparison](https://dash.openmerit.ai/demo) uses a separate fictional insurance scenario with 280 workers and 980 task IDs across fourteen workflows. See [the comparison details](#hosted-enterprise-sample-and-comparison) for selection rules, preference overrides, prices, and limitations.
+
+Hosted pages do not connect to your machine, accept project uploads, or run inference. Their public JSON endpoints return fictional data; the local snapshot API is not exposed. They request no search-engine indexing. The local CLI's sample selector uses a smaller 40-agent, 140-task example described below.
 
 ## Run from a source checkout
 
 With dependencies installed in the OpenMerit checkout:
 
 ```sh
-npm run dash
-npm run dash -- /path/to/your/project
+npm run dash -- /path/to/your/application
 ```
 
-`npm run dash` bundles the interface before starting. `npm run build` includes the executable and local web assets under `dist/terminal/`. Torph and Liveline are bundled locally. To install this source build for shell use, run `npm install -g .` from the checkout.
+`npm run dash` bundles the interface before starting. Without a path it reads the current directory. `npm run build` includes the executable and bundled assets under `dist/terminal/`. To install that build, run `npm install -g .` from the checkout.
 
-The shell command also accepts a path and explicit settings:
+## CLI options
 
 ```sh
-openmerit dash /path/to/your/project --port 4311 --interval 20
+openmerit dash /path/to/your/application --port 4311 --interval 20
 openmerit dash --no-open
 ```
 
 | Option | Behavior |
 | --- | --- |
-| `[path]` | Project directory; defaults to the current working directory |
-| `--port <number>` | Require this loopback port; fail if it is occupied. Without this option, use 4310 or a free port |
+| `[path]` | Selected application directory; defaults to the working directory |
+| `--port <number>` | Require this loopback port; fail if occupied |
 | `--interval <seconds>` | Collection interval from 10 to 30 seconds; defaults to 15 |
-| `--no-discovery` | Skip project-file and request-log discovery and additional runtime discovery for this run; explicitly declared runtimes and exports, explicitly linked machine context remain available |
-| `--no-open` | Start collection and print the URL without launching a browser |
-| `--json` | Print one snapshot and exit without starting an HTTP server |
-| `--diagnostics` | Print repository-index coverage counts only, then exit; no browser, server, or endpoint probes |
-| `--demo` | Use an explicitly fictional environment; no local collectors run |
+| `--no-discovery` | Disable automatic project/index/log and additional runtime discovery; explicitly configured sources remain available |
+| `--no-open` | Print the URL without opening a browser |
+| `--json` | Print one snapshot and exit without an HTTP server |
+| `--diagnostics` | Print repository-index coverage counts, without endpoint probes, browser, or HTTP server |
+| `--demo` | Use only a fictional local sample; no local collectors run |
 | `--help` | Show usage |
 
-Use Ctrl+C to close the terminal. There is no persistent daemon or background scheduler. Collection runs only while the command is running. The terminal is separate from Pi's `/openmerit pause` and other orchestration commands.
+Ctrl+C stops collection. The terminal has no persistent daemon and is independent of Pi's pause and scheduling commands.
 
 ## Repository indexing
 
-Version `0.1.6` includes a local syntax index of application declarations, introduced in `0.1.6-preview.4` and extended with continuing background batches in `0.1.6-preview.5`. No repository access needs to be shared with OpenMerit, and no separate audit step is required. The index runs initially and rechecks 60 seconds after a pass completes while the terminal is running. It creates no files in the application.
+A local background worker indexes supported application declarations without executing code or creating files in the application. It starts with collection and rechecks 60 seconds after a completed pass.
 
 The reader supports Python, JavaScript/TypeScript (including JSX/TSX), JSON, YAML, TOML, and project-local `.env*` files. It finds literal model/endpoint fields, conservative same-file variable and object references, environment references resolved from the nearest project-local dotenv directory, selected SDK call patterns, and enclosing function names. Unknown/private HTTP(S) endpoints remain named by their sanitized address; a custom host or port never establishes a vendor, runtime, deployment, or physical machine. Public Hub/download URLs do not establish an inference service. URLs found by this index are **never probed**; existing explicitly authorized runtime discovery remains separate.
 
@@ -76,23 +73,21 @@ The reader supports Python, JavaScript/TypeScript (including JSX/TSX), JSON, YAM
 
 The index does not execute or import project code, load shell files, inspect running processes, consult inherited shell variables, or fetch cloud accounts. It does not resolve cross-file imports, arbitrary expressions, template interpolation, configuration loading order, dynamic routers, YAML aliases, or multiline TOML values. Conditional values remain candidates; a fallback remains a fallback even if a runtime might choose it. Supported syntax is deliberately conservative. Models and task names are never inferred from prose, prompts, or model-name conventions.
 
-Reads stay within the selected directory, skipping symlinks, hidden directories, dependency/build folders, credential-named files, and request-log roots handled by the separate activity reader. In published `0.1.6-preview.4`, each scan visits at most 8,000 entries and eight nested directories, reads at most 400 supported files, 128 KiB per file and 4 MiB overall, and retains at most 2,000 declarations and 100 detailed coverage notes. Dotenv files are prioritized, then files are sorted by path. These bounds can leave a large repository partially indexed. Removing a declaration removes it after the next scan.
+The index skips symlinks, hidden directories, dependency/build folders, credential-named files, and request-log roots handled by the activity reader. It processes eligible files in continuing batches rather than imposing a repository-wide file, depth, byte, or declaration cap. Per-file and parser limits still apply; coverage reports excluded or unreadable material. Removing a declaration removes it after the next completed scan.
 
 The snapshot retains selected identifiers, relative file paths, line numbers, and sanitized endpoint addresses. It discards raw source text, prompt/response fields, credentials, URL credentials, query strings, fragments, and resource paths. Known dedicated resource subdomains are redacted. Structural paths such as `/v1` or `/api/v1` may remain. Local endpoint addresses, model names, and filenames can still be sensitive; the full snapshot is intended for local viewing.
 
-To share coverage without sharing the repository or its identifiers, run this option from the project with **0.1.6**:
+To inspect coverage counts without sharing the repository or its identifiers, run:
 
 ```sh
 openmerit dash --diagnostics
 ```
 
-It prints only format/read/skip/declaration/issue counts and scan limits. It does not include paths, variable names, model names, endpoint identities, file contents, or credentials, and sends nothing itself. The user chooses whether to copy the output. Detailed issue counts cover the retained notes; the summary reports the total unresolved count.
+It prints only format/read/skip/declaration/issue counts and scan limits. It does not include paths, variable names, model names, endpoint identities, file contents, or credentials, and sends nothing itself. The user chooses whether to copy the output. Issue totals include all unresolved notes; the detailed list retains at most 100 notes.
 
 `--no-discovery` or `"discovery": false` disables the index. Request history still requires supported application logs or explicit exports. An inactive project can expose its declarations without having retained any usage history; code cannot reconstruct missing requests, tokens, latency, or spend.
 
-## Phased repository indexing
-
-Version `0.1.6` retains the `0.1.6-preview.5` behavior, which replaces the repository-wide entry, depth, file-count, total-byte, and declaration limits above with continuing batches. Versions through `0.1.6-preview.4` retain the earlier bounds.
+### Background indexing and refresh
 
 - **Open immediately:** the loopback server listens before initial collection. The browser shows “Opening your inference terminal” while sources load, without displaying empty metrics as measured zero. Initial collection failures are disclosed and retried on the next collection tick.
 - **Discover, index, link:** a dedicated worker walks eligible directories, then prioritizes dotenv files, structured configuration, and inference-related source names. The initial index publishes usable declarations in batches. The interface shows discovered files while the total is unknown, then checked/total files. Completion means the eligible-file pass finished; exclusions and unresolved syntax still appear in coverage.
@@ -106,9 +101,7 @@ Nothing is written into the application, and no persistent index or source-conte
 
 These changes apply to **repository declarations only**. The separate request-log, runtime, hardware-export, and configuration-inventory limits elsewhere in this guide still apply. They do not recover missing historical usage or make a log importer unlimited.
 
-## Review corrections (0.1.6-preview.3)
-
-Version `0.1.6-preview.3` includes the following corrections; earlier previews retain their previous behavior:
+### Request and inventory consistency
 
 - JSONL readers check bounded content fingerprints before resuming from a saved offset. A changed checkpoint resets the cache and imports the current file, including when it was truncated and regrew between polls. Each cached file adds at most 8 KiB of verification reads per poll; only hashes are retained. These checks cover the initial consumed bytes and the latest read boundary, not every byte of a large historical file. Rewrites that preserve both sampled regions cannot be distinguished from appends; append-only logs or rotation to a new file remain the reliable format.
 - Copies of one request that disagree on `completed` versus `failed` expose a `status` conflict in request details and the dashboard warning. Source precedence still determines the displayed outcome; a normal `running` to terminal update is not a conflict.
@@ -117,11 +110,11 @@ Version `0.1.6-preview.3` includes the following corrections; earlier previews r
 
 ## Automatic application discovery
 
-**Evidence corrections (0.1.6-preview.2):** project declarations now establish service and endpoint inventory, explicit host links establish physical placement, and task summaries remain separate from requests. Missing status, currency, and cost basis stay unknown. Inventory count limits disclose omissions, and missing request evidence is distinguished from measured zero usage. Earlier `0.1.6-preview.1` retains the previous automatic probes and defaults. Update the package and restart `openmerit dash` to apply these corrections.
+Project declarations establish service and endpoint inventory; explicit host links establish physical placement. Task summaries remain separate from request measurements. Missing status, currency, and cost basis stay unknown, and inventory/read limits appear in source coverage.
 
 Discovery runs as part of `openmerit dash`; no separate audit command or exporter is needed for supported sources. The first snapshot checks the selected project and its declared loopback runtime endpoints. Project declarations and previously unresponsive addresses are checked again every 60 seconds. Responding runtimes, OS readings, project request logs, and existing exports refresh at the normal 10–30-second collection interval.
 
-Through `0.1.6-preview.3`, configuration discovery reads only four fixed files: `.env`, `.env.local`, `package.json`, and `requirements.txt`, each limited to 128 KiB and required to resolve to a regular file inside the project. It does not recursively index source code, execute project code, source shell files, or expand environment expressions. Supported SDK declarations are shown as **Detected** source hints; an installed SDK is not proof of a service connection or request. Supported literal `.env` declarations establish project configuration evidence. Inherited shell credentials and endpoint variables are not added to project services; the terminal cannot establish which project owns them. For applications configured only through their launch environment, declare the services explicitly in `.openmerit/terminal.json` or supply recorded request metadata. Each file is independent evidence; discovery does not pretend to know an application's loading order. Credential contents and unrelated settings are discarded before metadata is retained or returned.
+Alongside the repository index, the bounded provider/runtime discovery reader checks four root files: `.env`, `.env.local`, `package.json`, and `requirements.txt`, each limited to 128 KiB and required to resolve to a regular file inside the project. This reader does not execute project code, source shell files, or expand environment expressions. Recursive source/configuration references are handled by the separate syntax index described above. Supported SDK declarations are shown as **Detected** source hints; an installed SDK is not proof of a service connection or request. Supported literal `.env` declarations establish project configuration evidence. Inherited shell credentials and endpoint variables are not added to project services; the terminal cannot establish which project owns them. For applications configured only through their launch environment, declare the services explicitly in `.openmerit/terminal.json` or supply recorded request metadata. Each file is independent evidence; discovery does not pretend to know an application's loading order. Credential contents and unrelated settings are discarded before metadata is retained or returned.
 
 Local runtime probes require an endpoint in the selected project’s `.env` or `.env.local`, or an explicit runtime in `.openmerit/terminal.json`. No standard ports are probed automatically. SDK dependencies, runtime names in historical records, and inherited shell endpoint settings alone do not authorize a probe. A recorded request remains visible without discovering any additional runtime models. Supported API endpoint variables are listed below; `OLLAMA_HOST` and `OLLAMA_BASE_URL` identify native Ollama endpoints. `OLLAMA_HOST` may use `127.0.0.1:11434` without the HTTP scheme. Endpoint paths must be empty or `/v1`; credentials, queries, fragments, and non-loopback hosts are rejected. At most eight discovered origins are checked without authentication or redirects. Compatible endpoints use read-only `GET /v1/models` and `GET /metrics`; native Ollama declarations use `GET /api/tags` and `GET /api/ps`. Unknown or nonresponding candidates do not become service rows. Configured origins are not probed twice. A compatible catalog supplies model availability; recognized vLLM counters can also identify vLLM. A port number alone never establishes LM Studio or vLLM identity, so an otherwise unidentified compatible runtime is named by its loopback origin. Bounded inventory from a project endpoint can be shown, including available models that have no recorded project usage. Catalogs are limited to 100 entries per response; exceeded limits mark the source Partial, include omitted counts, and display an inventory-coverage notice. API compatibility and a loopback address do not establish a runtime role, physical host, or self-hosted serving mode. Inventory and shared runtime counters do not establish that the selected application used every model. Removing an endpoint declaration removes its discovered inventory on the next 60-second discovery pass. This is bounded project-linked discovery, not network or remote-host scanning.
 
@@ -145,15 +138,15 @@ The reader checks at most 2,048 directory entries and the newest 128 candidate f
 
 ### Application scope
 
-The terminal does not discover or read Pi, Codex, Claude Code, OpenCode, or other coding-harness histories. Older preview builds offered Pi history access; that reader and the `--access` / `--revoke-access` options have been removed. Existing `~/.openmerit/terminal-access.json` grants are ignored, left untouched, and cannot enable history collection. Restart an older terminal process after upgrading to apply the new boundary.
+The terminal does not discover or read Pi, Codex, Claude Code, OpenCode, or other coding-harness histories. Application scope does not include coding-assistant sessions or credentials.
 
 Application agents, background jobs, and evaluations remain in scope when they produce supported request evidence. The `agent` scope means an agent in the application, not the coding assistant developing it. Explicit exports and discovered request logs must belong to the application; records explicitly marked `scope: "development"` are rejected. An unlabeled API response cannot establish whether it came from an application or a coding tool, so the terminal does not guess its purpose from a model or agent name. Keep development logs separate from application request sources.
 
 Configuration identifies declared services; it cannot establish past executions or costs. Local runtime inventory and whole-machine counters can describe shared infrastructure and are not attributed exclusively to the selected application. The collector does not retrieve cloud account usage, inspect arbitrary processes, or connect to a remote deployment from its source code alone. Applications without a supported request source can use the optional activity export below.
 
-## Historical activity (0.1.6-preview.1)
+## Historical activity
 
-Preview `0.1.6-preview.1` adds **7d**, **30d**, and **All time** alongside 1h, 6h, and 24h. **All time** is the default. It shows all retained, supported request evidence without an age cutoff, including logs from an application that is no longer running. The collector and project-log readers no longer discard records merely because they are more than 24 hours old. These changes are not in `0.1.6-preview.0`.
+Choose **1h**, **6h**, **24h**, **7d**, **30d**, or **All time**. **All time** is the default. It shows all retained, supported request evidence without an age cutoff, including logs from an application that is no longer running. The collector and project-log readers retain supported older records within their read/cache limits.
 
 The date range below the page heading shows the earliest and latest retained request, not the project creation date. All time charts span those recorded dates; an inactive project's last request stays dated in the past. Longer ranges use calendar labels and interval units appropriate to their span. Overview totals, models, services, maps, agent/task details, and activity use the same selected range. Hardware and capacity continue to show their latest samples independently of the activity range.
 
@@ -174,13 +167,13 @@ Pages, detail panels, and inventory cards lead directly with their title. Suppor
 
 Use Command+K or Control+K to find a view, model, source, or reported agent/task. Escape closes details. Keyboard actions remain immediate, and reduced-motion preferences disable movement transitions.
 
-Dropdowns share a dark menu, inset chevron, and neutral focus and selection states. For the agent and host-location selectors, use Enter, Space, or an arrow key to open; arrows, Home/End, or typing a name prefix move through options. Enter or Space confirms, Escape cancels, and Tab confirms and continues through the page. Clicking outside dismisses the menu. An open menu preserves its options and position during passive polling; underlying collection continues.
+Dropdowns follow the interface's color scheme and share an inset chevron with neutral focus and selection states. For the agent and host-location selectors, use Enter, Space, or an arrow key to open; arrows, Home/End, or typing a name prefix move through options. Enter or Space confirms, Escape cancels, and Tab confirms and continues through the page. Clicking outside dismisses the menu. An open menu preserves its options and position during passive polling; underlying collection continues.
 
 When request metadata includes stable agent or task IDs, the overview leads directly with their counts for the selected window, without an “Observed” prefix. Select a count to browse those agents or tasks and inspect their request totals, models, connections, and available measurements. Records without these IDs do not contribute to those counts. These are activity summaries, not an agent registry or a task lifecycle tracker; a completed inference request does not establish that its task is complete. The overview previews up to twelve models and ten recent requests. The Models view contains the full retained model inventory; Activity lists the latest 100 requests.
 
 ### Where names come from
 
-The terminal does not infer agent roles, task descriptions, service identities, or physical locations from arbitrary source code, prompts, or model output. Being present in a repository does not establish that an agent ran. The terminal reads supported metadata exports, explicit configuration, runtime inventory, and the operating system. Version `0.1.6-preview.4` also indexes supported code/configuration declarations as references; those references do not create running agents or request history. Application logs or exports must supply their existing request identities; the terminal does not generate nicer descriptions to fill gaps.
+The terminal does not infer agent roles, task descriptions, service identities, or physical locations from arbitrary source code, prompts, or model output. Being present in a repository does not establish that an agent ran. The terminal reads supported metadata exports, explicit configuration, runtime inventory, and the operating system. The syntax index records supported code/configuration declarations as references; those references do not create running agents or request history. Application logs or exports must supply their existing request identities; the terminal does not generate nicer descriptions to fill gaps.
 
 | Identity | Source and fallback |
 | --- | --- |
@@ -221,15 +214,58 @@ Torph animates changed totals, table measurements, status labels, and source cou
 
 The environment selector opens `inference-sample`, a fictional enterprise workspace with 40 agents, 140 tasks, 21 models, and 9,600 requests over 24 hours. It includes all five path classifications, traffic bursts, running and failed requests, missing measurements, partial and unavailable sources, three self-hosted runtimes, 27 distinct deployments, 14 shared capacity pools, and an hour of host history. Its six hosts include a workstation and five rack servers across two racks, with eleven accelerators listed. Four hosts have recent readings, one is stale, and one has no measurement. Seven loaded models are placed explicitly across three hosts. Six cloud models have separate `us-1` and `eu-1` example deployments. The capacity sample includes twelve recent readings, one stale reading, and one missing reading. Shared gateway pools connect consumers across reported agent IDs. Names resemble exported configuration keys (for example, `support.review` and `llm-gateway`); some agent/task display names are absent and the UI uses their IDs. These identifiers are fictional, not discovered in the current repository. The sample workstation has no site label; server sites and racks are fictional exporter fields. Baseten, Groq, and Cerebras illustrate serving services distinct from model creators; their sample configuration evidence, deployment modes, and quotas are fictional. Model rates, latency, allocation, and utilization values are illustrative; the sample is for inspecting the interface at scale, not a performance benchmark. Sample data never enters the real snapshot or project files. A command started with `--demo` only exposes a sample workspace; restart without that flag to collect real data.
 
+### Hosted enterprise sample and comparison
+
+The hosted sample at [dash.openmerit.ai](https://dash.openmerit.ai/) has a separate enterprise scenario: 9.6 million fictional requests over 30 days, represented by 9,600 groups of 1,000 requests, with 200 agent IDs, 699 task IDs and 21 models. Tokens and costs are **group totals**. Overview, model/service counts, activity, maps and inspectors use the represented request count. These sample weights never affect live request records. The local `--demo` and sample selector keep the smaller scenario described above.
+
+The comparison at [`dash.openmerit.ai/demo`](https://dash.openmerit.ai/demo) is an **insurance-tech-demo**: 280 fictional workers, 980 task identifiers, and fourteen insurance application workflows. Claim intake, policy extraction, coverage review support, document verification, renewals and correspondence use plausible application declarations. These names were not discovered from a real insurance repository. Both stages preserve the configured service set, workloads, request counts and token volume. Qualified assignments can move between those services; their request paths and service attribution update accordingly. No real claims, underwriting decisions or inference calls are involved.
+
+The comparison is the leading panel, with larger Before/After token costs, a prominent stage switch, and the modeled difference beside its evidence link. Overrides remain visible below the totals; workload decisions expand within the panel. On narrow screens the controls and result stack without hiding the evidence. This visual hierarchy does not change selection, pricing or request totals.
+
+**After OpenMerit** includes a preference selector: default within the existing stack, bias ZDR, bias open weights, no open weights, bias USA models, or bias Chinese models. Every workload searches the full reviewed catalog across the six configured services. Default chooses the lowest modeled token cost among qualifying replacements and retains the baseline if none qualify.
+
+Other preferences first select the lowest-cost qualifying match. If none exists, an already matching baseline is retained. Otherwise, a compatible matching offer may override the benchmark and efficiency criteria. These **Preference overrides** are labeled in the comparison, workload decisions and evidence drawer. The drawer lists every unmet criterion and preserves missing comparisons and known regressions. Overrides are ranked by fewest axes with known regressions, then fewest axes with missing evidence, presence of a verified efficiency improvement, and modeled cost; offer ID breaks ties. This is a disclosed demo fallback, not a measured claim that the selected model is best for the task. Missing evidence never becomes a passing score.
+
+Configured services, compatible output contracts and known token prices remain required. **No open weights** still requires every assigned model to be explicitly closed-weight. Only when no compatible closed offer exists does a workload remain unassigned and the full-plan cost stay unavailable. Other biases fall back to the default qualifying choice or baseline if no compatible preference match exists, with the mismatch visible. Overrides are included in all After views and modeled costs; request and token volumes stay fixed.
+
+The primary criteria require equal or better selected public benchmark scores for tool use, capabilities and reasoning, plus lower modeled token cost or a lower paired latency measurement. The catalog records benchmark suite, metric, score, source and publisher configuration. Missing or non-comparable evidence prevents qualification even when a preference override selects the model. Decision workflows additionally require accuracy and calibration evidence; an override must disclose missing or regressing decision metrics too. Jev is a typed-decision model, not a general text generator; the configured-service and output-contract constraints keep it out of the After plan.
+
+The reviewed catalog contains 26 model/service offers across OpenAI, Anthropic, Google, Groq, Baseten and Mistral; Jev is separately reviewed as an out-of-stack candidate. Model identity is distinct from serving service. Multiple benchmark cohorts remain separate, and every matched metric on each required axis is checked: a passing score cannot hide a matched regression. Model-card scores refer to their published configurations, not measured endpoint performance.
+
+Nine workflows qualify for a change, including four application workers that generate actuarial/reporting SQL and reconciliation/reserve-analysis scripts. These are inference inside the fictional insurance product, not coding-harness activity. Their selected capability screen is code generation: LiveCodeBench v6 in the [GLM-4.7 comparison](https://huggingface.co/zai-org/GLM-4.7), or LiveCodeBench Pro in the [Gemini 3 Flash comparison](https://storage.googleapis.com/deepmind-media/gemini/gemini_3_flash_model_evaluation.pdf). Each comparison also supplies tool-use and GPQA reasoning scores for its candidate and Claude Sonnet 4.5 baseline. Percentage and Elo scores remain separate, and these code-generation results are not substituted for general knowledge or visual-document capability.
+
+Both GLM-4.7 on Baseten and Gemini 3 Flash qualify for those four workloads. The scenario declares fixed fictional profiles: SQL workers average 14K/700 and 18K/900 input/output tokens per request; script workers average 5K/2K and 6K/2.4K, with deterministic ±15% variation. Profiles include billed reasoning output and remain identical across stages and preferences. Published rates favor Gemini for these input-heavy SQL profiles and GLM for the longer script outputs. No preference changes prices, token volumes or recorded benchmark scores. Exceptions to the benchmark and efficiency requirements remain explicit.
+
+The fixed 30-day scenario calculates $344,689.33 Before. Default selects eight model changes and one additional service-only change, with five retained baselines, producing $177,125.39 After (48.6% lower modeled token cost). Under preference overrides the complete After projections are:
+
+| Preference | Modeled token cost | Workloads with an override |
+| --- | ---: | ---: |
+| Default | $177,125.39 | 0 |
+| ZDR | $38,281.42 | 8 |
+| Open weights | $50,147.49 | 7 |
+| No open weights | $171,146.32 | 3 |
+| USA | $162,436.44 | 2 |
+| Chinese | $40,953.91 | 9 |
+
+Lower cost in a plan containing overrides is not a quality-preserving improvement claim. All six plans now cover fourteen workloads and 9.6 million requests. Closed weights assigns GPT-4.1 mini to claim tagging, fraud review support and broker correspondence with missing-comparison exceptions; claim tagging also loses the verified efficiency improvement and costs more than its baseline. The other eleven closed assignments need no override. Equivalent model mixes, if any, are disclosed instead of changed artificially.
+
+Public benchmark scores are publisher-reported point estimates in the linked configurations, **not an OpenMerit insurance-task evaluation**, statistical proof of parity, or a claim of superiority across every benchmark. For example, GPT-4.1's ComplexFuncBench score is below GPT-4o's even though it passes the selected τ-bench retail, MMLU and AIME 2024 screen. The DeepSeek comparison uses AutomationBench, Agent’s Last Exam and GPQA Diamond from its published instruct-model comparison; it is not a claim about every reasoning benchmark. Application quality and latency remain untested. The dated catalog is not exhaustive: retained means no qualifying alternative in the reviewed evidence.
+
+ZDR means an endpoint/account retention policy, not a property of weights. This scenario only treats Baseten's documented synchronous default policy as confirmed; other providers' conditional ZDR eligibility is not assumed enabled. Router, gateway and application logging require separate checks. USA and Chinese refer to the model developer's origin, not hosting location or data residency. Open weights does not imply a particular license or privacy policy.
+
+Both sides calculate token cost from published standard USD input/output rates checked on 10 October 2026. **Workload decisions** shows each Before/After model, serving service, qualifying-alternative count, per-workload Before/After token cost for the selected window, and preference match. Choices that differ from Default appear first, followed by workloads with more qualifying alternatives. **Swaps & evidence** exposes selected and rejected candidates, rates, public benchmark comparisons and methodology. Inputs are below 33K tokens per represented request; the scenario assumes uncached text, no batch discount, and no regional premium, tool charge, gateway fee, tax or infrastructure cost. Token counts, including billed reasoning output where applicable, are held fixed for comparison; actual tokenization, reasoning budgets and generated output can differ. The broader sample leaves costs unknown when no verified token price is included. It never treats local infrastructure as free.
+
+The hosted browser builds deterministic fictional records from a small same-origin seed response. `/api/sample` and `/api/demo?stage=before|after&preference=default|zdr|open|closed|usa|china` expose complete sample snapshots; `/api/sample-seed` supplies the recipe version and timestamp. The comparison metadata includes plan feasibility, model-versus-service changes, preference matches, `equivalentPreferences` for identical model/service assignments, and window totals. An incomplete After snapshot keeps `simulation.feasible: false`, contains request/model/deployment rows only for qualifying assignments, and has no projected complete-plan After cost. Its unresolved assignments and full Before totals remain in comparison metadata. Each window reports both baseline `requests` and `qualifiedRequests`; individual qualifying records and workload totals retain their After costs. Unresolved assignments retain `after: null`, without substituting their open-weight baseline. Unknown preferences fall back to default. Stage and preference survive tab navigation and are shareable in the page URL. No collector, local files, credentials or inference provider is contacted. The comparison holds its dataset fixed while you explore; reload to start a fresh scenario. The hosted scenarios are separate from local collection; exploring them does not change your installed package or application.
+
 ### Optional map preview
 
-The test preview includes an opt-in React Flow + ELK map experiment. Add `?map=flow` before the page hash in your local terminal URL (for example, `http://127.0.0.1:4310/?map=flow#overview`). To try fictional enterprise data instead, use `?sample=1&map=flow#overview`. The existing map remains the default; **Original map** removes the preview flag immediately. No data migration or server restart is needed to switch views.
+The terminal includes an opt-in React Flow + ELK map experiment. Add `?map=flow` before the page hash in your local terminal URL (for example, `http://127.0.0.1:4310/?map=flow#overview`). To try fictional enterprise data instead, use `?sample=1&map=flow#overview`. The existing map remains the default; **Original map** removes the preview flag immediately. No data migration or server restart is needed to switch views.
 
 The preview keeps the existing card dimensions and uses ELK to arrange all collapsed groups. Layout is recalculated only when the set of nodes or relationships changes, not when request counts refresh. The graph scrolls at a fixed scale rather than automatically zooming to fit.
 
 **Browse all** opens a searchable inventory of every workload, entry connection, and model/service pair in the selected scope. Selecting a stack filters that inventory to its exact members; the visible result count always shows the filter coverage. Selecting a member spotlights its actual recorded paths and brings a matching member to the front of each related stack without expanding the diagram. This does not change the total request counts. **Details** opens the existing inspector; **Clear spotlight** or Escape clears the selection. The browser sits beside the map on wide screens and below it on narrower screens, with independently scrolling lists. Keyboard users can navigate its tabs with arrows, Home, and End.
 
-This is a presentation experiment within the preview, not a new data source. Its separately bundled scripts and styles are loaded from the local server only when the preview is selected. It makes no inference calls and adds no collection permissions. There is additional download and initial layout cost; very large and densely connected environments still need user evaluation. The original map remains available for comparison and rollback.
+This is an optional presentation experiment, not a new data source. Its separately bundled scripts and styles are loaded from the local server only when the preview is selected. It makes no inference calls and adds no collection permissions. There is additional download and initial layout cost; very large and densely connected environments still need user evaluation. The original map remains available for comparison and rollback.
 
 ## Collection sources
 
@@ -337,7 +373,7 @@ Configuration, inventory, and activity evidence are distinct. A configured servi
 
 ### Inference services and model creators
 
-The terminal recognizes configuration evidence for Baseten, Groq, and Cerebras alongside the existing services below. Expanded coverage in `0.1.6-preview.3` is documented under **Additional inference endpoints**. The configuration reader described here uses supported literal declarations in the selected project’s `.env` and `.env.local` files. Version `0.1.6-preview.4` additionally indexes supported code and nested configuration as described under Repository indexing. Global shell variables do not establish project ownership. Shell histories and credential stores are not scanned; source indexing follows the bounded syntax rules above. A configured service does not establish an active connection, model inventory, usage, available capacity, or a hardware allocation.
+The terminal recognizes configuration evidence for Baseten, Groq, and Cerebras alongside the existing services below. The full set is documented under **Additional inference endpoints**. The configuration reader described here uses supported literal declarations in the selected project’s `.env` and `.env.local` files. The repository index additionally reads supported code and nested configuration as described under Repository indexing. Global shell variables do not establish project ownership. Shell histories and credential stores are not scanned; source indexing follows the bounded syntax rules above. A configured service does not establish an active connection, model inventory, usage, available capacity, or a hardware allocation.
 
 | Service | Credential-variable names | Optional endpoint-variable names |
 | --- | --- | --- |
@@ -359,7 +395,7 @@ For example, an export may report `model: "gpt-oss-120b"`, `modelCreator: "OpenA
 
 ### Additional inference endpoints
 
-Version `0.1.6-preview.3` recognizes 39 cloud/API/router/gateway service identities, including the two Hugging Face products described below. Earlier previews do not include this expansion. Detection still reads only supported literal declarations in the selected project's `.env` and `.env.local`; the file's presence does not prove the application loaded it. No installation, API request, account access, or additional permission is needed to recognize these declarations.
+The provider/runtime discovery reader recognizes 39 cloud/API/router/gateway service identities, including the two Hugging Face products described below. It reads supported literal declarations in the selected project's `.env` and `.env.local`; the file's presence does not prove the application loaded it. The separate repository index can identify additional nested declarations and conservative source references without probing their URLs. No installation, API request, account access, or additional permission is needed to recognize these declarations.
 
 Any recognized endpoint can be supplied through `OPENAI_BASE_URL` or `OPENAI_API_BASE`, even when the application uses an OpenAI-compatible SDK for another service. The following additional variable names are accepted. Endpoint aliases are supported application conventions, not a claim that every vendor SDK reads them automatically. A nonempty credential declaration is recorded only as **Configured**; its value is discarded. An endpoint override suppresses that group's credential-based vendor fallback, including when the URL is unknown or malformed.
 
@@ -419,7 +455,7 @@ Configured services add no historical requests, model catalog, agent names, toke
 
 ### Hugging Face endpoints
 
-Version `0.1.6-preview.3` recognizes literal endpoint declarations in the selected project's `.env` and `.env.local`. Earlier previews do not include this support.
+The provider/runtime reader recognizes literal endpoint declarations in the selected project's `.env` and `.env.local`.
 
 | Endpoint host | Service in the terminal | Type |
 | --- | --- | --- |
@@ -439,7 +475,7 @@ These declarations identify two configured services; they do not establish that
 
 The terminal makes no remote calls, reads no Hugging Face token cache, and fetches no account history or model catalog. Models, requests, tokens, costs, agents, and deployment links still require recorded metadata or explicit inventory. Supported request logs can use `provider: "Hugging Face Endpoints"` or `route: "Hugging Face Router"` to identify the corresponding service. A generic `Hugging Face` name remains unclassified; the terminal does not choose a product or serving provider for it. The two products share a locally bundled official logo but remain separate service identities.
 
-The recognized formats follow Hugging Face's [environment-variable documentation](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables), [dedicated endpoint guide](https://huggingface.co/docs/huggingface_hub/guides/inference_endpoints), and [inference router documentation](https://huggingface.co/docs/inference-providers/index). In `0.1.6-preview.3`, URLs in source code and arbitrary environment-variable names are outside the declaration scan. Version `0.1.6-preview.4` can index supported literal source declarations and custom dotenv names referenced by supported inference fields; dynamic expressions and cross-file imports remain unresolved.
+The recognized formats follow Hugging Face's [environment-variable documentation](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables), [dedicated endpoint guide](https://huggingface.co/docs/huggingface_hub/guides/inference_endpoints), and [inference router documentation](https://huggingface.co/docs/inference-providers/index). The repository index also reads supported literal source declarations and custom dotenv names referenced by supported inference fields; dynamic expressions and cross-file imports remain unresolved.
 
 ### Deployments and shared capacity
 
@@ -582,7 +618,7 @@ There are no favicon lookups, third-party image services, or remote asset reques
 
 Collectors run once per interval with no overlapping collection cycles. Each runtime request has a timeout and a response-size bound. Browser refreshes read the cached snapshot, so additional tabs do not multiply runtime polling. The interface shows a delayed-collection notice when its snapshot is older than three intervals. A successful source check does not mean that source has recent inference activity.
 
-File readers handle partial lines, split UTF-8 writes, and ordinary file rotation. Activity and audit file reads are bounded to 4 MiB, and each reader retains up to 10,000 unique records. Discovered project logs have the stricter per-file and aggregate limits described above. A large initial file imports its tail. Truncation and malformed records are disclosed in source details. From `0.1.6-preview.1`, valid retained records have no age cutoff; future timestamps remain excluded. The released `0.1.6-preview.0` still limits results to the most recent 24 hours. An unavailable or removed file is not kept looking connected.
+File readers handle partial lines, split UTF-8 writes, and ordinary file rotation. Activity and audit file reads are bounded to 4 MiB, and each reader retains up to 10,000 unique records. Discovered project logs have the stricter per-file and aggregate limits described above. A large initial file imports its tail. Truncation and malformed records are disclosed in source details. Valid retained records have no age cutoff; future timestamps remain excluded. An unavailable or removed file is not kept looking connected.
 
 Inventory bounds remain in place: eight explicit runtimes, eight discovered endpoint candidates after deduplication, 100 declared inventory entries, 200 deployments, 100 declared/exported pools, 16 pool references per inventory/deployment entry, 100 models per endpoint catalog or native Ollama list, 64 exported hosts, 16 accelerators and 100 loaded-model entries per host. Reaching a count bound with additional entries sets **Partial** coverage and reports omitted counts in Sources; the dashboard displays **Inventory is incomplete**. Retained counts mean entries in view, not complete environment totals. Oversized files or responses remain unavailable rather than being silently accepted.
 
@@ -604,7 +640,7 @@ From the checkout:
 node packages/terminal/src/cli.mjs dash /path/to/project --json
 ```
 
-While the server is running, `GET /api/snapshot` returns the cached JSON object. Schema version 1 contains `mode`, `observedAt`, `interval`, `project`, `models`, `inventory`, `records`, `taskSummaries`, `sources`, `runtimeStats`, `deployments`, `capacityPools`, `hosts`, `machine`, and `discovery`. The optional `discovery` object reports whether discovery is enabled, its last check time, the 60-second configuration discovery interval, candidate count, contributing request sources, and reconciliation coverage. Request records can include `sourceIds`, a supported log `evidenceFormat`, and `conflictingFields`. Colliding local IDs from demonstrably distinct records receive a stable namespaced ID while retaining `originalId`. The host, deployment, and pool fields extend the preview version-1 snapshot; older snapshots without them show unknown coverage. `machine` remains the current host for compatibility; `hosts` is the retained project-linked hardware inventory. Sources affected by inventory limits expose `truncated: true` and a `limits` array of `{ collection, limit, omitted }`; their state is `partial`. The UI shows an inventory-coverage notice and per-source omitted counts. Task summaries never establish request coverage. Unknown status, cost basis, and currency are represented by `"unknown"`. Fields containing unsupported measurements use `null` or are absent. `GET /api/sample` returns only fictional data. `GET /api/info` describes the process's mode and interval.
+While the server is running, `GET /api/snapshot` returns the cached JSON object. Schema version 1 contains `mode`, `observedAt`, `interval`, `project`, `models`, `inventory`, `records`, `taskSummaries`, `sources`, `runtimeStats`, `deployments`, `capacityPools`, `hosts`, `machine`, and `discovery`. The optional `discovery` object reports whether discovery is enabled, its last check time, the 60-second configuration discovery interval, candidate count, contributing request sources, and reconciliation coverage. Request records can include `sourceIds`, a supported log `evidenceFormat`, and `conflictingFields`. Colliding local IDs from demonstrably distinct records receive a stable namespaced ID while retaining `originalId`. The host, deployment, and pool fields are optional version-1 snapshot fields; snapshots without them show unknown coverage. `machine` remains the current host for compatibility; `hosts` is the retained project-linked hardware inventory. Sources affected by inventory limits expose `truncated: true` and a `limits` array of `{ collection, limit, omitted }`; their state is `partial`. The UI shows an inventory-coverage notice and per-source omitted counts. Task summaries never establish request coverage. Unknown status, cost basis, and currency are represented by `"unknown"`. Fields containing unsupported measurements use `null` or are absent. `GET /api/sample` returns only fictional data. `GET /api/info` describes the process's mode and interval.
 
 The server binds to `127.0.0.1`, accepts read-only GET requests, checks Host and Origin, rejects cross-site browser requests, and serves only its bundled assets and explicit API routes. It provides no general file-serving endpoint. Local processes and other users with access to this machine may still reach the loopback service; this is not a multi-user authentication boundary.
 

package/docs/interactive-demo.md

@@ -0,0 +1,118 @@
+# Interactive delivery-resolution experiment
+
+This guide describes the separately deployed website experiment, build `guided-20261002-v9`. Its application source is preserved on `codex/main-before-terminal-2026-10-12`; main’s `site/demo/` remains the historical recorded player. The hosted experiment and the terminal’s fictional [before/after sample](https://dash.openmerit.ai/demo) are separate products.
+
+Revision `guided-20261002-v9` of the [interactive demo](/demo/) presents five business scenarios: a delivery policy update, privacy update, an in-house deployment preference, peak-time response needs and an incomplete customer request. The agent has one prepared goal: resolve a damaged delivery before a customer's event. Four sequential model calls read the store policy, assess the affected order and delivery options, choose a resolution, and draft a customer reply. The fourth call is a real model response. No refund, shipment or customer message is executed.
+
+## Run one request
+
+Select **Run request**. John is a member with a $240 order. One $90 desk lamp arrived damaged two days ago; the other items are fine. He needs a replacement before an event in three days. Stock is available, but standard delivery takes five days. Express delivery takes two days and costs the store $12.
+
+The agent must use the affected item's price rather than the order total, apply damage coverage despite the final-sale label, check stock and the delivery deadline, apply the member express allowance, and respect the human-approval limit. Express coverage starts off. John’s standard delivery would miss his event, so the initial correct offer is a $90 item-only refund. Enabling express coverage changes the correct offer to a free two-day replacement, with the store covering the $12 shipping cost. Other prepared cases require an item-only refund, review or a decline.
+
+The starting assignment is Gemini 3.1 Flash Lite through OpenRouter for all four tasks. This is a fixed starting example, not an OpenMerit-selected or approved setup. Each output is checked. It runs once per session; its four calls and cost count toward the session allowance. A passing opening request does not consume a comparison cycle. Since revision v8, a failed business check, response-time check or provider call automatically starts the existing full comparison under the same rules. That recovery consumes one fresh comparison and uses the existing call/spend allowance, preserving verification resources.
+
+The task blocks light up blue as calls run, green for passed checks, and red for failures. A completed passing request shows its proposed resolution and the actual fourth model's **Draft reply · not sent**. A failed opening request remains in the evidence. Since revision v8 the demo immediately compares alternatives and keeps scenario controls locked; it does not ask the visitor to diagnose the failure or relax a business rule. A qualifying result is a proposal, still requiring **Use this setup** and fresh verification. If the bounded search fails, no setup is applied. Drafts and request metrics are hidden during a new run, after a rule change, or when the draft is not valid for the displayed session state.
+
+## Decision instructions and recovery
+
+The resolution prompt separates damage claims from ordinary returns. Damage claims explicitly ignore the final-sale label and ordinary membership window, then apply damage expiry, human approval, stock and on-time shipping in order. The model must return the reason belonging to the selected branch. The application still passes real upstream output to later calls and compares the result against an independent expected outcome; no answer repair or relaxed grading is introduced.
+
+This addresses the observed v7 error where a covered damaged item was declined as final sale, or refunded with an inconsistent `final_sale` reason. Live decision-only canaries test eight cases on each of the starting OpenRouter model and direct OpenAI model, including an undamaged final-sale counterexample; these are separate from whole-agent qualification. Passing one request or one task is not proof that a complete setup qualifies. In the subsequent local v8 browser validation, a mixed setup with three hosted open-weight assignments and a direct OpenAI resolution assignment passed all four comparison cases, both fresh verification cases and John’s rerun. Earlier failed candidates remained in the evidence. This is a small prepared-suite result, not a guarantee for other requests or future runs.
+
+## Explore business scenarios
+
+This revision has five yes/no scenarios under **What changed?**, all off initially. Run the opening request, then select a scenario. Each selection immediately starts reassessment; it does not directly choose a model.
+
+| Scenario | What changes in the experiment |
+| --- | --- |
+| **We now cover urgent replacements** — Policy update | Members receive express delivery costing the store at most $15 when it can meet their deadline and standard shipping cannot. The policy extraction, expected resolution and reply checks change. John's confirmed damage claim changes from a $90 item-only refund to a free two-day replacement. |
+| **Customer data needs tighter protection** — Privacy update | The fictional company allows customer-data tasks only through its approved endpoint, direct OpenAI. Public-policy calls can still use OpenRouter. This is endpoint eligibility, not a claim that a provider guarantees privacy; hosted open weights are subject to the same restriction. The business-facing card does not name or prescribe a model. |
+| **We want to run this ourselves later** — In-house option | Prefer compatible reviewed open-weight assignments that pass the complete suite, then measured cost or response time. These calls still use OpenRouter in the cloud. This does not measure local deployment feasibility. |
+| **The support queue is filling up** — Peak time | Prioritize the lowest measured worst-case response time, with a five-second limit per complete test instead of twenty seconds. This represents the business need for faster answers; the demo does not generate a traffic burst, measure throughput or establish a production capacity guarantee. |
+| **The customer gives us less to go on** — Incomplete request | John's message no longer establishes whether the lamp is damaged, opened or unused. The facts call must return `condition: unknown`, the resolution must be `clarify / missing_context`, and the reply must ask one relevant question without offering a refund or replacement. The displayed customer message changes too. |
+
+The incomplete-request scenario changes both comparison and fresh verification inputs. Other cases still contain explicit damage, so a setup that always asks a question cannot pass. Missing condition takes precedence over a remedy or approval decision: the agent must establish what happened first. Enabling a delivery benefit cannot authorize a replacement when condition is unknown. Inputs supplied to the models never include the hidden expected condition or repaired outputs. Every proposed four-call setup must pass the entire suite; a stronger model is not assigned by name.
+
+There is no scenario-to-model mapping. OpenMerit selects from setups that satisfy endpoint eligibility, business checks and the response limit using measured tradeoffs. A scenario can change the correct customer outcome without requiring a different model. Models that still pass and remain the best measured choice stay in place; no swap is staged for the demo.
+
+Human approval above $150 per affected item remains a fixed store rule in the UI, rather than a scenario toggle. Exactly $150 does not trigger it, and the whole order total is not substituted for the affected item's price. Damage claims retain a 14-day inclusive window and override final-sale labels. A missing receipt requires review once condition is known. Prefer an in-stock replacement that meets the deadline: standard shipping first, then covered member express shipping when enabled. Otherwise offer an item-only refund. There are no sliders or model-picking controls. The backend retains bounded inputs for prepared experiments, including older approval/model restrictions. Older builds require a fresh session.
+
+All five switches and **Reset scenarios** are disabled immediately while a request starts, comparisons run, approval is verified, or John's request is rerun. They visibly dim and explain when they unlock. A lost connection keeps them locked until the session status is recovered. Approval stays near the result, above the scenarios.
+
+Changing delivery policy or customer context requires fresh comparison evidence. Measurements can be reused only for exactly the same business policy and prepared cases in the same build. Privacy changes recheck endpoint eligibility; the deployment preference and response target are reevaluated against applicable measurements. Reuse is labelled in the explanation and trace. The one-case opening request is never substituted for four-case comparison evidence. Changing a scenario pauses the previous setup until approval and verification.
+
+Changed task assignments show the previous and proposed model. **Why this choice?** reports whether the earlier setup failed a task/business check, missed the deadline, or remained a qualifying but less preferred measured tradeoff. An individual step's passing output is not proof that the entire setup qualifies. A comparison with no fitting initial setup automatically starts a broader search under unchanged business rules.
+
+With the in-house preference off, the first comparison tests four complete configurations: OpenRouter AutoRouter, Gemini through OpenRouter, direct OpenAI models, and a mixed assignment. Later policy comparisons include a previously activated expanded assignment in place of the original Gemini starting setup. Other prepared assignments, cached applicable results, new task combinations and catalogue models are available to broader search. Keeping the initial batch to four leaves room for successive business-rule changes within the existing session limits; every fresh batch also reserves verification resources. AutoRouter is an inference option, not an opponent. Its selection is restricted to the disclosed shortlist; served models are retained in the trace and downloaded evidence. No configuration is forced to win.
+
+## Hosted open-weight scenario
+
+**We want to run this ourselves later** uses a reviewed shortlist: [Qwen 3.5 9B](https://huggingface.co/Qwen/Qwen3.5-9B), [Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), [Gemma 3 12B](https://huggingface.co/google/gemma-3-12b-it), and [Mistral Small 3.2 24B](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506). This is a curated candidate set, not a popularity ranking or a hardware recommendation. The live OpenRouter catalogue must still list each candidate with compatible structured outputs, context size and prices. A release with an announced catalogue expiration is skipped. Unreviewed models are not inferred to have open weights from their vendor name.
+
+Task cards use the shorter **Open weights** badge. A note below the scenarios states that all calls run in the cloud. The read-only inspector, model-call trace and downloaded evidence retain **Open weights · Hosted on OpenRouter** and the actual provider. The demo measures hosted API charges and wall-clock latency. It does not measure local hardware, memory, throughput, deployment feasibility or local data privacy. Hosting on OpenRouter still sends inputs to cloud providers, and endpoint restrictions continue to apply to every private step.
+
+Enabling the scenario tests compatible open-weight assignments first, using permitted direct models on any tasks whose endpoint restrictions exclude OpenRouter. All four business cases and the response-time limit must still pass. A derived `preference_fit` score records the fraction of the four assignments using reviewed open-weight releases; it is metadata-based preference evidence, not a quality or privacy score. The coordinator includes it in the verified frontier. Selection prefers the highest fraction, then the existing cost or live-chat timing priority. A more expensive qualifying open-weight setup can therefore win without inventing a saving.
+
+When the initial candidates fail, broader search can retain successful open-weight tasks and test another model on the weakest task, then use the existing bounded fallback search. Every combination faces the complete suite. A mixed setup or a setup with no open-weight assignments may be proposed when it is the qualifying result found within the allowance; the explanation reports the actual count. If catalogue discovery fails, other permitted options remain available. No switch forces a specific model to pass or relaxes a business check. Turning the preference off restores the usual cost/timing selection and can reuse evidence for unchanged business cases. Approval and fresh verification remain required.
+
+The pool stays bounded at eight catalogue candidates, including compatible reviewed open-weight releases and direct OpenAI candidates. Existing call, spend and verification reservations are unchanged. In the 2 October v7 local live check, all four reviewed families were called successfully, but none of the tested complete or mixed setups passed every default-policy case. The best mixed setups passed three of four cases. No setup was approved; successful dispatch is not evidence of a qualifying agent.
+
+## Automatic broader search
+
+A comparison with no qualifying initial configuration automatically enters **Searching other models**. Search can also investigate a better open-weight fit when an existing qualifying setup uses fewer open-weight tasks than the endpoint rules permit; the UI distinguishes this preference search from a failed business check. The response-time limit, express policy, approval requirement, business fixtures and grading remain unchanged during the search. No additional switch or approval is needed for evaluation within the already authorized session allowance.
+
+The harness reads OpenRouter's model catalogue and the configured OpenAI account's model list. OpenRouter candidates must support this text/structured-output request format, have a sufficient context window and stay within the existing $1 input/$5 output per million token dispatch ceiling with no request fee. Catalogue latency ordering is a research lead, not a measurement of this workflow. Direct OpenAI discovery uses account availability intersected with adapters that have checked request semantics and token rates: GPT-5.4 nano and GPT-4o mini supplement the original GPT-4.1 options. Other OpenAI models are not yet supported by this demo. Catalogue failures are recorded independently; one unavailable source does not hide the other source's candidates.
+
+The bounded pool contains up to eight additional compatible models. Task-level checks and timings can suggest a new combination of previously tried models; untested prepared assignments and newly discovered models can also be tried. Every proposed combination is run through all four comparison cases with real sequential calls. It cannot qualify on estimated latency, catalogue rankings or successful individual steps alone. Each round uses prior task results to suggest a combination, then considers remaining prepared assignments and compatible catalogue models. No assignment is automatically promoted just because it is new.
+
+The search stops at the first qualifying additional setup (or first improvement in open-weight coverage during a preference search), after at most six extra setups per comparison, when its compatible options are tested, or when session/call/spend limits prevent another complete test. Twelve calls and a conservative spend allowance are preserved for approval verification and the original request rerun. The comparison and its expansion count as one fresh comparison. Failed configurations and catalogue sources remain in the evidence download. A failed fresh verification or rerun also disqualifies that assignment on subsequent comparisons wherever the recorded failure still violates the current requirements; earlier passing comparison evidence cannot silently override it. Unchanged workload results can be reused on subsequent rule changes; their current restrictions and time limit are always rechecked.
+
+While search runs, controls stay locked and the read-only IDE and trace follow the new model assignments. If no setup qualifies, the UI identifies the stopping reason and, when available, the fastest correct setup's slowest measured case against the deadline. Earlier task blocks are labelled **Earlier scenario**, not **Passed**. Search exhaustion does not prove that no model outside this bounded search could qualify. The demo never silently relaxes the requirement or applies a failed configuration.
+
+## Read and approve the choice
+
+The compact page shows one customer request and four task blocks. During a comparison, a caption identifies the prepared case currently being tested, so the blocks are not presented as calls for John alone. Amber marks a proposed changed assignment. **Why this choice?** contains the explanation, measurements, other candidate results and session usage. Labels accompany every semantic color.
+
+The UI pairs the demo on the left with a read-only **Behind the scenes** view on the right; the two panels are centered together on desktop and stack on smaller screens. Taller workflows scroll normally. The inspector uses a muted gray surface, softer borders and restrained syntax/status colors so the interactive panel remains the focus. Text, live highlights and errors remain readable. One code view shows the current constraints, prepared steps, model assignments and checks. The active call is highlighted and brought into view automatically. Changed rules and proposed model assignments are marked in amber. These are views of configuration and session data, not editable source files. The run trace stays visible underneath as a compact monospace console, with timestamps, source labels and colored RUN/OK/WARN/ERROR/INFO markers. The original event messages remain intact. All actions, including **Download run evidence** under **Why this choice?**, remain on the left. The right side has no tabs, buttons or editing controls and starts no model calls.
+
+The view distinguishes the starting assignment, a proposal awaiting approval, verification in progress, and a verified setup. Earlier rules, stopped sessions and lost connections are labelled explicitly. A comparison can run two setups concurrently; the code follows the latest reported task while the trace retains events from both. Returned output is shown only when the displayed request has completed and the rules still match. Scrolling back preserves the reader’s position; a new action on the left resumes automatic code following. The trace follows new events while the reader is at its bottom.
+
+Each configuration faces four synthetic cases: John's event deadline, an out-of-stock item requiring a refund, a high-value item that tests the approval switch, and a missing receipt with instructions to falsely claim the entire order was refunded. Two routes may run concurrently; calls and cases within each route are sequential. Policy, extracted facts, resolution and reply commitment fields are checked exactly against independent expected values under the current rules. The draft also receives bounded wording checks for required outcome wording, amounts and common false action claims. These checks do not constitute a full semantic evaluation of its prose.
+
+An upstream mistake is retained and passed to subsequent calls; it is not repaired with an expected answer. The application supplies arithmetic comparisons calculated from the actual extracted policy and facts, including delivery timing and express coverage. The third model applies rule precedence and chooses the resolution; the fourth model receives that actual resolution and writes the draft. These helpers and inventory/delivery snapshots are prepared application code and fictional inputs, not live commerce integrations.
+
+The prepared harness calculates the comparison; the actual OpenMerit coordinator validates the typed evidence and frontier and records its lifecycle. Model and harness activity are labelled separately from OpenMerit events. This demo does not run Pi as an autonomous investigator, invent integrations, or exhaustively search all model assignments. Its broader search uses the bounded strategy described above.
+
+The frontier compares task success, observed maximum whole-request time and API cost on the same cases. Local v7 also includes derived open-weight preference fit when that scenario is enabled. After the enabled open-weight preference is applied, live chat recommends the lowest observed maximum time; otherwise it recommends the lowest measured API cost within the time limit. Ties use the other metric, then a stable configuration ID. The core validates the frontier and the evidence-backed preference; the prepared harness calculates the recommendation. Reported average latency is separate. Business rules can increase cost or latency, and no saving is guaranteed. **Use this setup** records the exact approval, applies the selected configuration, and makes fresh calls on two separate boundary cases: exactly day 14, exactly the review threshold and exactly $15 express cost; and expired damage coverage on day 15. Every completed case makes four calls. The guided flow then reruns John’s original damaged-delivery request with fresh calls. It does not display a verification fixture as John’s request. The core’s reference is a measured candidate within the comparison; the separate one-case opening example is not treated as that four-case baseline. If the comparison reference remains best, the core records its use and fresh checks without claiming a core model-swap transition. Failed verification stops the application; it does not restore a setup that may violate new requirements.
+
+**Run John’s request again**, under the explanation, reruns the same known fictional case with fresh model calls. It is not an unseen validation set. Changing a requirement disables this action until reassessment. A failed live example pauses further work.
+
+## Measurements and data
+
+Times are observed wall-clock durations, including all four sequential calls and small harness overhead. The suite is too small to estimate production reliability or latency distributions. The requirement uses the observed maximum, not a promised service level.
+
+OpenRouter cost is the provider-reported charge. Direct OpenAI cost is an estimate calculated from actual input, cached-input, and output token counts using checked published rates for each supported direct model (GPT-4.1 mini/nano, GPT-4o mini and GPT-5.4 nano). It is not an invoice. The cost and latency cards beneath the result cover John’s displayed request, labelled as a comparison case when proposed. Values stay neutral because a rule change may increase or decrease either metric. They appear after a completed measurement and are hidden while a run is in progress or the rules have changed. Expanded comparison measurements report the four-case average and maximum separately. Changes in John’s request cost and time are shown under the explanation; different business rules can increase or decrease either. The session ledger also includes the opening request, other candidates, verification and live examples. Container infrastructure and engineering are excluded. Missing cost evidence is not treated as zero.
+
+All requests are constructed from fictional fixtures. The browser never receives API keys or container authorization tokens. Model output, synthetic inputs, measurements, and an event trace are available through **Download run**. The full experiment and core audit artifacts are also written inside the temporary container. The demo is cloud-only; it does not claim local inference or measure local hardware.
+
+## Session limits
+
+A session lasts up to 30 minutes and permits at most three fresh comparisons, 12 comparison/review cycles, 240 model calls, and $2 of conservative inference reservations. There are at most two concurrent sessions and 12 admitted sessions per UTC day shared across the site. There is no separate three-session limit for people on the same network. A session that closes or expires without reserving any model calls returns its admission slot; this also applies when a container fails to start or starts with an outdated runtime. Startup attempts remain limited to 36 per UTC day. Once a model call is reserved, that session counts toward the daily allowance even if the call fails. Reservations are retained conservatively. Temporary API keys and service availability can make live inference unavailable.
+
+If the shared allowance is exhausted, new sessions open at 00:00 UTC (4 a.m. Dubai). Existing unexpired sessions can continue within their own limits.
+
+If the connection drops while starting, the page checks whether a session was created. If none exists, **Run request** becomes available again; no model call is implied by a startup or reconnecting message. Expired sessions and sessions from an earlier demo version offer **Start a new session**. Completed evidence belongs to its original session.
+
+**End session** closes admission and destroys the container. A session cannot resume its live work after a process restart. It keeps its stopped status rather than presenting stale approval as active. These limits and the small prepared suite make this an experiment, not production readiness evidence. The earlier recorded demo and its guides remain archived.
+
+## Provider references
+
+- [OpenRouter AutoRouter](https://openrouter.ai/docs/guides/routing/routers/auto-router)
+- [OpenAI structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
+- [GPT-4.1 mini rates](https://developers.openai.com/api/docs/models/gpt-4.1-mini)
+- [GPT-4.1 nano rates](https://developers.openai.com/api/docs/models/gpt-4.1-nano)
+- [OpenRouter model catalogue](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties)
+- [OpenAI available-model list](https://developers.openai.com/api/reference/resources/models/methods/list)
+- [GPT-5.4 nano request support and rates](https://developers.openai.com/api/docs/models/gpt-5.4-nano)
+- [GPT-4o mini request support and rates](https://developers.openai.com/api/docs/models/gpt-4o-mini)

package/docs/lifecycle.md

@@ -3,9 +3,9 @@
 ## Stages
 
 1. **Unconfigured** — no confirmed profile exists.
-2. **Establishing evidence** — Pi confirms the exact incumbent application model, infers a task-specific metric and sampling plan, explains that plan and estimated cost for user confirmation, then creates and verifies evals and observability. In the unreleased source checkout, setup can also record sourced preliminary price and reputation leads; published `openmerit@0.1.5` does not yet do this.
+2. **Establishing evidence** — Pi confirms the exact incumbent application model, infers a task-specific metric and sampling plan, explains that plan and estimated cost for user confirmation, then creates and verifies evals and observability. Setup can also record sourced preliminary price and reputation leads.
 3. **Collecting baseline** — normal product work accumulates evidence.
-4. **Baseline ready** — in the unreleased source checkout, OpenMerit core checks stored application metric windows against every required objective and its sample threshold. Pi still authors graders and collectors and reports the measurements.
+4. **Baseline ready** — OpenMerit core checks stored application metric windows against every required objective and its sample threshold. Pi still authors graders and collectors and reports the measurements.
 5. **Discovering candidates** — Pi researches credible candidates within budget, refreshing any preliminary setup leads and considering new ones. This stage still follows baseline sufficiency.
 6. **Running challenger trials** — Pi runs the frozen incumbent and challenger set over the same cases under one recorded experiment manifest.
 7. **Calculating frontier** — Pi calculates and explains the Pareto frontier.
@@ -17,13 +17,13 @@
 
 ## Event-driven nudges
 
-OpenMerit remains quiet when no action is justified. On first interactive entry to an unconfigured project, the Pi adapter performs a bounded source preflight. The unreleased source checkout repeats detection after a settled build turn, allowing Pi to build an application in an empty directory before OpenMerit automatically establishes the profile. Later nudges become due when durable application-target signals satisfy the configured task-count or elapsed-time cadence, a metric crosses a regression threshold, the model catalogue changes under an enabled policy, a post-swap window completes, an intent enables the next stage, rollback policy covers a regression, or the user invokes a command.
+OpenMerit remains quiet when no action is justified. On first interactive entry to an unconfigured project, the Pi adapter performs a bounded source preflight. OpenMerit repeats detection after a settled build turn, allowing Pi to build an application in an empty directory before OpenMerit automatically establishes the profile. Later nudges become due when durable application-target signals satisfy the configured task-count or elapsed-time cadence, a metric crosses a regression threshold, the model catalogue changes under an enabled policy, a post-swap window completes, an intent enables the next stage, rollback policy covers a regression, or the user invokes a command.
 
 Every signal has an idempotency key. A due intent is persisted before dispatch so duplicate webhooks, restarts, and retrying adapters do not double-count tasks or silently lose pending work.
 
 ## Evaluation budget
 
-Candidate work can cost money. The contract passes the confirmed currency, spend, candidate count, runs per candidate, and expiration constraints to Pi. The harness is responsible for staying within those limits; OpenMerit does not independently meter spending or cut off provider requests. See [budgets and permissions](budgets.md).
+Candidate work can cost money. The contract passes the confirmed currency, spend, candidate count, runs per candidate, and expiration constraints to Pi. OpenMerit tracks reported challenger spend, rejects results over the allowance, and blocks another challenger intent when it is exhausted. The harness must constrain individual provider requests; OpenMerit cannot cancel a request already in flight or independently meter provider billing. See [budgets and permissions](budgets.md).
 
 ## Supervised graduation