openmerit 0.1.7-preview.3 → 0.1.7-preview.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +1 -1
- package/docs/commands.md +1 -1
- package/docs/scout.md +77 -35
- package/docs/security.md +2 -2
- package/docs/testing.md +16 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.7-preview.4 — 2026-10-17
|
|
4
|
+
|
|
5
|
+
- Update the packaged Scout guide for the independently deployed task-relevance beta, including metadata filters, existing ambiguity responses, measured-evidence preference and bounded matching.
|
|
6
|
+
- Document retained execution-test failures, incomplete coverage and provisional Pareto comparisons; no improvement in exact task accuracy or globally optimal membership is claimed.
|
|
7
|
+
- Preserve the package runtime, CLI commands, exports and legal metadata. Publish to `preview`; stable `latest` remains unchanged.
|
|
8
|
+
|
|
3
9
|
## 0.1.7-preview.3 — 2026-10-16
|
|
4
10
|
|
|
5
11
|
- Update packaged Scout API/MCP and agent-evaluation guides for the sole `/scout/` service and current request, inventory, data-handling and reliability contracts.
|
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboa
|
|
|
22
22
|
|
|
23
23
|
## Compare models with Pi
|
|
24
24
|
|
|
25
|
-
The **npm preview `0.1.7-preview.
|
|
25
|
+
The **npm preview `0.1.7-preview.4`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.4/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
|
|
26
26
|
|
|
27
27
|
With Pi 0.87 and a model provider configured:
|
|
28
28
|
|
package/docs/commands.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
|
|
4
4
|
|
|
5
|
-
The npm `0.1.7-preview.
|
|
5
|
+
The npm `0.1.7-preview.4` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
|
|
6
6
|
|
|
7
7
|
Since `0.1.7-preview.2`, `openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]` asks for a budget after scope. Choose $10, a custom amount, or no fixed cap; saved choices and cumulative usage persist. See [budget selection](agent-evaluations.md#budget-selection).
|
|
8
8
|
|
package/docs/scout.md
CHANGED
|
@@ -4,23 +4,71 @@ Describe what you need a model to do. Scout returns up to five model-and-endpoin
|
|
|
4
4
|
|
|
5
5
|
Scout is a separate beta product. It does not change the OpenMerit npm package, Pi coordinator, or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Evidence architecture — hosted beta, 16 October 2026
|
|
8
|
+
|
|
9
|
+
Scout separates semantic interpretation from exact metadata checks and numerical selection. The hosted service uses this evidence architecture after publication of its validated serving generation. The prompt and results interface is unchanged; no npm release is included.
|
|
10
|
+
|
|
11
|
+
| Stage | Hosted implementation |
|
|
12
|
+
| --- | --- |
|
|
13
|
+
| Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
|
|
14
|
+
| Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
|
|
15
|
+
| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and versioned Vectorize benchmark embeddings. The active pointer changes only after the complete generation is ready. |
|
|
16
|
+
| Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
|
|
17
|
+
| Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
|
|
18
|
+
| Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
|
|
19
|
+
| Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. Numerical Pareto comparisons apply only within comparable evaluation scopes with known quality, cost and latency. |
|
|
20
|
+
|
|
21
|
+
Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC reference; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
|
|
22
|
+
|
|
23
|
+
Quality remains a relative benchmark signal, not predicted accuracy on an arbitrary task. Correlated benchmark families are grouped. Incomplete or incomparable objectives remain provisional and cannot establish a global three-objective frontier. Narrow typed tasks can still use a task-fit shortlist with unknown quality. Comparable direct calibration observations supplement quality evidence without excluding other eligible models. Calibration episode charges and timing do not replace the common token-workload reference costs and endpoint-bound timing; unknown values remain unknown.
|
|
24
|
+
|
|
25
|
+
The UI and HTTP/MCP inputs remain unchanged. This path has no request-time web research, catalogue refresh, evidence preparation or recommended-model execution. Interpretation, query embedding and bounded Decisions checks are external calls; latency is measured end to end rather than assumed instantaneous. The matching budget is six seconds. A provider failure returns an unavailable response instead of silently relaxing restrictions.
|
|
26
|
+
|
|
27
|
+
OpenAI receives the submitted task for structured interpretation, Decisions and query embeddings. Query vectors are used transiently; only public benchmark embeddings enter Vectorize. Public model/endpoint facts and observations enter private D1, with source records in private R2. Parallel receives public research subjects, not visitor prompts. Existing private prompt-and-evidence analytics and three-month sampled retention remain unchanged. Credentials remain server-side Cloudflare secrets.
|
|
28
|
+
|
|
29
|
+
The canonical paths remain `/scout/`, `/scout/api/recommend`, `/scout/api/status` and `/scout/mcp`. There are no versioned preview pages or API aliases. The dated sections below retain historical implementation and validation records; this architecture and the current Ranking section supersede conflicting earlier policies.
|
|
30
|
+
|
|
31
|
+
Acceptance retained the earlier provider failures rather than counting them as successes. The final remote filter/control run initially passed 18 of 27 cases; nine reached the six-second deadline. Each of those nine passed one focused recheck without changing the matcher. Ten additional ambiguous-output cases passed on the revised interpreter. These checks establish interpretation, eligibility and API/MCP behavior for the tested prompts, not universal reliability or actual recommended-model accuracy. The isolated real-feed refresh completed R2/D1/Vectorize publication with new Parallel purchases disabled; the normal production workflow was then exercised separately. The production refresh published successfully after a staging-memory correction. Public HTTP/MCP checks initially passed six of eight cases; the provider and invoice timeouts both passed one focused recheck. Successful live observations were approximately 2.4–6.4 seconds. Six research jobs remain queued for later enrichment. These are separate attempts, not an eight-case clean first pass.
|
|
32
|
+
|
|
33
|
+
## Task relevance corrections — hosted beta, 17 October 2026
|
|
34
|
+
|
|
35
|
+
These corrections are deployed as a limited hosted beta. Hourly evidence documents retain verified research descriptions and directly selected methodology alongside feed metadata; embedding caches change when that evidence changes. The already verified descriptions are indexed directly, without a second model-generated summary; original citation excerpts remain in the source records rather than the semantic search text. Structured extraction separates the downstream work from model restrictions before Decisions interprets its capabilities and output contract. Name resolution can run alongside Decisions. This avoids interpreting model selection itself as a downstream capability; it does add a sequential interpretation dependency within the existing six-second budget.
|
|
36
|
+
|
|
37
|
+
Query vectors describe the downstream work, excluding model-selection restrictions. They retrieve methodology candidates, with prepared memberships as a fallback when the query has no matches and a backstop for required capabilities missing from the retrieved set. The query order and hourly model/benchmark availability map propose up to sixteen methodology records. Decisions checks each record independently against the actual workload, without mixing unrelated methodologies in one input. Both general relevance and a deliberately scored required ability must be sufficiently supported. A broader ability can be a component proxy without testing the exact domain or answer format; incidental prerequisites cannot qualify. Ambiguous labels, broad capability overlap and unavailable checks cannot grant quality evidence. Fine-grained scope is checked against the full workload and methodology rather than asking every discovered capability again on each search. Core task/format interpretation stays bounded as the hourly directory grows.
|
|
8
38
|
|
|
9
|
-
|
|
39
|
+
Classification and judgment searches consider applicable benchmark evidence before an unmeasured native endpoint contract. Without comparable candidates they retain an explicitly unmeasured task-fit fallback; contract compatibility alone does not demonstrate accuracy. Spatial pipelines keep their separate compatibility checks. Ambiguous output representations can use the existing clarification response, with model restrictions preserved. No task-name, object-name or model-specific exception is added.
|
|
10
40
|
|
|
11
|
-
The
|
|
41
|
+
The Artificial Analysis parser locates serialized records by their data shape rather than a component-property name and never executes downloaded scripts. Omitted historical observations are retained only for models still listed and unchanged benchmark definitions, keeping their original provenance and historical status. Missing endpoint-specific timing and incomparable quality scopes remain unknown or provisional; restoring a feed does not authorize copying model-level timing to an unverified route. Matching retains its six-second deadline, and the hourly schedule and UI are unchanged. Default selection orders supported frontier points by quality and known cost/time, without reserving slots for the cheapest or fastest point. Explicit cost and latency priorities and metadata preferences still apply.
|
|
12
42
|
|
|
13
|
-
|
|
43
|
+
The initial release was withheld after real-task validation; the owner subsequently approved a limited beta release with the evidence and accuracy limitations retained. Earlier development attempts exposed source-relevance, retrieval and interpretation errors; a passing interpretation/filter run alone did not authorize deployment. A fresh execution set made **38 OpenRouter requests**, with **$0.223226** in reported charges and no unknown-charge calls. Five shortlisted models per task received two public examples each; bird and invoice tasks also included two comparison models from the earlier shortlist. The source images, answer keys, scorer and selected pool were frozen before dispatch. These executions used recorded OpenRouter routes, which do not establish latency or cost on a different recommended host.
|
|
14
44
|
|
|
15
|
-
|
|
45
|
+
| Fresh task check | Shortlisted models | Comparison models | Release gate |
|
|
46
|
+
| --- | --- | --- | --- |
|
|
47
|
+
| Bird species names in alphabetical order | 19/20 required items correct | 8/8 correct | Lower sampled accuracy |
|
|
48
|
+
| Book versus poster labels | 20/20 labels correct | No separate comparison pool | Passed |
|
|
49
|
+
| Invoice number and total, strict JSON | 18/20 fields correct | 8/8 correct | Lower sampled accuracy |
|
|
50
|
+
|
|
51
|
+
One model confused two bird species; two invoice candidates added an extra zero to a printed total. All outputs parsed successfully, so schema compliance did not establish accuracy. These failures remain recorded; no model-specific exception, held-out score or answer key was added to ranking. The final downstream-interpretation correction passed 11/11 matching cases, including exact provider/family/context/date filters, known matches, no-match and ambiguous-output cases. Observed client durations were approximately 2.2–5.9 seconds, with no percentile guarantee. That correction changed some bird/category shortlist membership, so the frozen execution results do not certify the final pool; the invoice pool remained the same and failed the original accuracy-only gate. That gate compared five shortlisted models with two comparison models rather than evaluating quality, cost and latency together. In this sample the comparison models had approximately 4.1 times the mean request charge for birds and 3.4 times for invoices. Lower mean accuracy alone does not establish a worse Pareto shortlist. The beta makes no claim of improved task accuracy or globally optimal five-model membership. Broader coverage, comparable deployment measurements and independent validation of all three objectives remain follow-up work. The held-out outcomes remain excluded from ranking.
|
|
52
|
+
|
|
53
|
+
Publication retained the latest completed hourly research and a healthy refreshed feed set. The final rollout check passed 9/11 cases on its first attempt: bird matching reached the deadline and explicit SVG generation returned no shortlist. Both passed one focused recheck without a matcher change. After deployment, the public HTTP suite again passed 9/11: the explicit SVG request returned no shortlist, and a Qwen-family code-authoring request admitted tool-calling evidence. Creator/family/context/date eligibility checks and impossible-context no-match passed. The live browser bird prompt displayed five recommendations without overflow. These are retained semantic and latency limitations of the beta, not a clean first-pass acceptance or proof of best-five performance.
|
|
54
|
+
|
|
55
|
+
## Latency improvements — hosted beta, 17 October 2026
|
|
16
56
|
|
|
17
|
-
The
|
|
57
|
+
The hosted implementation keeps the existing task and format Decisions, exact metadata comparisons and numerical selection. Numeric and date vocabularies are omitted from interpretation input because their bounds are arbitrary; small canonical vocabularies remain in the first call. Larger vocabularies are deferred until an extracted restriction references them; a model-assisted resolution checks against the complete published vocabulary without changing unrelated conditions. Unknown mandatory names remain unresolved and cannot widen the result set. Stable reference data has an explicit provider prompt-cache boundary separate from the prompt and UTC reference time. Cache availability is measured rather than guaranteed.
|
|
18
58
|
|
|
19
|
-
|
|
59
|
+
Hourly staging prepares content-addressed specialist packs alongside profile indexes. Searches select the same scoped profiles and use those packs instead of many repository reads. A version-checked derived pack index can also be prepared for an existing immutable catalogue without repeating research. Existing generations remain readable through the original shard path when no matching pack index is available. Source research and the hourly trigger are unchanged.
|
|
20
60
|
|
|
21
|
-
|
|
61
|
+
Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache adds no external prompt storage and is not semantic similarity caching.
|
|
62
|
+
|
|
63
|
+
Private search records now include fixed stage durations, pending stages at a deadline, cache status and allowlisted provider status/error codes. They do not store raw errors, stack traces, headers or credentials. API results include `runtime` timings/cache status; cache hits report their current serving duration and zero new interpretation time. The page remains unchanged. A six-second matching deadline remains a reliability boundary; these changes do not establish a latency guarantee. The full repository release checks passed, including 318 Scout tests. An initial candidate timed out on books/posters; that failure was retained, and the interpretation was then reduced further. The revised semantic suite passed 17 of 17 cases, including cold repetitions and exact-cache reuse, and four final cold checks passed after the independent database reads were parallelized. These small samples do not establish a production percentile or a latency guarantee. Post-deployment HTTP/MCP checks passed five of five on their first attempts: supplied-label vision, creator restriction, impossible context, JSON invoice extraction through MCP and an exact repeat. Observed end-to-end durations were approximately 2.0–4.3 seconds for new searches and 146 ms for the repeated prompt served from the same isolate cache. The 127 UI source files and five live scripts/styles were unchanged. These are individual observations, not percentile guarantees.
|
|
64
|
+
|
|
65
|
+
## Default architecture — hosted beta, 16 October 2026
|
|
22
66
|
|
|
23
|
-
|
|
67
|
+
Scout maintains separate model, endpoint, benchmark and observation records. Structured feeds, resumable discovery and bounded Parallel research prepare source-backed public metadata each hour. Unreviewed discovery listings remain outside the executable catalogue. New or changed methodology, metadata and embeddings use source-addressed caches. An incomplete generation cannot replace the active catalogue; searches continue against the last committed version.
|
|
68
|
+
|
|
69
|
+
A search runs Decisions task interpretation, Structured Outputs metadata extraction and a transient query embedding. It applies exact restrictions in D1, retrieves prepared evidence and relevant specialist profiles, and compares verified eligible configurations locally. There is no request-time web research, source preparation or model execution. The model pool has no pre-frontier top-K cap; evidence retrieval and specialist checks remain bounded. Missing objectives, incomplete benchmark coverage and approximate evidence retrieval prevent a claim of a globally verified optimum.
|
|
70
|
+
|
|
71
|
+
The trigger remains hourly at `7 * * * *` UTC, with up to 24 Parallel jobs, refresh concurrency six and seven-day research rechecks. Discovery and enrichment are incremental; the trigger does not promise complete world coverage or a revisit to every source each hour. Relevant specialist checks remain bounded to 64 prepared profiles. Matching has a six-second budget and includes external provider/storage calls. Successful focused remote checks were approximately 2.4–5.4 seconds; earlier timeouts are disclosed above. Instantaneous results are not guaranteed.
|
|
24
72
|
|
|
25
73
|
## Matching corrections — hosted beta, 15 October 2026
|
|
26
74
|
|
|
@@ -93,7 +141,7 @@ The initial deployed snapshot processed all 146 benchmark series: 136 have suppo
|
|
|
93
141
|
|
|
94
142
|
Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
|
|
95
143
|
|
|
96
|
-
The comparison table keeps recommendations in Scout rank order.
|
|
144
|
+
The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
|
|
97
145
|
|
|
98
146
|
The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
|
|
99
147
|
|
|
@@ -113,7 +161,7 @@ Scout exposes one read-only tool, `scout_models`, with a required prompt:
|
|
|
113
161
|
{"prompt":"Upload an invoice and extract six fields as JSON"}
|
|
114
162
|
```
|
|
115
163
|
|
|
116
|
-
No API key is required from the caller. Scout sends the task description to OpenAI Decisions
|
|
164
|
+
No API key is required from the caller. Scout sends the task description to OpenAI for Decisions interpretation, structured metadata extraction and a transient query embedding, then compares its collected evidence. It does not execute the recommended models. The MCP endpoint supports protocol versions `2025-03-26`, `2025-06-18`, and `2025-11-25`, including initialization, tool discovery, and tool calls. It uses JSON responses without sessions or a long-lived SSE stream. Browser requests must have the same origin; ordinary agent clients omit the Origin header. An unsupported protocol version is rejected. Agent configuration is client-specific; connecting an endpoint does not automatically make an agent use it.
|
|
117
165
|
|
|
118
166
|
Tell the agent: “Before choosing an inference model for this feature, call Scout with the task description. Compare the five candidates, read their evidence and warnings, and evaluate the shortlist on our own examples.”
|
|
119
167
|
|
|
@@ -136,14 +184,14 @@ Recommendation `id` preserves legacy router IDs; a new protocol receives a disti
|
|
|
136
184
|
The response contains:
|
|
137
185
|
|
|
138
186
|
- `recommendations`: up to five distinct model IDs with rank, provider, `endpoint` (provider model ID, protocol, configuration, deployment, provenance and limits), reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
|
|
139
|
-
- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `judge`, `detect`, `segment`, `visual_unspecified`, `other`), `outputContract` (labels, decision, boxes, masks, text, or unspecified), `outputFormat` (`json`, `other`, `none`, or `unspecified`; only specifically established formats can match format-specific calibration), `outputCardinality`, optional `clarification`, and `interpreter: "openai-
|
|
187
|
+
- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `judge`, `detect`, `segment`, `visual_unspecified`, `other`), `outputContract` (labels, decision, boxes, masks, text, or unspecified), `outputFormat` (`json`, `other`, `none`, or `unspecified`; only specifically established formats can match format-specific calibration), `outputCardinality`, optional `clarification`, and `interpreter: "openai-hybrid"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
|
|
140
188
|
- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Compatible vision deployments may instead enter a task-fit shortlist; they are not counted twice. Unsupported task types can omit this field.
|
|
141
189
|
- `specialistRepositories`: bounded Hub catalogue count, separate from the evidence-backed `models` count.
|
|
142
190
|
- `calibration`: released task profiles, scopes, sample sizes, measurement dates and expiry status.
|
|
143
191
|
- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any hourly source unchecked for more than two hours makes the index stale. Versioned endpoint contracts disclose their manual review date separately; a scheduled metadata refresh does not renew those price claims.
|
|
144
192
|
- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task. `endpoints` counts hosted route listings, including reviewed typed contracts; `independentModels` counts catalogue identities with metadata independent of OpenRouter. Neither number implies credentials or verified account access.
|
|
145
193
|
- `wordCloud`: capability vocabulary with benchmark and model coverage counts.
|
|
146
|
-
- `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final order
|
|
194
|
+
- `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final shortlist order. `ranking.frontier` reports compared configurations, selected distinct models and whether the objective scope is comparable or partial. The compatibility `score` can be `null`; there is no weighted quality/cost/time composite in frontier selection.
|
|
147
195
|
- `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.
|
|
148
196
|
|
|
149
197
|
`pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. The same rule applies to MCP responses. `pricing.basis` identifies the price source and calculation assumptions.
|
|
@@ -189,40 +237,34 @@ Missing evaluations mean **unknown**, not poor performance. Task-fit candidates
|
|
|
189
237
|
|
|
190
238
|
## Ranking
|
|
191
239
|
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
Strong JSON/schema filtering follows the semantic format answer and its confidence check, without rereading the prompt for keywords. Uncertain format judgments remain disclosed and cannot inherit narrow calibration evidence. Low-confidence consequential classifications and unspecified retention/tool interfaces are exposed as uncertainty. This is still a probabilistic classifier: paraphrases can differ, ambiguous requests can be misread, and unfamiliar languages or formats can be misinterpreted. Ambiguous visual separation returns `task.clarification`; HTTP/MCP clients should ask the output question, and the page offers three explicit choices. Model and endpoint inventories are the only supported selection constraints.
|
|
240
|
+
[OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work, capability requirements, operation, answer space, serialization and tradeoff priority. It distinguishes writing source from executing it, supplied categories from open identification, text-bearing objects from reading their contents, and session state from persistent memory. Typed answers are validated and checked for consistency. Uncertain answers remain disclosed; they cannot authorize narrow calibration evidence. No prompt keyword rules or model-name bonuses enter selection.
|
|
195
241
|
|
|
196
|
-
|
|
242
|
+
A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC reference. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
|
|
197
243
|
|
|
198
|
-
|
|
244
|
+
Image tasks require declared image input and relevant visual evidence in performance ranking. Document reading needs document evidence. JSON/schema tasks need declared `response_format` or `structured_outputs`; required tool execution needs tools support and relevant tool evidence. Persistent memory requires external-memory evidence rather than a large context window. [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) do not establish built-in memory or performance on the visitor's storage system. Recognized image/video/audio generation and transcription remain outside the supported output index. Ambiguous spatial output produces the existing clarification choices.
|
|
199
245
|
|
|
200
|
-
|
|
246
|
+
Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality stays unknown. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
|
|
201
247
|
|
|
202
|
-
|
|
248
|
+
For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by their hourly capability memberships. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
|
|
203
249
|
|
|
204
|
-
|
|
250
|
+
Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A relative quality signal is not predicted accuracy or a statistical confidence interval.
|
|
205
251
|
|
|
206
|
-
|
|
252
|
+
Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after seven days. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
|
|
207
253
|
|
|
208
|
-
|
|
254
|
+
All eligible verified catalogue configurations with relevant evidence enter numerical comparison before model deduplication. A configuration can dominate another only within a comparable quality scope, with known quality, cost and time: it must be at least as good in all three and strictly better in one. Missing objectives and incomparable coverage remain provisional. This cannot establish a global three-objective frontier over every model in the world.
|
|
209
255
|
|
|
210
|
-
|
|
211
|
-
Each comparison also considers relative cost and completion time. The default balanced priority uses **60% quality, 20% cost, 20% latency**. An explicit quality priority uses 90/5/5; cost uses 60/35/5; latency uses 60/5/35. These are transparent defaults, not learned optimal weights. Decisions can infer a priority from the prompt; there are no new controls or extra interpretation calls. Broad or ambiguous preferences can still be misclassified.
|
|
256
|
+
Up to five distinct models represent the supported tradeoffs. Explicit priorities and soft metadata preferences order eligible frontier points, followed by supported quality and known cost/time. Default selection reserves no cheapest or fastest slot. Preferences do not broaden eligibility. There is no weighted quality/cost/time composite. Fewer than five points can remain even when more models are eligible; the API reports eligibility separately from the selected count. Stable IDs resolve exact ties.
|
|
212
257
|
|
|
213
|
-
|
|
258
|
+
Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Supplied-label choices use a 16-token output reference. Reference cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. Decisions has no output-token charge. Unknown or self-hosted operating cost is not inferred to be zero.
|
|
214
259
|
|
|
215
|
-
Reference
|
|
260
|
+
Reference completion time is **time to the first answer token + visible output tokens / throughput**, only when measurements identify the selected endpoint. Multiple published configurations retain paired timing before taking a median. Timing is not borrowed across provider routes, and early reasoning tokens are not confused with the first answer. Missing timing stays unknown. The page's request duration measures Scout matching, not a recommended model executing the visitor's task.
|
|
216
261
|
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
Matching is deliberately narrow; code-function tests do not establish repository-level engineering skill, and small tool loops or key-value memory tasks do not establish long-horizon agent reliability. A profile label and quality basis are returned with every measured shortlist. No model-name bonuses, prompt-example rules, or task answer keys enter the ranking.
|
|
220
|
-
|
|
221
|
-
The goal is an ordered five-model shortlist, not just a first-place pick. Regression checks cover all five positions, all ten within-five pairs, cutoff membership, provider-name invariance, unknown cost/time, capability and modality boundaries, expiry, and rejection of held-out observations during evidence building. Paid evaluation uses separate calibration and held-out cases across multiple task types. Held-out measurements compare both membership and ordering within the tested candidate pool; they do not establish a global optimum or a perfect ordering for every prompt. Test the five candidates on your own examples before choosing a production model.
|
|
222
|
-
If fewer than five eligible models have evidence, Scout returns fewer and explains why. There are no visitor accounts, visitor-facing prompt history, vector database, user selection controls, or language-model reranker. Private team search records are described below.
|
|
262
|
+
The shortlist requires evaluation on the visitor's own examples. Public benchmarks, small calibration samples and semantic applicability checks do not prove the optimal membership or order of five models for an arbitrary task. There are no visitor accounts, public prompt history or additional filter controls. Public methodology embeddings are stored in Vectorize; visitor query vectors are transient. Private prompt-and-evidence analytics are described below.
|
|
223
263
|
|
|
224
264
|
## Current validation limits
|
|
225
265
|
|
|
266
|
+
The evidence architecture is validated against the published catalogue, not a million-model production load. Synthetic 10,000-configuration checks cover comparable and disconnected evidence scopes; they do not establish request latency with dense, mixed benchmark coverage. Discovery remains incremental and neither coverage counts nor a shortlist establish a globally verified optimum.
|
|
267
|
+
|
|
226
268
|
The identification correction was checked on the original bird-name prompt, plant-species identification, supplied indoor/outdoor choices and supplied ticket categories with strict JSON, using real prompt interpretation. Regression checks also cover missing/uncertain label sets, explicit endpoint scopes, specialist discovery, visual evidence and classification calibration boundaries. These checks verify interpretation and eligibility, not the accuracy of model-generated bird or plant names.
|
|
227
269
|
|
|
228
270
|
The earlier 15 October endpoint-catalogue revision passed **20/20 fresh task-interpretation cases** with expectations frozen before dispatch. They covered image/text labels, predicates, rubric scores, single/all-instance boxes, masks, ambiguous separation, visual/text extraction, code, SQL, tool loops, memory, prose and unsupported generation. A prior 18-case development check exposed two routing errors; those cases were retired before the final set. Unit checks cover output-contract eligibility, exact endpoint scoping, name invariance, duplicate endpoints and unknown prices. This validates tested routing boundaries, not best-five task performance.
|
|
@@ -270,7 +312,7 @@ The first test exposed cross-task request failures; those test outcomes were not
|
|
|
270
312
|
|
|
271
313
|
## Refresh and data handling
|
|
272
314
|
|
|
273
|
-
The task description is sent to Scout and then to OpenAI Decisions
|
|
315
|
+
The task description is sent to Scout and then to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, a transient query embedding and, when relevant, specialist compatibility/applicability checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. Scout stores the submitted prompt alongside its recommendations in private Cloudflare Analytics Engine records. Runtime request logging remains disabled. The public index, published calibration aggregates and the limited search analytics described below are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
|
|
274
316
|
|
|
275
317
|
Scout records one server-side Analytics Engine event for each valid recommendation attempt on the website/JSON API or MCP. It keeps the original submitted prompt (up to 4,000 characters, including whitespace), fixed task categories, capability/input/output requirements, priority and workload buckets together with the ordered model/endpoint shortlist (up to five), its quality/cost/time values and their calibration/reference/unknown basis, attached source evidence, ranking and interpreter versions, index date, scope category, freshness, outcome and server duration. Clarifications, unsupported tasks, empty results, admission limits and interpretation errors are distinct outcomes. A failure before interpretation has an unknown task category. Invalid requests rejected before recommendation, status checks and MCP discovery are not search events. Website and other JSON API calls share the `api` channel; MCP uses `mcp`.
|
|
276
318
|
|
|
@@ -297,7 +339,7 @@ npm run scout
|
|
|
297
339
|
|
|
298
340
|
Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted Parallel research workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
|
|
299
341
|
|
|
300
|
-
`npm run scout:refresh` fetches the eight hourly public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur
|
|
342
|
+
`npm run scout:refresh` fetches the eight hourly public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted hourly Workflow additionally prepares and publishes its D1/Vectorize serving generation.
|
|
301
343
|
|
|
302
344
|
## Hosting and verification
|
|
303
345
|
|
package/docs/security.md
CHANGED
|
@@ -47,10 +47,10 @@ Repository metadata and usage caches live in process memory and disappear when t
|
|
|
47
47
|
|
|
48
48
|
## Hosted Scout analytics
|
|
49
49
|
|
|
50
|
-
Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI Decisions
|
|
50
|
+
Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
|
|
51
51
|
|
|
52
52
|
Search records are accessible only through authenticated Cloudflare account queries. Scout exposes no analytics dashboard, report generator, report file, sign-in, or public history/query endpoint. Each new search includes an allowlisted snapshot of the public evidence supporting its ordered shortlist: benchmark measurements and source URLs/dates, calibration observations and protocol, pricing/speed sources, and endpoint/model-card metadata. Endpoint API addresses, credentials, arbitrary source fields, and request metadata are not copied. Source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions. The dataset is sampled and retained by Cloudflare for three months. Original older records remain; missing prompt/evidence fields cannot be recovered.
|
|
53
53
|
|
|
54
54
|
## Scout research storage
|
|
55
55
|
|
|
56
|
-
Scout uses Parallel for hourly public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.
|
|
56
|
+
Scout uses Parallel for hourly public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.
|
package/docs/testing.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
# Testing and validation
|
|
2
2
|
|
|
3
|
+
## Preview.4 release checks — 17 October 2026
|
|
4
|
+
|
|
5
|
+
The `0.1.7-preview.4` candidate passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
|
|
6
|
+
|
|
7
|
+
The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy. Registry installation and post-publication VM results are recorded separately after publication.
|
|
8
|
+
|
|
9
|
+
## Published preview.3 acceptance — 16 October 2026
|
|
10
|
+
|
|
11
|
+
Published `openmerit@0.1.7-preview.3` under npm's `preview` tag after all 596 repository tests and the syntax, type, build, documentation and whitespace checks passed. Stable `latest` remains `0.1.6`. Registry integrity matches the tested tarball. Runtime files are unchanged from `0.1.7-preview.2`; this release updates packaged Scout guides and excludes internal website/deployment notes. Scout's hosted research and matching service deploys independently of npm.
|
|
12
|
+
|
|
13
|
+
A registry installation in a separate directory in the existing Ubuntu ARM64 Lima VM, with Node 22.23.1, passed all 202 acceptance assertions, including SHA-256 verification of all 101 packaged files. Public core/protocol/Pi imports and the installed Pi extension load passed. The existing global installation was retained.
|
|
14
|
+
|
|
15
|
+
Seven workload descriptions were tested through both the live HTTP API and the official MCP SDK: books/posters classification, open bird identification, safety labels, safety labels with explanation/rewrite, SQL generation with a runtime schema, photographed-invoice JSON extraction, and unsupported audio transcription. The operation, modality and output requirements agreed across channels. Inventory restrictions, invalid requests, MCP discovery/ping, and endpoint selection passed. Installed `eval init` and `eval plan --scope existing --budget 1` produced two frozen task shortlists limited to the declared connections; no execution, budget ledger or background work was started.
|
|
16
|
+
|
|
17
|
+
Across these 14 prompt requests, observed median wall time was 1.43 seconds and the slowest was 4.14 seconds. Source status reported neither stale nor degraded data. This verifies packaging and API/MCP integration, not model task accuracy, universal semantic reliability, a guaranteed response time, or a new hourly-refresh trial. The earlier hosted refresh acceptance remains separate. No website deployment is included in this npm release.
|
|
18
|
+
|
|
3
19
|
## Standard verification
|
|
4
20
|
|
|
5
21
|
```sh
|