omp-llm-role 1.0.0 → 1.1.0

package/docs/dev/data-sources.md

@@ -5,7 +5,6 @@ llm-stats id, and where it is cached. All fetching lives in `src/engine.ts`
 (plus `src/availability.ts` for the key/catalog/probe surfaces).
 
 - Module map and pipeline: [`architecture.md`](architecture.md).
-- Normative behavior contract: [`spec.md`](spec.md).
 
 ## llm-stats.com — quality
 
@@ -22,12 +21,27 @@ payload; there is no public JSON API.
   `context`, `release_date`, `multimodal`, `license`, `input_price`/`output_price`
   (unused), `throughput`/`latency` (unused), the `index_*` scores (0–100:
   general, reasoning, math, code, agents, search, vision, tool_calling,
-  long_context) and the benchmark scores (0–1: `gpqa_score`,
-  `aime_2025_score`, `swe_bench_verified_score`, `arc_agi_v2_score`,
-  `mrcr_v2_score`, `terminal_bench_score`, `tau_bench_retail_score`).
+  long_context, communication, finance, healthcare, legal) and the benchmark
+  scores (0–1: `gpqa_score`, `aime_2025_score`, `swe_bench_verified_score`,
+  `arc_agi_v2_score`, `mrcr_v2_score`, `terminal_bench_score`,
+  `tau_bench_retail_score`, plus `hle_score`, `simpleqa_score`, `mmmu_score`,
+  `mmmu_pro_score`, `mmmlu_score`, `browsecomp_score`, `swe_bench_pro_score`,
+  `mcp_atlas_score`, `apex_agents_score`, `osworld_score`, `scicode_score`,
+  `screenspot_pro_score`, `charxiv_r_score`, `frontiermath_score`,
+  `toolathlon_score`, `coding_arena_score`). `buildModels` maps the nine
+  short-key indices plus the four new `index_*` fields (communication, finance,
+  healthcare, legal) and the seven short-key benchmark scores plus the fifteen
+  new `*_score` benchmarks — 13 indices and 22 benchmarks in all. The new
+  metrics keep the **raw leaderboard field name as the metric key**
+  (`index_communication`, `simpleqa_score`, …), unlike the original nine/seven
+  short keys. `coding_arena_score`, `latency` and `context` are fetched but
+  discarded (see the research note on the agentic-engineering benchmarks for the
+  coverage/collinearity of the unmapped ones).
 - **Build**: `buildModels(rows)` maps each row to a `Model`; `metrics` carries
   the index/benchmark values, with `price`/`throughput`/`website`/`writing`
-  left `null` for later enrichment.
+  left `null` for later enrichment. The row's `release_date` is carried onto
+  `Model.releaseDate` (null when absent) and used by the focus-metric freshness
+  axis (scoring.md, the nine-axis gate).
 - **Cache**: `cache/llm-stats-fetched-rankings.json` —
   `{ fetchedAt, source, modelCount, rankings[] }` (pretty-printed; each row
   gains a `rank` = array position). Fresh while `fetchedAt` is the current UTC
@@ -48,14 +62,16 @@ Two payloads, both joined to llm-stats by slug suffix: the llm-stats `model_id`
 - **Shape**: `data.endpoint_perf[endpointId]` → `{ p50_throughput (output tok/s),
   p50_latency (ms) }` over the last **30 minutes** of routed traffic;
   `data.models[]` rows link a slug to one endpoint id — the route currently
-  getting the traffic — each carrying `pricing.prompt`/`pricing.completion`
-  (USD/token strings, discounts already applied). The payload also carries
-  `analytics`, `benchmarks`, `benchmark_ranges`, `categories`,
+  getting the traffic — each carrying `pricing.prompt`/`pricing.completion` and
+  `pricing.input_cache_read` (USD/token strings, discounts already applied). The
+  payload also carries `analytics`, `benchmarks`, `benchmark_ranges`, `categories`,
   `modality_counts`.
 - **Parse**: `parseFindData(v)` narrows `find` (models, `endpoint_perf`,
-  `endpoint_price` blended to $/M at 3:1 in:out) and keeps the full `data`
-  object verbatim for the cache; it also extracts the Design Arena mirror
-  (`extractDesignElo`, see below).
+  `endpoint_price` blended to $/M at 3:1 in:out, the parallel
+  `endpoint_weight_price` input-only $/M map — the router's sort key — and the
+  `endpoint_cache_read_price` cache-read input-only $/M map; see "The
+  1/price² blend") and keeps the full `data` object verbatim for the cache; it
+  also extracts the Design Arena mirror (`extractDesignElo`, see below).
 - **Cache**: `cache/openrouter-fetched-data.json` —
   `{ fetchedAt, source, modelCount, data }` where `data` is the verbatim
   `find?fmt=cards` response data (sorted keys). Throughput/price maps are
@@ -65,35 +81,75 @@ Two payloads, both joined to llm-stats by slug suffix: the llm-stats `model_id`
 
 - **URL**: `https://openrouter.ai/<slug>` (RSC flight payload), fetched only for
   slugs that match a leaderboard model (~150 of ~540).
-- **Shape**: every provider route of a model — pricing, service tier, status and
-  routed-traffic p50 stats — dehydrated as React-Query state (sometimes twice,
-  one copy without stats; records merge by endpoint id, the stats-carrying copy
-  winning). `parseModelPage(html)` narrows each row to
-  `OpenRouterEndpointRecord` (`id`, `providerSlug`, `serviceTier`, `status`,
-  `free`, `variant`, `price`, `tput`, `latency`).
+- **Shape**: every provider route of a model — pricing, service tier, status,
+  routed-traffic p50 stats and the per-endpoint capability ceilings — dehydrated
+  as React-Query state (sometimes twice, one copy without stats; records merge by
+  endpoint id, the stats-carrying copy winning). `parseModelPage(html)` narrows
+  each row to `OpenRouterEndpointRecord` (`id`, `providerSlug`, `serviceTier`,
+  `status`, `free`, `variant`, `price`, `weightPrice`, `cacheReadPrice`
+  (endpoint `pricing.input_cache_read`; `null` when absent), `tput`, `latency`,
+  `contextLength` (endpoint `context_length`), `maxCompletionTokens` (endpoint
+  `max_completion_tokens`) and `supportsTools` (`supported_parameters` contains
+  `tools`)). The three capability fields are `null` when the field is absent —
+  the role endpoint filters keep a `null` (missing data is not a capability
+  failure).
 - **Fetch**: `fetchEndpointPages(slugs)` with an 8-worker pool
   (`OPENROUTER_PAGE_CONCURRENCY`), one retry per page, permanent 404/410 gives
   up. A page that fails is absent and that model keeps the single-route
   fallback.
 - **Cache**: `cache/openrouter-endpoints-fetched-data.json` —
   `{ fetchedAt, source, slugCount, slugs }` where `slugs` maps each fetched
-  model slug to its narrowed per-provider route records.
+  model slug to its narrowed per-provider route records. `readEndpointsCache`
+  rejects a cache whose records lack `weightPrice` (the pre-#21 shape),
+  `contextLength` (the pre-#18 shape) **or** `cacheReadPrice` (the pre-#30
+  shape) and refetches, rather than silently dropping every route from the
+  blend, treating every ceiling as unstated, or leaving a role's `cacheHitRate`
+  inert (a missing cache-read price reads as the full input price).
+- **Per-model routes**: `buildOpenRouterEnrichment` carries the eligible
+  standard-tier pool on `OrEnrichment.routes`, and `applyOpenRouterData` sets it
+  on `Model.routes` — the pool the role endpoint filters gate. The
+  find-route fallback carries the same field (its capability fields are `null`,
+  since the find payload has no ceilings).
 
 ### The 1/price² blend
 
 `buildOpenRouterEnrichment(data, pages)` computes the expected price/throughput
 of one request under OpenRouter's default price-based load balancing (a request
 goes to ONE provider, picked among the stable standard-tier routes with
-probability proportional to `1/price²`):
+probability proportional to `1/price²`). The weight basis is the **input
+(prompt) price** — the router's sort key — while the reported price is the
+**billed 3:1 in:out blend** under that distribution.
 
 - **Pool**: the page's per-provider routes, plus any find-row endpoint the page
   does not list (the page is normally a superset).
 - **Eligible**: standard tier (`serviceTier === null`), `status === 0`, not
-  `:free`, `price > 0`, not `:batch`. flex/priority tiers are excluded (only the
-  `:floor`/`:nitro` variants make them eligible); degraded routes (`status ≠ 0`)
-  are fallbacks and drop out.
-- **Blend**: `price = Σ(1/p²)·p / Σ(1/p²)`; `throughput = Σ(1/p²)·t / Σ(1/p²)`
-  renormalized over the routes that have p50 data; latency likewise.
+  `:free`, `price > 0`, `weightPrice > 0`, not `:batch`. flex/priority tiers are
+  excluded (only the `:floor`/`:nitro` variants make them eligible); degraded
+  routes (`status ≠ 0`) are fallbacks and drop out.
+- **Weight basis**: the input-only $/M (`pricing.prompt * 1e6`, carried as
+  `weightPrice` on each route and `endpoint_weight_price` in the find payload).
+  OpenRouter's docs say only "weighted by inverse square of the price" and never
+  define the basis; the input price is the measured one. Tarun Chitra (Robot
+  Ventures), *Caching Cheaters on OpenRouter* (2026-08-14,
+  https://robvc.com/research/caching-cheaters): 64/64 informative provider
+  splits picked the cheapest-**input** provider (including menus where that
+  provider quoted the most expensive output price), output price carries ≲6
+  cents per dollar of input, and the inverse-square exponent recovered from
+  1,648 choices is r = 1.968 (95% CI [1.856, 2.098]); cached-token price adds no
+  measurable predictive power. The authors' caveat: behaviour-measured, not
+  source-read.
+- **Blend**: `price = Σ(1/p_in²)·p_billed / Σ(1/p_in²)` — the w-weighted mean of
+  the billed 3:1 blend (what a caller pays), weighted by the input price;
+  `throughput = Σ(1/p_in²)·t / Σ(1/p_in²)` renormalized over the routes that
+  have p50 data; latency likewise. A route with no input price drops from the
+  pool (`1/p²` blows up at 0, and the router cannot score a route without one).
+- **Cache-read pricing**: a role may declare `cacheHitRate` (0–1, opt-in), and
+  `blendRoutePool(routes, cacheHitRate)` then blends each route's
+  `cacheReadPrice` (the endpoint's `pricing.input_cache_read`) into its billed
+  price, `billed = price + 3·(h·cacheRead + (1−h)·input − input)/4`, **without**
+  changing the `1/p_in²` weight basis (the router still sorts on the listed input
+  price). A route with no cache-read price keeps its full input price; `h = 0`
+  is byte-identical to the uncached blend. See scoring.md, *Cache-read pricing*.
 - **Fallback** (no eligible route carries throughput): throughput = highest-p50
   variant (`:free` included — it rescues otherwise-unranked models), price =
   cheapest billed route (a $0 free tier never sets the price).
@@ -101,6 +157,51 @@ probability proportional to `1/price²`):
   `m.throughput`, `m.metrics.price`/`throughput`, and `m.thinking`
   (`supports_reasoning`).
 
+A role with a `providerPin` does **not** use this blend: it prices the
+model's matching route directly — the route's billed 3:1 price and its p50
+throughput (scoring.md, *Route-aware pricing*). The blend is the unpinned
+default.
+
+## OpenRouter — key metadata (tier gate)
+
+Bearer-authenticated GETs with the omp-resolved key decide which variant tier the
+run may use:
+
+- `GET https://openrouter.ai/api/v1/key` → `is_free_tier`, `limit_remaining`,
+  `limit`, `free_model_daily_requests.remaining`.
+- `GET https://openrouter.ai/api/v1/credits` → `total_credits`, `total_usage`.
+
+```
+billedUsable = !is_free_tier && limit_remaining > 0 && (total_credits - total_usage) > 0
+freeUsable   = free_model_daily_requests.remaining > 0
+```
+
+`tierGate` (`src/availability.ts`) maps these to `"billed" | "free" | "none"`:
+billed → billed variants only (exclude `:free`, `:batch`); else free → `:free`
+variants only (exclude `:batch`); else `"none"` aborts the run. The key endpoint
+carries **no** model/provider restriction field — the account's allowed-providers
+whitelist is invisible to it (see the keyed catalog below and the probe in
+[`architecture.md`](architecture.md)).
+
+## OpenRouter — keyed catalog (availability)
+
+The account setting **"Filter the model catalog for API keys"** (openrouter.ai/settings)
+makes the Bearer-authenticated catalog a per-key allowlist.
+
+- **URLs**: `GET https://openrouter.ai/api/v1/models` twice — once unauthenticated
+  (the public catalog) and once with `Authorization: Bearer <key>` (the keyed catalog).
+  `fetchKeyAvailability` (`src/availability.ts`) issues both in parallel.
+- **Signal**: set membership only — the response carries no per-model access flag. Both
+  id lists are normalized with `rankingIdOf` (suffix after the last `/`, minus `:free`,
+  minus a trailing `-latest`), the same identity space as the engine's `Model.id`, so
+  aliases never produce a false "blocked". The keyed list being a non-empty **proper
+  subset** of the public one means the setting is on (`active`); equality is `no-filter`;
+  an empty list or any transport/status/shape failure is `unavailable`.
+- **Consumers**: the updater fetches it **once per run** and shares it across roles; the
+  explorer fetches it on boot and on `POST /api/refresh`. It is **not cached to disk** —
+  every consumer re-fetches, and `fetchKeyAvailability` **never throws** (a failure
+  degrades to `unavailable` with empty sets, so the probe walk / `unknown` marks stand).
+
 ## Design Arena — design quality
 
 Two routes, never averaged (same Elo family; mean diff −1.0 website … −5.1 svg,
@@ -191,25 +292,88 @@ mapping to a `KNOWN_METRICS` key. A source whose data the engine already fetches
 ### The generic llm-stats benchmark
 
 `GET https://api.zeroeval.com/leaderboard/benchmarks/<id>` returns
-`{ benchmark_id, benchmark_name, max_score, entries[] }`; each entry carries
-`model_id` (the bare llm-stats id, so the join is **direct**) and
+`{ benchmark_id, benchmark_name, max_score, total_models, entries[] }`; each
+entry carries `model_id` (the bare llm-stats id, so the join is **direct**) and
 `normalized_score` (0-1). Any `llm-stats.com/benchmarks/<id>` link (or a bare
 benchmark id) resolves to a source whose metric is `bench:<normalized-id>`, cache
 file `bench-<normalized-id>-fetched-data.json`, transform `identity`, and a
 capability fill of 0.195. The raw id is preserved for the fetch URL (the real id
 `alpacaeval-2.0` normalizes to the metric `bench:alpacaeval-2_0`), so the source
 is persisted as a declaration (below) — the metric key alone cannot reconstruct
-the raw id.
+the raw id. `catalogBenchmarkDeclaration` builds that declaration for a dotted
+catalog id at discovery/authoring time, carrying the raw id in `fetch.url`.
+
+Each entry also carries the trust flags `self_reported` and `verified` (both
+booleans), the `multimodal` flag, the org fields `organization_id`/
+`provider_id`, and a **second, independent** set of the price/throughput/context
+quantities: `input_cost_per_million`, `output_cost_per_million`, `speed_rps`
+(requests/s) and `context_window` (tokens). `parseBenchmarkPayloadMeta(payload)`
+reads them independently of `source.parse` (a declared/writing source reads no
+entry flags) and returns
+`{ trust: { selfReported, verified, covered }, modality: { multimodal, covered },
+provenance: { dominantOrg, dominantShare, orgs, covered }, crossSource }` — the
+trust
+`covered` is the count of entries carrying a boolean `self_reported` (the share
+denominator), `selfReported` the count of those whose `self_reported` is `true`,
+and `verified` the count of those whose `verified` is `true`; the modality
+`covered` is the count of entries carrying a boolean `multimodal` and `multimodal`
+the count of those that are `true`; the provenance `covered` is the count of
+entries carrying a non-empty org string (each row's `organization_id`, falling
+back to `provider_id`), `orgs` the distinct org count, and `dominantOrg`/
+`dominantShare` the most frequent org and its count / `covered`; `crossSource` is
+a map keyed by the entry's `model_id` of `{ input, output, speed, context }`
+(each a finite number, else `null`), `null` when no entry carried any of the four
+(the four summaries have independent denominators)
+(measured 2026-10-07: `verified` is uniformly `false`; `self_reported` varies —
+gpqa 19/20, deepswe-1.1 14/20, terminal-bench-4.0 10/20, automationbench-aa 0/1).
+`trust`/`modality`/`provenance`/`crossSource` are each `null` when no entry
+carried the respective field, and the whole meta is `null` when the payload has
+no llm-stats entries shape.
+`BenchmarkPayload` carries it as `meta`, `BenchmarkScores` as `meta`, and the
+scores cache persists it. The scores-cache reader carries the same shape guard as
+the catalog reader: a cache lacking the `meta` key (written before the
+payload-meta fields landed) is rejected and refetched, so the four
+payload-derived focus axes never go silently inert; `meta: null` (a
+declared/writing payload) is still accepted. The
+focus-metric trust, modality, provenance and cross-source axes (scoring.md)
+consume it via `cachedSourceInfo`.
+
+The endpoint caps `entries` at `BENCHMARK_ENTRY_CAP` (20) regardless of
+`limit`/`offset`/`page`/`per_page`, so a generic metric can never load more and
+no pagination loop helps. `total_models` still reports the full set, so
+`loadBenchmarkScores` returns `loaded` (the entries actually read) alongside
+`total`; `loaded < total` marks a capped load (annotated to stderr) and the
+post-fetch coverage count is the loadable `loaded`, never the catalog's
+`model_count`.
 
 ### The benchmark catalog (discovery)
 
 `GET https://api.zeroeval.com/leaderboard/benchmarks` (the same backend, no
 `/<id>`) returns the full catalog as a top-level JSON array of ~745 rows:
 `{ benchmark_id, name, description, categories[], modality, max_score, verified,
-model_count, is_community }`. `parseBenchmarkCatalog` narrows it to
-`{ id, name, description, categories, modelCount }`; `loadBenchmarkCatalog`
-follows the same daily cache chain (cache file
-`benchmark-catalog-fetched-data.json`). `/create-agent` uses it to discover the
+model_count, is_community, updated_at, version_count, latest_version_row_count,
+star_count, dataset_id, dataset_org_id, dataset_slug }`. `parseBenchmarkCatalog`
+narrows it to
+`{ id, name, description, categories, modelCount, isCommunity, modality,
+updatedAt, versionCount, latestVersionRowCount, starCount, datasetId,
+datasetOrgId, datasetSlug }` (the
+row's `is_community`, defaulting `false` when absent/non-boolean — consumed by
+the trust axis as `community`, 20/745 true — the row's `modality` as an open
+string, `null` when absent/non-string, so an unknown value survives, the
+maintenance fields `updated_at`/`version_count`/`latest_version_row_count`/
+`star_count`, each `null` when absent, non-string (for `updated_at`) or
+non-finite (for the three counts), and the provenance fields `dataset_id`/
+`dataset_org_id`/`dataset_slug`, each `null` when absent, non-string or empty —
+`dataset_org_id` is the benchmark's owning organisation, the provenance axis's
+owner, with `dataset_slug` as its fallback);
+`loadBenchmarkCatalog` follows the same
+daily cache chain (cache file `benchmark-catalog-fetched-data.json`). The catalog
+cache reader carries a shape guard mirroring `readEndpointsCache`: a cache whose
+sampled entry lacks `isCommunity`, `modality`, any of the four maintenance
+fields or any of the three provenance fields (written before the field landed) is
+rejected and refetched, so a stale
+cache never reports `undefined`; an empty
+catalog is still `null`. `/create-agent` uses it to discover the
 benchmarks relevant to a purpose: `discoverBenchmarks` filters by coverage
 (`modelCount >= 3`), ranks the survivors by IDF-weighted lexical overlap with the
 purpose (a category that names the purpose's skill is the strongest signal), caps
@@ -282,10 +446,10 @@ fresh checkout needs no setup.
 | `cache/openrouter-fetched-data.json` | `{ fetchedAt, source, modelCount, data }` | non-fatal (affected models unranked) |
 | `cache/openrouter-endpoints-fetched-data.json` | `{ fetchedAt, source, slugCount, slugs }` | non-fatal (single-route fallback) |
 | `cache/designarena-fetched-data.json` | `{ fetchedAt, source, categories }` | non-fatal (OR mirror alone) |
-| `cache/writing-fetched-data.json` | `{ fetchedAt, source, scores }` | non-fatal (`writing` unfilled) |
-| `cache/bench-<id>-fetched-data.json` | `{ fetchedAt, source, scores }` | non-fatal (the external metric is unfilled) |
-| `cache/benchmark-catalog-fetched-data.json` | `{ fetchedAt, source, entries }` | non-fatal (discovery skipped) |
-| `cache/<declared-id>-fetched-data.json` | `{ fetchedAt, source, scores }` | non-fatal (the external metric is unfilled) |
+| `cache/writing-fetched-data.json` | `{ fetchedAt, source, scores, total, meta }` | non-fatal (`writing` unfilled) |
+| `cache/bench-<id>-fetched-data.json` | `{ fetchedAt, source, scores, total, meta }` | non-fatal (the external metric is unfilled) |
+| `cache/benchmark-catalog-fetched-data.json` | `{ fetchedAt, source, entries }` (each entry carries `isCommunity`, `modality`, the maintenance fields `updatedAt`/`versionCount`/`latestVersionRowCount`/`starCount` and the provenance fields `datasetId`/`datasetOrgId`/`datasetSlug`) | non-fatal (discovery skipped) |
+| `cache/<declared-id>-fetched-data.json` | `{ fetchedAt, source, scores, total, meta }` | non-fatal (the external metric is unfilled) |
 
 The last two are the registry's generic scores cache (`loadBenchmarkScores`),
 shared by the generic llm-stats benchmark and every declared source.
@@ -324,9 +488,9 @@ the source does not cover.
   the routes that have p50 data, so a stable route with no recent requests
   contributes price weight but no throughput. A degraded cheapest route
   (`status ≠ 0`) drops out entirely until it recovers, and the price can jump.
-  The blend weights by the documented `1/p²` formula, not by the page's observed
-  request counts (which aggregate ALL OpenRouter traffic, `:nitro`/`:floor` and
-  `sort` users included).
+  The blend weights by the `1/p_in²` formula (input price, see "The 1/price²
+  blend"), not by the page's observed request counts (which aggregate ALL
+  OpenRouter traffic, `:nitro`/`:floor` and `sort` users included).
 - **The find route and the page record for the same endpoint id can disagree**
   by a few percent (price revisions, status flips, p50 windows); the page copy
   wins the pool merge, the find row joins only when the page does not list its