omp-llm-role 1.2.0 → 1.4.0

package/docs/dev/writes-and-state.md

@@ -134,41 +134,139 @@ gating — that this plugin never reads (zero occurrences in `src/`, `web/`,
 `tests/`). The plugin's `features.<id>` keys are plugin settings, not omp
 feature flags; do not merge the two.
 
-## The provider-pin auto-configuration (spec #58)
+**`tokenProfile` (spec #81).** A global setting (`tokenProfile.inputTokens` /
+`tokenProfile.outputTokens`, flat dotted keys like the rest) — the per-task
+token profile the explorer's `$/task` column falls back to when a role has no
+recorded usage. `null` (the default) = unset, and the column reads `—`. It is
+validated to `null` or a record whose two fields are finite numbers ≥ 0
+(`tokenProfile: must be null or an object with inputTokens/outputTokens` /
+`tokenProfile.<key>: must be a number >= 0`). It is **display-only**: it never
+feeds the ranking, the reported `$/M` price, or any write. The recorded usage
+takes precedence over it (`resolveTokenProfile`), so once a role has ledger rows
+the setting is ignored for that role.
+
+## The provider-pin auto-configuration (spec #58, routability-verified by #90)
 
 When a role's **expanded** def sets `preferOwnProvider: true` (the
 `providerPinning` capability's recommended bundle) and carries **no** manual
-`providerPin`, the updater binds the role's requests to the chosen model's best
-route by writing that route's provider slug as the selector's `@<slug>` suffix:
+`providerPin`, the updater binds the role's requests to one provider route by
+writing that route's slug as the selector's `@<slug>` suffix:
 
 ```
 openrouter/<id>@<slug>:<level>
 ```
 
 omp's `splitUpstreamRouting` parses the trailing `@<slug>` and applies
-`compat.openRouterRouting = { only: [slug] }`, so the price the ranking assumed
-is the price the request pays. The slug is `bestRoute(chosenModel, def)`'s
-`providerSlug` — the max-value-for-money candidate route (see scoring.md,
-*Route ordering*). The pin is re-derived on every run, so it follows the ranking when the
-chosen model changes. A manual `roles.<role>.providerPin` **wins** over the
-automatic choice and stays a **hard** gate (a model with no matching route is
-ineligible); clearing it resumes the automatic choice.
+`compat.openRouterRouting = { only: [slug] }`. The pin is the best route **the
+probe accepts**, which is the route the role was priced on — the route the walk
+tries first — so the price the ranking assumed is the price the request pays
+*only while the walk does not skip that candidate*; a skipped candidate means the
+route that replaced it pays, at its own price. The pin is re-derived on every
+run, so it follows the ranking when the chosen model changes.
+
+**The walk — the pin is probed before it is written.** The slug is **not** the
+score-max route unconditionally: the updater walks the role's route order
+(`rankedRoutes(chosenModel, def)` — the `vfm` order the chain also uses; see
+scoring.md, *Route ordering*) and probes each candidate with `probeRoute`: a
+one-token completion carrying `provider: { only: [slug] }`, the same request
+skeleton and the same narrow no-allowed-providers pattern as `probeModel`. The
+verdict vocabulary is `probeModel`'s:
+
+- `blocked` — the narrow no-allowed-providers 404 — skips the route and the walk
+  continues to the next candidate;
+- `ok` and **`unknown` are both usable**. A non-404 status (a 429, a 5xx), a 404
+  whose body does not carry the pattern, and a transport throw (a timeout, a
+  connection reset) are all `unknown`: a transient blip must never silently unpin
+  a role, and a rate limit is omp's runtime-fallback domain, not this plugin's.
+
+Verdicts are cached per run by `catalogId@slug`, so a route shared by several
+roles is probed once, and the walk is capped per role at **`ROUTE_PROBE_BUDGET =
+12`** *new* probes (cached verdicts are free) — a pathological pool must not turn
+the daily update into a long probe walk.
+
+**Why a real request.** OpenRouter's endpoints API and its routing table
+disagree: an endpoint can list `status: 0` with a full `supported_parameters`
+list and 100 % uptime, pass the whitelist gate and carry a usable billed price,
+and still 404 for `provider.only` — and the exclusion is dynamic (the same
+request can succeed minutes later). The routing table is visible only in the 404
+body of a request that failed, so a request is the only reliable signal. This is
+exactly the failure the endpoint gates cannot see (spec #90: the live config held
+`deepseek/deepseek-v4.1-flash@relace`, which 404'd on every revert).
+
+**No routable candidate → no pin.** If every candidate is `blocked` (or the
+budget runs out with no clean candidate), no pin is written: the selector stays
+**bare** and OpenRouter routes the request itself, instead of omp failing on an
+unroutable `@<slug>`. `Decision.pinDropped` is true in that case (see *State,
+history, lock*), and the run notes it.
+
+A manual `roles.<role>.providerPin` **wins** over the automatic choice and stays
+a **hard** gate (a model with no matching route is ineligible). It is **never
+probed and never replaced** — the probe applies only to the route the capability
+itself chooses; clearing the manual pin resumes the automatic choice.
 
 **Chain shape.** The primary's chain key is the selector minus the trailing
-`:level` — so it **includes** the pin (`openrouter/<id>@<slug>`). Its values, in
-order:
+`:level` — so it **includes** the pin (`openrouter/<id>@<slug>`, or the bare
+`openrouter/<id>` when the pin was dropped). Its values, in order:
 
-1. the same model on **every** gate-passing route except the pinned best, in
+1. the same model on **every** gate-passing route except the pinned route, in
    score order (the `vfm` route score desc — see scoring.md, *Route ordering*),
 2. then, for each of the `fallbackChainDepth` fallback models, that model on
    **every** gate-passing route, in score order.
 
-Length = `(R_primary − 1) + Σ R_fallback`, where `R` is the number of distinct
-gate-passing provider slugs for that model — data-driven, not a fixed cap.
-Entries dedup by chain value, so two routes sharing a provider slug collapse to
-one. Every entry carries the same `@<slug>` suffix (and the role's thinking level
-where the target advertises it). A model with no gate-passing route contributes
-nothing. The gates are the engine's `rankedRoutes` set: non-degraded
+**Which models are chained (spec #95).** The `fallbackChainDepth` fallback
+**models** come from the role's whole **ranked candidate list** — the
+tier-eligible ranking rows, in value order — not from the probe window (which is
+`1 + fallbackChainDepth` models, the same size as the chain, so a reorder of it
+can never reach a cheaper model at rank 4 or deeper). Two classes of candidate
+are excluded, with the same predicates the probe walk uses: **key-blocked**
+models (the account's key cannot use them) and models whose probe verdict is
+`blocked`. An **unprobed** candidate stays — the chain's deeper entries are
+never probe-verified (they are not a new class: chain entries are already
+route-level unprobed, and omp fails over on an unroutable entry). `after` is
+that list **below the chosen model** — the leader on a fresh pick, the kept
+current candidate on a kept role (`kept-eligible` / `kept-margin`) — so the
+`after` set and the floor both key off the model the selector actually names —
+the previous `pool.slice(chosenIdx + 1)` keyed off the same index in the
+probe-clean pool — and candidates ranked above the chosen are never chained. The
+set is then split:
+
+```
+cheaper = after where priceEff <= chosen.priceEff AND q >= chosen.q · fallbackQualityFloor
+pricier = after where priceEff > chosen.priceEff            (never floor-filtered)
+chain   = [...cheaper, ...pricier].slice(0, fallbackChainDepth)
+```
+
+so a cheaper-or-equal fallback keeps the role's quality (the floor is a fraction
+of the chosen model's **own** `q`, so it reads the same on every role's quality
+scale) and a cheap-but-weak candidate set cannot starve the chain (the pricier
+bucket always fills any remaining depth with the next-best by value). Both
+buckets keep value order, so a cheaper candidate deep in the ranking is reached
+only when fewer than `fallbackChainDepth` cheaper-or-equal candidates clear the
+floor — a bounded search, not a cheapest-first sort. A candidate priced exactly
+at the chosen model's price whose `q` is below the floor is in **neither** bucket
+and drops out. `fallbackQualityFloor` (a number in `[0, 1]`, default `0.9`; `0`
+disables the floor, `1` requires an exact quality match — validated, never
+silently clamped) is documented in the README's settings table. The floor gates
+**chain membership only**: the primary choice, the hysteresis decision, the
+ranking, the probe walk's stop condition and budget, the chain key, the route
+expansion, the thinking levels, the dedup and the prune-on-pin-change are all
+unchanged.
+
+The exclusion is **by pinned slug, not by position**: the probe can pin a route
+other than the score leader (so the leader rides as a fallback entry while the
+pinned route does not), and with a **dropped pin** (the auto-pin applied, but no
+candidate was routable) **no** route is excluded — the primary contributes every
+gate-passing route. Length = `(R_primary − 1) + Σ R_fallback`, or `R_primary + Σ
+R_fallback` with a dropped pin, where `R` is the number of distinct gate-passing
+provider slugs for that model — data-driven, not a fixed cap. Entries dedup by
+chain value, so two routes sharing a provider slug collapse to one. Every entry
+carries the same `@<slug>` suffix (and the role's thinking level where the target
+advertises it).
+**Chain entries are not probed** — only the primary pin is: the chain carries
+many routes and one unroutable entry is survivable (omp fails over), so probing
+it would multiply the request cost for no benefit (out of scope, spec #90). A
+model with no gate-passing route contributes nothing. The gates are the engine's
+`rankedRoutes` set: non-degraded
 (`status === 0`), the endpoint capability filters (`tools` / `minContextTokens` /
 `minOutputTokens`), a usable billed price, and the `maxPriceUsdPerM` cap
 (thinking-adjusted). The soft `filters.preferredMinThroughput` floor is a
@@ -179,16 +277,28 @@ pool to allowed providers before ranking, which is what makes `rankedRoutes`
 whitelist-aware. The probe walk target is unchanged (`1 + fallbackChainDepth` —
 it counts models, not routes).
 
+**Outside the auto-pin.** The chosen model contributes nothing when the auto-pin
+does not apply: with a manual `providerPin` (`autoPinApplies` is false, so the
+route probe does not run either) or with the capability off, the chain carries
+**one entry per `fallbackChainDepth` fallback model** —
+`openrouter/<fallbackId>@<manualSlug>` with a manual pin, the bare
+`openrouter/<fallbackId>` with none — each with the role's thinking level where
+the target advertises it, and no route of the chosen model at all.
+
 **Prune on a pin change.** Because the chain key includes the pin, a pin change
 changes the key; the updater's chain prune (`state.pluginWrittenChainKeys`) drops
 the old key so a stale pinned chain does not linger.
 
-**No-ops.** The auto-pin writes nothing when: the capability is off
-(`preferOwnProvider` not `true`); a manual `providerPin` is set; the role is
-locked or disabled; the selector is not an OpenRouter selector; the model has no
-route data; no route passes the role's endpoint filters; or the best route has no
-usable billed price. With the capability off the written config is byte-identical
-to before.
+**No-ops.** The auto-pin writes nothing — and **the route probe is never
+consulted** — when: the capability is off (`preferOwnProvider` not `true`); a
+manual `providerPin` is set; the role is locked or disabled; the selector is not
+an OpenRouter selector; the model has no route data; no route passes the role's
+endpoint filters; or no candidate has a usable billed price (the last three leave
+`rankedRoutes` empty, so there is no candidate to probe, and `pinDropped` stays
+false — the pin is simply absent). The case where the model *has* candidates but
+none is routable is different: that is the **dropped pin** above — still no pin,
+but recorded (`pinDropped`). With the capability off the written config is
+byte-identical to before.
 
 ## State, history, lock
 
@@ -209,6 +319,7 @@ settings entry the global behavior is unchanged.
 |---|---|---|
 | `llm-role-state.json` | `{ lastRunDay, managedRoles, roleLastSelector, pluginWrittenChainKeys, previousModelRoles, managedDisabledAgents }` | day gate, managed-role set, last selectors, plugin-owned chain keys, pre-write `modelRoles` snapshot, **managed `task.disabledAgents` names** |
 | `llm-role-history.jsonl` | one JSON row per completed run: `{ ts, trigger, keyMeta{isFreeTier, limitRemaining, creditsRemaining}, decisions[] }` | append-only decision log |
+| `llm-role-usage.jsonl` | one JSON row per assistant turn: `{ ts, role, model, provider, input, output, cacheRead, cacheWrite, cost }` | the per-role usage ledger (spec #81) the explorer's per-task price resolves its token profile from |
 | `llm-role-projects.json` | `{ projects: [{ root, lastUsed }] }` | the plugin's memory of the projects where `/project-roles` was used (the explorer's scope list) |
 | `.llm-role-refresh.lock` | `O_EXCL` create, holds `"<pid> <iso>"` | serializes concurrent session starts; stale (> 60 s) locks are unlinked and retried once |
 | `llm-role-error-reports.json` | `{ reports: [{ fingerprint, label, title, firstSeen, lastSeen, createdDay?, offeredDay?, count, action, issueNumber?, issueUrl? }], dayCount: { day, count } }` | what the opt-in `errorReporting` path filed, offered or suppressed |
@@ -241,6 +352,24 @@ settings entry the global behavior is unchanged.
   that feeds the 3-creations-per-day cap. Nothing is written when
   `errorReporting` is `off`.
 
+- `llm-role-usage.jsonl` is the per-role usage ledger (spec #81,
+  `src/usage-ledger.ts`). Unlike the two ledgers above it is **not** global: it
+  follows the plugin's state location — `<cwd>/.omp` in project mode,
+  `agentDir()` otherwise — so a project's usage stays separate from the
+  user-level profile. The extension records one row per assistant turn on
+  `turn_end` and `session_shutdown` (the backstop for a headless one-turn run),
+  read from `sessionManager.getBranch()` — the same entries the session jsonl
+  persists — into `resolveProjectDir() ?? agentDir()`, deduped by `ts|model` so
+  the two hooks cannot double-record. The file is **append-only** (a crash never
+  corrupts it) and **bounded**: `appendUsageRow` compacts to the newest
+  `USAGE_LEDGER_CAP = 2000` rows via a temp file + atomic rename, so a crash
+  mid-compaction leaves the original intact. Recording is **best-effort** — a
+  failure never breaks a session, and `appendUsageRow` never throws. Reading is
+  tolerant: a missing, unreadable or malformed file degrades to `[]`, and an
+  unparseable or wrongly-shaped line is skipped. The explorer reads the active
+  scope's ledger (a project scope its own `<root>/.omp`), so the per-task column
+  follows the scope.
+
 - `previousModelRoles` is the full snapshot of the last `modelRoles` block
   **before** the plugin changed it — a manual rollback aid; there is no rollback
   command.
@@ -264,7 +393,13 @@ settings entry the global behavior is unchanged.
   ids the role's endpoint filters dropped (no capable standard-tier route), rendered
   `; endpoint-blocked: …` on the decision line; `pinBlocked[]` records the ranking ids
   the role's provider pin dropped (no route matching the pin), rendered
-  `; pin-blocked: …`.
+  `; pin-blocked: …`. `routeBlockedSlugs[]` records the provider slugs the auto-pin's
+  route probe rejected on the chosen model, in walk order and deduped by slug
+  (two routes of one model can share a slug), rendered `; route-blocked: …`;
+  `pinDropped` is `true` when the auto-pin applies and the model has gate-passing
+  routes but no candidate survived the probe (or the `ROUTE_PROBE_BUDGET` ran
+  out), rendered `; pin dropped (no routable route)` — the selector is bare in
+  that case. Both are empty/false when no probe ran.
 - Role removal (`/remove-agent`) goes through `removeRoleSettings` in
   `src/role-settings.ts` — the same validate/backup/atomic-write path as
   `writeRoleSettings`, deleting the role's flat dotted keys and any nested entry.