omp-llm-role 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -8
- package/docs/dev/architecture.md +25 -4
- package/docs/dev/ci.md +12 -1
- package/docs/dev/current-state.md +63 -26
- package/docs/dev/data-sources.md +33 -4
- package/docs/dev/explorer.md +22 -4
- package/docs/dev/quirks.md +32 -0
- package/docs/dev/releasing.md +31 -30
- package/docs/dev/scoring.md +98 -35
- package/docs/dev/testing.md +70 -23
- package/docs/dev/writes-and-state.md +56 -0
- package/docs/llm-role-rankings.md +127 -127
- package/package.json +82 -22
- package/src/availability.ts +87 -9
- package/src/cli/create-role.ts +1 -1
- package/src/engine.ts +157 -76
- package/src/explorer/explain.ts +24 -6
- package/src/explorer/server.ts +15 -2
- package/src/extension.ts +58 -15
- package/src/features.ts +9 -7
- package/src/provider-display.ts +67 -0
- package/src/settings.ts +8 -2
- package/src/updater.ts +91 -13
- package/web/app.js +24 -1
package/README.md
CHANGED
|
@@ -48,6 +48,16 @@ omp plugin install omp-llm-role@gott50-plugins
|
|
|
48
48
|
session actually booted on the previous `default` selector (an explicit
|
|
49
49
|
`omp --model X` is never clobbered), and only on a real change. Disable with
|
|
50
50
|
`activateDefaultOnEmptySession=false`.
|
|
51
|
+
- **Provider in the model display.** When the live session model carries a
|
|
52
|
+
provider pin — a hand-set `providerPin`, or the automatic pin the
|
|
53
|
+
`providerPinning` capability writes — the status line's model segment shows
|
|
54
|
+
the route next to the model name, e.g. `DeepSeek V4.1 Flash @coreweave/fp8`.
|
|
55
|
+
The suffix is applied on session start, refreshed when a fallback switches the
|
|
56
|
+
model, and re-derived at the start of every turn, so it self-heals after omp
|
|
57
|
+
silently restores the primary model following a fallback cooldown. An unpinned
|
|
58
|
+
model's display is unchanged, and the suffix is display-only: it rides the
|
|
59
|
+
model object's `name`, so attribution, usage recording and model comparison
|
|
60
|
+
are unaffected.
|
|
51
61
|
- **Surgical writes.** Only managed role lines, managed chain keys, and the
|
|
52
62
|
plugin-managed `task.disabledAgents` names change; comments, blank lines and
|
|
53
63
|
unknown keys stay byte-identical, and a no-change run writes nothing.
|
|
@@ -126,7 +136,7 @@ Per-role knobs (`roles.<name>.*`):
|
|
|
126
136
|
| `filters.minOutputTokens` | `0` (off) | Drop routes whose endpoint output ceiling is below this (tokens) |
|
|
127
137
|
| `filters.maxPriceUsdPerM` | `0` (off) | Drop a model whose thinking-adjusted blend exceeds this ($/M) |
|
|
128
138
|
| `thinking` | shipped level | Thinking level appended to the selector (`off`…`max`, `auto`) |
|
|
129
|
-
| `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible |
|
|
139
|
+
| `providerPin` | `""` (unset) | Pin the role's requests to one OpenRouter provider route (`@<slug>`, may be tiered like `deepinfra/fp8`). The role is then priced and gated by that route — its billed price and p50 throughput, not the `1/price²` blend — and a model with no matching route is ineligible. A manual pin **overrides** the automatic pin the `providerPinning` capability would write |
|
|
130
140
|
| `cacheHitRate` | `0` (off) | Assumed cache-hit rate (0–1) for the role's input tokens; each route's effective input price becomes `h·cacheRead + (1−h)·input` before the 3:1 billed blend and the thinking factor. The `1/price²` weight basis stays the **listed** input price (the router's sort key), so a cheap cache on a lightly-weighted route cannot dominate; a route with no cache-read price keeps its full input price. `0`/absent = off (byte-identical ranking) |
|
|
131
141
|
| `lambda` | derived | Explicit λ ($ per quality point) override |
|
|
132
142
|
| `locked` | `false` | Leave the role alone: rank it but never rewrite its selector or chain (explorer toggle) |
|
|
@@ -142,7 +152,7 @@ defaults.
|
|
|
142
152
|
|---|---|---|
|
|
143
153
|
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | Routes that cannot call tools and routes whose output ceiling is below 16 384 tokens leave the pool — a reasoning model is not cut off mid-thought |
|
|
144
154
|
| `cachePricing` | `cacheHitRate: 0.5` | The role is priced on a cache-heavy agent loop's invoice (the endpoint's `cacheReadPrice` blended in) rather than the sticker price |
|
|
145
|
-
| `providerPinning` | `preferOwnProvider: true` |
|
|
155
|
+
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | Each request is bound to the model's **best provider route** — the max-value-for-money route (the `vfm` score, with a soft 50 tok/s floor that deprioritizes a pathologically slow route without dropping it) — the selector gains `@<slug>` — and the fallback chain carries the primary model on every gate-passing route except the pinned best, then each fallback model on every gate-passing route, so a down provider fails over to another provider of the same model before the chain switches models. The model is priced **and measured** on that route (its billed price and p50 throughput, so `q` moves too), falling back to the default blend only when it has no candidate route |
|
|
146
156
|
| `costCap` | `filters.maxPriceUsdPerM: 10` | The priciest tail of the eligible pool leaves it, so the pick cannot be an expensive outlier |
|
|
147
157
|
|
|
148
158
|
Precedence is one rule:
|
|
@@ -156,10 +166,13 @@ and otherwise when the global `features.<id>` is `true` — so a per-role `false
|
|
|
156
166
|
beats a global `true`. The preset fills only knobs the role does not already
|
|
157
167
|
set, and an explicit key is merged last, so it always wins. `providerPinning`'s
|
|
158
168
|
recommended setting is a **soft per-model policy** (`preferOwnProvider`), not a
|
|
159
|
-
hard `providerPin`: it
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
169
|
+
hard `providerPin`: it prices a model on its **best route** (the max-`vfm` route
|
|
170
|
+
— the value-for-money score, with the soft 50 tok/s floor — among the
|
|
171
|
+
filter-passing, non-degraded, priced routes the account's
|
|
172
|
+
allowed-providers whitelist permits) — the route's billed price
|
|
173
|
+
**and** its p50 throughput, so `q` moves as well as `priceEff` — and never makes
|
|
174
|
+
a model ineligible (the hard `providerPin` stays available by hand, and overrides
|
|
175
|
+
the automatic choice). `minContextTokens` is deliberately **not** part of `endpointCeilings` —
|
|
163
176
|
the complaint it would answer is an endpoint advertising a *fraction of the
|
|
164
177
|
model's own window*, which an absolute token floor cannot express, so it stays a
|
|
165
178
|
hand-set knob.
|
|
@@ -465,6 +478,13 @@ The plugin keeps only models the OpenRouter key can actually run:
|
|
|
465
478
|
whitelist** is invisible to the key endpoint and the unauthenticated catalog, and a
|
|
466
479
|
real request's 404 is the only reliable signal. Blocked candidates are excluded from
|
|
467
480
|
selection and chains; a role with no clean candidate is left untouched.
|
|
481
|
+
- **Allowed-providers whitelist** — the account's OpenRouter **allowed-providers
|
|
482
|
+
privacy whitelist** (openrouter.ai/settings/privacy) is learned once per run (a
|
|
483
|
+
canary request that bills no tokens) and every model's route pool is narrowed to
|
|
484
|
+
allowed providers before ranking, so a pin or a fallback hop is never written on a
|
|
485
|
+
provider the account cannot use. The run notes `OpenRouter allowed-providers
|
|
486
|
+
whitelist active — N providers; M routes pruned` (or `… whitelist: no whitelist`).
|
|
487
|
+
With no whitelist set — or when the harvest fails — nothing changes.
|
|
468
488
|
- **Explorer key marking** — the explorer overlays every ranked model with a
|
|
469
489
|
**usable** / **key-blocked** / **unknown** badge derived from the
|
|
470
490
|
key-authenticated `GET /api/v1/models` (the account setting **"Filter the model
|
|
@@ -485,8 +505,10 @@ The plugin keeps only models the OpenRouter key can actually run:
|
|
|
485
505
|
[docs/dev/quirks.md](docs/dev/quirks.md)).
|
|
486
506
|
- **Provider whitelist.** The account's allowed-providers whitelist blocks many
|
|
487
507
|
models, so the report's #1 and the written selector can differ — judge weights
|
|
488
|
-
on the reachable pool.
|
|
489
|
-
|
|
508
|
+
on the reachable pool. The updater prunes *routes* to the whitelist (see
|
|
509
|
+
Availability), but the report has no token and is not whitelist-aware.
|
|
510
|
+
- **Provider pinning opts out of auto-Exacto.** A `providerPin` — or the
|
|
511
|
+
automatic pin the `providerPinning` capability writes — puts a trailing
|
|
490
512
|
`@<slug>` on the selector, which omp applies as OpenRouter
|
|
491
513
|
`provider: { only: [slug] }` routing. `only` is exclusive, so a pinned role's
|
|
492
514
|
requests bypass OpenRouter's automatic Exacto tool routing (which routes tool
|
package/docs/dev/architecture.md
CHANGED
|
@@ -29,13 +29,14 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
|
|
|
29
29
|
| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
|
|
30
30
|
| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
|
|
31
31
|
| `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
|
|
32
|
-
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe,
|
|
32
|
+
| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, the keyed-catalog availability primitive, and the allowed-providers whitelist harvest. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `fetchAllowedProviders`, `providerAllowed`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
|
|
33
33
|
| `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
|
|
34
34
|
| `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
|
|
35
35
|
| `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
|
|
36
36
|
| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
|
|
37
|
-
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
37
|
+
| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Also subscribes to `session_start`, `retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` to keep the live model's display name suffixed with its provider pin (`src/provider-display.ts`); each handler mutates the live model's `name` in place and never calls `pi.setModel` (which would wipe omp's in-flight fallback state and clobber the thinking level). Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
|
|
38
38
|
| `src/guards.ts` | The package's one type guard: `isRecord`. |
|
|
39
|
+
| `src/provider-display.ts` | The provider display (spec #77): keep the live session model's `name` suffixed with its upstream provider pin, so omp's model segment renders e.g. `DeepSeek V4.1 Flash @coreweave/fp8`. Pure and dual-runtime (no SDK import). Exports `providerPinOf` (the single `compat.openRouterRouting.only[0]`, else `compat.vercelGatewayRouting.only[0]`; null unless exactly one non-empty string), `providerPinOfSelector` (the trailing `@<slug>` of a selector, after stripping a thinking suffix), `withProviderSuffix` (strip any previous ` @<slug>` then append, idempotent), and `applyProviderDisplay` (mutate a record's `name`; no-op otherwise). |
|
|
39
40
|
| `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
|
|
40
41
|
| `src/role-archetypes.ts` | Purpose → weight archetype table (10 sets) + keyword fitting. Exports `ARCHETYPES`, `FALLBACK_ARCHETYPE`, `fitArchetype`, `archetypeById`, and the `Archetype`/`ArchetypeMatch` types. |
|
|
41
42
|
| `src/agent-file.ts` | Agent `.md` rendering/placement. Exports `renderAgentFile`, `writeAgentFile`, `removeAgentFile`, `isReadOnlyTools`, `userAgentsDir`, `projectAgentsDir`, `READ_ONLY_TOOLS`, `RESERVED_AGENT_NAMES`, `AGENT_NAME_RE`, and the `AgentFileSpec`/`AgentWriteResult`/`AgentRemoveResult` types. |
|
|
@@ -116,7 +117,22 @@ The omp extension injects the production `Deps`; tests inject fakes.
|
|
|
116
117
|
fetchKeyMeta(token)` → `tierGate` → `"billed" | "free" | "none"`; `"none"`
|
|
117
118
|
aborts.
|
|
118
119
|
5. **Catalog + probe.** `deps.getCatalog()` → `filterCatalog(catalog, tier)`;
|
|
119
|
-
`enrichThinkingLevels(rank.models, catalog)`.
|
|
120
|
+
`enrichThinkingLevels(rank.models, catalog)`.
|
|
121
|
+
**Whitelist prune.** The updater then harvests the account's
|
|
122
|
+
allowed-providers privacy whitelist once per run
|
|
123
|
+
(`deps.getAllowedProviders ? deps.getAllowedProviders(token, canaryModelId)
|
|
124
|
+
: fetchAllowedProviders(token, canaryModelId)`, a canary 404 that bills no
|
|
125
|
+
tokens) and, when it is non-null, replaces every model's route pool with the
|
|
126
|
+
routes whose `providerSlug` matches a whitelist entry (`providerAllowed`,
|
|
127
|
+
prefix match `slug === entry || slug.startsWith(entry + "/")`,
|
|
128
|
+
non-mutating). The prune is
|
|
129
|
+
the single change point — `roleRoute`, `bestRoute`, `rankedRoutes` and
|
|
130
|
+
`endpointFilteredModel` all read `m.routes` — so the soft basis, the auto-pin
|
|
131
|
+
and the fallback chains all become whitelist-aware. A `null` whitelist (no
|
|
132
|
+
whitelist set, harvest failed) leaves every pool untouched. The run notes
|
|
133
|
+
`OpenRouter allowed-providers whitelist active — N providers; M routes pruned`
|
|
134
|
+
(or `… whitelist: no whitelist`).
|
|
135
|
+
Per role, candidates are
|
|
120
136
|
`resolveVariant`-resolved and then verified by a bounded `probeModel` walk
|
|
121
137
|
(current selector first, then rank order; budget `PROBE_BUDGET = 12`; verdicts
|
|
122
138
|
cached per run). Only the narrow no-allowed-providers 404 disqualifies; every
|
|
@@ -202,6 +218,7 @@ type Deps = {
|
|
|
202
218
|
getSettings?(): Promise<Record<string, unknown>>;
|
|
203
219
|
getKeyMeta?(token: string): Promise<KeyMeta>;
|
|
204
220
|
probeModel?(token: string, catalogId: string): Promise<ProbeVerdict>;
|
|
221
|
+
getAllowedProviders?(token: string, modelId: string): Promise<Set<string> | null>;
|
|
205
222
|
// Host coupling (extension only): apply a concrete selector to the live session model.
|
|
206
223
|
applySessionModel?(selector: string, previous: string | null): Promise<void>;
|
|
207
224
|
};
|
|
@@ -221,7 +238,11 @@ before `catalogFromOmpModelsJson`.
|
|
|
221
238
|
`startExplorer`), `/create-agent` (architect → `createAgent` → in-process
|
|
222
239
|
`runUpdater`), `/remove-agent` (`removeAgent` → in-process `runUpdater`),
|
|
223
240
|
`/project-roles` (profile architect → `setupProject` → in-process
|
|
224
|
-
`runUpdater`), `session_shutdown` (closes the explorer handle).
|
|
241
|
+
`runUpdater`), `session_shutdown` (closes the explorer handle). The provider
|
|
242
|
+
display adds four event subscriptions — `session_start`,
|
|
243
|
+
`retry_fallback_applied`, `retry_fallback_succeeded` and `turn_start` — each
|
|
244
|
+
applying the live model's provider pin to its `name` (see
|
|
245
|
+
[`quirks.md`](quirks.md) for why `turn_start` is the self-healing hook).
|
|
225
246
|
- **CLI**: `src/cli/llm-role-rank.ts` (report), `src/cli/create-role.ts`
|
|
226
247
|
(authoring). The updater and explorer have no CLI shim — they run through
|
|
227
248
|
`/refresh-roles` and `/explore-roles`.
|
package/docs/dev/ci.md
CHANGED
|
@@ -141,10 +141,21 @@ exit 1, so the command is safe to run unattended.
|
|
|
141
141
|
is that version's CHANGELOG section. This is also the recovery path for a
|
|
142
142
|
failed publish: the tag exists, so no new version is needed.
|
|
143
143
|
|
|
144
|
+
A successful `npm publish` is **not** immediately visible: npm prints "Your
|
|
145
|
+
package is being processed and may take a few minutes to become available",
|
|
146
|
+
and registry reads can serve the pre-publish packument for several minutes
|
|
147
|
+
(third-party mirrors such as jsDelivr lag longer). The step's log is the
|
|
148
|
+
discriminator — a direct publish prints `publishing to
|
|
149
|
+
https://registry.npmjs.org/ with tag latest`, `Signed provenance statement`
|
|
150
|
+
and `+ <pkg>@<version>`; a staged publish is a separate `npm stage publish`
|
|
151
|
+
and prints none of those.
|
|
152
|
+
|
|
144
153
|
The dispatch resolves the workflow **from the ref**, so the tag's commit must
|
|
145
154
|
contain `.github/workflows/ci.yml`. A tag cut before this workflow existed
|
|
146
155
|
cannot be dispatched (`HTTP 422: Workflow does not have 'workflow_dispatch'
|
|
147
|
-
trigger`) — the recovery path only works for tags cut after it landed.
|
|
156
|
+
trigger`) — the recovery path only works for tags cut after it landed. The
|
|
157
|
+
workaround is to repoint the tag at a commit that contains the workflow (done
|
|
158
|
+
once, for `v1.1.0` → `4f78ce2`).
|
|
148
159
|
|
|
149
160
|
`tests/ci.test.ts` asserts the offline half: the workflow's structure (jobs,
|
|
150
161
|
triggers, permissions, the command it runs), the hook/workflow command parity,
|
|
@@ -1,15 +1,45 @@
|
|
|
1
|
-
# Current state (2026-10-
|
|
1
|
+
# Current state (2026-10-11)
|
|
2
2
|
|
|
3
3
|
Dated snapshot of the ranking and the landed work as of the date in the heading —
|
|
4
4
|
matched/priced/eligible counts, per-role leaders, the open defects, and the
|
|
5
5
|
per-issue work log below. It is refreshed by
|
|
6
6
|
regenerating the report (`node src/cli/llm-role-rank.ts --out docs/llm-role-rankings.md`) and
|
|
7
7
|
copying the new numbers here; release prose belongs in
|
|
8
|
-
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The
|
|
9
|
-
below are the 2026-10-
|
|
10
|
-
#17/#18/#19/#20 wave and the #34–#40 focus-axis/capability-flag work;
|
|
11
|
-
|
|
8
|
+
[`../../CHANGELOG.md`](../../CHANGELOG.md), not in this file. The model-level
|
|
9
|
+
ranking numbers below are the 2026-10-09 measurement (the input-price blend basis
|
|
10
|
+
#21, the #17/#18/#19/#20 wave and the #34–#40 focus-axis/capability-flag work);
|
|
11
|
+
the provider-pin numbers are the 2026-10-11 measurement (spec #71; see the top
|
|
12
|
+
bullet). Neither is re-derived on read.
|
|
12
13
|
|
|
14
|
+
- Provider ordering: value-for-money route score (2026-10-11, spec #71): the
|
|
15
|
+
auto-pin route basis changed from the route's own value (`q_route −
|
|
16
|
+
λ·priceEff_route`) to a min/max-normalized **value-for-money score** `vfm =
|
|
17
|
+
wT·tNorm − wP·pNorm` (`wT = wTput/(wPrice+wTput)`, `wP =
|
|
18
|
+
wPrice/(wPrice+wTput)`; `tNorm`/`pNorm` min/max over the candidate set), with a
|
|
19
|
+
new **soft** `filters.preferredMinThroughput` floor (routes at or above sort
|
|
20
|
+
first; below-floor routes are kept and sorted last) that the `providerPinning`
|
|
21
|
+
preset sets to 50. `routeValue` is deleted. The model-level ranking is
|
|
22
|
+
unchanged — the report is not regenerated (the ranking CLI does not apply
|
|
23
|
+
capability features and no shipped role sets `preferOwnProvider`). Measured
|
|
24
|
+
2026-10-11 on the current endpoints cache with the live lock's expanded defs
|
|
25
|
+
(all four capabilities on, `cacheHitRate 0.9`; `deepseek/deepseek-v4.1-flash`,
|
|
26
|
+
11 allowed providers, 12 allowed routes):
|
|
27
|
+
`morph/fp8` ($0.0840,
|
|
28
|
+
103 tok/s) wins `default`, `smol`, `commit`, `task`, `advisor`; `coreweave/fp8`
|
|
29
|
+
($0.1978, 185.5 tok/s) wins `slow`, `tiny`, `designer`; `decart/fp4` ($0.0639,
|
|
30
|
+
72 tok/s) wins `vision`, `plan`.
|
|
31
|
+
|
|
32
|
+
- Auto-pin chain expansion (2026-10-10, issue #69): with `providerPinning` on,
|
|
33
|
+
the fallback chain no longer stops at three providers per model — the primary
|
|
34
|
+
model contributes every gate-passing route except the pinned best, then each
|
|
35
|
+
`fallbackChainDepth` fallback model contributes every gate-passing route, in
|
|
36
|
+
score order (the `vfm` route score — spec #71 replaced the original
|
|
37
|
+
route-value order) and deduped by provider slug, so a down provider fails over
|
|
38
|
+
to another provider of the same model before the chain switches models. Chain
|
|
39
|
+
length is data-driven (`(R_primary − 1) + Σ R_fallback`) instead of the fixed
|
|
40
|
+
`2 + 3·fallbackChainDepth`; the manual-`providerPin` and no-pin paths are
|
|
41
|
+
unchanged. The ranking numbers are unchanged — the chain is a write-side
|
|
42
|
+
shape, not a ranking input.
|
|
13
43
|
- Opt-in error reporting (2026-10-08, issue #48): a new `errorReporting` setting
|
|
14
44
|
(`off` — the default — `ask`, `auto`) turns a genuinely unexpected failure — a
|
|
15
45
|
command handler throwing, or a `runUpdater` abort whose `RunResult.defect` is
|
|
@@ -73,26 +103,33 @@ bullets) and are not re-derived on read.
|
|
|
73
103
|
`src/features.ts`) is the single source of truth and `expandFeatures` the
|
|
74
104
|
single precedence implementation (`effective def = shipped default ⊕
|
|
75
105
|
preset(enabled flags) ⊕ explicit user keys`; a per-role `false` beats a global
|
|
76
|
-
`true`; an explicit knob always wins). Grounding on the 2026-10-
|
|
77
|
-
(`default` role,
|
|
106
|
+
`true`; an explicit knob always wins). Grounding on the 2026-10-09 cache
|
|
107
|
+
(`default` role, 146 eligible; the `providerPinning` row is re-measured on the
|
|
108
|
+
2026-10-11 cache):
|
|
78
109
|
|
|
79
110
|
| Flag | Recommended settings | Measured effect |
|
|
80
111
|
|---|---|---|
|
|
81
|
-
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` |
|
|
82
|
-
| `cachePricing` | `cacheHitRate: 0.5` |
|
|
83
|
-
| `providerPinning` | `preferOwnProvider: true` | per-model policy, never changes eligibility;
|
|
84
|
-
| `costCap` | `filters.maxPriceUsdPerM: 10` |
|
|
112
|
+
| `endpointCeilings` | `filters.tools: true`, `filters.minOutputTokens: 16384` | 146 → 135 eligible, leader unchanged (`deepseek-v4.1-flash`, 0.37 $/M) |
|
|
113
|
+
| `cachePricing` | `cacheHitRate: 0.5` | 146 eligible, leader unchanged, its `priceEff` 0.3732 → 0.3522 (−5.6 %) |
|
|
114
|
+
| `providerPinning` | `preferOwnProvider: true`, `filters.preferredMinThroughput: 50` | per-model policy, never changes eligibility; prices each model on its **best route** (max `vfm` among the filter-passing, non-degraded, priced routes, cap-respecting; the soft 50 tok/s floor sorts below-floor routes last; ties by slug then id) — 144 of 144 routed models have a candidate route (2026-10-11), so the soft basis covers the whole routed field |
|
|
115
|
+
| `costCap` | `filters.maxPriceUsdPerM: 10` | 146 → 137 eligible (the top ≈6 %), leader unchanged; pool p90 7.8, p95 11.1 $/M, every leader 0.20–3.71 $/M |
|
|
85
116
|
|
|
86
117
|
`providerPinning`'s recommended setting is a **soft per-model policy**
|
|
87
118
|
(`preferOwnProvider`), not a hard `providerPin`: a hard pin would make a model
|
|
88
|
-
with no matching route ineligible and empty
|
|
89
|
-
policy
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
`
|
|
94
|
-
|
|
95
|
-
|
|
119
|
+
with no matching route ineligible and empty most of a role's pool, so the
|
|
120
|
+
policy prices a model on its **best route** — the max-`vfm` route (the
|
|
121
|
+
value-for-money score, with the soft 50 tok/s floor sorting below-floor routes
|
|
122
|
+
last) among the filter-passing, non-degraded, priced routes (thinking-adjusted
|
|
123
|
+
price ≤ `maxPriceUsdPerM`; ties by slug then id) — and falls back to the
|
|
124
|
+
`1/price²` blend only when the model has no candidate route. The basis is soft:
|
|
125
|
+
it never changes eligibility, and on the 2026-10-11 cache every one of the 144
|
|
126
|
+
routed models has a candidate route, so the capability covers the whole routed
|
|
127
|
+
field (the old own-lab slug-prefix rule matched only 51 of 149). `minContextTokens`
|
|
128
|
+
is deliberately not in the `endpointCeilings` bundle (an absolute floor cannot
|
|
129
|
+
express the video's fraction-of-the-model's-own-window complaint). The flags
|
|
130
|
+
are settable from `create-role.ts`, `/create-agent`, `/project-roles`
|
|
131
|
+
(`--feature`/`--list-features`) and the explorer's Features panel; the ranking
|
|
132
|
+
numbers are unchanged until a flag is turned on.
|
|
96
133
|
|
|
97
134
|
- Cross-source axis (2026-10-07, issue #38): the focus-metric gate gained the
|
|
98
135
|
cross-source axis — the zeroeval per-benchmark payload's **second, independent**
|
|
@@ -616,18 +653,18 @@ bullets) and are not re-derived on read.
|
|
|
616
653
|
0.663, margin 0.039 > `switchMargin`), because the incumbent has no Design
|
|
617
654
|
Arena data and takes the neutral fill while MiMo carries a measured 0.97.
|
|
618
655
|
The other nine roles are untouched (only `designer` weights `website`).
|
|
619
|
-
-
|
|
620
|
-
2026-10-
|
|
656
|
+
- 408 llm-stats models; OpenRouter matched 155/408 (throughput), 153 priced (the
|
|
657
|
+
2026-10-08 snapshot read 155/407 — the daily cache moved, not the code; the
|
|
621
658
|
#34–#40 axes and flags leave every ranking byte-identical until a flag is
|
|
622
659
|
turned on).
|
|
623
660
|
- Eligible per role: 146 (vision 75, designer 89, image-input filter).
|
|
624
661
|
- Value-ranking leaders (this report, thinking-adjusted prices; the *ranking*
|
|
625
662
|
leaders, before the account's provider whitelist drops the blocked ones):
|
|
626
|
-
`default` DeepSeek-V4.1-Flash (0.
|
|
627
|
-
`slow`
|
|
628
|
-
Muse Spark 1.3 (0.
|
|
629
|
-
|
|
630
|
-
Muse Spark 1.3 (0.
|
|
663
|
+
`default` DeepSeek-V4.1-Flash (0.810), `smol` DeepSeek-V4.1-Flash (0.729),
|
|
664
|
+
`slow` DeepSeek-V4.1-Flash (0.832), `vision` DeepSeek-V4.1-Flash (0.730),
|
|
665
|
+
`plan` Muse Spark 1.3 (0.793), `commit` DeepSeek-V4.1-Flash (0.762), `tiny`
|
|
666
|
+
Muse Spark 1.1 (0.791), `task` DeepSeek-V4.1-Flash (0.754), `advisor`
|
|
667
|
+
Muse Spark 1.3 (0.787), `designer` DeepSeek-V4.1-Flash (0.787).
|
|
631
668
|
- Thinking-adjusted pricing landed (2026-09-27): the suffix table moved into
|
|
632
669
|
`DEFAULT_ROLES` as a per-role `thinking` field, and the price axis scales by
|
|
633
670
|
the level's factor for thinking-capable models — `slow` (`:max`, ×7.86)
|
package/docs/dev/data-sources.md
CHANGED
|
@@ -159,8 +159,11 @@ probability proportional to `1/price²`). The weight basis is the **input
|
|
|
159
159
|
|
|
160
160
|
A role with a `providerPin` does **not** use this blend: it prices the
|
|
161
161
|
model's matching route directly — the route's billed 3:1 price and its p50
|
|
162
|
-
throughput (scoring.md, *Route-aware pricing*).
|
|
163
|
-
|
|
162
|
+
throughput (scoring.md, *Route-aware pricing*). A role with the soft
|
|
163
|
+
`preferOwnProvider` policy (the `providerPinning` capability) likewise prices
|
|
164
|
+
each model on its **best route** rather than the blend, falling back to the blend
|
|
165
|
+
only when the model has no candidate route (scoring.md, *Capability presets*).
|
|
166
|
+
The blend is the default for every other model.
|
|
164
167
|
|
|
165
168
|
## OpenRouter — key metadata (tier gate)
|
|
166
169
|
|
|
@@ -180,8 +183,8 @@ freeUsable = free_model_daily_requests.remaining > 0
|
|
|
180
183
|
billed → billed variants only (exclude `:free`, `:batch`); else free → `:free`
|
|
181
184
|
variants only (exclude `:batch`); else `"none"` aborts the run. The key endpoint
|
|
182
185
|
carries **no** model/provider restriction field — the account's allowed-providers
|
|
183
|
-
whitelist is invisible to it (see the
|
|
184
|
-
[`architecture.md`](architecture.md)).
|
|
186
|
+
whitelist is invisible to it (see the allowed-providers whitelist below and the
|
|
187
|
+
probe in [`architecture.md`](architecture.md)).
|
|
185
188
|
|
|
186
189
|
## OpenRouter — keyed catalog (availability)
|
|
187
190
|
|
|
@@ -202,6 +205,32 @@ makes the Bearer-authenticated catalog a per-key allowlist.
|
|
|
202
205
|
every consumer re-fetches, and `fetchKeyAvailability` **never throws** (a failure
|
|
203
206
|
degrades to `unavailable` with empty sets, so the probe walk / `unknown` marks stand).
|
|
204
207
|
|
|
208
|
+
## OpenRouter — allowed-providers whitelist (availability)
|
|
209
|
+
|
|
210
|
+
The account's **allowed-providers privacy whitelist**
|
|
211
|
+
(openrouter.ai/settings/privacy) restricts which provider routes a request may
|
|
212
|
+
use. It is invisible to `/api/v1/key` and to the catalog endpoints; the only
|
|
213
|
+
place it appears is the 404 body of a request that fails because of it.
|
|
214
|
+
|
|
215
|
+
- **URL**: `POST https://openrouter.ai/api/v1/chat/completions` with
|
|
216
|
+
`provider.only: ["__omp-llm-role-canary__"]` (a nonexistent provider slug) on a
|
|
217
|
+
valid catalog model. `fetchAllowedProviders(token, modelId, fetchImpl?)`
|
|
218
|
+
(`src/availability.ts`) issues it.
|
|
219
|
+
- **Signal**: the 404 body names the whitelist after "permits only:" — the
|
|
220
|
+
message is parsed into a `Set<string>` of bare lab names (`coreweave`,
|
|
221
|
+
`baseten`, …). A non-404 response, a body without the clause (no whitelist
|
|
222
|
+
set), or any transport/shape failure returns `null`. The canary 404s, so it
|
|
223
|
+
bills no tokens.
|
|
224
|
+
- **Match rule**: a route's `providerSlug` is allowed when it equals a whitelist
|
|
225
|
+
entry or starts with `<entry>/` — the whitelist carries bare lab names while
|
|
226
|
+
route slugs are tiered (`coreweave/fp8`, `baseten/fp8`).
|
|
227
|
+
- **Consumer**: the updater harvests it **once per run** (with a valid catalog
|
|
228
|
+
model id) and prunes every model's route pool to allowed providers before
|
|
229
|
+
ranking (architecture.md, *Data flow*). It is **not cached to disk** — every
|
|
230
|
+
run re-learns it, so a privacy-setting change takes effect on the next refresh
|
|
231
|
+
— and `fetchAllowedProviders` **never throws** (a failure degrades to `null`,
|
|
232
|
+
which leaves every route pool untouched).
|
|
233
|
+
|
|
205
234
|
## Design Arena — design quality
|
|
206
235
|
|
|
207
236
|
Two routes, never averaged (same Elo family; mean diff −1.0 website … −5.1 svg,
|
package/docs/dev/explorer.md
CHANGED
|
@@ -99,8 +99,12 @@ Endpoints:
|
|
|
99
99
|
expansion the preview would omit the preset the baseline includes and every
|
|
100
100
|
row's Δ would be a lie. The posted def is the **authored** def (flags + explicit
|
|
101
101
|
keys); the expansion is applied only for the ranking call, never persisted.
|
|
102
|
-
- `handleExport`
|
|
103
|
-
|
|
102
|
+
- `handleExport` validates every dirty role first — `validateRole` plus a
|
|
103
|
+
**provider-pin check**: a `providerPin` that matches no route in the loaded
|
|
104
|
+
dataset is rejected (`role "<name>": provider pin "<slug>" matches no route in
|
|
105
|
+
the loaded dataset`) before any write, so an exported role cannot carry an
|
|
106
|
+
unroutable pin. It then calls `writeRoleSettings(activeScope.lockPath,
|
|
107
|
+
body.roles)` and returns its result verbatim plus the written `lockPath`.
|
|
104
108
|
- Unknown `/api/*` → 404; wrong method → 405; non-API non-GET → 405.
|
|
105
109
|
- `readBody` caps at `MAX_BODY = 1 MiB` (413); `readJson` → 400 on invalid JSON.
|
|
106
110
|
- `serveStatic` resolves against `webDir` and rejects traversal (`full` must be
|
|
@@ -144,7 +148,12 @@ Pure, no I/O; all ranking math is delegated to `src/engine.ts`.
|
|
|
144
148
|
`gapAbove`/`gapToTop`/`above`, `closing` (the
|
|
145
149
|
per-metric raw target that would close the gap, via `inverseCardinal`, with
|
|
146
150
|
unreachable/extrapolated notes), `dominators` (models both cheaper and at
|
|
147
|
-
least as good on `q`, top 3),
|
|
151
|
+
least as good on `q`, top 3), `ownProvider` (true when the model was priced on
|
|
152
|
+
the soft best route rather than the blend), `autoPin` (the provider slug the
|
|
153
|
+
updater would write — `bestRoute(model, def)?.providerSlug`, the max-`vfm`
|
|
154
|
+
route (scoring.md, *Route ordering*), when the role sets
|
|
155
|
+
`preferOwnProvider` and carries no manual `providerPin`, else `null`), and the
|
|
156
|
+
availability overlay `key`/`keyReason`
|
|
148
157
|
(the same verdict as `rankRows`, plus the reason it reads that way). It mirrors
|
|
149
158
|
`rankRole`'s capability fill so the contributions sum exactly to `q`.
|
|
150
159
|
|
|
@@ -277,7 +286,16 @@ a scraped third-party site, so the DOM is built with
|
|
|
277
286
|
changes call `scheduleRecompute()` (120 ms debounce).
|
|
278
287
|
- **Explain**: `selectModel(id)` → `POST /api/explain {role, def, modelId}` →
|
|
279
288
|
`renderExplain()` (composition table, cost line, "why not higher",
|
|
280
|
-
dominators, and the availability line).
|
|
289
|
+
dominators, and the availability line). The cost line appends `· best route`
|
|
290
|
+
when the model was priced on the soft best route (`ownProvider`), and a
|
|
291
|
+
**pin line** shows the pin in force: `pin: <slug> (manual)` when the edited def
|
|
292
|
+
carries a `providerPin`, else `auto pin: <slug>` from the explanation's
|
|
293
|
+
`autoPin` (read-only — the manual field is the only way to set one).
|
|
294
|
+
- **Provider pin field**: the editor renders a text input for
|
|
295
|
+
`roles.<role>.providerPin` (a tiered slug like `deepinfra/fp8` is kept
|
|
296
|
+
verbatim; empty clears the key, resuming the automatic choice). It is the
|
|
297
|
+
manual override of the auto-pin; Export rejects a slug that matches no route in
|
|
298
|
+
the loaded dataset (see *Export path*).
|
|
281
299
|
- **Availability overlay**: each row's `key` renders as a three-state badge in
|
|
282
300
|
the `key` column; the **hide key-blocked** checkbox beside the filter/top-n
|
|
283
301
|
controls drops `blocked` rows from the table (a client-side filter — the
|
package/docs/dev/quirks.md
CHANGED
|
@@ -146,6 +146,22 @@ things that only bite someone changing or debugging the code.
|
|
|
146
146
|
update (or a `link` to a changed tree), a session that was already running
|
|
147
147
|
still executes the module it loaded at start; a write from that stale code can
|
|
148
148
|
revert a newer write. Restart the session after updating the plugin.
|
|
149
|
+
- **omp's restore-to-primary is silent to extensions.** After a fallback
|
|
150
|
+
cooldown, omp restores the primary model at prompt admission
|
|
151
|
+
(`#maybeRestoreRetryFallbackPrimary`) with **no** session event, and
|
|
152
|
+
`model_changed` is not delivered to extensions — so a display derived from the
|
|
153
|
+
fallback events alone would keep the fallback's provider pin after a restore.
|
|
154
|
+
The provider display (`src/provider-display.ts`) therefore re-derives the pin
|
|
155
|
+
from the live model on `turn_start`, which *is* extension-delivered and fires
|
|
156
|
+
after the restore. The same re-derive covers a manual `/model` change (with a
|
|
157
|
+
one-turn lag). The primary and a fallback may be the same registry object
|
|
158
|
+
(same model id, different route), so the strip-then-append is idempotent and
|
|
159
|
+
reads the pin from the live model's routing block, never a cached value.
|
|
160
|
+
`retry_fallback_applied`/`retry_fallback_succeeded` are also emitted for
|
|
161
|
+
**advisor** fallbacks, where only the advisor's model changes and the session
|
|
162
|
+
model is untouched, so the event's selector is trusted only when it names the
|
|
163
|
+
live session model (`liveModelIsSelector`); otherwise the pin is re-derived
|
|
164
|
+
from the live model's routing block.
|
|
149
165
|
- **Testing a registered slash command needs `--mode rpc` or a TUI** — `omp -p`
|
|
150
166
|
(print mode) sends a leading `/name` as a *prompt* to the model, so the handler
|
|
151
167
|
never runs and `ctx.ui.notify` is a no-op anyway. RPC mode does dispatch them:
|
|
@@ -154,6 +170,22 @@ things that only bite someone changing or debugging the code.
|
|
|
154
170
|
stderr; `get_available_commands` confirms registration. Verified this way for
|
|
155
171
|
`/create-agent` and `/explore-roles` (omp 18.4.8).
|
|
156
172
|
|
|
173
|
+
## Repository plumbing
|
|
174
|
+
|
|
175
|
+
- **The https credential cannot *introduce* a workflow change.** `origin` is
|
|
176
|
+
https and its credential is a `gh` OAuth token with scopes
|
|
177
|
+
`admin:public_key, gist, read:org, repo` — no `workflow` — so a push that adds
|
|
178
|
+
or modifies `.github/workflows/*` is rejected ("refusing to allow an OAuth App
|
|
179
|
+
to create or update workflow … without `workflow` scope"), and the REST
|
|
180
|
+
contents API is rejected the same way. The check is a diff against the ref's
|
|
181
|
+
base, not a property of the ref: a new branch or tag whose commit already
|
|
182
|
+
contains the workflow, with `main` also containing it, pushes fine over https
|
|
183
|
+
(verified 2026-10-08). So only the commit that lands a workflow edit needs the
|
|
184
|
+
SSH route — `git push git@github.com:Gott50/omp-llm-role.git main` — and
|
|
185
|
+
because that bypasses the `origin` remote it leaves `refs/remotes/origin/main`
|
|
186
|
+
stale, which makes `scripts/release.ts cut` refuse ("HEAD is not origin/main")
|
|
187
|
+
until `git fetch origin` runs.
|
|
188
|
+
|
|
157
189
|
## Historical
|
|
158
190
|
|
|
159
191
|
- **Bare roles were priced as if thinking were free** (fixed 2026-09-30).
|
package/docs/dev/releasing.md
CHANGED
|
@@ -116,14 +116,35 @@ path, plus `marketplace update` + `upgrade` to a bumped catalog version), and
|
|
|
116
116
|
|
|
117
117
|
## Status
|
|
118
118
|
|
|
119
|
-
- **CI/CD landed 2026-10-08** (`fa7e546
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
against version `1.1.0` failed
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
119
|
+
- **CI/CD landed 2026-10-08** (`fa7e546`, docs follow-ups `4f78ce2`, `1d20ef9`,
|
|
120
|
+
`9244fdc`, `3c0125a`): every pull request and every `main` push runs the suite
|
|
121
|
+
on GitHub (verified green on PR #57 and on the `main` pushes), and a `v*` tag
|
|
122
|
+
push publishes to npm over OIDC and creates the GitHub Release. The guard was
|
|
123
|
+
verified with a negative control — tag `v0.0.1` against version `1.1.0` failed
|
|
124
|
+
in `notes` before any publish.
|
|
125
|
+
- **1.1.0 published by CD 2026-10-09** (run 37897647817,
|
|
126
|
+
`gh workflow run ci.yml --ref v1.1.0`): the guard and the npm floor passed,
|
|
127
|
+
`npm publish` completed over OIDC trusted publishing with a provenance
|
|
128
|
+
attestation (`npm audit signatures`: 0 invalid, 0 missing), and the GitHub
|
|
129
|
+
Release was created with the 1.1.0 CHANGELOG section as its body
|
|
130
|
+
(`https://github.com/Gott50/omp-llm-role/releases/tag/v1.1.0`). Verified:
|
|
131
|
+
`dist-tags.latest` = `1.1.0`, npm's `gitHead` = `4f78ce2` (the tag's commit),
|
|
132
|
+
`_npmUser` = `GitHub Actions` carrying the trusted-publisher OIDC config, the
|
|
133
|
+
provenance statement in the transparency log at
|
|
134
|
+
`https://search.sigstore.dev/?logIndex=3159690884`, and the tarball is 56
|
|
135
|
+
files / 308.4 kB. The `v1.1.0` tag was repointed to
|
|
136
|
+
`4f78ce2` first — its original commit predates the workflow file, so the
|
|
137
|
+
dispatch was rejected (see [`ci.md`](ci.md)).
|
|
138
|
+
- **npm route smoke-tested for 1.1.0** (2026-10-09, omp 18.8.6, throwaway
|
|
139
|
+
`HOME`, cwd outside the real home tree): `omp plugin install omp-llm-role`
|
|
140
|
+
installed 1.1.0 from the registry (`omp plugin doctor`: 4 ok, 0 warnings,
|
|
141
|
+
0 errors) and the post-install session run (`omp -p "say ok"` with
|
|
142
|
+
`OPENROUTER_API_KEY` from `omp token openrouter`) wrote `modelRoles`
|
|
143
|
+
(9 roles), `retry.fallbackChains` and `task.disabledAgents: [designer]` into
|
|
144
|
+
that HOME's `config.yml`, landed `llm-role-state.json` +
|
|
145
|
+
`llm-role-history.jsonl`, and answered.
|
|
146
|
+
- **1.1.0 cut 2026-10-08** (`c013918`, tagged `v1.1.0` — repointed to `4f78ce2`,
|
|
147
|
+
see the publish bullet). `package.json` and
|
|
127
148
|
`.omp-plugin/marketplace.json` bumped in lockstep (plus `package-lock.json`
|
|
128
149
|
via `npm install --package-lock-only` — the repo keeps it in lockstep, see
|
|
129
150
|
`8811309`), `## [Unreleased]` renamed to `## [1.1.0] - 2026-10-08`, and the
|
|
@@ -141,26 +162,6 @@ path, plus `marketplace update` + `upgrade` to a bumped catalog version), and
|
|
|
141
162
|
`modelRoles` (9 roles), `retry.fallbackChains` and
|
|
142
163
|
`task.disabledAgents: [designer]` into that HOME's `config.yml`, landed
|
|
143
164
|
`llm-role-state.json` + `llm-role-history.jsonl`, and the session answered.
|
|
144
|
-
The npm route has **not** been smoke-tested for 1.1.0 — it is still serving
|
|
145
|
-
1.0.0.
|
|
146
|
-
- **npm publish pending a valid token.** The `~/.npmrc` token is rejected
|
|
147
|
-
(`npm whoami` → 401 "your authentication token seems to be invalid",
|
|
148
|
-
`npm publish` → 404 on `PUT /omp-llm-role`), so `omp-llm-role@1.1.0` is not
|
|
149
|
-
on the registry yet, `dist-tags.latest` still points at `1.0.0`, and **npm's
|
|
150
|
-
`gitHead` for 1.1.0 does not exist yet** — the tag currently stands on its
|
|
151
|
-
own. The token is the only credential source (no repo `.npmrc`, no
|
|
152
|
-
`/opt/homebrew/etc/npmrc`, no `NPM_TOKEN` in the environment or the shell
|
|
153
|
-
profiles, no keychain entry) and it is a literal value, not a `${NPM_TOKEN}`
|
|
154
|
-
indirection, so the cause is a rejected/revoked/expired token — **not** the
|
|
155
|
-
2026-07-31 GAT restriction, which covers account/org/package-management
|
|
156
|
-
actions and not `whoami`, and not direct publish either (that loss is
|
|
157
|
-
targeted January 2027). `npm ping` and `npm owner ls` still succeed because
|
|
158
|
-
they are unauthenticated reads. Fix: the release workflow — OIDC trusted
|
|
159
|
-
publishing, no token — once the trusted publisher is registered (see
|
|
160
|
-
[`ci.md`](ci.md)). The tag was cut before the publish (the documented order is
|
|
161
|
-
publish-then-tag) precisely so the pinned git ref and the changelog link
|
|
162
|
-
resolve meanwhile; that also means the tag's commit predates the workflow
|
|
163
|
-
file, so the workflow cannot be dispatched against it.
|
|
164
165
|
- **1.0.0 on all three channels** (2026-10-03): `omp-llm-role@1.0.0` on npm
|
|
165
166
|
(`dist-tags.latest` = `1.0.0`), tagged `v1.0.0` at `fe98b76` (npm's
|
|
166
167
|
`gitHead`); all three routes verified end-to-end against the real URLs
|
|
@@ -169,7 +170,7 @@ path, plus `marketplace update` + `upgrade` to a bumped catalog version), and
|
|
|
169
170
|
landed the three daily caches in the installed copy.
|
|
170
171
|
- **Stray registry placeholder.** The registry also carries a `0.0.0-stage`
|
|
171
172
|
version ("Temporary package placeholder for staged publishing", 334 B,
|
|
172
|
-
published 2026-10-03). It is not from this tree
|
|
173
|
-
`1.
|
|
173
|
+
published 2026-10-03). It is not from this tree and does not affect installs
|
|
174
|
+
(`latest` is `1.1.0`).
|
|
174
175
|
- **Next release: one owed.** `## [Unreleased]` carries the CI/CD entry; the
|
|
175
176
|
number comes from the Unreleased scope, not from the open issues.
|