omp-llm-role 1.0.0 → 1.1.0

package/docs/dev/architecture.md

@@ -4,7 +4,7 @@ Maintainer map of the `omp-llm-role` plugin: how the modules fit together, the
 runtime contract they must satisfy, and the invariants a change must not break.
 
 - User-facing install/usage: [`../../README.md`](../../README.md).
-- Normative behavior contract: [`spec.md`](spec.md).
+- How it is verified: [`testing.md`](testing.md).
 - Where the numbers come from: [`data-sources.md`](data-sources.md).
 
 ## Module map
@@ -25,32 +25,34 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
 
 | File | Role / key exports |
 |---|---|
-| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
-| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
-| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. |
+| `src/engine.ts` | Ranking engine shared by CLI and plugin. Fetch/cache chain per source, cardinal transforms, value scoring. Exports `loadRankData`, `computeRankings`, `buildModels`, `rankRole`, `paretoFrontier`, `cardinalMetric`, `roleLambda`, `thinkingPriceFactor`, `CAPABILITY_FILL`, `SUFFIX_LEVELS`, `META_LEVELS`, `THINKING_TOKEN_OVERHEAD`, `parseFindData`, `parseModelPage`, `narrowEndpointRecord`, `buildOpenRouterEnrichment`, `applyOpenRouterData`, `hasEndpointFilters`, `routePassesEndpointFilters`, `modelPassesEndpointFilters`, `endpointFilterDrops`, `applyDesignPercentiles`, `applyWritingScores`, `extractFlight`, `extractJsonArray`, the cache readers/writers, and the `Model`/`RoleDef`/`Ranked`/`RankData`/`OpenRouterEndpointRecord` types. Re-exports `normalizeDesignId` and `parseWritingEvidence` from the benchmark-source registry. |
+| `src/benchmark-sources.ts` | The benchmark-source registry: the single source of truth for which sources exist, what metric each feeds, and how to fetch/parse/join each. Exports `BENCHMARK_SOURCES` (the shipped static sources), `resolveBenchmarkSource`, `declaredSourceForLink`, `sourceForMetric`, `parseBenchmarkPayload`, `parseBenchmarkPayloadMeta`, `loadBenchmarkScores`, `applyBenchmarkScores`, `joinBenchmarkScores`, `normalizeMetricKey`, `externalMetricKey`, `normalizeDesignId`, `parseWritingEvidence`, `parseLlmStatsBenchmark`, `parseBenchmarkCatalog`, `loadBenchmarkCatalog`, `cachedSourceInfo`, `catalogMetric`, `SHIPPED_CATALOG_METRICS`, the declarative-source surface (`SourceDeclaration`, `validateDeclaration`, `parseSourceDeclaration`, `loadDeclaredSources`, `saveDeclaredSource`, `declaredSourcesPath`, `declarationToSource`, `executeDeclaration`, `llmStatsBenchmarkDeclaration`, `genericBenchmarkSource`, `dryRunDeclaration`), the fetch helpers (`fetchJson`, `fetchText`), and the `BenchmarkSource`/`JoinRule`/`BenchmarkScores`/`BenchmarkPayloadMeta`/`FocusSourceInfo`/`BenchmarkCatalogEntry`/`DeclarationDryRun` types. Imports `engine.ts` for types only, so there is no runtime cycle. |
+| `src/settings.ts` | Shipped defaults and the settings merge/validate path. Exports `DEFAULT_ROLES`, `DEFAULT_SETTINGS`, `SHIPPED_AGENTS`, `KNOWN_METRICS`, `isKnownMetric` (a shipped key or an external `<ns>:<local>`), `PLUGIN_SETTINGS_PATH`, `ACTIVATE_DEFAULT_KEY`, `deriveSettingsSchema`, `readPluginSettingsMap`, `resolveSettings`, `roleUniverse`, `deepMergeInto`, `findProjectAnchor`, `projectLockPath`, and the `ResolvedSettings`/`PluginSettingSchema`/`UniverseEntry`/`RoleKind` types. `readPluginSettingsMap` merges the project lock file (`projectLockPath(cwd)` = `<cwd>/.omp/plugins/omp-plugins.lock.json`, omp's project dir with **no walk-up**) over the user-level one; `findProjectAnchor` (the walk-up `.omp`/`.git` anchor) is kept only for the `/project-roles` divergence warning. |
 | `src/agent-pins.ts` | Agent → pinned-role derivation. Exports `parseAgentPin` (first `@<role>` in the `model:` frontmatter) and `discoverAgentPins` (scans the shipped, user and project agent dirs; project > user > plugin). Drives the `task.disabledAgents` sync. |
-| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict` types. |
+| `src/availability.ts` | Key tier gate, catalog filter, variant resolution, provider-allowlist probe, and the keyed-catalog availability primitive. The keyed source is the Bearer-authenticated `GET /api/v1/models` (the account setting "Filter the model catalog for API keys" makes it a per-key allowlist; set membership is the only signal), shared by the explorer and the updater's companion fast path. Exports `fetchKeyMeta`, `tierGate`, `filterCatalog`, `rankingIdOf`, `currentRankingId`, `resolveVariant`, `enrichThinkingLevels`, `probeModel`, `computeKeyAvailability`, `fetchKeyAvailability`, `catalogFromOmpModelsJson`, `THINKING_LEVELS`, and the `CatalogEntry`/`KeyMeta`/`Tier`/`ProbeVerdict`/`KeyAvailability` types. |
 | `src/config-edit.ts` | Surgical line-oriented YAML patch for `modelRoles` + `retry.fallbackChains` + `task.disabledAgents`, plus the atomic writer. Exports `parseConfig`, `patchConfig`, `writeConfigAtomic`, `ConfigEditError`, and the `ConfigPatch` type. No runtime YAML dependency. |
 | `src/state.ts` | State/history/lock files under the agent dir, and agent-dir resolution. Exports `agentDir`, `loadState`, `saveState`, `appendHistory`, `acquireLock`, `releaseLock`, `freshState`, and the `PluginState` type (which carries `managedDisabledAgents` — the `task.disabledAgents` names the plugin added). |
-| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. |
-| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, and `session_shutdown`. Owns `extDeps` (the extension's `Deps`) and the live-session model hook. |
+| `src/error-report.ts` | Opt-in reporting of **unexpected** plugin errors as GitHub issues on `Gott50/omp-llm-role` (the `errorReporting` setting). The plugin's only GitHub client and the only home for a fingerprint, a redaction rule, the agent-dir ledger (`ERROR_REPORT_LEDGER_FILE`) and the issue body. **Entry point:** `reportUnexpectedError(input, deps)` — the feature's one test seam (per-call input + fully injected `deps`; tests supply a fake `fetch`, clock, env, dir and token resolver). Also exports `defaultErrorReportDeps()` (production deps: resolved settings, global `fetch`, real clock, `agentDir`, the shipped manifest, `gh auth token`, the OS opener) and `resetErrorReportGuard()` (the in-process dedupe guard's test hook). Never throws, never rejects; a no-op when `errorReporting` is `off`. |
+| `src/updater.ts` | Orchestration: rank → tier gate → probe → hysteresis → chains → agent-disable sync → config write. Exports `runUpdater`, the `Deps`/`Decision`/`RunResult`/`Trigger`/`DecisionReason` types. Resolves the project scope (`<cwd>/.omp` when it carries an `omp-llm-role` settings entry) and scopes the config/state/history/lock to it. `agentDisablePatch` also removes a previously-managed `task.disabledAgents` name whose agent file is gone. `RunResult.defect` is set on the exception-abort paths only (a `ConfigEditError`/self-check mismatch and the catch-all throw); the enumerated environment aborts leave it unset. |
+| `src/extension.ts` | omp extension entry (default export). Registers `session_start` (awaited day-gated run), `/refresh-roles`, `/explore-roles` (in-process explorer), `/create-agent` (architect + benchmark-link resolution + catalog discovery), `/remove-agent`, `/project-roles` (profile architect + `setupProject` + in-process updater), and `session_shutdown`. Owns `extDeps` (the extension's `Deps`) and the live-session model hook. Every unexpected command-handler throw and every `runUpdater` defect is routed through `src/error-report.ts`: `reportCommandError` for the six command-handler catch sites and `reportDefect` for a `runUpdater` result whose `defect` is set — each keeps the pre-existing one-line warning and adds the reporting module's own lines. |
 | `src/guards.ts` | The package's one type guard: `isRecord`. |
 | `src/role-settings.ts` | The one validated role write path (validate → merge → backup → atomic write) and its removal half. Exports `validateRole`, `mergeExport`, `writeRoleSettings`, `mergeRemove`, `removeRoleSettings`, `timestamp`. Shared by the explorer's Export, `src/cli/create-role.ts`, `/create-agent` and `/remove-agent`. |
 | `src/role-archetypes.ts` | Purpose → weight archetype table (10 sets) + keyword fitting. Exports `ARCHETYPES`, `FALLBACK_ARCHETYPE`, `fitArchetype`, `archetypeById`, and the `Archetype`/`ArchetypeMatch` types. |
 | `src/agent-file.ts` | Agent `.md` rendering/placement. Exports `renderAgentFile`, `writeAgentFile`, `removeAgentFile`, `isReadOnlyTools`, `userAgentsDir`, `projectAgentsDir`, `READ_ONLY_TOOLS`, `RESERVED_AGENT_NAMES`, `AGENT_NAME_RE`, and the `AgentFileSpec`/`AgentWriteResult`/`AgentRemoveResult` types. |
-| `src/agent-architect.ts` | omp's agent-creation architect, run in-process (extension-only). Exports `generateAgentSpec`, `parseAgentSpec`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions` type. Imports `@oh-my-pi/pi-coding-agent` at the package root. |
-| `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Extension-only (imports `@oh-my-pi/pi-coding-agent`). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
+| `src/agent-architect.ts` | omp's in-process architects: the agent-creation architect (`/create-agent`, `/agents` hub) and the project-profile architect (`/project-roles`). Exports `generateAgentSpec`, `generateProjectProfile`, `parseAgentSpec`, `parseProjectProfile`, `extractJsonObject`, `extractAssistantText`, and the `ArchitectOptions`/`ProjectArchitectOptions` types. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the runner), so the module is importable under plain Node. |
+| `src/benchmark-author.ts` | Benchmark-source authoring for an unknown link: fetch the link, run an in-process architect (`src/prompts/benchmark-source-architect.md`) to propose a declarative source spec, and validate it. Imports `@oh-my-pi/pi-coding-agent` **dynamically** (inside the function). Exports `authorBenchmarkSource`, `parseSourceDeclarationJson`, and the `BenchmarkAuthorOptions` type. |
 | `src/benchmark-discovery.ts` | Judge-backed benchmark relevance for `/create-agent` discovery: one `noul` (yes/no probability) question per catalog candidate, batched into a single judgment through omp's configured `judge` role (TypeSafe jev or its fallback chain). Extension-only (imports `@oh-my-pi/pi-coding-agent` and its `./judgment` subpath — the deeper `./judgment/standalone` path does not resolve under omp's SDK injection). The SDK is imported dynamically so a resolution failure degrades to "no discovery" instead of failing the whole extension load. Exports `judgeBenchmarkRelevance`. |
-| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `discoverBenchmarks`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types. |
+| `src/agent-create.ts` | `/create-agent` core: purpose → archetype → validated role → agent `.md`, plus the shared input parser and report formatter both hosts use. Exports `createAgent`, `resolveRole`, `parseCreateAgentInput` (flag form or free text), `parseCreateAgentArgs`, `tokenizeArgs`, `extractBenchmarks`, `extractBenchmarkLinks`, `applyFocusBenchmarks`, `assessFocusMetric`, `belowBarReason`, `focusCoverageOk`, `countMetricCoverage`, `discoverBenchmarks`, `checkWeightMath`, `formatBenchmarks`, `formatArchetypes`, `formatCreateAgentReport`, `CREATE_AGENT_USAGE`, and the request/result types (incl. `FocusMetricAssessment`/`FocusCoverageEntry`/`FocusAssessor`). |
 | `src/agent-remove.ts` | `/remove-agent` core: delete an agent `.md` and its role, plus the shared flag parser and report formatter. Exports `removeAgent`, `parseRemoveAgentArgs`, `formatRemoveAgentReport`, `REMOVE_AGENT_USAGE`, and the request/result types. Refuses shipped default roles. |
+| `src/project-setup.ts` | `/project-roles` core: a discovered `ProjectProfile` → a project-scoped role set (kept shipped roles, dropped ones, new fitted roles) written to the project plugin settings lock file, plus the project agents for the new roles. All-or-nothing (every check — names, metrics, weight math, the `--force` gate, agent-file collisions — before any write), `--dry-run`. Exports `setupProject`, `parseProjectRolesArgs`, `parseRolesSpec`, `applyProfileOverrides`, `formatProjectRolesReport`, `PROJECT_ROLES_USAGE`, and the `ProjectProfile`/`ProposedRole`/`SetupProjectOpts`/`SetupProjectResult` types (the result carries an `archetypes` map). The plan report prints the capability flags each new role will author (`features: <id>=<on>`). Dual-runtime safe (no SDK import); the extension command does the LLM work and calls it. |
 
 ### `src/explorer/`
 
 | File | Role / key exports |
 |---|---|
 | `src/explorer/boot.ts` | Shared explorer launcher: bind/port fallback, lock-file roles, browser open, close. Exports `startExplorer`, `EXPLORER_DEFAULT_PORT`, and the `ExplorerHandle`/`ExplorerBootOpts` types. Used by `/explore-roles`. |
-| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap`, `POST /api/rank`, `POST /api/explain`, `POST /api/export`, `POST /api/refresh`. |
-| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets. Exports `rankRows`, `explainModel`, `inverseCardinal`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta` types. |
+| `src/explorer/server.ts` | Zero-dependency HTTP surface (static SPA + JSON API). Exports `createExplorerServer` and the `ExplorerOpts` type. Endpoints: `GET /api/bootstrap` (carries `focusAssessments`), `POST /api/rank`, `POST /api/explain`, `POST /api/export`, `POST /api/refresh`. |
+| `src/explorer/explain.ts` | Pure explanation layer: rank rows with baseline deltas, per-model decomposition, inverse-cardinal targets, and the nine-axis focus assessment. Exports `rankRows`, `explainModel`, `inverseCardinal`, `focusMetricsOf`, `focusAssessments`, `METRIC_META`, `metricMeta`, `metricMetaFor`, `weightableMetrics`, and the `RankRow`/`Explanation`/`Contribution`/`Closing`/`MetricMeta` types. |
 
 ### Assets and non-code
 
@@ -58,6 +60,7 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
 |---|---|
 | `src/prompts/agent-creation-architect.md`, `src/prompts/agent-creation-user.md` | omp's architect prompts, shipped verbatim; read by `src/agent-architect.ts` via `new URL("./prompts/…", import.meta.url)`. |
 | `src/prompts/benchmark-source-architect.md`, `src/prompts/benchmark-source-user.md` | The benchmark-source architect prompts; read by `src/benchmark-author.ts`. |
+| `src/prompts/project-profile-architect.md`, `src/prompts/project-profile-user.md` | The `/project-roles` profile-discovery architect prompts; read by `src/agent-architect.ts`. |
 | `web/index.html`, `web/app.js`, `web/style.css` | Explorer SPA — no framework, no build step, no external requests. Served by `src/explorer/server.ts`. |
 | `agents/designer.md` | The shipped `designer` subagent, discovered from the plugin's extension root (`<ext>/agents/*.md`). Opt-in: kept in `task.disabledAgents` until `roles.designer.enabled=true`. |
 | `skills/omp-llm-role-create-agent/SKILL.md` | Shipped skill, discovered from the plugin's `skills/` root. Hand-driven equivalent of `/create-agent`. |
@@ -70,20 +73,35 @@ the extension's `/refresh-roles`, `/explore-roles` and `/create-agent`.
 | `skills-lock.json` | Lock for the local `.agents/skills/` collection. |
 | `.agents/skills/` | Local dev-skill collection (Matt Pocock skills); not part of the plugin's shipped surface. |
 | `docs/agents/domain.md`, `docs/agents/issue-tracker.md`, `docs/agents/triage-labels.md` | Repo process docs (domain glossary, issue tracker, triage labels). |
-| `docs/dev/spec.md` | Normative spec (moved from the repo root). |
+| `docs/dev/testing.md` | The test-suite seam map and the live checks. |
 | `docs/dev/architecture.md`, `docs/dev/data-sources.md` | This file and the data-source map. |
-| `tests/` | `node --test tests/` fixtures (tier gate, variant resolution, config edit, hysteresis, chain pruning, chain suffixes, explorer, role creation, writing metric, benchmark sources, thinking-price, openrouter-blend, agent disable, agent pins, settings schema, session model, role lock, role enable, probe gate, create-agent, create-role, **remove-agent**). |
+| `tests/` | `node --test tests/` fixtures — the seam map is in [`testing.md`](testing.md). |
 | `docs/llm-role-rankings.md` | Generated report (regenerate with `--out`); the README's worked example. |
 | `cache/*.json` | Daily UTC caches (gitignored; see [`data-sources.md`](data-sources.md)). |
 
+## Variant resolution (ranking id → selector)
+
+A ranking row (an llm-stats bare id) resolves to an `openrouter/*` catalog id by
+matching the id after the last `/` (org prefix — including `~`-prefixed —
+ignored), or after stripping a trailing `-latest`. Among the matches,
+`resolveVariant` emits in order: **exact id** (bare or dated, whatever equals the
+ranking id) → **newest dated id** (max trailing `-MMDD`/date suffix) → **bare
+id** → `~org/…-latest` alias (last resort — a valid selector beats no update).
+Never `:batch`; `:free` only on the free-tier branch of the tier gate. Ties
+break lexicographically. The emitted selector is always `openrouter/<catalogId>`.
+
 ## Data flow
 
 One pipeline, driven by `runUpdater(trigger, deps, opts)` in `src/updater.ts`.
 The omp extension injects the production `Deps`; tests inject fakes.
 
 1. **Trigger.** `session_start` (awaited, day-gated) or `/refresh-roles`
-   (forced). `runUpdater` loads `loadState()` and the raw
-   settings (`deps.getSettings?.() ?? readPluginSettingsMap()`), then
+   (forced). `runUpdater` resolves the **project scope** first: when
+   `<cwd>/.omp/plugins/omp-plugins.lock.json` carries an `omp-llm-role` settings
+   entry, the config, the settings read, and the state/history/lock are scoped to
+   `<cwd>/.omp` (the global `~/.omp/agent/config.yml` is never written in project
+   mode); otherwise the global agent dir is used. It then loads `loadState()` and
+   the raw settings (`deps.getSettings?.() ?? readPluginSettingsMap()`), then
    `resolveSettings(raw)`; any validation error aborts with no write.
 2. **Agent-disable sync.** `agentDisablePatch` computes the
    `task.disabledAgents` adds/removes from `roleUniverse` + `discoverAgentPins`,
@@ -101,7 +119,10 @@ The omp extension injects the production `Deps`; tests inject fakes.
    `enrichThinkingLevels(rank.models, catalog)`. Per role, candidates are
    `resolveVariant`-resolved and then verified by a bounded `probeModel` walk
    (current selector first, then rank order; budget `PROBE_BUDGET = 12`; verdicts
-   cached per run). Blocked candidates are excluded and recorded on the decision.
+   cached per run). Only the narrow no-allowed-providers 404 disqualifies; every
+   other failure (5xx, timeout, unknown model) counts as usable and stays in
+   omp's runtime-fallback domain. Blocked candidates are excluded and recorded on
+   the decision.
 6. **Hysteresis.** `no-current` → adopt; ineligible current → `adopted`; best
    beats current by `switchMargin` → `switched`; inside the margin but
    `cheaperInsideMargin` → `switched-cost`; else `kept-margin` / `kept-eligible`.
@@ -154,10 +175,16 @@ Rules every module must satisfy:
   `undefined` at runtime, not a compile error. Run the script after edits and
   sanity-check stderr match/eligible counts.
 
-The one deliberate exception is `src/agent-architect.ts`: it imports
-`@oh-my-pi/pi-coding-agent` (the **package root** only — subpath imports do not
-resolve in the compiled binary) and is therefore **extension-only**. The tests
-import `src/agent-create.ts`, which never imports the architect.
+The one deliberate exception is the SDK import in `src/agent-architect.ts` and
+`src/benchmark-author.ts`: they import `@oh-my-pi/pi-coding-agent` (the
+**package root** only — subpath imports do not resolve in the compiled binary)
+**dynamically, inside the runner function**. A static import would be fatal to
+the whole `extension.ts` load (it would kill every plugin command, not just the
+architect); dynamic, the modules — and `src/extension.ts` — stay importable
+under plain Node (the command registration is testable) and a resolution failure
+degrades to the caller's error path. The architect is still **extension-only in
+practice**: it is only called from the extension. The tests import
+`src/agent-create.ts`, which never imports the architect.
 
 ## Injected `Deps`
 
@@ -193,7 +220,8 @@ before `catalogFromOmpModelsJson`.
   (`runUpdater("manual", …, { force: true })`), `/explore-roles` (in-process
   `startExplorer`), `/create-agent` (architect → `createAgent` → in-process
   `runUpdater`), `/remove-agent` (`removeAgent` → in-process `runUpdater`),
-  `session_shutdown` (closes the explorer handle).
+  `/project-roles` (profile architect → `setupProject` → in-process
+  `runUpdater`), `session_shutdown` (closes the explorer handle).
 - **CLI**: `src/cli/llm-role-rank.ts` (report), `src/cli/create-role.ts`
   (authoring). The updater and explorer have no CLI shim — they run through
   `/refresh-roles` and `/explore-roles`.
@@ -202,12 +230,18 @@ before `catalogFromOmpModelsJson`.
 
 `src/engine.ts` is the shared engine; `src/cli/llm-role-rank.ts` is a thin CLI over it.
 
-- `loadRankData(opts?: { refresh?; url?; roles? }): Promise<RankData>` — the
-  per-source cache → fetch → stale-cache chain. `opts.roles` names the roles the
-  caller will rank: a role-exclusive source is fetched only when some role
+- `loadRankData(opts?: { refresh?; url?; roles?; extraMetrics? }): Promise<RankData>`
+  — the per-source cache → fetch → stale-cache chain. `opts.roles` names the roles
+  the caller will rank: a role-exclusive source is fetched only when some role
   weights its metric — Design Arena (`website`), the writing leaderboard
   (`writing`), and every external metric (a declared source or the generic
-  llm-stats benchmark). `undefined` = fetch all (what the explorer passes).
+  llm-stats benchmark). `opts.extraMetrics` adds metrics no role weights and also
+  forces the `website`/`writing` fetches. `undefined` roles = fetch every declared
+  source. The explorer passes the resolved roles **plus** the shipped keys
+  (`Object.keys(KNOWN_METRICS)`), so its dataset is a superset of any def the UI
+  can rank — without it a role weighting an undeclared generic benchmark
+  (`bench:<id>`) would rank on a dataset missing that metric and disagree with the
+  updater.
 - `src/benchmark-sources.ts` is the registry: `resolveBenchmarkSource(link)` maps
   a link to a source, `sourceForMetric(metric)` maps a metric back, and
   `loadBenchmarkScores`/`applyBenchmarkScores` are the shared cache chain and
@@ -257,7 +291,8 @@ stderr `openrouter: matched N/<pool> models (throughput), M priced`, the
 - **One write path per artifact.** Roles go through `writeRoleSettings` (and
   `removeRoleSettings` for deletion); the config through `patchConfig` +
   `writeConfigAtomic`; agent files through `writeAgentFile` (and
-  `removeAgentFile` for deletion). Nothing else writes those files.
+  `removeAgentFile` for deletion); the error-report ledger through `writeLedger`
+  in `src/error-report.ts` (temp + rename). Nothing else writes those files.
 - **Surgical config edits.** Only managed role lines, managed chain keys and the
   plugin-managed `task.disabledAgents` names change; comments, blank lines and
   unknown keys stay byte-identical. A no-change run writes nothing.
@@ -276,3 +311,25 @@ stderr `openrouter: matched N/<pool> models (throughput), M priced`, the
   field already sets and prices the effort.
 - **The engine is the only ranking math.** The explorer never reimplements
   `value = q − λ·$/M`; it calls `rankRole`/`explainModel`.
+
+## Non-goals
+
+- **First-party provider selectors.** The plugin emits `openrouter/*` selectors
+  only — the ranking's price/throughput are OpenRouter-derived and the key is an
+  OpenRouter key.
+- **Non-OpenRouter scoring sources.**
+- **A `/rollback` command.** `llm-role-state.json` snapshots the previous
+  `modelRoles` block as a manual rollback aid; there is no command.
+- **Auto-tuning `switchMargin`.**
+- **A generic benchmark scraper** that auto-detects a payload with no user
+  review — the declarative source + authoring step is the mechanism.
+- **A UI for browsing/editing benchmark sources** — the source file is edited by
+  hand or by the authoring step; the explorer's role editor gains the external
+  metrics already in use.
+- **Changing the cardinal transform classes** — external metrics are normalized
+  to 0–1 at parse time.
+- **Re-ranking or re-fetching beyond the daily UTC cache chain.**
+
+Weight editing is the explorer's Export, `src/cli/create-role.ts` or
+`/create-agent` (all validate through `resolveSettings`); role/agent removal is
+`/remove-agent` (through `removeRoleSettings`).

package/docs/dev/ci.md

@@ -0,0 +1,152 @@
+# CI/CD
+
+The GitHub Actions pipeline that gates every change and delivers every release.
+The release *policy* — when to bump, SemVer over the public surface, the
+CHANGELOG rules, the three channels — lives in [`releasing.md`](releasing.md);
+this doc owns the machinery.
+
+## The workflow
+
+`.github/workflows/ci.yml` — one file, two jobs. One file because the npm
+trusted-publisher registration names the workflow **filename**, so a second file
+would need a second registration.
+
+| Job | Runs on | Does |
+|---|---|---|
+| `test` | every `pull_request`, every push to `main`, every `v*` tag push, and `workflow_dispatch` | `npm ci`, then `node --test tests/` |
+| `release` | `v*` tag pushes (and a `workflow_dispatch` against a tag ref) | verifies the tag, publishes to npm over OIDC, creates the GitHub Release |
+
+`release` declares `needs: test`, so a red tree cannot be published: the same
+ref must be green first. `test` runs **exactly** the command the local
+pre-commit hook runs (`.githooks/pre-commit`), and `tests/ci.test.ts` fails when
+the two drift apart — "green locally" and "green in CI" cannot diverge.
+
+The Node version comes from `.node-version` at the repo root (the exact local
+toolchain; bumping it is a normal commit). `engines` is deliberately not in
+`package.json`: the plugin executes inside omp's runtime, so an `engines` field
+would only produce install warnings for users.
+
+`test` is the whole gate — no lint, no format, no typecheck, no coverage. The
+repo runs Node type-stripping without a typecheck (see [`quirks.md`](quirks.md)).
+
+## The tag → publish contract
+
+Pushing a `v<version>` tag is the delivery trigger. The `release` job:
+
+1. checks out the tag;
+2. runs `node scripts/release.ts notes "$GITHUB_REF_NAME"` as the **guard** —
+   tag ↔ package version ↔ lockstep ↔ CHANGELOG section — and writes the release
+   notes to a file. A mistyped tag, a half-applied lockstep bump or a missing
+   CHANGELOG section fails here, *before* anything is published;
+3. ensures npm ≥ 11.5.1 (the trusted-publishing floor);
+4. `npm publish` — authenticated by GitHub OIDC, with a provenance attestation
+   generated automatically. No `NPM_TOKEN`, no `.npmrc` step, no third-party
+   publish action;
+5. `gh release create <tag> --notes-file <notes>` — the GitHub Release body is
+   the CHANGELOG section, never retyped.
+
+Because the publish runs from the tagged commit, npm's `gitHead` equals the tag.
+
+### One-time setup: register the trusted publisher
+
+Required before the first CD run, and it cannot be done from the repo. On
+npmjs.com → the package → Settings → Trusted publishing → GitHub Actions:
+
+- **Organization or user**: `Gott50`
+- **Repository**: `omp-llm-role`
+- **Workflow filename**: `ci.yml` (the filename only, with the extension)
+- **Allowed actions**: tick **direct publishing (`npm publish`)**.
+
+That last tick is the trap. `npm stage publish` is always allowed, and a
+configuration created after 2026-09-03 is automatically set to allow *only*
+staged publishing — so a publisher registered today rejects the job's
+`npm publish` **after** the guard has already passed, the worst possible failure
+point. Staged publishing is not a substitute: it needs a separate
+`npm stage approve` with 2FA, which breaks unattended CD. Do **not** tick
+`npm dist-tag`; CD never moves a dist-tag.
+
+Floors and constraints (npm's trusted-publishing docs): npm CLI ≥ 11.5.1, Node
+≥ 22.14.0 (the pinned `.node-version` clears both), and GitHub-**hosted** runners
+only — hence `ubuntu-latest`.
+
+## The release script
+
+`scripts/release.ts` — Node builtins only, no dependencies, outside the `files`
+whitelist so it never ships in the tarball. Three subcommands, each with a
+documented exit code:
+
+| Command | Exit 0 | Exit 1 |
+|---|---|---|
+| `due` | `## [Unreleased]` has at least one entry under an allowed section | it is empty (message on stdout) |
+| `notes <tag>` | prints **only** the CHANGELOG section body, ready for `--notes-file` | the tag is not `v<semver>`, disagrees with the package version, the lockstep fields disagree, the CHANGELOG has no `## [<version>] - <date>` heading, or that section has no entries (message on stderr) |
+| `cut <patch\|minor\|major> [--dry-run]` | the cut ran | a refusal (message on stderr) |
+
+`due` is the agent's branch point: "not owed" is not an error, so a script can
+branch on the code without parsing markdown.
+
+### The lockstep set
+
+Exactly four fields, bumped together:
+
+- `package.json` `version`
+- `.omp-plugin/marketplace.json` `plugins[0].version`
+- `package-lock.json` `version`
+- `package-lock.json` `packages[""].version`
+
+The marketplace catalog's own `metadata.version` is **deliberately excluded** —
+`releasing.md` locks the plugin entry, not the catalog. The two numbers look
+like they should agree; they do not.
+
+### What `cut` does, in order
+
+1. refuse a dirty tree;
+2. refuse an existing tag;
+3. refuse an empty `## [Unreleased]`;
+4. refuse when HEAD is not `origin/main` (so the tag always points at a pushed
+   commit and npm's `gitHead` resolves);
+5. compute the new version from the bump argument;
+6. write the four version fields;
+7. rename `## [Unreleased]` to `## [<version>] - <YYYY-MM-DD>` (UTC) and insert
+   a fresh empty `## [Unreleased]` above it;
+8. rewrite `[Unreleased]: …/compare/v<version>...HEAD` and prepend
+   `[<version>]: …/releases/tag/v<version>` to the footer link block;
+9. regenerate `docs/llm-role-rankings.md`
+   (`node src/cli/llm-role-rank.ts --all --out docs/llm-role-rankings.md`);
+10. commit `Release <version>`, staging only the touched paths (never `.`);
+11. push `main`, create the annotated tag `v<version>`, push the tag.
+
+The tag push is the CD trigger. `--dry-run` writes nothing: it prints the target
+version, the file plan and the changelog diff, and skips the network-dependent
+report regeneration with a note.
+
+`cut` does not run the suite itself. The commit it makes goes through
+`.githooks/pre-commit` when `core.hooksPath` is wired, and the `release` job
+re-verifies with `needs: test` on the tag — a second definition of the gate
+inside the script would be a second thing to keep in sync.
+
+There is no `--force` and no interactive prompt: every refusal is a message plus
+exit 1, so the command is safe to run unattended.
+
+## Verifying the pipeline
+
+- **The gate:** push a branch and open a PR, or push to `main`; `gh run watch`
+  the `test` job green.
+- **The guard (negative control):** push a tag that disagrees with the package
+  version (e.g. `v0.0.1` against `1.1.0`); the `release` job must fail in the
+  `notes` guard *before* any publish and leave the registry untouched. Delete
+  the tag afterwards.
+- **The publish:** run the release job against an already-cut tag —
+  `gh workflow run ci.yml --ref v<version>` — then confirm npm's `gitHead`
+  equals the tag commit, `dist-tags.latest` moved, and the GitHub Release body
+  is that version's CHANGELOG section. This is also the recovery path for a
+  failed publish: the tag exists, so no new version is needed.
+
+  The dispatch resolves the workflow **from the ref**, so the tag's commit must
+  contain `.github/workflows/ci.yml`. A tag cut before this workflow existed
+  cannot be dispatched (`HTTP 422: Workflow does not have 'workflow_dispatch'
+  trigger`) — the recovery path only works for tags cut after it landed.
+
+`tests/ci.test.ts` asserts the offline half: the workflow's structure (jobs,
+triggers, permissions, the command it runs), the hook/workflow command parity,
+the tag-trigger ↔ CHANGELOG-link agreement, the lockstep fields, and the
+script's exit codes and refusals against fixture repositories.