claude-spotter 1.4.17 → 1.4.18

package/CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## 1.4.18
+
+auditor model の更新を model 名の場当たり的な置換から切り離し、versioned policy と再現可能な比較 eval を
+導入する。反復評価24/24 exactの `gpt-5.6-terra × medium` をowner裁定でproductionへ昇格した。
+`latest` aliasやCodex CLIの暗黙既定、失敗時fallback、eval artifactからの自動昇格は使わない。
+
+### 変更点
+
+- **model policy**: 意味論的な auditor role、production selection、`gpt-5.6-luna × low` /
+  `gpt-5.6-terra × low` / `gpt-5.6-terra × medium` の評価 profile、policy version、検証状態を
+  単一 module に集約した。medium追加時にpolicy versionを2、production昇格時に3へ上げた。
+- **backend / diagnostics**: model と reasoning effort を backend 生成時に一度だけ解決し、成功・失敗の
+  structured result と diagnostics に effective selection、選択元、policy version、検証状態を残す。
+  model invocation が失敗しても別 model へ retry しない。
+- **比較 eval**: `spotter auditor model-matrix` を追加した。同じ versioned fixture を固定順序で実行し、
+  fixture hash、Codex CLI version、schema / exact match、false positive / negative、p50 / p95、timeout、
+  catalog 外 name と anomaly を bounded artifact に記録する。Codex JSONL `turn.completed.usage` から
+  token数だけを抽出し、raw event本文は保存しない。ChatGPTプランの金額costはAPI価格で代用せず
+  `not-available-chatgpt-plan` とし、artifact自身がmodel昇格を許可することはない。
+- **usage-limit diagnostics**: Codex CLIが実測済みの利用上限文言で非ゼロ終了した場合をgenericな
+  `E_CODEX_CLI_EXIT` から `E_CODEX_CLI_USAGE_LIMIT` へ分離した。認証失効を先に判定し、一般的な429は
+  誤分類しない。Hookはリセット時刻まで待つかCodexプランを確認する復旧案内を表示し、fallbackは行わない。
+- **model unavailable diagnostics**: Codex CLIが指定modelをChatGPTアカウントで利用できないと返した場合を
+  `E_CODEX_CLI_MODEL_UNAVAILABLE`へ分離した。認証・利用上限を優先し、providerのstdout/stderrはredact、
+  effective selectionとexit codeは診断用に保持する。別modelへのfallbackは行わない。
+- **公式model更新監視**: OpenAI公式のlatest-model / ChatGPT pricing Markdownの両方に揃った完全3種familyだけを
+  数値版比較し、週次workflowが評価提案Issueを重複なく作る。policy書換え・model呼出・自動昇格は行わない。
+- **運用SLO / Stop実測**: UserPromptSubmit / Stopのp50・p95・timeout率と品質gateを日本語で正本化した。
+  CLIのStop continuationはmax-1を確認した一方、App background/app-server taskはStop非発火だったため、
+  active Appのblock挙動が未確認のまま既存契約を変えず、pending deliveryを維持する。
+
+### 検証
+
+`auditor-model-matrix-cmd` / `auditor-cmd` / `cli` の targeted test 29 / 29 pass、全体
+439 / 437 pass / 0 fail / 2 skip。
+循環・非文字列 backend 応答、model selection 不一致、version 出力の秘密混入、dirty fixture、filtered
+hallucination を敵対的に再検証し、blocker 0 を確認した。代表 fixture の repeat=1 operational smoke は
+baseline / Luna / Terra の全12件が Codex CLI usage limit で `E_CODEX_CLI_EXIT` となったため、model 品質・
+latency・availability の比較には使わなかった。Pro20回復後のrepeat=3を2回実行し、Terra lowは合算
+23/24 exactでbaseline 18/24、Luna low 17/24より最良。usage対応runではtoken usageを36/36取得した。
+Terra lowにも見逃しが1件出たためmediumを追加評価した。Terra mediumは2回のrepeat=3で合計24/24 exact、
+FP/FN 0、timeout 0となり、owner裁定でproductionへ採用した。ChatGPTプランの金額costは取得不能として
+明示した。採用runは全体p95 4.361秒で、stage別SLOとtimeout/workload感度も
+[`docs/04_operational-slo.md`](https://github.com/kitepon-rgb/Spotter/blob/main/docs/04_operational-slo.md)へ固定した。
+
 ## 1.4.17
 
 v1.4.16 で実装済みだった stale Unix socket recovery を、v1.4.16 tag を改変せずこの patch の実配布候補に含める。
@@ -25,8 +70,10 @@ daemon 異常死後の orphan socket を次回起動前に安全に除去する
 
 `1c67698` の clean worktree から npm pack / temp prefix install / CLI version / Hook install・reinstall を
 smoke し、`node --test` は 383 / 381 pass / 0 fail / 2 skip。targeted test と adversarial review も
-blocker 0。OS CI matrix は macOS / Linux / Windows × Node 22.5.0 / 22.x の全6件が green。
-publish 後の npm / GitHub Release / fresh global install 三者一致確認は残る。
+blocker 0。v1.4.17-only README / CHANGELOG を載せた local release candidate は `6ea6a2b`。
+CLI help、58-entry pack、同じ full suite を再確認した。OS CI matrix はmacOS / Linux / Windows ×
+Node 22.5.0 / 22.xの全6件がgreen。最終 SHA `7987f2a`をtag / npm / GitHub Releaseへ公開し、
+npm `latest`とfresh global installの三者一致を確認した。
 
 ## 1.4.16
 

package/README.ja.md

@@ -172,6 +172,8 @@ spotter codex-hook install
                          # Codex native hooks の修復 / 明示登録 (通常は spotter install が実行)
 spotter codex-hook diagnostics
                          # Codex hook の登録/readiness を診断。trust は /hooks で review
+spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v1.json
+                         # pinned auditor model profile を再現可能に比較する experimental eval
 spotter uninstall        # hook 登録を解除 (~/.spotter は残す)
 ```
 
@@ -190,9 +192,11 @@ Primary auditor backend policy: Claude hooks の auto selection は PATH に Cod
 `SPOTTER_AUDITOR_BACKEND` の明示 override はどちらの host でも優先し、runtime failure では別 backend へ
 hidden fallback しません。
 Codex 側の SessionStart hook は `.spotter/tool-db.codex.json` を bg refresh し、Claude DB には触れません。
-Codex CLI auditor の子プロセスは、hook 判定を安く速く保つため既定で `gpt-5.4-mini` と
-`model_reasoning_effort="low"` を明示指定します。実測や制御された実験では
-`SPOTTER_CODEX_CLI_MODEL` / `SPOTTER_CODEX_CLI_REASONING_EFFORT` で上書きできます。
+Codex CLI auditor は versioned product policy を使い、production は反復 fixture 評価を通過した
+`gpt-5.6-terra × medium`。`gpt-5.6-luna × low` / `gpt-5.6-terra × low` は比較 profile として残し、
+profile から production へ自動昇格しません。`latest` alias や
+親 Codex の default を暗黙継承せず、失敗時に別 model へ retry しません。制御された実験では
+`SPOTTER_CODEX_CLI_MODEL` / `SPOTTER_CODEX_CLI_REASONING_EFFORT` で上書きでき、diagnostics は unverified と表示します。
 明示 smoke には `SPOTTER_AUDITOR_BACKEND=codex-sidecar` も使えます。
 
 ## 設計ドキュメント
@@ -207,7 +211,7 @@ Codex CLI auditor の子プロセスは、hook 判定を安く速く保つため
 
 - v1.4.8 以降、Claude / Codex 両 host で `Stop` hook は **遅延配送 (deferred delivery)** に統一されています。`Stop` で見落としツールを検出した場合、Spotter は `<projectRoot>/.spotter/pending/<sessionId>.json` に指摘を積み、次の same-session `UserPromptSubmit` で `additionalContext` として配信します。当ターンの最初の応答は transcript にそのまま残ります
 - pending ファイルは Claude / Codex が同じパス (`.spotter/pending/`) を共有します。host-neutral 設計です
-- **Haiku の JSON スキーマ違反は v0.5.0 以降「想定済み異常」として session renew + `role_collapse_reset` で回復**します。**v1.4.15 以降、auditor/daemon の失敗はプロンプトをブロックしません**: `UserPromptSubmit` は `[Spotter からの警告]` を出して exit 0。v1.4.17 では `Stop` 失敗も warning pending に積み、次の same-session prompt で1回配信します。直後に session が終わる場合だけ、配送先となる次 prompt がありません
+- **Haiku の JSON スキーマ違反は v0.5.0 以降「想定済み異常」として session renew + `role_collapse_reset` で回復**します。**v1.4.15 以降、auditor/daemon の失敗はプロンプトをブロックしません**: `UserPromptSubmit` は `[Spotter からの警告]` を出して exit 0。`Stop` 失敗も warning pending に積み、次の same-session prompt で1回配信します。直後に session が終わる場合だけ、配送先となる次 prompt がありません
 
 <details>
 <summary><strong>📋 最近のハイライト</strong></summary>

package/README.md

@@ -173,6 +173,8 @@ spotter codex-hook install
                          # repair / explicitly register Codex native hooks (normally handled by spotter install)
 spotter codex-hook diagnostics
                          # check Codex hook registration/readiness; trust is reviewed with /hooks
+spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v1.json
+                         # experimental reproducible comparison of pinned auditor model profiles
 spotter uninstall        # remove hooks from this project (leaves ~/.spotter intact)
 ```
 
@@ -191,10 +193,12 @@ otherwise the Haiku-compatible path. Codex native hooks automatically select Cod
 `SPOTTER_AUDITOR_BACKEND` override wins on either host; runtime failure never triggers a hidden fallback.
 The Codex SessionStart hook refreshes `.spotter/tool-db.codex.json` in the background
 without touching the Claude DB.
-Codex CLI auditor child processes explicitly use `gpt-5.4-mini` with
-`model_reasoning_effort="low"` by default so hook checks stay cheap and fast;
+Codex CLI auditor child processes use a versioned product policy. The production selection is
+`gpt-5.6-terra × medium`, promoted after repeated fixture evaluation. `gpt-5.6-luna × low` and
+`gpt-5.6-terra × low` remain comparison profiles; profiles never trigger automatic upgrades.
+Spotter does not inherit a `latest` alias or the parent Codex default, and an invocation failure never retries another model.
 `SPOTTER_CODEX_CLI_MODEL` and `SPOTTER_CODEX_CLI_REASONING_EFFORT` can override
-those values for smoke tests or controlled experiments.
+the production values for controlled experiments; diagnostics mark overrides as unverified.
 `SPOTTER_AUDITOR_BACKEND=codex-sidecar` is available for explicit sidecar auditor smoke.
 
 ## Design docs
@@ -209,7 +213,7 @@ those values for smoke tests or controlled experiments.
 
 - The `Stop` hook fires **after** the first answer has already been streamed. Spotter therefore queues a finding for the next same-session prompt instead of rewriting that answer. Detection accuracy in `UserPromptSubmit` (the *pre-response* stage) remains the primary quality axis
 - `Stop` hook is **deferred** for both Claude and Codex hosts as of v1.4.8. When Spotter finds a missed tool at `Stop`, it appends the finding to `<projectRoot>/.spotter/pending/<sessionId>.json` and surfaces it on the next same-session `UserPromptSubmit` as `additionalContext`. The original assistant message stays as the turn's final transcript entry — no `decision:"block"` re-generation cycle. The same pending file is shared by Claude and Codex (host-neutral path)
-- **Since v0.5.0, JSON schema violations from Haiku are treated as expected anomalies** (session renew + `role_collapse_reset`). **Since v1.4.15, an auditor/daemon failure no longer blocks the prompt**: `UserPromptSubmit` emits a loud `[Spotter からの警告]` and exits 0. In v1.4.17, a `Stop` failure is queued as the same kind of warning and delivered once on the next same-session prompt. If the session ends immediately, no later prompt exists and that final warning cannot be surfaced
+- **Since v0.5.0, JSON schema violations from Haiku are treated as expected anomalies** (session renew + `role_collapse_reset`). **Since v1.4.15, an auditor/daemon failure no longer blocks the prompt**: `UserPromptSubmit` emits a loud `[Spotter からの警告]` and exits 0. A `Stop` failure is queued as the same kind of warning and delivered once on the next same-session prompt. If the session ends immediately, no later prompt exists and that final warning cannot be surfaced
 
 <details>
 <summary><strong>📋 Recent highlights</strong></summary>

package/bin/spotter.mjs

@@ -50,6 +50,8 @@ Usage:
                                         (experimental) run primary auditor backend once
   spotter auditor matrix --stage STAGE --input FILE
                                         (experimental) compare primary auditor backend matrix
+  spotter auditor model-matrix --fixtures FILE
+                                        (experimental) evaluate pinned Codex auditor profiles
   spotter daemon start --session-id ID  (internal) run session daemon
   spotter hook <event>                  (internal) hook dispatch
                                         events: session-start | user-prompt |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-spotter",
-  "version": "1.4.17",
+  "version": "1.4.18",
   "description": "Audit agent running alongside Claude Code that catches missed tool calls — 気づく役と実行する役の分離",
   "type": "module",
   "bin": {

package/scripts/check-codex-model-policy.mjs

@@ -0,0 +1,151 @@
+import { CODEX_AUDITOR_MODEL_POLICY } from '../src/core/codex-auditor-model-policy.mjs';
+
+export const LATEST_MODELS_URL = 'https://developers.openai.com/api/docs/guides/latest-model.md';
+export const PRICING_URL = 'https://learn.chatgpt.com/docs/pricing.md';
+export const MAX_BODY_BYTES = 1024 * 1024;
+export const FETCH_TIMEOUT_MS = 15_000;
+
+const ROLES = ['sol', 'terra', 'luna'];
+const PROPOSAL = '同じfixtureでLuna low/Terra lowを比較し、品質不足時のみTerra mediumを評価する。productionの自動昇格・書換えは行わない。';
+
+export class ModelPolicyCheckError extends Error {
+  constructor(code) {
+    super(code);
+    this.name = 'ModelPolicyCheckError';
+    this.code = code;
+  }
+}
+
+function fail(code) { throw new ModelPolicyCheckError(code); }
+
+function versionKey(major, minor) { return `${Number(major)}.${Number(minor)}`; }
+
+export function compareVersions(left, right) {
+  const [leftMajor, leftMinor] = left.split('.').map(Number);
+  const [rightMajor, rightMinor] = right.split('.').map(Number);
+  return leftMajor - rightMajor || leftMinor - rightMinor;
+}
+
+export function extractCompleteFamilies(text, source) {
+  const regex = source === 'latest'
+    ? /\bgpt-(\d+)\.(\d+)-(sol|terra|luna)\b/gi
+    : /\bGPT-(\d+)\.(\d+)\s+(Sol|Terra|Luna)\b/g;
+  const byVersion = new Map();
+  for (const match of text.matchAll(regex)) {
+    const version = versionKey(match[1], match[2]);
+    const roles = byVersion.get(version) ?? new Set();
+    roles.add(match[3].toLowerCase());
+    byVersion.set(version, roles);
+  }
+  return [...byVersion.entries()]
+    .filter(([, roles]) => ROLES.every((role) => roles.has(role)))
+    .map(([version]) => ({ version, models: ROLES.map((role) => `gpt-${version}-${role}`) }))
+    .sort((a, b) => compareVersions(a.version, b.version));
+}
+
+async function readBoundedBody(response) {
+  const declaredLength = Number(response.headers?.get?.('content-length'));
+  if (Number.isFinite(declaredLength) && declaredLength > MAX_BODY_BYTES) fail('E_MODEL_POLICY_CHECK_BODY_TOO_LARGE');
+  if (!response.body?.getReader) {
+    const text = await response.text();
+    if (Buffer.byteLength(text) > MAX_BODY_BYTES) fail('E_MODEL_POLICY_CHECK_BODY_TOO_LARGE');
+    return text;
+  }
+  const reader = response.body.getReader();
+  const chunks = [];
+  let size = 0;
+  try {
+    for (;;) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      size += value.byteLength;
+      if (size > MAX_BODY_BYTES) {
+        await reader.cancel();
+        fail('E_MODEL_POLICY_CHECK_BODY_TOO_LARGE');
+      }
+      chunks.push(value);
+    }
+  } finally {
+    reader.releaseLock?.();
+  }
+  return new TextDecoder().decode(Buffer.concat(chunks));
+}
+
+export async function fetchMarkdown(url, { fetchFn = fetch, timeoutMs = FETCH_TIMEOUT_MS } = {}) {
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), timeoutMs);
+  try {
+    let response;
+    try {
+      response = await fetchFn(url, { signal: controller.signal, headers: { accept: 'text/markdown,text/plain;q=0.9' } });
+    } catch {
+      fail('E_MODEL_POLICY_CHECK_FETCH_FAILED');
+    }
+    if (!response || response.status !== 200) fail('E_MODEL_POLICY_CHECK_HTTP_STATUS');
+    return await readBoundedBody(response);
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+function policyProduction(policy) {
+  const model = policy?.production?.model;
+  const match = typeof model === 'string' && /^gpt-(\d+)\.(\d+)-(sol|terra|luna)$/.exec(model);
+  if (!match) fail('E_MODEL_POLICY_CHECK_INVALID_POLICY');
+  return { model, version: versionKey(match[1], match[2]) };
+}
+
+export async function runModelPolicyCheck({
+  fetchFn = fetch,
+  now = () => new Date(),
+  policy = CODEX_AUDITOR_MODEL_POLICY,
+  timeoutMs = FETCH_TIMEOUT_MS,
+} = {}) {
+  const production = policyProduction(policy);
+  const [latestMarkdown, pricingMarkdown] = await Promise.all([
+    fetchMarkdown(LATEST_MODELS_URL, { fetchFn, timeoutMs }),
+    fetchMarkdown(PRICING_URL, { fetchFn, timeoutMs }),
+  ]);
+  const latestFamilies = extractCompleteFamilies(latestMarkdown, 'latest');
+  const pricingFamilies = extractCompleteFamilies(pricingMarkdown, 'pricing');
+  if (latestFamilies.length === 0 || pricingFamilies.length === 0) fail('E_MODEL_POLICY_CHECK_REQUIRED_FAMILY_MISSING');
+  const latestDetected = latestFamilies.at(-1);
+  const pricingDetected = pricingFamilies.at(-1);
+  if (latestDetected.version !== pricingDetected.version) fail('E_MODEL_POLICY_CHECK_SOURCE_MISMATCH');
+  const pricingVersionSet = new Set(pricingFamilies.map(({ version }) => version));
+  const commonFamilies = latestFamilies.filter(({ version }) => pricingVersionSet.has(version));
+  const detectedFamily = latestDetected;
+  const status = compareVersions(detectedFamily.version, production.version) > 0 ? 'update-available' : 'current';
+  const diagnostics = status === 'current'
+    && commonFamilies.some(({ version }) => version !== production.version)
+    ? ['E_MODEL_POLICY_CHECK_NON_PRODUCTION_CANDIDATE_PRESENT']
+    : [];
+  return {
+    schema: 'spotter.codex_model_update_check.v1',
+    checkedAt: now().toISOString(),
+    policy: { version: policy.policyVersion, production: policy.production.model },
+    sources: [LATEST_MODELS_URL, PRICING_URL],
+    detectedFamily,
+    candidates: commonFamilies,
+    status,
+    proposal: status === 'update-available' ? PROPOSAL : null,
+    diagnostics,
+  };
+}
+
+export async function main({ stdout = process.stdout, stderr = process.stderr, run = runModelPolicyCheck } = {}) {
+  try {
+    const artifact = await run();
+    stdout.write(`${JSON.stringify(artifact)}\n`);
+    return 0;
+  } catch (error) {
+    const code = error instanceof ModelPolicyCheckError ? error.code : 'E_MODEL_POLICY_CHECK_UNEXPECTED';
+    stderr.write(`${code}\n`);
+    return 1;
+  }
+}
+
+if (import.meta.url === new URL(process.argv[1], 'file:').href) {
+  const exitCode = await main();
+  process.exitCode = exitCode;
+}

package/src/cli/auditor-cmd.mjs

@@ -2,6 +2,7 @@ import { readFile } from 'node:fs/promises';
 import { resolve } from 'node:path';
 import { createAuditorBackend } from '../core/auditor-backend.mjs';
 import { readLocal } from '../tool-db/refresh.mjs';
+import { runAuditorModelMatrixCommand } from './auditor-model-matrix-cmd.mjs';
 
 const AUDITOR_USAGE = `spotter auditor — experimental primary auditor smoke commands
 
@@ -10,6 +11,8 @@ Usage:
                         [--project DIR] [--host-agent claude|codex|automation|unknown]
                         [--backend haiku|codex-cli|codex-sidecar|auto]
   spotter auditor matrix --stage user_input|turn_end --input FILE [--project DIR]
+  spotter auditor model-matrix --fixtures FILE [--profile baseline|luna|terra|terra-medium]...
+                               [--repeat N] [--project DIR] [--output FILE]
 
 Input JSON:
   user_input: {"user_input":"..."} or {"userInput":"..."}
@@ -32,6 +35,14 @@ export async function runAuditorCommand({ argv = process.argv.slice(2) } = {}) {
     await runAuditorMatrixCommand({ argv: argv.slice(1) });
     return;
   }
+  if (sub === 'model-matrix') {
+    if (argv.slice(1).includes('--help') || argv.slice(1).includes('-h')) {
+      process.stdout.write(`Usage: spotter auditor model-matrix --fixtures FILE [--profile baseline|luna|terra|terra-medium]...\n       [--repeat N] [--project DIR] [--output FILE]\n`);
+      return;
+    }
+    await runAuditorModelMatrixCommand({ argv: argv.slice(1) });
+    return;
+  }
   process.stderr.write(`unknown auditor subcommand: ${sub}\n${AUDITOR_USAGE}`);
   process.exit(2);
 }

package/src/cli/auditor-model-matrix-cmd.mjs

@@ -0,0 +1,181 @@
+import { createHash } from 'node:crypto';
+import { execFile } from 'node:child_process';
+import { promisify } from 'node:util';
+import { readFile, writeFile } from 'node:fs/promises';
+import { isAbsolute, relative, resolve, sep } from 'node:path';
+import { version } from '../version.mjs';
+import { createCodexCliAuditorBackend } from '../core/codex-cli-backend.mjs';
+import { CODEX_AUDITOR_MODEL_POLICY, resolveCodexAuditorModelSelection } from '../core/codex-auditor-model-policy.mjs';
+
+const execFileAsync = promisify(execFile);
+const ALLOWED_PROFILES = ['baseline', 'luna', 'terra', 'terra-medium'];
+const DEFAULT_PROFILES = ['baseline', 'luna', 'terra'];
+export const MAX_MODEL_MATRIX_RUNS = 300;
+
+export async function runAuditorModelMatrixCommand({
+  argv = [], env = process.env, now = () => Date.now(), readFileFn = readFile, writeFileFn = writeFile,
+  createBackendFn = createCodexCliAuditorBackend, resolveSelectionFn = resolveCodexAuditorModelSelection,
+  getCodexCliVersionFn = getCodexCliVersion, generatedAt = () => new Date().toISOString(), writeOutput = (text) => process.stdout.write(text),
+} = {}) {
+  const opts = parseArgs(argv);
+  const raw = await readFileFn(opts.fixturesPath);
+  const fixtureBytes = Buffer.isBuffer(raw) ? raw : Buffer.from(raw);
+  const fixture = parseAndValidateFixture(fixtureBytes.toString('utf8'));
+  if (fixture.cases.length * opts.repeat * opts.profiles.length > MAX_MODEL_MATRIX_RUNS) {
+    throw new Error(`model-matrix run count exceeds maximum ${MAX_MODEL_MATRIX_RUNS}`);
+  }
+  const selections = Object.fromEntries(opts.profiles.map((profile) => [profile, resolveSelectionFn({ env, profile })]));
+  const backends = Object.fromEntries(opts.profiles.map((profile) => {
+    const backend = createBackendFn({ catalog: fixture.catalog, projectRoot: opts.projectRoot, env, modelProfile: profile });
+    if (!sameSelection(selections[profile], backend?.modelSelection)) {
+      throw new Error(`backend model selection does not match preflight profile: ${profile}`);
+    }
+    return [profile, backend];
+  }));
+  const cliVersion = normalizeCodexCliVersion(await getCodexCliVersionFn({ env }));
+  const runs = [];
+  let order = 0;
+  for (const item of fixture.cases) for (let repeat = 1; repeat <= opts.repeat; repeat += 1) for (const profile of opts.profiles) {
+    const startedAt = now();
+    const modelSelection = selections[profile];
+    try {
+      const backend = backends[profile];
+      const judgment = await backend.judge({ ...toAuditorInput(item), meta: { caseId: item.id, repeat, profile } });
+      if (!sameSelection(modelSelection, judgment?.meta?.modelSelection)) {
+        throw new Error('judgment model selection does not match backend model selection');
+      }
+      const rawFindings = Array.isArray(judgment?.findings) ? judgment.findings : [];
+      const rawActualTools = rawFindings.map((finding) => finding?.toolName);
+      const actualTools = rawActualTools.filter(isCleanString);
+      const invalidFindingCount = rawActualTools.length - actualTools.length;
+      const actualPass = typeof judgment?.pass === 'boolean' ? judgment.pass : null;
+      const rawDroppedTools = Array.isArray(judgment?.meta?.diagnostics?.droppedCatalogExternalNames)
+        ? judgment.meta.diagnostics.droppedCatalogExternalNames
+        : [];
+      const rawAnomalies = Array.isArray(judgment?.anomalies) ? judgment.anomalies : [];
+      const droppedTools = cleanStrings(rawDroppedTools);
+      const anomalyTypes = cleanStrings(rawAnomalies.map((anomaly) => anomaly?.type));
+      const tokenUsage = safeTokenUsage(judgment?.meta?.diagnostics?.tokenUsage);
+      const schemaSuccess = actualPass !== null && Array.isArray(judgment?.findings)
+        && invalidFindingCount === 0 && new Set(actualTools).size === actualTools.length
+        && actualPass === (actualTools.length === 0);
+      runs.push(successRun({
+        order: ++order,
+        item,
+        repeat,
+        profile,
+        modelSelection,
+        durationMs: now() - startedAt,
+        schemaSuccess,
+        actualPass,
+        actualTools,
+        invalidFindingCount,
+        droppedTools,
+        droppedToolCount: rawDroppedTools.length,
+        anomalyTypes,
+        anomalyCount: rawAnomalies.length,
+        tokenUsage,
+      }));
+    } catch (err) {
+      runs.push({ order: ++order, caseId: item.id, repeat, profile, status: 'error', durationMs: now() - startedAt,
+        schemaSuccess: false, exactMatch: false, expected: item.expected, actual: { pass: null, missingTools: [], invalidFindingCount: 0, droppedCatalogExternalNames: [], droppedCatalogExternalNameCount: 0, anomalies: [], anomalyCount: 0 }, falsePositiveTools: [], falseNegativeTools: [], modelSelection,
+        tokenUsage: null, error: safeError(err) });
+    }
+  }
+  const usage = summarizeTokenUsage(runs, opts.profiles);
+  const artifact = {
+    schema: 'spotter.auditor_model_matrix.v1', generatedAt: generatedAt(), packageVersion: version,
+    fixture: { schema: fixture.schema, path: safeFixturePath(opts.fixturesPath, opts.projectRoot), sha256: createHash('sha256').update(fixtureBytes).digest('hex'), cases: fixture.cases.length, catalogCount: fixture.catalog.length },
+    codexCli: cliVersion, policy: { schema: CODEX_AUDITOR_MODEL_POLICY.schema, version: CODEX_AUDITOR_MODEL_POLICY.policyVersion },
+    profiles: Object.fromEntries(Object.entries(selections).map(([profile, selection]) => [profile, {
+      model: selection.effectiveModel,
+      reasoningEffort: selection.effectiveReasoningEffort,
+      verifiedAt: selection.effectiveVerifiedAt,
+      verificationScope: selection.effectiveVerificationScope ?? null,
+      status: selection.effectiveStatus ?? null,
+      selection,
+    }])), runs, summary: summarize(runs, opts.profiles), usageStatus: usage.status, tokenUsage: usage.summary,
+    costStatus: 'not-available-chatgpt-plan', cost: null,
+    evaluation: { repeat: opts.repeat, profiles: opts.profiles, maxRuns: MAX_MODEL_MATRIX_RUNS },
+    executionOrdering: 'case-repeat-profile', promotionEligible: false, blockingReasons: blockingReasons(runs, usage.status),
+  };
+  const json = JSON.stringify(artifact, null, 2) + '\n';
+  if (opts.outputPath) await writeFileFn(opts.outputPath, json, 'utf8');
+  writeOutput(json);
+  return artifact;
+}
+
+function parseArgs(argv) {
+  const opts = { fixturesPath: null, profiles: [], repeat: 1, projectRoot: process.cwd(), outputPath: null };
+  for (let i = 0; i < argv.length; i += 1) {
+    const arg = argv[i]; const value = () => { const v = argv[++i]; if (!v || v.startsWith('--')) throw new Error(`${arg} requires a value`); return v; };
+    if (arg === '--fixtures') opts.fixturesPath = resolve(value());
+    else if (arg === '--profile') opts.profiles.push(value());
+    else if (arg === '--repeat') opts.repeat = Number(value());
+    else if (arg === '--project') opts.projectRoot = resolve(value());
+    else if (arg === '--output') opts.outputPath = resolve(value());
+    else throw new Error(`unknown auditor model-matrix option: ${arg}`);
+  }
+  if (!opts.fixturesPath) throw new Error('--fixtures FILE is required');
+  if (!Number.isInteger(opts.repeat) || opts.repeat < 1) throw new Error('--repeat must be a positive integer');
+  opts.profiles = opts.profiles.length ? opts.profiles : [...DEFAULT_PROFILES];
+  if (new Set(opts.profiles).size !== opts.profiles.length || opts.profiles.some((profile) => !ALLOWED_PROFILES.includes(profile))) throw new Error('--profile must be baseline, luna, terra, or terra-medium without duplicates');
+  return opts;
+}
+
+function parseAndValidateFixture(raw) {
+  let fixture; try { fixture = JSON.parse(raw); } catch { throw new Error('fixture must be valid JSON'); }
+  objectOnly(fixture, ['schema', 'catalog', 'cases'], 'fixture');
+  if (fixture.schema !== 'spotter.auditor_model_fixtures.v1') throw new Error('unsupported fixture schema');
+  if (!Array.isArray(fixture.catalog) || !Array.isArray(fixture.cases) || fixture.cases.length === 0) throw new Error('fixture catalog and non-empty cases are required');
+  const catalogNames = new Set();
+  for (const tool of fixture.catalog) { objectOnly(tool, ['name', 'description'], 'catalog tool'); clean(tool.name, 'catalog name'); clean(tool.description, 'catalog description'); if (catalogNames.has(tool.name)) throw new Error('duplicate catalog name'); catalogNames.add(tool.name); }
+  const ids = new Set();
+  for (const item of fixture.cases) {
+    objectOnly(item, ['id', 'stage', 'input', 'expected'], 'case'); clean(item.id, 'case id');
+    if (ids.has(item.id)) throw new Error('duplicate case id'); ids.add(item.id);
+    if (!['user_input', 'turn_end'].includes(item.stage)) throw new Error('case stage is invalid');
+    validateInput(item.stage, item.input); objectOnly(item.expected, ['pass', 'missingTools'], 'case expected');
+    if (typeof item.expected.pass !== 'boolean' || !Array.isArray(item.expected.missingTools)
+      || new Set(item.expected.missingTools).size !== item.expected.missingTools.length
+      || item.expected.missingTools.some((name) => typeof name !== 'string' || !catalogNames.has(name))
+      || item.expected.pass !== (item.expected.missingTools.length === 0)) throw new Error('case expected is invalid or references a catalog-external tool');
+  }
+  return fixture;
+}
+function objectOnly(value, keys, label) { if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some((key) => !keys.includes(key))) throw new Error(`${label} has invalid fields`); }
+function clean(value, label) { if (typeof value !== 'string' || value.length === 0 || value.trim() !== value) throw new Error(`${label} must be a clean non-empty string`); }
+function validateInput(stage, input) { if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('case input is invalid'); if (stage === 'user_input') { objectOnly(input, ['userInput'], 'user_input input'); clean(input.userInput, 'userInput'); } else { objectOnly(input, ['finalResponse', 'usedTools'], 'turn_end input'); clean(input.finalResponse, 'finalResponse'); if (!Array.isArray(input.usedTools) || input.usedTools.some((tool) => typeof tool !== 'string' || tool.length === 0 || tool.trim() !== tool) || new Set(input.usedTools).size !== input.usedTools.length) throw new Error('usedTools is invalid'); } }
+function toAuditorInput(item) { return item.stage === 'user_input' ? { stage: item.stage, userInput: item.input.userInput } : { stage: item.stage, finalResponse: item.input.finalResponse, usedTools: item.input.usedTools }; }
+function isCleanString(value) { return typeof value === 'string' && value.length > 0 && value.trim() === value; }
+function cleanStrings(values = []) { return [...new Set(values.filter(isCleanString))]; }
+function successRun({ order, item, repeat, profile, modelSelection, durationMs, schemaSuccess, actualPass, actualTools, invalidFindingCount, droppedTools, droppedToolCount, anomalyTypes, anomalyCount, tokenUsage }) { const expectedTools = item.expected.missingTools; const fp = [...actualTools.filter((tool) => !expectedTools.includes(tool)), ...droppedTools.filter((tool) => !expectedTools.includes(tool))]; const fn = expectedTools.filter((tool) => !actualTools.includes(tool)); return { order, caseId: item.id, repeat, profile, status: 'success', durationMs, schemaSuccess, exactMatch: schemaSuccess && actualPass === item.expected.pass && actualTools.length === expectedTools.length && fp.length === 0 && fn.length === 0 && droppedToolCount === 0 && anomalyCount === 0, expected: item.expected, actual: { pass: actualPass, missingTools: actualTools, invalidFindingCount, droppedCatalogExternalNames: droppedTools, droppedCatalogExternalNameCount: droppedToolCount, anomalies: anomalyTypes, anomalyCount }, falsePositiveTools: fp, falseNegativeTools: fn, modelSelection, tokenUsage }; }
+const SAFE_RUN_ERRORS = Object.freeze({
+  E_CODEX_CLI_AUTH: { name: 'AuditorBackendError', message: 'codex-cli authentication failed; run codex login' },
+  E_CODEX_CLI_EXIT: { name: 'AuditorBackendError', message: 'codex-cli invocation exited unsuccessfully' },
+  E_CODEX_CLI_MODEL_POLICY: { name: 'CodexAuditorModelPolicyError', message: 'codex-cli model policy validation failed' },
+  E_CODEX_CLI_NO_FINAL_JSON: { name: 'AuditorBackendError', message: 'codex-cli did not produce a final JSON result' },
+  E_CODEX_CLI_SCHEMA: { name: 'AuditorBackendError', message: 'codex-cli returned an invalid auditor result' },
+  E_CODEX_CLI_SPAWN: { name: 'AuditorBackendError', message: 'codex-cli invocation could not start' },
+  E_CODEX_CLI_TIMEOUT: { name: 'AuditorBackendError', message: 'codex-cli invocation timed out' },
+  E_CODEX_CLI_USAGE_LIMIT: { name: 'AuditorBackendError', message: 'codex-cli usage limit reached; wait for reset or change plan' },
+  E_AUDITOR_MODEL_MATRIX: { name: 'Error', message: 'auditor model-matrix run failed' },
+});
+const SAFE_DIAGNOSTIC_ENUMS = Object.freeze({
+  completionReason: new Set(['last_message_before_process_close']),
+  lastMessageCheck: new Set(['schema_valid', 'missing_last_message', 'schema_invalid_last_message']),
+  processCountMethod: new Set(['direct_child_spawn', 'direct_child_exec_file', 'spawn_failed', 'not_available', 'not_instrumented']),
+});
+function safeFixturePath(fixturePath, projectRoot) { const path = relative(projectRoot, fixturePath); if (path === '') return '.'; if (path === '..' || path.startsWith(`..${sep}`) || isAbsolute(path)) return '<external-fixture>'; return path.split(sep).join('/'); }
+function safeError(err) { const code = Object.hasOwn(SAFE_RUN_ERRORS, err?.code) ? err.code : 'E_AUDITOR_MODEL_MATRIX'; const safe = SAFE_RUN_ERRORS[code]; const diagnostics = compact(err?.diagnostics); const stage = ['user_input', 'turn_end', 'configuration'].includes(err?.stage) ? err.stage : 'unknown'; return { code, name: safe.name, message: safe.message, backend: 'codex-cli', stage, ...(diagnostics ? { diagnostics } : {}) }; }
+function compact(value) { if (!value || typeof value !== 'object' || Array.isArray(value)) return null; const out = {}; for (const key of ['durationMs', 'processCount', 'exitCode']) if (typeof value[key] === 'number' && Number.isFinite(value[key])) out[key] = value[key]; for (const key of ['stdoutTruncated', 'stderrTruncated']) if (typeof value[key] === 'boolean') out[key] = value[key]; for (const [key, allowed] of Object.entries(SAFE_DIAGNOSTIC_ENUMS)) if (allowed.has(value[key])) out[key] = value[key]; for (const key of ['stdout', 'stderr']) if (typeof value[key] === 'string' || Buffer.isBuffer(value[key])) out[`${key}Bytes`] = Buffer.byteLength(value[key]); return Object.keys(out).length ? out : null; }
+export function percentile(values, ratio) { if (!values.length) return null; const sorted = [...values].sort((a, b) => a - b); return sorted[Math.ceil(sorted.length * ratio) - 1]; }
+function summarize(rows, profiles) { const one = (items) => { const durations = items.map((item) => item.durationMs); const timeout = items.filter((item) => item.error?.code === 'E_CODEX_CLI_TIMEOUT').length; return { total: items.length, success: items.filter((item) => item.status === 'success').length, error: items.filter((item) => item.status === 'error').length, schemaSuccess: items.filter((item) => item.schemaSuccess).length, exactMatch: items.filter((item) => item.exactMatch).length, falsePositiveTools: items.reduce((sum, item) => sum + item.falsePositiveTools.length, 0), falseNegativeTools: items.reduce((sum, item) => sum + item.falseNegativeTools.length, 0), invalidFindingCount: items.reduce((sum, item) => sum + item.actual.invalidFindingCount, 0), anomalyCount: items.reduce((sum, item) => sum + item.actual.anomalyCount, 0), droppedCatalogExternalNameCount: items.reduce((sum, item) => sum + item.actual.droppedCatalogExternalNameCount, 0), durationMs: { p50: percentile(durations, 0.5), p95: percentile(durations, 0.95) }, timeoutCount: timeout, timeoutRate: items.length ? timeout / items.length : 0 }; }; return { ...one(rows), byProfile: Object.fromEntries(profiles.map((profile) => [profile, one(rows.filter((row) => row.profile === profile))])) }; }
+function safeTokenUsage(value) { const keys = ['inputTokens', 'cachedInputTokens', 'outputTokens', 'reasoningOutputTokens', 'totalTokens']; if (!value || typeof value !== 'object' || keys.some((key) => !Number.isSafeInteger(value[key]) || value[key] < 0) || value.totalTokens !== value.inputTokens + value.outputTokens) return null; return Object.fromEntries(keys.map((key) => [key, value[key]])); }
+function summarizeTokenUsage(runs, profiles) { const observed = runs.filter((run) => run.tokenUsage); const one = (items) => { const withUsage = items.filter((item) => item.tokenUsage); const total = (key) => withUsage.reduce((sum, item) => sum + item.tokenUsage[key], 0); const totals = { inputTokens: total('inputTokens'), cachedInputTokens: total('cachedInputTokens'), outputTokens: total('outputTokens'), reasoningOutputTokens: total('reasoningOutputTokens'), totalTokens: total('totalTokens') }; return { observedRuns: withUsage.length, totalRuns: items.length, totals, totalTokensPerRun: { p50: percentile(withUsage.map((item) => item.tokenUsage.totalTokens), 0.5), p95: percentile(withUsage.map((item) => item.tokenUsage.totalTokens), 0.95) } }; }; const status = observed.length === 0 ? 'not-available' : observed.length === runs.length ? 'complete' : 'partial'; return { status, summary: observed.length === 0 ? null : { ...one(runs), byProfile: Object.fromEntries(profiles.map((profile) => [profile, one(runs.filter((run) => run.profile === profile))])) } }; }
+function blockingReasons(runs, usageStatus) { const reasons = []; if (usageStatus !== 'complete') reasons.push('usage_not_available'); reasons.push('cost_not_available'); if (runs.some((run) => run.status === 'error')) reasons.push('run_error'); if (runs.some((run) => !run.schemaSuccess)) reasons.push('schema_failure'); if (runs.some((run) => !run.exactMatch)) reasons.push('exact_mismatch'); if (runs.some((run) => run.actual.anomalyCount > 0 || run.actual.droppedCatalogExternalNameCount > 0)) reasons.push('quality_anomaly'); return reasons; }
+function sameSelection(expected, actual) { if (!expected || !actual || typeof expected !== 'object' || typeof actual !== 'object') return false; const keys = ['effectiveModel', 'effectiveReasoningEffort', 'modelSource', 'effortSource', 'policySchema', 'policyVersion', 'policyVerifiedAt', 'policyVerificationScope', 'effectiveVerifiedAt', 'effectiveStatus', 'effectiveVerificationScope', 'role', 'availability']; return keys.every((key) => expected[key] === actual[key]); }
+function byteCount(value) { return typeof value === 'string' || Buffer.isBuffer(value) ? Buffer.byteLength(value) : 0; }
+function safeVersionCode(value) { return typeof value === 'string' && /^E_[A-Z0-9_]{1,80}$/.test(value) ? value : 'E_CODEX_CLI_VERSION_INVALID'; }
+function normalizeCodexCliVersion(value) { const status = value?.status; if (status === 'available' && typeof value.version === 'string' && value.version.length <= 200 && !/[\r\n]/.test(value.version) && /^codex(?:-cli)?\s+v?\d+(?:\.\d+){1,3}(?:[-+][A-Za-z0-9.-]+)?$/.test(value.version)) return { status: 'available', version: value.version }; if (status === 'unavailable') return { status: 'unavailable', ...(value?.code ? { code: safeVersionCode(value.code) } : {}) }; return { status: 'error', code: safeVersionCode(value?.code), stdoutBytes: byteCount(value?.stdout ?? value?.version), stderrBytes: byteCount(value?.stderr) }; }
+async function getCodexCliVersion({ env = process.env } = {}) { try { const { stdout } = await execFileAsync('codex', ['--version'], { timeout: 5_000, encoding: 'utf8', env }); return { status: 'available', version: stdout.trim() }; } catch (err) { return { status: err?.code === 'ENOENT' ? 'unavailable' : 'error', code: err?.code === 'ENOENT' ? 'E_CODEX_CLI_NOT_FOUND' : 'E_CODEX_CLI_VERSION_COMMAND', stdout: err?.stdout, stderr: err?.stderr }; } }