claude-spotter 1.4.18 → 1.4.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/README.ja.md +61 -12
- package/README.md +65 -12
- package/bin/spotter.mjs +59 -4
- package/package.json +1 -1
- package/src/cli/auditor-cmd.mjs +1 -1
- package/src/cli/auditor-model-matrix-cmd.mjs +15 -8
- package/src/cli/codex-hook-cmd.mjs +152 -100
- package/src/cli/doctor.mjs +42 -1
- package/src/cli/install.mjs +15 -0
- package/src/core/auditor-backend.mjs +7 -0
- package/src/core/auditor-context.mjs +198 -0
- package/src/core/auditor-prompt-data.mjs +38 -0
- package/src/core/auditor-response.mjs +7 -14
- package/src/core/codex-cli-backend.mjs +35 -17
- package/src/core/codex-risk-dispatch.mjs +9 -1
- package/src/core/codex-sidecar-runner.mjs +14 -1
- package/src/core/hook-event-log.mjs +3 -3
- package/src/daemon/daemon.mjs +26 -2
- package/src/daemon/haiku-caller.mjs +11 -3
- package/src/hooks/lib.mjs +3 -65
- package/src/hooks/parent-output-projector.mjs +80 -0
- package/src/hooks/pending-context.mjs +11 -49
- package/src/hooks/stop.mjs +15 -75
- package/src/hooks/user-prompt.mjs +143 -71
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,45 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.4.20 — 2026-07-13
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- **Throughline監査文脈のopt-in配布。** 新規projectは既定`disabled`のまま、ownerがabsoluteなThroughline commandとrepeatableな先頭引数で有効化できる。freshな完了L2 user/assistant pairだけを`N=2`、body 600文字、total 4,000文字に制限し、Codex CLIへstdinで渡す。Haikuはこのcontext経路では呼ばず、fresh以外でもAIを呼ばない。connector障害は固定警告に限定する。親出力は安全なcatalog tool ID由来の固定・非命令形助言だけで、L2、reason、provider rawを反射しない。7日・30 fresh resultのproduction default昇格gateは未完了のため、配布後もproject opt-inを維持する。
|
|
8
|
+
|
|
9
|
+
## 1.4.19
|
|
10
|
+
|
|
11
|
+
親セッションの暴走を誘発できたHook出力の信頼境界を修正する。監査用AIは内部で構造化判定を返すだけとし、
|
|
12
|
+
その自由文やprovider出力を親モデルへ渡さない。親向け出力はSpotterプログラムが検証済みtool IDから
|
|
13
|
+
決定論的に生成する、採否を親が独立判断できる非命令形の助言へ限定する。
|
|
14
|
+
|
|
15
|
+
### 変更点
|
|
16
|
+
|
|
17
|
+
- **共通parent-output projector**: Claude / Codexの両Hookが同じprojectorを使う。tool IDは
|
|
18
|
+
ASCII grammar、160文字、5件、重複排除、安定sort、助言全体2,000文字の上限を持ち、改行・制御文字・
|
|
19
|
+
backtick・Markdown・超長大IDを拒否する。監査用AIの`reason` / `raw`はAPI入力に持たない。
|
|
20
|
+
- **非命令形のUserPrompt助言**: `additionalContext`はcatalog照合済みtool IDと固定テンプレートだけから
|
|
21
|
+
作る。「使え」「補正せよ」「ユーザーへ伝えよ」といった命令や監査用AIの説明文を含めない。
|
|
22
|
+
- **failure分離**: backend codeをallow-list済みの固定状態へ写像し、固定`systemMessage`・固定stderr・
|
|
23
|
+
構造Hook eventへ出す。backend message、provider stdout / stderr、未知code本文を反射しない。この保証は
|
|
24
|
+
Hook出力生成までで、全Codex App/background面でのUI可視性は未保証。
|
|
25
|
+
- **Stop持ち越し廃止**: finding / failureを`.spotter/pending/`へ新規保存せず、次の無関係な
|
|
26
|
+
UserPromptSubmitへ配送しない。findingは構造Hook event、failureは固定診断として当該Stopで完結する。
|
|
27
|
+
旧same-session pendingは内容を読まずにunlinkし、`ENOENT`以外の失敗も固定診断だけにする。
|
|
28
|
+
- **安全性維持**: `decision:"block"`、Stop継続、UserPromptSubmit exit 2の範囲拡大は行わず、既存の
|
|
29
|
+
再帰Hook / daemon proliferation防止、marker、short prompt、non-blocking failure契約を維持する。
|
|
30
|
+
|
|
31
|
+
### 検証
|
|
32
|
+
|
|
33
|
+
projector / Claude / Codexのtargeted testとfull suiteを実行し、447件中445 pass・0 fail・既存2 skip。
|
|
34
|
+
AI/backend/provider sentinel、unsafe ID、legacy pending非読取unlink、Stop次turn非配送、catalog/transcript
|
|
35
|
+
読込例外の固定degradationを確認した。敵対的再監査は初回BLOCKER 2件(Codex読込例外の境界外、
|
|
36
|
+
UI可視性の過剰主張)を検出・修正し、再監査BLOCKER 0。`npm pack --dry-run`は62 files、global
|
|
37
|
+
`spotter 1.4.19`へlocal installし、projector smoke・Hook diagnostics・global UserPromptSubmit smokeを
|
|
38
|
+
確認した。公開前release gateでfull test、pack、秘密混入を再検証し、公開SHA`5393919`のCIは
|
|
39
|
+
macOS/Linux/Windows × Node 22.5/22.xの6/6 green。`v1.4.19` tag、npm `latest`、
|
|
40
|
+
[GitHub Release](https://github.com/kitepon-rgb/Spotter/releases/tag/v1.4.19)、registry由来global installを
|
|
41
|
+
1.4.19へ揃え、global projector smokeとHook diagnosticsもgreen。
|
|
42
|
+
|
|
3
43
|
## 1.4.18
|
|
4
44
|
|
|
5
45
|
auditor model の更新を model 名の場当たり的な置換から切り離し、versioned policy と再現可能な比較 eval を
|
package/README.ja.md
CHANGED
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
|
|
16
16
|
Claude には「使えるツールがあるのに、使うべきタイミングで使わない」という構造的な弱点があります。記録すべき決定を memory / caveat MCP に残さない、docs lookup MCP を呼ばずに古い知識で応答する、ブラウザ自動化 MCP で確認せず UI 状態を推測する — **「分からないと自覚できない」から、ツールを取りに行けない**。
|
|
17
17
|
|
|
18
|
-
Spotter はツールカタログを完全に把握した別の監査エージェントで、ユーザー入力と主役 AI の応答を並走監査します。自動選択では Claude host は Codex CLI があればそれを、なければ session-scoped Haiku を選び、Codex host は Codex CLI を既定にします。明示 backend override は host より優先しますが、runtime failure で別 backend
|
|
18
|
+
Spotter はツールカタログを完全に把握した別の監査エージェントで、ユーザー入力と主役 AI の応答を並走監査します。自動選択では Claude host は Codex CLI があればそれを、なければ session-scoped Haiku を選び、Codex host は Codex CLI を既定にします。明示 backend override は host より優先しますが、runtime failure で別 backend へ黙って切り替えません。応答前は検証済みtool IDだけを固定・非命令形の助言へ変換でき、応答後のfindingは構造eventに留めて後続turnへ注入しません。監査用AIの自由文が親セッションへ入ることはありません。**主役 AI が自覚して自己監査する**設計は本プロダクトの存在意義を破壊するため、hook 経由でその意思と独立に検出します。
|
|
19
19
|
|
|
20
20
|
<p align="center">
|
|
21
21
|
<img src=".github/concept.svg" alt="Claude が答え、Spotter が見ている" width="80%">
|
|
@@ -62,6 +62,7 @@ Codex 側では現行の `[features].hooks = true` を有効化し、互換の
|
|
|
62
62
|
Spotter が所有する Codex handler は現行の同期 command schema で生成します。install / upgrade 後は `/hooks` で review して新しい Codex session を開いてください。`spotter codex-hook diagnostics` は登録と readiness を診断しますが、trust を内部状態から推測しません。
|
|
63
63
|
|
|
64
64
|
Spotter を upgrade した後、release note で hook 設定変更が案内されている場合は、各 install 済みプロジェクトで `spotter install` を再実行してください。global package update でコード経路は変わりますが、既存 `.claude/settings.json` の timeout 値は自動では書き換わりません。
|
|
65
|
+
`v1.4.19`はruntimeの出力変換だけを変更するため、install済みprojectで`spotter install`をやり直す必要はありません。global packageを更新し、新しいClaude/Codexセッションを開いてください。
|
|
65
66
|
|
|
66
67
|
```bash
|
|
67
68
|
spotter uninstall # このプロジェクトの hook 登録を解除
|
|
@@ -88,20 +89,19 @@ spotter codex-hook install
|
|
|
88
89
|
|
|
89
90
|
### 1 ターンの監査フロー
|
|
90
91
|
|
|
91
|
-
Claude Code と Codex
|
|
92
|
-
|
|
93
|
-
|
|
92
|
+
Claude Code と Codex は同じ安全なparent-output projectorを使います。監査用AIの自由文は内部に留め、
|
|
93
|
+
`UserPromptSubmit`では検証済みtool IDだけを固定・非命令形の助言へ変換します。`Stop` findingは
|
|
94
|
+
構造eventに記録し、後のturnへ注入しません。
|
|
94
95
|
|
|
95
96
|
```mermaid
|
|
96
97
|
flowchart TD
|
|
97
98
|
U([User 発話]) --> UPH[UserPromptSubmit hook<br/>Spotter が発話とカタログから一次判定]
|
|
98
|
-
UPH --> BT[
|
|
99
|
+
UPH --> BT[Host model<br/>検証済みtool IDの<br/>固定助言を受け取る場合がある]
|
|
99
100
|
BT --> BA([Claude の最初の応答])
|
|
100
101
|
BA --> SH[Stop hook<br/>応答と使用済みツールから最終チェック]
|
|
101
102
|
SH --> DEC{見落とし<br/>あり?}
|
|
102
103
|
DEC -->|なし| DONE([完了])
|
|
103
|
-
DEC -->|あり|
|
|
104
|
-
SB --> NEXT([次の UserPromptSubmit で<br/>additionalContext として配信])
|
|
104
|
+
DEC -->|あり| EVT[構造Hook eventへ記録<br/>次turnへは注入しない]
|
|
105
105
|
```
|
|
106
106
|
|
|
107
107
|
### カタログの収集経路
|
|
@@ -146,6 +146,54 @@ flowchart LR
|
|
|
146
146
|
|
|
147
147
|
両者に共通するのは **「主体に頼らない仕組み」**。併用できます。
|
|
148
148
|
|
|
149
|
+
### 任意の Throughline auditor context canary
|
|
150
|
+
|
|
151
|
+
新規の project install では auditor context は **disabled** が既定です。これは project が所有する
|
|
152
|
+
opt-in であり、Spotter 全体の production 既定ではありません。現在の候補は Spotter リポジトリだけの
|
|
153
|
+
canary です。auditor-context オプションなしで `spotter install` を再実行した場合、既存プロジェクトの
|
|
154
|
+
設定は保持されます。無効化へ戻すには次を実行します。
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
spotter install -y --auditor-context disabled
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
project を opt-in するには、絶対パスの Throughline 実行ファイルと、必要なら繰り返し指定できる
|
|
161
|
+
`--throughline-arg` を設定します。
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
spotter install -y --auditor-context throughline \
|
|
165
|
+
--throughline-command /absolute/path/to/throughline
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Windows では shell injection を避けるため `.cmd` / `.bat` wrapper を意図的に拒否します。絶対パスの
|
|
169
|
+
`node.exe` を command にし、絶対パスの `throughline.mjs` を繰り返し引数として渡してください。
|
|
170
|
+
|
|
171
|
+
```powershell
|
|
172
|
+
spotter install -y --auditor-context throughline `
|
|
173
|
+
--throughline-command 'C:\Program Files\nodejs\node.exe' `
|
|
174
|
+
--throughline-arg 'C:\absolute\path\to\throughline\bin\throughline.mjs'
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
この connector の AI 呼出しは Codex CLI のみです。コンテキストを argv には載せず、AI へは stdin 経由で
|
|
178
|
+
渡します。Haiku はこの context path に未対応であり、呼び出しません。Throughline 結果が `fresh` の時だけ
|
|
179
|
+
AI 呼出し候補になり、それ以外の status では AI を呼びません。enabled connector の障害は hidden fallback
|
|
180
|
+
ではなく固定 warning として出します。
|
|
181
|
+
|
|
182
|
+
Throughline から渡すのは fresh な完了済み L2 user/assistant pair だけです。直近 2 pair (N=2)、各 body は
|
|
183
|
+
600 文字、合計は 4,000 文字に制限します。Spotter は Throughline の L2、`reason`、`raw` を親へ反射せず、
|
|
184
|
+
親には安全な catalog tool ID から作る固定・非命令形の助言だけを渡します。`spotter doctor` は command / args
|
|
185
|
+
を表示せず、auditor-context mode と固定の availability detail だけを表示します。
|
|
186
|
+
|
|
187
|
+
v2 model-matrix では context choice を明示できます。
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v2.json \
|
|
191
|
+
--recent-turns 2 --body-cap 600
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
評価結論は現在 N=2 / 600 です。ただし rollout 承認ではありません。Spotter リポジトリ canary の
|
|
195
|
+
7 日・30 fresh result gate は未完了です。
|
|
196
|
+
|
|
149
197
|
## よく使うコマンド
|
|
150
198
|
|
|
151
199
|
```bash
|
|
@@ -172,7 +220,7 @@ spotter codex-hook install
|
|
|
172
220
|
# Codex native hooks の修復 / 明示登録 (通常は spotter install が実行)
|
|
173
221
|
spotter codex-hook diagnostics
|
|
174
222
|
# Codex hook の登録/readiness を診断。trust は /hooks で review
|
|
175
|
-
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.
|
|
223
|
+
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v2.json --recent-turns 2 --body-cap 600
|
|
176
224
|
# pinned auditor model profile を再現可能に比較する experimental eval
|
|
177
225
|
spotter uninstall # hook 登録を解除 (~/.spotter は残す)
|
|
178
226
|
```
|
|
@@ -209,15 +257,16 @@ profile から production へ自動昇格しません。`latest` alias や
|
|
|
209
257
|
|
|
210
258
|
## 既知の制約
|
|
211
259
|
|
|
212
|
-
-
|
|
213
|
-
-
|
|
214
|
-
- **Haiku の JSON スキーマ違反は v0.5.0 以降「想定済み異常」として session renew + `role_collapse_reset` で回復**します。**v1.4.
|
|
260
|
+
- `Stop` hookは最初の応答がstream済みになった後で発火します。v1.4.19以降は応答の書換え・継続強制・次turnへの監査文配送をせず、findingを構造Hook eventへ記録します。pre-responseの`UserPromptSubmit`精度が引き続き主軸です
|
|
261
|
+
- `UserPromptSubmit.additionalContext`は受動的metadataではなくモデル可視contextです。v1.4.19以降はcatalog一致・grammar検証済みtool IDだけから決定論的に生成し、監査用AIのreason、backend message、provider stdout/stderrを反射しません
|
|
262
|
+
- **Haiku の JSON スキーマ違反は v0.5.0 以降「想定済み異常」として session renew + `role_collapse_reset` で回復**します。**v1.4.19以降、auditor/daemonの失敗はモデルcontextにせずnon-blockingを維持**します。allow-list済み固定`systemMessage`・固定stderr・構造Hook eventだけを出してexit 0にします
|
|
215
263
|
|
|
216
264
|
<details>
|
|
217
265
|
<summary><strong>📋 最近のハイライト</strong></summary>
|
|
218
266
|
|
|
267
|
+
- **親出力をルールベース化** (v1.4.19) — 監査用AIの自由文は親セッションへ入りません。検証済みtool IDだけがUserPromptSubmitの任意助言になり、Stop findingは無関係な次turnへ持ち越されません
|
|
219
268
|
- **daemon は異常死しても復活する** (v1.4.16) — daemon が graceful shutdown を経ず死んでも (マシンスリープ / 強制終了 / `SessionEnd` 前の crash)、残った Unix socket が以後の起動を塞がなくなった。`startDaemon` が bind 前に orphan socket を除去するので、次の `UserPromptSubmit` の auto-resurrect が `EADDRINUSE` で crash-loop せずに成功し、「そのセッションが永久に未監査」になる事態を防ぐ
|
|
220
|
-
- **失敗は声に出して縮退、host
|
|
269
|
+
- **失敗は声に出して縮退、hostを固めない** (v1.4.15) — この版でbackend failureによるpromptのsilent消去を止めた。v1.4.19以降もnon-blocking挙動は維持し、旧model可視警告文は固定`systemMessage`・stderr・構造event診断へ置換した
|
|
221
270
|
- **プラグイン形式の MCP サーバー対応** — `plugin:everything-claude-code:context7` のように名前に内部コロンを含むサーバーを正しくパースし、配下のツールをカタログに取り込めるようになった (旧版はこの形式のサーバーをすべて単一の `"plugin"` に潰して、Claude の監査から silent に脱落させていた)
|
|
222
271
|
- **プロジェクト単位の監査隔離** — daemon が監査に使うのはローカル DB のみ。グローバル DB は description 再利用キャッシュに役割限定。**他プロジェクト**でインストールしたツールが現プロジェクトの監査に混入することはない
|
|
223
272
|
- **手放しでカタログ維持** — `spotter install` が Claude DB を自動 seed、Claude / Codex それぞれの SessionStart が host-local DB を bg refresh する。手書き管理は一切不要
|
package/README.md
CHANGED
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
|
|
16
16
|
Claude has a structural blind spot: **it can't reach for a tool it doesn't realize it needs**. It may skip a project memory MCP when a decision should be recorded, answer from stale memory instead of a docs-lookup MCP, or reason about UI state without a browser-automation MCP. The model can't always tell when it doesn't know — so the tool stays unused.
|
|
17
17
|
|
|
18
|
-
Spotter runs a separate auditor with the full tool catalog and checks both the user's prompt and the primary agent's reply. Automatic selection uses Codex CLI on a Claude host when available, otherwise the session-scoped Haiku path; on a Codex host it defaults to Codex CLI. An explicit backend override takes precedence, but a runtime failure never silently switches backend.
|
|
18
|
+
Spotter runs a separate auditor with the full tool catalog and checks both the user's prompt and the primary agent's reply. Automatic selection uses Codex CLI on a Claude host when available, otherwise the session-scoped Haiku path; on a Codex host it defaults to Codex CLI. An explicit backend override takes precedence, but a runtime failure never silently switches backend. Before the primary reply, validated tool IDs may become fixed, non-directive advice; after the reply, findings remain structured events and are not injected into a later turn. Auditor prose never enters the parent session. **The primary agent is never asked to self-audit** — that would defeat the premise. Detection happens through hooks, independent of the primary agent's intent.
|
|
19
19
|
|
|
20
20
|
<p align="center">
|
|
21
21
|
<img src=".github/concept.svg" alt="Claude answers · Spotter watches" width="80%">
|
|
@@ -62,6 +62,7 @@ For Codex, install enables the current `[features].hooks = true` flag and still
|
|
|
62
62
|
Installer-owned Codex handlers use the current synchronous command schema. After install or upgrade, review them with `/hooks`, then open a fresh Codex session; `spotter codex-hook diagnostics` reports registration/readiness but does not guess hook trust.
|
|
63
63
|
|
|
64
64
|
After upgrading Spotter, re-run `spotter install` in each installed project when release notes mention hook setting changes. The global package update changes the code path, but existing `.claude/settings.json` timeout values are not rewritten automatically.
|
|
65
|
+
`v1.4.19` changes runtime output projection only, so already installed projects do not need another `spotter install`; update the global package and open a fresh Claude/Codex session.
|
|
65
66
|
|
|
66
67
|
```bash
|
|
67
68
|
spotter uninstall # remove hooks from this project
|
|
@@ -88,20 +89,19 @@ spotter codex-hook install
|
|
|
88
89
|
|
|
89
90
|
### Audit flow per turn
|
|
90
91
|
|
|
91
|
-
Claude Code and Codex
|
|
92
|
-
|
|
93
|
-
|
|
92
|
+
Claude Code and Codex share the same safe parent-output projector. Auditor prose stays
|
|
93
|
+
internal; only validated tool IDs can become fixed, non-imperative advice on
|
|
94
|
+
`UserPromptSubmit`. `Stop` records structured findings without injecting them into a later turn.
|
|
94
95
|
|
|
95
96
|
```mermaid
|
|
96
97
|
flowchart TD
|
|
97
98
|
U([User prompt]) --> UPH[UserPromptSubmit hook<br/>Spotter audits prompt against catalog]
|
|
98
|
-
UPH --> BT[
|
|
99
|
+
UPH --> BT[Host model<br/>may receive fixed advisory<br/>with validated tool IDs]
|
|
99
100
|
BT --> BA([Claude's first answer])
|
|
100
101
|
BA --> SH[Stop hook<br/>Spotter re-audits answer + tools used]
|
|
101
102
|
SH --> DEC{Missed<br/>tool?}
|
|
102
103
|
DEC -->|No| DONE([Done])
|
|
103
|
-
DEC -->|Yes|
|
|
104
|
-
SB --> NEXT([Surfaces as additionalContext<br/>on next UserPromptSubmit])
|
|
104
|
+
DEC -->|Yes| EVT[Record structured Hook event<br/>no next-turn injection]
|
|
105
105
|
```
|
|
106
106
|
|
|
107
107
|
### Catalog discovery
|
|
@@ -146,6 +146,58 @@ The audited catalog is host-local: Claude uses `<project>/.spotter/tool-db.json`
|
|
|
146
146
|
|
|
147
147
|
Both share the principle of **"don't rely on the primary agent to do it itself."** They compose well — you can run them together.
|
|
148
148
|
|
|
149
|
+
### Optional Throughline auditor context canary
|
|
150
|
+
|
|
151
|
+
Fresh project installs keep auditor context **disabled**. This is a project-owned
|
|
152
|
+
opt-in, not a production-wide default: the current candidate is a canary in the
|
|
153
|
+
Spotter repository only. Re-running `spotter install` without auditor-context
|
|
154
|
+
options preserves the project's existing setting. Disable it again with:
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
spotter install -y --auditor-context disabled
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
To opt a project in, configure a direct absolute Throughline executable and any
|
|
161
|
+
leading arguments with repeatable `--throughline-arg`:
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
spotter install -y --auditor-context throughline \
|
|
165
|
+
--throughline-command /absolute/path/to/throughline
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
On Windows, `.cmd` and `.bat` wrappers are deliberately rejected to avoid shell
|
|
169
|
+
injection. Point at an absolute `node.exe` and pass the absolute
|
|
170
|
+
`throughline.mjs` path as a repeated argument instead:
|
|
171
|
+
|
|
172
|
+
```powershell
|
|
173
|
+
spotter install -y --auditor-context throughline `
|
|
174
|
+
--throughline-command 'C:\Program Files\nodejs\node.exe' `
|
|
175
|
+
--throughline-arg 'C:\absolute\path\to\throughline\bin\throughline.mjs'
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
The connector is Codex CLI-only. It sends no context in argv: the bounded
|
|
179
|
+
projection is supplied to that AI over stdin. Haiku does not support this
|
|
180
|
+
context path and is never called for it. Only a `fresh` Throughline result is
|
|
181
|
+
eligible for an AI call; every other status skips AI. An enabled connector
|
|
182
|
+
failure becomes a fixed warning, not a hidden fallback.
|
|
183
|
+
|
|
184
|
+
Throughline contributes only fresh, completed L2 user/assistant pairs: two
|
|
185
|
+
recent pairs (N=2), each body capped at 600 characters and 4,000 characters in
|
|
186
|
+
total. Spotter never reflects Throughline L2, `reason`, or `raw` to the parent.
|
|
187
|
+
The parent receives only fixed non-imperative advice built from safe catalog tool
|
|
188
|
+
IDs. `spotter doctor` displays the auditor-context mode and a fixed availability
|
|
189
|
+
detail without printing its command or arguments.
|
|
190
|
+
|
|
191
|
+
The v2 model-matrix can make the context choice explicit:
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v2.json \
|
|
195
|
+
--recent-turns 2 --body-cap 600
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Current evaluation conclusion is N=2 / 600. It is not a rollout approval: the
|
|
199
|
+
Spotter-repository canary still lacks its 7-day and 30-fresh-result gates.
|
|
200
|
+
|
|
149
201
|
## Common commands
|
|
150
202
|
|
|
151
203
|
```bash
|
|
@@ -173,7 +225,7 @@ spotter codex-hook install
|
|
|
173
225
|
# repair / explicitly register Codex native hooks (normally handled by spotter install)
|
|
174
226
|
spotter codex-hook diagnostics
|
|
175
227
|
# check Codex hook registration/readiness; trust is reviewed with /hooks
|
|
176
|
-
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.
|
|
228
|
+
spotter auditor model-matrix --fixtures test/fixtures/auditor-model-matrix.v2.json --recent-turns 2 --body-cap 600
|
|
177
229
|
# experimental reproducible comparison of pinned auditor model profiles
|
|
178
230
|
spotter uninstall # remove hooks from this project (leaves ~/.spotter intact)
|
|
179
231
|
```
|
|
@@ -211,15 +263,16 @@ the production values for controlled experiments; diagnostics mark overrides as
|
|
|
211
263
|
|
|
212
264
|
## Known limitations
|
|
213
265
|
|
|
214
|
-
- The `Stop` hook fires **after** the first answer has already been streamed. Spotter
|
|
215
|
-
- `
|
|
216
|
-
- **Since v0.5.0, JSON schema violations from Haiku are treated as expected anomalies** (session renew + `role_collapse_reset`). **Since v1.4.
|
|
266
|
+
- The `Stop` hook fires **after** the first answer has already been streamed. Spotter records a structured finding but does not rewrite the answer, force a continuation, or inject auditor text into the next prompt. Detection accuracy in `UserPromptSubmit` (the *pre-response* stage) remains the primary quality axis
|
|
267
|
+
- `UserPromptSubmit.additionalContext` is model-visible context, not passive metadata. Since v1.4.19 it is generated only by a deterministic projector from catalog-matched, grammar-checked tool IDs. Auditor reasons, backend messages, and provider stdout/stderr are never reflected into it
|
|
268
|
+
- **Since v0.5.0, JSON schema violations from Haiku are treated as expected anomalies** (session renew + `role_collapse_reset`). **Since v1.4.19, an auditor/daemon failure remains non-blocking without becoming model context**: Claude and Codex emit only an allow-listed fixed `systemMessage`, fixed stderr, and a structured Hook event, then exit 0
|
|
217
269
|
|
|
218
270
|
<details>
|
|
219
271
|
<summary><strong>📋 Recent highlights</strong></summary>
|
|
220
272
|
|
|
273
|
+
- **Rule-based parent output boundary** (v1.4.19) — auditor AI prose cannot enter parent-session Hook output. Validated tool IDs become optional fixed advice on `UserPromptSubmit`; `Stop` findings stay structured and are never carried into an unrelated next turn
|
|
221
274
|
- **Daemon recovers after an ungraceful death** (v1.4.16) — if the daemon dies without graceful shutdown (machine sleep, force-quit, crash before `SessionEnd`), the Unix socket it leaves behind no longer bricks every restart. `startDaemon` removes the orphaned socket before binding, so the next `UserPromptSubmit` auto-resurrect succeeds instead of crash-looping on `EADDRINUSE` and leaving the session permanently unaudited
|
|
222
|
-
- **Failures degrade loudly, never freeze the host** (v1.4.15) —
|
|
275
|
+
- **Failures degrade loudly, never freeze the host** (v1.4.15) — this release stopped backend failure from silently erasing a prompt. Since v1.4.19, the non-blocking behavior remains but the old model-visible warning text is replaced by fixed `systemMessage`, stderr, and structured event diagnostics
|
|
223
276
|
- **Plugin-scoped MCP servers** — names like `plugin:everything-claude-code:context7` (with internal colons) are now parsed correctly and their tools enter the catalog. Earlier versions silently collapsed all plugin MCP servers into a single literal `"plugin"`, dropping their tools from Claude's audit
|
|
224
277
|
- **Per-project / per-host audit isolation** — the daemon audits against the local DB only; global DBs are host-specific description caches. Tools discovered in *other* projects or another host can never bleed into this project's audit set
|
|
225
278
|
- **Zero-touch catalog** — `spotter install` seeds the Claude DB automatically; Claude and Codex SessionStart hooks keep their host-local DBs fresh in the background. You never have to maintain the tool list by hand
|
package/bin/spotter.mjs
CHANGED
|
@@ -21,7 +21,9 @@ import { runSessionEnd } from '../src/hooks/session-end.mjs';
|
|
|
21
21
|
const USAGE = `spotter — Claude Code tool-call auditor
|
|
22
22
|
|
|
23
23
|
Usage:
|
|
24
|
-
spotter install [-y]
|
|
24
|
+
spotter install [-y] [--auditor-context disabled|throughline]
|
|
25
|
+
[--throughline-command ABS] [--throughline-arg VALUE]
|
|
26
|
+
register hooks in <cwd>/.claude/settings.json
|
|
25
27
|
and create <cwd>/.spotter/marker.json
|
|
26
28
|
(run inside each project you want audited)
|
|
27
29
|
spotter install --user [-y] legacy: register globally in ~/.claude/settings.json
|
|
@@ -75,9 +77,8 @@ async function main() {
|
|
|
75
77
|
const [cmd, ...rest] = argv;
|
|
76
78
|
switch (cmd) {
|
|
77
79
|
case 'install': {
|
|
78
|
-
const
|
|
79
|
-
|
|
80
|
-
await runInstall({ target, autoYes });
|
|
80
|
+
const options = parseInstallArgs(rest);
|
|
81
|
+
await runInstall(options);
|
|
81
82
|
return;
|
|
82
83
|
}
|
|
83
84
|
case 'uninstall': {
|
|
@@ -140,6 +141,60 @@ async function main() {
|
|
|
140
141
|
}
|
|
141
142
|
}
|
|
142
143
|
|
|
144
|
+
function parseInstallArgs(argv) {
|
|
145
|
+
let target = 'project';
|
|
146
|
+
let autoYes = false;
|
|
147
|
+
let mode = null;
|
|
148
|
+
let command = null;
|
|
149
|
+
const args = [];
|
|
150
|
+
for (let index = 0; index < argv.length; index += 1) {
|
|
151
|
+
const arg = argv[index];
|
|
152
|
+
if (arg === '--user') {
|
|
153
|
+
if (target === 'user') throw invalidInstallArgs();
|
|
154
|
+
target = 'user';
|
|
155
|
+
} else if (arg === '-y' || arg === '--yes') {
|
|
156
|
+
if (autoYes) throw invalidInstallArgs();
|
|
157
|
+
autoYes = true;
|
|
158
|
+
} else if (arg === '--auditor-context') {
|
|
159
|
+
if (mode !== null) throw invalidInstallArgs();
|
|
160
|
+
mode = argv[++index];
|
|
161
|
+
if (mode !== 'disabled' && mode !== 'throughline') throw invalidInstallArgs();
|
|
162
|
+
} else if (arg === '--throughline-command') {
|
|
163
|
+
if (command !== null) throw invalidInstallArgs();
|
|
164
|
+
command = argv[++index];
|
|
165
|
+
if (typeof command !== 'string' || command.length === 0) throw invalidInstallArgs();
|
|
166
|
+
} else if (arg === '--throughline-arg') {
|
|
167
|
+
const value = argv[++index];
|
|
168
|
+
if (typeof value !== 'string' || value.length === 0) throw invalidInstallArgs();
|
|
169
|
+
args.push(value);
|
|
170
|
+
} else {
|
|
171
|
+
throw invalidInstallArgs();
|
|
172
|
+
}
|
|
173
|
+
}
|
|
174
|
+
if (mode === null && (command !== null || args.length > 0)) throw invalidInstallArgs();
|
|
175
|
+
if (mode === 'disabled' && (command !== null || args.length > 0)) throw invalidInstallArgs();
|
|
176
|
+
if (mode === 'throughline' && (command === null || !isAbsoluteCommand(command) || isShellWrapper(command))) throw invalidInstallArgs();
|
|
177
|
+
const auditorContext = mode === null ? undefined : mode === 'disabled'
|
|
178
|
+
? { mode: 'disabled' }
|
|
179
|
+
: { mode: 'throughline', command, args };
|
|
180
|
+
return { target, autoYes, auditorContext };
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
function isAbsoluteCommand(value) {
|
|
184
|
+
return value.startsWith('/') || /^[A-Za-z]:[\\/]/.test(value);
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
function isShellWrapper(value) {
|
|
188
|
+
return /\.(?:cmd|bat)$/i.test(value);
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
function invalidInstallArgs() {
|
|
192
|
+
const err = new Error('invalid install arguments');
|
|
193
|
+
err.stack = '';
|
|
194
|
+
err.exitCode = 2;
|
|
195
|
+
return err;
|
|
196
|
+
}
|
|
197
|
+
|
|
143
198
|
main().catch((err) => {
|
|
144
199
|
process.stderr.write(`spotter: ${err.stack || err.message || err}\n`);
|
|
145
200
|
process.exit(err.exitCode ?? 2);
|
package/package.json
CHANGED
package/src/cli/auditor-cmd.mjs
CHANGED
|
@@ -37,7 +37,7 @@ export async function runAuditorCommand({ argv = process.argv.slice(2) } = {}) {
|
|
|
37
37
|
}
|
|
38
38
|
if (sub === 'model-matrix') {
|
|
39
39
|
if (argv.slice(1).includes('--help') || argv.slice(1).includes('-h')) {
|
|
40
|
-
process.stdout.write(`Usage: spotter auditor model-matrix --fixtures FILE [--profile baseline|luna|terra|terra-medium]...\n [--repeat N] [--project DIR] [--output FILE]\n`);
|
|
40
|
+
process.stdout.write(`Usage: spotter auditor model-matrix --fixtures FILE [--profile baseline|luna|terra|terra-medium]...\n [--repeat N] [--recent-turns 0|1|2|3] [--body-cap CHARS] [--project DIR] [--output FILE]\n`);
|
|
41
41
|
return;
|
|
42
42
|
}
|
|
43
43
|
await runAuditorModelMatrixCommand({ argv: argv.slice(1) });
|
|
@@ -4,7 +4,7 @@ import { promisify } from 'node:util';
|
|
|
4
4
|
import { readFile, writeFile } from 'node:fs/promises';
|
|
5
5
|
import { isAbsolute, relative, resolve, sep } from 'node:path';
|
|
6
6
|
import { version } from '../version.mjs';
|
|
7
|
-
import { createCodexCliAuditorBackend } from '../core/codex-cli-backend.mjs';
|
|
7
|
+
import { CODEX_AUDITOR_PROMPT_VERSION, createCodexCliAuditorBackend } from '../core/codex-cli-backend.mjs';
|
|
8
8
|
import { CODEX_AUDITOR_MODEL_POLICY, resolveCodexAuditorModelSelection } from '../core/codex-auditor-model-policy.mjs';
|
|
9
9
|
|
|
10
10
|
const execFileAsync = promisify(execFile);
|
|
@@ -21,6 +21,7 @@ export async function runAuditorModelMatrixCommand({
|
|
|
21
21
|
const raw = await readFileFn(opts.fixturesPath);
|
|
22
22
|
const fixtureBytes = Buffer.isBuffer(raw) ? raw : Buffer.from(raw);
|
|
23
23
|
const fixture = parseAndValidateFixture(fixtureBytes.toString('utf8'));
|
|
24
|
+
const contextOptions = resolveContextOptions(opts, fixture.schema);
|
|
24
25
|
if (fixture.cases.length * opts.repeat * opts.profiles.length > MAX_MODEL_MATRIX_RUNS) {
|
|
25
26
|
throw new Error(`model-matrix run count exceeds maximum ${MAX_MODEL_MATRIX_RUNS}`);
|
|
26
27
|
}
|
|
@@ -40,7 +41,7 @@ export async function runAuditorModelMatrixCommand({
|
|
|
40
41
|
const modelSelection = selections[profile];
|
|
41
42
|
try {
|
|
42
43
|
const backend = backends[profile];
|
|
43
|
-
const judgment = await backend.judge({ ...toAuditorInput(item), meta: { caseId: item.id, repeat, profile } });
|
|
44
|
+
const judgment = await backend.judge({ ...toAuditorInput(item, contextOptions), meta: { caseId: item.id, repeat, profile } });
|
|
44
45
|
if (!sameSelection(modelSelection, judgment?.meta?.modelSelection)) {
|
|
45
46
|
throw new Error('judgment model selection does not match backend model selection');
|
|
46
47
|
}
|
|
@@ -87,6 +88,7 @@ export async function runAuditorModelMatrixCommand({
|
|
|
87
88
|
schema: 'spotter.auditor_model_matrix.v1', generatedAt: generatedAt(), packageVersion: version,
|
|
88
89
|
fixture: { schema: fixture.schema, path: safeFixturePath(opts.fixturesPath, opts.projectRoot), sha256: createHash('sha256').update(fixtureBytes).digest('hex'), cases: fixture.cases.length, catalogCount: fixture.catalog.length },
|
|
89
90
|
codexCli: cliVersion, policy: { schema: CODEX_AUDITOR_MODEL_POLICY.schema, version: CODEX_AUDITOR_MODEL_POLICY.policyVersion },
|
|
91
|
+
auditorPromptVersion: CODEX_AUDITOR_PROMPT_VERSION,
|
|
90
92
|
profiles: Object.fromEntries(Object.entries(selections).map(([profile, selection]) => [profile, {
|
|
91
93
|
model: selection.effectiveModel,
|
|
92
94
|
reasoningEffort: selection.effectiveReasoningEffort,
|
|
@@ -96,7 +98,7 @@ export async function runAuditorModelMatrixCommand({
|
|
|
96
98
|
selection,
|
|
97
99
|
}])), runs, summary: summarize(runs, opts.profiles), usageStatus: usage.status, tokenUsage: usage.summary,
|
|
98
100
|
costStatus: 'not-available-chatgpt-plan', cost: null,
|
|
99
|
-
evaluation: { repeat: opts.repeat, profiles: opts.profiles, maxRuns: MAX_MODEL_MATRIX_RUNS },
|
|
101
|
+
evaluation: { repeat: opts.repeat, profiles: opts.profiles, maxRuns: MAX_MODEL_MATRIX_RUNS, recentTurns: contextOptions.recentTurns, bodyCap: contextOptions.bodyCap },
|
|
100
102
|
executionOrdering: 'case-repeat-profile', promotionEligible: false, blockingReasons: blockingReasons(runs, usage.status),
|
|
101
103
|
};
|
|
102
104
|
const json = JSON.stringify(artifact, null, 2) + '\n';
|
|
@@ -106,7 +108,7 @@ export async function runAuditorModelMatrixCommand({
|
|
|
106
108
|
}
|
|
107
109
|
|
|
108
110
|
function parseArgs(argv) {
|
|
109
|
-
const opts = { fixturesPath: null, profiles: [], repeat: 1, projectRoot: process.cwd(), outputPath: null };
|
|
111
|
+
const opts = { fixturesPath: null, profiles: [], repeat: 1, projectRoot: process.cwd(), outputPath: null, recentTurns: null, bodyCap: null };
|
|
110
112
|
for (let i = 0; i < argv.length; i += 1) {
|
|
111
113
|
const arg = argv[i]; const value = () => { const v = argv[++i]; if (!v || v.startsWith('--')) throw new Error(`${arg} requires a value`); return v; };
|
|
112
114
|
if (arg === '--fixtures') opts.fixturesPath = resolve(value());
|
|
@@ -114,10 +116,14 @@ function parseArgs(argv) {
|
|
|
114
116
|
else if (arg === '--repeat') opts.repeat = Number(value());
|
|
115
117
|
else if (arg === '--project') opts.projectRoot = resolve(value());
|
|
116
118
|
else if (arg === '--output') opts.outputPath = resolve(value());
|
|
119
|
+
else if (arg === '--recent-turns') opts.recentTurns = Number(value());
|
|
120
|
+
else if (arg === '--body-cap') opts.bodyCap = Number(value());
|
|
117
121
|
else throw new Error(`unknown auditor model-matrix option: ${arg}`);
|
|
118
122
|
}
|
|
119
123
|
if (!opts.fixturesPath) throw new Error('--fixtures FILE is required');
|
|
120
124
|
if (!Number.isInteger(opts.repeat) || opts.repeat < 1) throw new Error('--repeat must be a positive integer');
|
|
125
|
+
if (opts.recentTurns !== null && (!Number.isInteger(opts.recentTurns) || opts.recentTurns < 0 || opts.recentTurns > 3)) throw new Error('--recent-turns must be 0, 1, 2, or 3');
|
|
126
|
+
if (opts.bodyCap !== null && (!Number.isInteger(opts.bodyCap) || opts.bodyCap <= 0)) throw new Error('--body-cap must be a positive integer');
|
|
121
127
|
opts.profiles = opts.profiles.length ? opts.profiles : [...DEFAULT_PROFILES];
|
|
122
128
|
if (new Set(opts.profiles).size !== opts.profiles.length || opts.profiles.some((profile) => !ALLOWED_PROFILES.includes(profile))) throw new Error('--profile must be baseline, luna, terra, or terra-medium without duplicates');
|
|
123
129
|
return opts;
|
|
@@ -126,7 +132,7 @@ function parseArgs(argv) {
|
|
|
126
132
|
function parseAndValidateFixture(raw) {
|
|
127
133
|
let fixture; try { fixture = JSON.parse(raw); } catch { throw new Error('fixture must be valid JSON'); }
|
|
128
134
|
objectOnly(fixture, ['schema', 'catalog', 'cases'], 'fixture');
|
|
129
|
-
if (
|
|
135
|
+
if (!['spotter.auditor_model_fixtures.v1', 'spotter.auditor_model_fixtures.v2'].includes(fixture.schema)) throw new Error('unsupported fixture schema');
|
|
130
136
|
if (!Array.isArray(fixture.catalog) || !Array.isArray(fixture.cases) || fixture.cases.length === 0) throw new Error('fixture catalog and non-empty cases are required');
|
|
131
137
|
const catalogNames = new Set();
|
|
132
138
|
for (const tool of fixture.catalog) { objectOnly(tool, ['name', 'description'], 'catalog tool'); clean(tool.name, 'catalog name'); clean(tool.description, 'catalog description'); if (catalogNames.has(tool.name)) throw new Error('duplicate catalog name'); catalogNames.add(tool.name); }
|
|
@@ -135,7 +141,7 @@ function parseAndValidateFixture(raw) {
|
|
|
135
141
|
objectOnly(item, ['id', 'stage', 'input', 'expected'], 'case'); clean(item.id, 'case id');
|
|
136
142
|
if (ids.has(item.id)) throw new Error('duplicate case id'); ids.add(item.id);
|
|
137
143
|
if (!['user_input', 'turn_end'].includes(item.stage)) throw new Error('case stage is invalid');
|
|
138
|
-
validateInput(item.stage, item.input); objectOnly(item.expected, ['pass', 'missingTools'], 'case expected');
|
|
144
|
+
validateInput(fixture.schema, item.stage, item.input); objectOnly(item.expected, ['pass', 'missingTools'], 'case expected');
|
|
139
145
|
if (typeof item.expected.pass !== 'boolean' || !Array.isArray(item.expected.missingTools)
|
|
140
146
|
|| new Set(item.expected.missingTools).size !== item.expected.missingTools.length
|
|
141
147
|
|| item.expected.missingTools.some((name) => typeof name !== 'string' || !catalogNames.has(name))
|
|
@@ -145,8 +151,9 @@ function parseAndValidateFixture(raw) {
|
|
|
145
151
|
}
|
|
146
152
|
function objectOnly(value, keys, label) { if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).some((key) => !keys.includes(key))) throw new Error(`${label} has invalid fields`); }
|
|
147
153
|
function clean(value, label) { if (typeof value !== 'string' || value.length === 0 || value.trim() !== value) throw new Error(`${label} must be a clean non-empty string`); }
|
|
148
|
-
function validateInput(stage, input) { if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('case input is invalid'); if (stage === 'user_input') { objectOnly(input, ['userInput'], 'user_input input'); clean(input.userInput, 'userInput'); } else { objectOnly(input, ['finalResponse', 'usedTools'], 'turn_end input'); clean(input.finalResponse, 'finalResponse'); if (!Array.isArray(input.usedTools) || input.usedTools.some((tool) => typeof tool !== 'string' || tool.length === 0 || tool.trim() !== tool) || new Set(input.usedTools).size !== input.usedTools.length) throw new Error('usedTools is invalid'); } }
|
|
149
|
-
function
|
|
154
|
+
function validateInput(schema, stage, input) { if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('case input is invalid'); if (stage === 'user_input') { objectOnly(input, schema === 'spotter.auditor_model_fixtures.v2' ? ['userInput', 'recentContext'] : ['userInput'], 'user_input input'); clean(input.userInput, 'userInput'); if (schema === 'spotter.auditor_model_fixtures.v2') { if (!Array.isArray(input.recentContext) || input.recentContext.length < 1 || input.recentContext.length > 3) throw new Error('recentContext must contain one to three turns'); for (const turn of input.recentContext) { objectOnly(turn, ['user', 'assistant'], 'recentContext turn'); clean(turn.user, 'recentContext user'); clean(turn.assistant, 'recentContext assistant'); } } } else { objectOnly(input, ['finalResponse', 'usedTools'], 'turn_end input'); clean(input.finalResponse, 'finalResponse'); if (!Array.isArray(input.usedTools) || input.usedTools.some((tool) => typeof tool !== 'string' || tool.length === 0 || tool.trim() !== tool) || new Set(input.usedTools).size !== input.usedTools.length) throw new Error('usedTools is invalid'); } }
|
|
155
|
+
function resolveContextOptions(opts, schema) { if (schema === 'spotter.auditor_model_fixtures.v1') { if (opts.recentTurns !== null || opts.bodyCap !== null) throw new Error('--recent-turns and --body-cap require a v2 fixture'); return { recentTurns: null, bodyCap: null }; } return { recentTurns: opts.recentTurns ?? 2, bodyCap: opts.bodyCap ?? 1200 }; }
|
|
156
|
+
function toAuditorInput(item, contextOptions) { if (item.stage !== 'user_input') return { stage: item.stage, finalResponse: item.input.finalResponse, usedTools: item.input.usedTools }; const input = { stage: item.stage, userInput: item.input.userInput }; if (contextOptions.recentTurns > 0 && item.input.recentContext) input.recentContext = item.input.recentContext.slice(-contextOptions.recentTurns).map((turn) => ({ user: turn.user.slice(-contextOptions.bodyCap), assistant: turn.assistant.slice(-contextOptions.bodyCap) })); return input; }
|
|
150
157
|
function isCleanString(value) { return typeof value === 'string' && value.length > 0 && value.trim() === value; }
|
|
151
158
|
function cleanStrings(values = []) { return [...new Set(values.filter(isCleanString))]; }
|
|
152
159
|
function successRun({ order, item, repeat, profile, modelSelection, durationMs, schemaSuccess, actualPass, actualTools, invalidFindingCount, droppedTools, droppedToolCount, anomalyTypes, anomalyCount, tokenUsage }) { const expectedTools = item.expected.missingTools; const fp = [...actualTools.filter((tool) => !expectedTools.includes(tool)), ...droppedTools.filter((tool) => !expectedTools.includes(tool))]; const fn = expectedTools.filter((tool) => !actualTools.includes(tool)); return { order, caseId: item.id, repeat, profile, status: 'success', durationMs, schemaSuccess, exactMatch: schemaSuccess && actualPass === item.expected.pass && actualTools.length === expectedTools.length && fp.length === 0 && fn.length === 0 && droppedToolCount === 0 && anomalyCount === 0, expected: item.expected, actual: { pass: actualPass, missingTools: actualTools, invalidFindingCount, droppedCatalogExternalNames: droppedTools, droppedCatalogExternalNameCount: droppedToolCount, anomalies: anomalyTypes, anomalyCount }, falsePositiveTools: fp, falseNegativeTools: fn, modelSelection, tokenUsage }; }
|