glassbox-trace 0.9.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -0
- package/dist/glassbox.html +6 -2
- package/package.json +1 -1
- package/src/adhere.mjs +8 -3
- package/src/claims.mjs +368 -0
- package/src/cli.mjs +35 -2
- package/src/trace-core.js +6 -2
package/README.md
CHANGED
|
@@ -39,6 +39,7 @@ glassbox compare 81c4 9f0a --out cmp.html …as one HTML with both sessions
|
|
|
39
39
|
glassbox watch live tail: the viewer follows the newest session as Claude Code writes it
|
|
40
40
|
glassbox collect ./reports --format md --out fleet.md
|
|
41
41
|
one report from a folder of redacted check reports, one per machine
|
|
42
|
+
glassbox claims 81c4 --format md said vs did: every claim the agent made about its own work, with the tool result behind it — or the gap
|
|
42
43
|
glassbox clean remove the temp files `open` and the hook leave behind
|
|
43
44
|
```
|
|
44
45
|
|
|
@@ -137,6 +138,32 @@ Glassbox adhere · /home/aldo/api
|
|
|
137
138
|
|
|
138
139
|
Nine rule shapes are checkable from tool calls — run X before commit/push, run X after changes, use A not B, never run, never touch, ask before, read before edit, commit message format, no new docs — and everything else is listed as *not checkable yet* with its line number, so the report never claims more than it measured. Commands are judged on their shell surface (a `git push --force` inside a document being written is not a force push). The first real run, on the session that built it, scored 22%. Details and the honest caveats in [docs/ADHERE.md](docs/ADHERE.md).
|
|
139
140
|
|
|
141
|
+
### Said vs did — `glassbox claims`
|
|
142
|
+
|
|
143
|
+
Every sentence in which the agent told you what it did — "all 76 tests pass", "committed as `753be02`", "live on npm", "nothing changed" — matched to the tool result that should be standing behind it, or to the gap where one should be:
|
|
144
|
+
|
|
145
|
+
```
|
|
146
|
+
glassbox claims the newest session (subagents included)
|
|
147
|
+
glassbox claims 81c4 --format md --out claims.md
|
|
148
|
+
glassbox claims --fail-on unverified exit 1 on any claim without a receipt (default: contradicted only)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
```
|
|
152
|
+
Glassbox claims · 38f81247
|
|
153
|
+
69 claims · 61 verified · 7 declared · 1 unverified · 0 contradicted
|
|
154
|
+
|
|
155
|
+
UNVERIFIED test t23 Your folder is as I left it: `master` is GitHub's `main` plus the loop guard and Action commit,…
|
|
156
|
+
↳ no test, build, validate or CI result in the window
|
|
157
|
+
VERIFIED test t3 58 of 58 pass.
|
|
158
|
+
↳ 58 passed, 0 failed (1 call)
|
|
159
|
+
VERIFIED ship t4 I committed it as `753be02` on branch `wait-aware-findings`: 6 files, 117 lines added and 6 rem…
|
|
160
|
+
↳ ship result in the window (1 call)
|
|
161
|
+
DECLARED declared t3 Not verified: the browser e2e test (`test/browser.e2e.mjs`) didn't run because Playwright isn't…
|
|
162
|
+
↳ the agent said so
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Six kinds of claim (tests pass, shipped, verified, fixed, nothing changed, file written), judged on the same agent's tool calls in that turn: a test claim needs a test run after the last edit — from `Bash`, `PowerShell` or Cowork's `device_bash` — with its counts read and compared; a shipped claim needs the commit, push, publish or upload result, carrying the sha or version the sentence names; "nothing changed" needs a `git status` after the thing that could have changed it. An agent that says "not verified" gets a *declared* row and no finding. The same ledger runs inside `check` as three rules — `contradicted-claim` (error), `unverified-claim` (warn), `stale-claim` (info) — so the Stop hook hands the sentence back to the agent that wrote it. On sixteen real sessions (367 claims; three of them checked by hand first) the ledger found one unverified sentence — "the tests still pass", 24 minutes after the last run — five fixes never re-run after their edit, and no contradictions. A verdict is about the sentence, never the code. Details in [docs/CLAIMS.md](docs/CLAIMS.md).
|
|
166
|
+
|
|
140
167
|
## What it flags
|
|
141
168
|
|
|
142
169
|
| rule | fires when |
|
|
@@ -157,6 +184,7 @@ Nine rule shapes are checkable from tool calls — run X before commit/push, run
|
|
|
157
184
|
| `slow-tool` | a tool call ≥ 60 s (warn at 5 min); one line per tool when repeated; blocking `TaskOutput` waits roll up per background task as info; time spent on the human (questions, permission prompts) is excluded |
|
|
158
185
|
| `permission-denied` | tool calls the human rejected, a permission rule or auto mode blocked, or an interrupt stopped (info) |
|
|
159
186
|
| `max-tokens`, `api-error`, `hook-error`, `compaction`, `thinking-heavy`, `subagent-share`, `long-turn` | what they say (`long-turn`: ≥ 30 tool calls after one prompt in the main conversation) |
|
|
187
|
+
| `contradicted-claim`, `unverified-claim`, `stale-claim` | the agent told the human something the transcript contradicts (error), has no receipt for (warn), or checked before its last edit / only partly (info) — see `glassbox claims` above |
|
|
160
188
|
|
|
161
189
|
Identical findings print once with a `×N` count in `check`. All mechanical, no AI. The thresholds were tuned on a corpus of 33 real sessions (481 findings → 142, median 2 per session); `node scripts/corpus-audit.mjs` re-runs that audit on your own sessions and prints counts only. Thresholds are in `TraceCore.DEFAULTS` and can be overridden when calling `diagnose(trace, opts)`. Every rule has a one-line "next time" in `TraceCore.ADVICE`, which is what the Markdown report and the Stop hook hand back to the agent.
|
|
162
190
|
|
|
@@ -188,6 +216,7 @@ src/viewer.html the UI; the build inlines trace-core and the demo
|
|
|
188
216
|
src/cli.mjs CLI library (session discovery, embed, check, compare, hook); bin/glassbox.mjs is the entry point
|
|
189
217
|
src/tail.mjs live tail: byte-offset tailer + loopback SSE server for `glassbox watch`
|
|
190
218
|
src/guard.mjs the PreToolUse loop guard (`hook --guard`)
|
|
219
|
+
src/claims.mjs said vs did: claims out of the agent's text, judged against the tool calls (`claims`, and three `check` rules)
|
|
191
220
|
scripts/build.mjs build
|
|
192
221
|
scripts/sanitize.mjs turn a real transcript into a shareable fixture (demo or structure mode)
|
|
193
222
|
scripts/corpus-audit.mjs run every rule over your local sessions; counts only (--baseline to diff two runs)
|
package/dist/glassbox.html
CHANGED
|
@@ -408,7 +408,7 @@ kbd{font:11px var(--mono);border:1px solid var(--line);border-bottom-width:2px;b
|
|
|
408
408
|
})(typeof globalThis !== 'undefined' ? globalThis : this, function () {
|
|
409
409
|
'use strict';
|
|
410
410
|
|
|
411
|
-
const VERSION = '0.
|
|
411
|
+
const VERSION = '0.10.0';
|
|
412
412
|
|
|
413
413
|
// ---------------------------------------------------------------------------
|
|
414
414
|
// Rate card (USD per million tokens). Prefix-matched against model ids so dated
|
|
@@ -1333,6 +1333,10 @@ kbd{font:11px var(--mono);border:1px solid var(--line);border-bottom-width:2px;b
|
|
|
1333
1333
|
'permission-denied': 'Before a call the user may not want (installs, network, deleting, publishing), say what it will do and ask; for calls they always allow, suggest a permissions allow rule so the prompt goes away.',
|
|
1334
1334
|
'duplicate-subagent-read': 'When several subagents need the same file, put the relevant excerpt in their briefs or have one agent summarise it once, instead of every agent reading it into its own context.',
|
|
1335
1335
|
'idle-cache-expiry': 'The cache expired while the session sat idle; resume sooner, use the 1-hour cache lifetime where available, or start a fresh session with a summary instead of re-sending the whole history.',
|
|
1336
|
+
// claims ledger (src/claims.mjs): said vs did
|
|
1337
|
+
'unverified-claim': "Say it after you've checked it: a test claim needs a test run after your last edit, a shipped claim needs the command's result in this transcript, a 'nothing changed' claim needs a git status after the thing that could have changed it. If you did not check, say 'not re-run' instead.",
|
|
1338
|
+
'contradicted-claim': 'The transcript disagrees with what you told the user; read the tool result you are summarising before you summarise it, and correct the statement now.',
|
|
1339
|
+
'stale-claim': 'The check you are citing ran before your last edit, or covers only part of what you said; run it again or say exactly what it covered.',
|
|
1336
1340
|
};
|
|
1337
1341
|
|
|
1338
1342
|
// Advice for a concrete finding: the rule's line, except where the evidence says otherwise.
|
|
@@ -1381,7 +1385,7 @@ kbd{font:11px var(--mono);border:1px solid var(--line);border-bottom-width:2px;b
|
|
|
1381
1385
|
}
|
|
1382
1386
|
|
|
1383
1387
|
// Rules whose detail quotes tool output or model text (everything else is names and numbers).
|
|
1384
|
-
const CONTENT_DETAIL = new Set(['failed-tool', 'api-error', 'hook-error', 'duplicate-subagent-read']);
|
|
1388
|
+
const CONTENT_DETAIL = new Set(['failed-tool', 'api-error', 'hook-error', 'duplicate-subagent-read', 'unverified-claim', 'contradicted-claim', 'stale-claim']);
|
|
1385
1389
|
function redactDetail(f) { return CONTENT_DETAIL.has(f.id) ? blank(f.detail) : f.detail; }
|
|
1386
1390
|
|
|
1387
1391
|
// ---------------------------------------------------------------------------
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "glassbox-trace",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.0",
|
|
4
4
|
"description": "Other tools show what happened in a Claude Code session; Glassbox tells you what went wrong. Timeline, token and cost burn, auto-diagnostics with evidence and advice, session compare, live tail, a fleet view, a secrets scanner for transcripts (fence), a CLAUDE.md compliance report (adhere), a CLI and a Claude Code Stop hook that feeds the findings back to the agent.",
|
|
5
5
|
"main": "src/trace-core.js",
|
|
6
6
|
"scripts": {
|
package/src/adhere.mjs
CHANGED
|
@@ -17,6 +17,10 @@ const core = require('./trace-core.js');
|
|
|
17
17
|
export const encodeProject = (p) => String(p).replace(/[\\/:]/g, '-');
|
|
18
18
|
const norm = (p) => String(p || '').replace(/\\/g, '/').toLowerCase();
|
|
19
19
|
const WRITE_TOOLS = new Set(['Edit', 'MultiEdit', 'Write', 'NotebookEdit']);
|
|
20
|
+
// Commands come from every shell tool Claude Code offers, not only Bash: PowerShell on Windows, and Cowork's
|
|
21
|
+
// device_bash on the user's machine. A `git push` from PowerShell is a push (0.9.1 missed all of them).
|
|
22
|
+
export const SHELL_TOOLS = new Set(['Bash', 'PowerShell']);
|
|
23
|
+
export const isShell = (c) => SHELL_TOOLS.has(c.name) || /device_bash$/.test(c.name || '');
|
|
20
24
|
const EDIT_TOOLS = new Set(['Edit', 'MultiEdit', 'NotebookEdit']);
|
|
21
25
|
const READ_TOOLS = new Set(['Read', 'NotebookRead']);
|
|
22
26
|
|
|
@@ -172,9 +176,10 @@ function judgeSession(rules, s, project) {
|
|
|
172
176
|
const turnPrompt = (i) => { const t = turns.find((x) => x.index === i && x.agent === 'main') || turns.find((x) => x.index === i); return t ? { text: t.promptText || '', kind: t.promptKind } : { text: '', kind: null }; };
|
|
173
177
|
const prevTurnAsked = (i) => { const prev = trace.requests.filter((r) => r.turnIndex === i - 1 && r.agent === 'main'); if (!prev.length) return false; const last = prev[prev.length - 1]; const texts = (last.blocks || []).filter((b) => b.type === 'text').map((b) => b.text).join(' ').trim(); return /\?\s*$/.test(texts); };
|
|
174
178
|
const short = (t) => String(t == null ? '' : t).replace(/\s+/g, ' ').trim().slice(0, 160);
|
|
175
|
-
|
|
176
|
-
const
|
|
177
|
-
const
|
|
179
|
+
// turn is 1-based, the number `check` prints and the viewer shows (turnIndex + 1).
|
|
180
|
+
const occ = (c, obeyed, detail) => ({ obeyed, session: s.id, turn: c.turnIndex + 1, prompt: turnPrompt(c.turnIndex).text.replace(/\s+/g, ' ').slice(0, 80), detail: short(detail), at: c.start });
|
|
181
|
+
const sessionOcc = (obeyed, detail, c) => ({ obeyed, session: s.id, turn: c ? c.turnIndex + 1 : null, prompt: c ? turnPrompt(c.turnIndex).text.replace(/\s+/g, ' ').slice(0, 80) : '', detail: short(detail), at: c ? c.start : trace.meta.start });
|
|
182
|
+
const bash = calls.filter((c) => isShell(c) && typeof c.input?.command === 'string');
|
|
178
183
|
const writes = calls.filter((c) => WRITE_TOOLS.has(c.name) && c.input && typeof c.input.file_path === 'string');
|
|
179
184
|
const out = rules.map(() => []);
|
|
180
185
|
rules.forEach((r, i) => {
|
package/src/claims.mjs
ADDED
|
@@ -0,0 +1,368 @@
|
|
|
1
|
+
// glassbox claims — said vs did.
|
|
2
|
+
//
|
|
3
|
+
// Every session ends with the agent's account of itself: "all 76 tests pass", "committed as 753be02", "live on
|
|
4
|
+
// npm", "nothing changed". This module pulls each such sentence out of the assistant's user-facing text, looks
|
|
5
|
+
// for the tool result that should be standing behind it, and gives it a verdict. It is pure: a parsed trace in,
|
|
6
|
+
// a ledger out. Nothing here knows whether the code is right — only whether the sentence has a receipt.
|
|
7
|
+
// Design: DESIGN.md §18, docs/CLAIMS.md.
|
|
8
|
+
|
|
9
|
+
// ---------------------------------------------------------------------------------------------
|
|
10
|
+
// Sentences
|
|
11
|
+
// ---------------------------------------------------------------------------------------------
|
|
12
|
+
const FENCE_RE = /```[\s\S]*?```/g;
|
|
13
|
+
const SPLIT_RE = /(?<=[.!?]|[.!?]["”’)])\s+(?=[A-Za-z"“`*(\-\d])|\n+/;
|
|
14
|
+
const ABBR_RE = /\b(?:e\.g|i\.e|vs|etc|approx|no|Mr|Dr|St)\.(?=\s)/g;
|
|
15
|
+
const strip = (s) => s.replace(/^[\s>*\-•·]+/, '').replace(/\*\*|__/g, '').replace(/^\d+[.)]\s+/, '').trim();
|
|
16
|
+
|
|
17
|
+
// User-facing texts: assistant text blocks (any agent) and, in Cowork, the message and file tools.
|
|
18
|
+
export function userFacingTexts(trace) {
|
|
19
|
+
const out = [];
|
|
20
|
+
const callsByReq = new Map();
|
|
21
|
+
for (const c of trace.toolCalls) { if (!callsByReq.has(c.requestId)) callsByReq.set(c.requestId, []); callsByReq.get(c.requestId).push(c); }
|
|
22
|
+
trace.requests.forEach((r, i) => {
|
|
23
|
+
for (const b of r.blocks) if (b.type === 'text' && b.text && b.text.trim()) out.push({ request: r, index: i, agent: r.agent, via: 'text', text: b.text });
|
|
24
|
+
for (const c of callsByReq.get(r.id) || []) {
|
|
25
|
+
const inp = c.input || {};
|
|
26
|
+
if (c.name === 'SendUserMessage' && typeof inp.message === 'string') out.push({ request: r, index: i, agent: r.agent, via: 'SendUserMessage', text: inp.message, call: c });
|
|
27
|
+
if (c.name === 'SendUserFile' && typeof inp.caption === 'string') out.push({ request: r, index: i, agent: r.agent, via: 'SendUserFile', text: inp.caption, call: c });
|
|
28
|
+
}
|
|
29
|
+
});
|
|
30
|
+
return out;
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
export function sentences(text) {
|
|
34
|
+
return String(text).replace(FENCE_RE, ' ').replace(ABBR_RE, (m) => m.replace(/\./g, '\u2024')).split(SPLIT_RE).map((x) => strip(x.replace(/\u2024/g, '.'))).filter((s) => s.length >= 8);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// ---------------------------------------------------------------------------------------------
|
|
38
|
+
// Classification
|
|
39
|
+
// ---------------------------------------------------------------------------------------------
|
|
40
|
+
// Honesty markers: the agent said it did not check. Recorded as `declared`, never a finding.
|
|
41
|
+
const DECLARED_RE = /\bnot\s+(?:verified|re-?run|checked|tested|run|confirmed|reproduced|pushed|published|deployed|uploaded|committed|merged|released)\b|\bunverified\b|\bI\s+(?:couldn't|could\s+not|can't|cannot|wasn't able to|was\s+unable\s+to)\b|\bcouldn't\s+(?:verify|check|run|find|reach|push|confirm)\b|\bhaven't\s+(?:run|verified|checked|tested)\b|\bdidn't\s+run\b|\bnot\s+done\b|\bI\s+got\s+(?:this|that|it)\s+wrong\b|\bfalse\s+positive\b|\bmy\s+(?:error|mistake)\b|\bI\s+was\s+wrong\b|\bcorrection\b|\bskipped\s+on\s+purpose\b|\bleft\s+(?:this|that|it)\s+to\s+you\b/i;
|
|
42
|
+
// Not a claim: plans, questions, conditionals, instructions to the human, descriptions of what a tool does.
|
|
43
|
+
const NOT_CLAIM_RE = /\b(?:I'll|I\s+will|will\b|let\s+me|going\s+to|about\s+to|next\s+(?:step|up|time)|should|would|could|may\b|might|if\b|whether|unless|once\s+you|when\s+you|to\s+do\b|todo|plan(?:ned|s)?\b|need(?:s|ed)?\s+to|want(?:s|ed)?\s+to|try(?:ing)?\s+to|please|you\s+can|you\s+could|you\s+should|run\s+this|paste|waiting\s+on|still\s+to\s+(?:go|do|upload|run)|before\s+I|now\s+I|here's\s+how|to\s+bring|to\s+pin|to\s+continue|instead|I\s+can\b|wait\s+for|needs?\b|has\s+to\b|have\s+to\b|I'd\b|we'd\b|you'd\b|we'll\b|we\s+will\b|let's\b|tell\s+me\b|ping\s+me\b|let\s+me\s+know|once\s+(?:the|that|this|it|you|we|they|your|I)\b|(?:it|that|this)\s+seems\b|seems\s+(?:like|to)\b|likely\b|probably\b|presumably\b|perhaps\b|maybe\b|I\s+(?:think|suspect|guess|assume|believe)\b|something\s+like\b|has\s+since\b|have\s+since\b|you\s+must\b|must\s+have\b|run\s+`[^`]*`\s+first|submit\s+(?:through|via|the)|(?:doesn't|won't|will\s+not|does\s+not|wouldn't)\s+(?:touch|push|change|include|affect|need|go))\b|\?\s*$|^\s*(?:want|shall|do|does|did|can|is|are)\s+(?:me|I|you|it|we|this|that)\b/i;
|
|
44
|
+
// Sentences about what the tool does, not what happened.
|
|
45
|
+
const DESCRIBES_RE = /\b(?:it|this|that|the\s+(?:guard|hook|rule|command|viewer|action|script|check))\s+(?:blocks?|fires?|reads?|writes?|prints?|checks?|runs?|serves?|counts?|redacts?|refuses?)\b|\bgets\s+\w+(?:ed|en)\b/i;
|
|
46
|
+
|
|
47
|
+
const KINDS = [
|
|
48
|
+
['test', /\b(?:tests?|suites?|specs?|e2e|audit|ci|checks?|lint(?:er)?|build|typecheck|type-check|validation|coverage)\b[^.]{0,80}?\b(?:pass(?:es|ed|ing)?|green|succeed(?:s|ed)?|ok|clean|no\s+failures|validates?|valid|passed)\b|\b(?:all|\d[\d,]*)\s*(?:of\s*\d[\d,]*\s*)?(?:tests?\s+)?(?:still\s+)?(?:pass(?:es|ed|ing)?|green)\b|\b\d[\d,]*\s*(?:\/\s*\d[\d,]*\s*)?(?:tests?|specs?)\s*(?:still\s+)?(?:pass|passing|passed|green|ok)\b|\b(?:plugin|manifest|schema|json-ld)\s+validates?\b|\b(?:ci|workflow|run\s+#?\d+|publish\s+#?\d+)\s+(?:is\s+|was\s+)?(?:green|succeeded|passed|completed)\b|\bexit(?:ed| code)?\s*0\b/i],
|
|
49
|
+
['ship', /^(?:both|all|everything|it|they)\s+(?:saved|written|uploaded|committed|pushed|deployed|published|live)\b|\b(?:all|both)\s+\d+\s+(?:saved|uploaded|live)\b|\ball\s+trimmed\s+and\s+live\b|\blive\s+and\s+(?:rendering|serving)\b|(?<!\b(?:the|a|an|its|my|your|this|our|that|one|each|these|those|two|three|four|five|six|last|previously)\s)\b(?:pushed(?!\s+back)|committed|commit(?:ted)?\s+(?:it\s+)?as\b|published|deployed|uploaded|merged|released|republished|shipped|posted)\b(?!\s+(?:blog|post|page|version|package|site|copy|report|article|files?|templates?|branch|folder|build|commits?)\b)|\b(?:is\s+(?:now\s+)?(?:live|released|on\s+npm|on\s+github|up(?!\s*(?:\d|\$|by\b|to\b|from\b|and\s+running\b)))|went\s+live|now\s+live|are\s+live|on\s+npm\b|in\s+production|is\s+on\s+main\b|(?:is|are|now|already|sits?)\s+(?:up\s+)?on\s+the\s+server)\b/i],
|
|
50
|
+
['state', /\b(?:nothing\s+(?:changed|was\s+changed|else\s+changed|left\s+uncommitted|was\s+left)|didn't\s+(?:disturb|touch|change)|(?:wasn't|weren't|not)\s+touched|untouched|unchanged|as\s+you\s+left\s+it|byte-identical|exactly\s+as\s+(?:you|I|we)\s+left|same\s+\d+\s+(?:staged|files)|no\s+(?:changes|diff|difference))\b/i],
|
|
51
|
+
['verify', /\b(?:all|both)\s+\d+\s+(?:done|ok|checked)\b|\b(?:verified|confirmed|double-checked|checked(?:\s+(?:that|it|the|all|every|against))?|validated|I\s+(?:ran|re-ran|tested|opened|loaded|looked|compared|hashed|checked)|renders\b|rendered\s+(?:fine|correctly|ok|as\s+expected)|loads\s+(?:fine|correctly|ok)|loaded\s+(?:fine|correctly|ok)|works?\s+(?:as\s+expected|fine|correctly|now|end-to-end)|working\s+(?:as\s+expected|fine|correctly|now)|round-?trips?|resolves?(?:d)?|matches?(?:\s+exactly)?|byte-exact|identical|(?:is|was|are|were|looks?|looked)\s+(?:fine|ok|good|correct|right)\b|has\s+(?:a\s+)?working\b|went\s+through\b|rendered(?=\s*[,.;]|\s+(?:fine|correctly|ok|properly|as\s+expected))|(?:took|measured|clocked|was|is|it's)\s+\d[\d.,]*\s*(?:ms|s|secs?|seconds?|minutes?|MB|KB|GB|bytes)\b|all\s+(?:green|\d+\s+done|\d+\s+ok)|\d+\s*\/\s*\d+,?\s*(?:zero|no)\s+errors?|tested\s+(?:it|this|that|locally|end\s+to\s+end)|no\s+errors?|zero\s+errors?|clean\b)\b/i],
|
|
52
|
+
['fix', /(?<!\b(?:the|a|its|my|your|this|our|that|one|each)\s)\bfixed\b(?!\s+(?:cost|price|fee|quote|rate|width|height|size|number|amount|set|point|term|period|version|one|copy|app))|\bfix\s+is\s+in\b|\b(?:resolved|patched|corrected|repaired)\b|\bno\s+longer\s+(?!needed|necessary|required|relevant|used|in\s+use|available|exists?|supported|part|the\s+case)\w+|\bnow\s+(?:works|handles|passes|renders|resolves|loads|decodes|parses|starts)\b|\bbug\s+(?:is\s+)?(?:fixed|gone)\b/i],
|
|
53
|
+
['write', /\b(?:wrote|written|created|saved|added|updated|delivered|copied|generated|rebuilt|regenerated|exported|installed|removed|deleted|renamed|moved|replaced|bumped|normalised|normalized|backed\s+up)\b[^.]{0,120}?(?:`[^`]+`|\b[\w./\\-]+\.(?:mjs|cjs|js|ts|json|md|html|css|py|yml|yaml|pdf|png|jpg|jpeg|zip|cmd|txt|jsonl|svg|mp4|csv|xlsx|docx)\b|\bfile|\bfolder|\brepo\b|\bbranch\b|\btag\b)/i],
|
|
54
|
+
];
|
|
55
|
+
const NUM_RE = /(?<![\w.\/#-])(?:\$\d[\d,]*(?:\.\d+)?|\d{1,3}(?:,\d{3})+(?:\.\d+)?|\d+(?:\.\d+)?(?:%|×|x)?|\b[0-9a-f]{7,40}\b)(?![\w-])/g;
|
|
56
|
+
const SHA_RE = /\b[0-9a-f]{7,40}\b/g;
|
|
57
|
+
const VERSION_RE = /\b\d+\.\d+\.\d+\b/g;
|
|
58
|
+
const URL_RE = /\b(?:https?:\/\/)?[\w.-]+\.(?:com|au|io|dev|org|net|app)(?:\/[\w./#?=&-]*)?/gi;
|
|
59
|
+
const PATH_RE = /`([^`\s]{2,120})`|((?:[A-Za-z]:)?[\w./\\-]+\.(?:mjs|cjs|js|ts|json|md|html|css|py|yml|yaml|pdf|png|jpg|jpeg|zip|cmd|txt|jsonl|svg|mp4|csv|xlsx|docx))\b/g;
|
|
60
|
+
|
|
61
|
+
// A negated claim word ("haven't written it to any file", "No bundle shipped", "never pushed") is not a claim of that
|
|
62
|
+
// kind; 0.10 has no matcher for negatives. "nothing changed" and "no longer" start their own match, so they pass.
|
|
63
|
+
const NEGATED_RE = /\b(?:not|never|haven't|hasn't|hadn't|didn't|wasn't|weren't|won't|isn't|aren't|don't|doesn't|nothing|no|without)\s+(?:\w+\s+){0,2}$/i;
|
|
64
|
+
function kindOf(s) {
|
|
65
|
+
for (const [kind, re] of KINDS) { const m = s.match(re); if (m && m.index != null && !NEGATED_RE.test(s.slice(Math.max(0, m.index - 40), m.index))) return kind; }
|
|
66
|
+
return null;
|
|
67
|
+
}
|
|
68
|
+
export function classify(s) {
|
|
69
|
+
if (/:\s*$/.test(s) || s.length < 8) return null; // a heading or a label, not a sentence
|
|
70
|
+
if (NOT_CLAIM_RE.test(s) || DESCRIBES_RE.test(s)) return null;
|
|
71
|
+
// Words inside short quotes are content the agent is showing, not asserting ("fixed quote, your deadline").
|
|
72
|
+
const kind = kindOf(s.replace(/["“][^"”\n]{1,80}["”]/g, '"…"'));
|
|
73
|
+
// "Not verified: …" / "I couldn't push" are declarations; a marker tacked on late ("…the audit I couldn't run here")
|
|
74
|
+
// does not turn a ship or test claim into one.
|
|
75
|
+
const dm = s.match(DECLARED_RE);
|
|
76
|
+
if (dm && (dm.index < s.length / 2 || !kind)) return 'declared';
|
|
77
|
+
// "Hook: a Stop hook installed as …" — an inventory line naming a thing, not an action. Only a write-shaped line
|
|
78
|
+
// with a noun label qualifies; "Fixed: the crash…" and "Verified: the README…" keep their kind.
|
|
79
|
+
const inv = kind === 'write' && s.match(/^([A-Z][\w' -]{0,24}):\s+(?:a|an|the)\b/);
|
|
80
|
+
if (inv && !/ed$|\b(?:done|live|ok)$/i.test(inv[1])) return null;
|
|
81
|
+
return kind;
|
|
82
|
+
}
|
|
83
|
+
// Small bare numbers are only worth checking when the sentence is a count ("58 of 58 pass", "27 pages").
|
|
84
|
+
const numbersIn = (s) => Array.from(new Set((s.match(NUM_RE) || []).map((n) => n.replace(/,/g, '')).filter((n) => !/^\d{1,2}$/.test(n) || /\b(?:pass|tests?|of|files?|pages?|commits?|sessions?|runs?)\b/i.test(s))));
|
|
85
|
+
|
|
86
|
+
// ---------------------------------------------------------------------------------------------
|
|
87
|
+
// Evidence
|
|
88
|
+
// ---------------------------------------------------------------------------------------------
|
|
89
|
+
const SHELL = (c) => c.name === 'Bash' || c.name === 'PowerShell' || /device_bash$/.test(c.name || '');
|
|
90
|
+
const cmdOf = (c) => { const i = c.input || {}; return typeof i.command === 'string' ? i.command : typeof i.script === 'string' ? i.script : ''; };
|
|
91
|
+
const inputText = (c) => { try { return JSON.stringify(c.input || {}); } catch (e) { return ''; } };
|
|
92
|
+
const TEST_CMD = /\b(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?(?:test|check|e2e|audit|lint|build|typecheck)\b|\bnode\s+--test\b|\bnode\s+(?:[\w./-]*\/)?test\/[\w.-]+\.m?js\b|\bpytest\b|\bpython3?\s+-m\s+(?:pytest|unittest)\b|\bcargo\s+test\b|\bgo\s+test\b|\bvitest\b|\bjest\b|\bmocha\b|\bplaywright\s+test\b|\btsc\b|\beslint\b|\bruff\b|\bmake\s+(?:test|check)\b|\bclaude\s+plugin\s+validate\b|\bdotnet\s+test\b|\bphpunit\b|\brspec\b|\bphp\s+-l\b|\bnode\s+(?:--check|-c)\b|\bpython3?\s+-m\s+py_compile\b|\bflake8\b|\bpylint\b|\bmypy\b|\bshellcheck\b|\bprettier\s+--check\b|\bbiome\s+(?:check|lint)\b|\bpython3?\s+(?:[\w./-]*\/)?(?:test_[\w-]+|[\w-]+_test|tests?)\.py\b|\bnode\s+(?:[\w./-]*\/)?[\w-]*test[\w-]*\.[cm]?js\b/i;
|
|
93
|
+
const CI_RESULT = /completed[:\s]+success|conclusion["']?\s*[:=]\s*["']?success|completed\s+successfully|\bci\s+#?\d+[^.\n]{0,40}\b(?:green|success)|"status":\s*"completed"[^}]{0,80}"conclusion":\s*"success"|publish\s+run[^\n]{0,40}success/i;
|
|
94
|
+
const CI_FAIL = /conclusion["']?\s*[:=]\s*["']?(?:failure|cancelled)|completed[:\s]+failure|\bfailed\b[^\n]{0,20}\bci\b/i;
|
|
95
|
+
const FAIL_MARK = /\bnot ok\b|# fail [1-9]|\b[1-9]\d* fail(?:ed|ing|ures?)?\b|\bFAIL(?:ED)?\b|AssertionError|Error:|\bTypeError|\bReferenceError|\bSyntaxError|Tests:\s+\d+\s+failed|✗|✖|\bexit(?:ed|\s*code)?\s*[=:]?\s*[1-9]\d*\b|Cannot find module|MODULE_NOT_FOUND/;
|
|
96
|
+
// A test-like run's identity: the runner and file it names, so a failing e2e run does not contradict "58 of 58 pass"
|
|
97
|
+
// said about the unit run beside it.
|
|
98
|
+
const testKey = (c) => { const m = cmdOf(c).match(TEST_CMD); return m ? m[0].toLowerCase().replace(/\s+/g, ' ') : (c.name + ':' + (c.resultText || '').slice(0, 40)); };
|
|
99
|
+
function runFailed(c) {
|
|
100
|
+
const r = c.resultText || ''; const k = testCounts(r);
|
|
101
|
+
return !!(c.isError || (k ? k.fail > 0 : FAIL_MARK.test(r)) || (CI_FAIL.test(r) && !CI_RESULT.test(r)));
|
|
102
|
+
}
|
|
103
|
+
const SHELL_EDIT_RE = /\bsed\s+-i\b|python3?\s+-\s*<<|\bcat\s*>>?\s|\btee\b|\bmv\b|\bcp\b|>\s*[\w./-]+\s*(?:&&|;|$)/;
|
|
104
|
+
const SHIP_CMD = /\bgit\s+(?:push|commit)\b|\bnpm\s+publish\b|\bgh\s+(?:pr|release)\b/;
|
|
105
|
+
// The sentence itself says the push (publish, upload, deploy) did not happen.
|
|
106
|
+
const SHIP_DECLINED_RE = /\b(?:push|publish|upload|deploy|release)(?:es|ed|ing)?\s+(?:was\s+|were\s+|got\s+|is\s+|has\s+been\s+)?(?:refused|rejected|failed|blocked|denied|not\s+done)\b|\bnot\s+pushed\b|\bcouldn't\s+(?:push|publish|upload|deploy)\b|\bwithout\s+pushing\b|\bpush\s+(?:is\s+)?(?:pending|waiting|still\s+to\s+do)\b/i;
|
|
107
|
+
const WRITE_TOOLS = new Set(['Write', 'Edit', 'MultiEdit', 'NotebookEdit']);
|
|
108
|
+
// The agent says it did the writing ("I wrote", "Saved to", "Rebuilt dist/"), as opposed to reporting someone else's write.
|
|
109
|
+
const MY_WRITE_RE = /^(?:I|I've|I'd|I\s+have|Also|And|Then|Now|Done|Saved|Wrote|Written|Created|Updated|Added|Delivered|Copied|Rebuilt|Regenerated|Generated|Exported|Installed|Removed|Deleted|Renamed|Moved|Replaced|Bumped|Backed|Both|All|Everything)\b|\b(?:I|I've|we|we've)\s+(?:also\s+|just\s+|now\s+)?(?:wrote|saved|created|added|updated|copied|delivered|generated|rebuilt|regenerated|exported|installed|removed|deleted|renamed|moved|replaced|bumped|backed|normali[sz]ed)\b/i;
|
|
110
|
+
const SHIP_TOOLS = new Set(['mcp__remote-devices__device_commit_files', 'Artifact', 'SendUserFile', 'mcp__claude-in-chrome__file_upload']);
|
|
111
|
+
// An edit that can undo a test result: code or tests inside the project's code folders. A report written to a
|
|
112
|
+
// scratch folder, or a site page, does not make an earlier test run stale.
|
|
113
|
+
const SOURCE_FILE = /(?:^|[\\/])(?:src|lib|test|tests|spec|specs|app|pkg|cmd|internal|hooks|bin|scripts)[\\/][^\\/]*\.(?:mjs|cjs|js|jsx|ts|tsx|py|rb|go|rs|java|kt|cs|php|c|cc|cpp|h|hpp|swift|sh|ps1|yml|yaml|json|toml)$|\.(?:test|spec)\.[cm]?[jt]sx?$|_test\.(?:py|go|rs)$/i;
|
|
114
|
+
|
|
115
|
+
// A result that reads like a test run whatever the command was: a summary line, PASS/FAIL rows, a lint's verdict.
|
|
116
|
+
const looksLikeRun = (r) => { r = String(r || ''); return !!testCounts(r) || /audit passed|browser test passed|Validation passed|No syntax errors detected|^\s*(?:PASS|FAIL)\b.*\n\s*(?:PASS|FAIL)\b/m.test(r); };
|
|
117
|
+
function testCounts(text) {
|
|
118
|
+
const t = String(text || '');
|
|
119
|
+
let m;
|
|
120
|
+
if ((m = t.match(/# pass (\d+)[\s\S]*?# fail (\d+)/))) return { pass: +m[1], fail: +m[2], total: +m[1] + +m[2] };
|
|
121
|
+
if ((m = t.match(/# fail (\d+)[\s\S]*?# pass (\d+)/))) return { pass: +m[2], fail: +m[1], total: +m[1] + +m[2] };
|
|
122
|
+
if ((m = t.match(/(\d+) passed(?:,\s*(\d+) failed)?/))) return { pass: +m[1], fail: +(m[2] || 0), total: +m[1] + +(m[2] || 0) };
|
|
123
|
+
if ((m = t.match(/Tests:\s+(?:(\d+) failed,\s*)?(\d+) passed,\s*(\d+) total/))) return { pass: +m[2], fail: +(m[1] || 0), total: +m[3] };
|
|
124
|
+
if ((m = t.match(/(\d+) passing(?:\s*\n\s*(\d+) failing)?/))) return { pass: +m[1], fail: +(m[2] || 0), total: +m[1] + +(m[2] || 0) };
|
|
125
|
+
return null;
|
|
126
|
+
}
|
|
127
|
+
const claimedCount = (s) => { const m = s.match(/\b(?:all\s+)?(\d[\d,]*)\s*(?:of\s*\d[\d,]*\s*)?(?:tests?|specs?)?\s*(?:still\s+)?(?:pass|green)|\b(\d[\d,]*)\s*\/\s*(\d[\d,]*)/i); return m ? +(m[1] || m[3] || m[2]).replace(/,/g, '') : null; };
|
|
128
|
+
|
|
129
|
+
// The window: this agent's calls in the same turn before the sentence (the last `cap` of them), plus the rest
|
|
130
|
+
// of the turn after it (a claim sometimes precedes its own check). A claim that opens a turn refers to the
|
|
131
|
+
// previous turn's work, so it gets that turn's tail.
|
|
132
|
+
function windowFor(trace, item, byReq, cap = 80) {
|
|
133
|
+
const reqs = trace.requests;
|
|
134
|
+
const before = [], after = [];
|
|
135
|
+
let opensTurn = true;
|
|
136
|
+
for (let i = 0; i < item.index; i++) { const r = reqs[i]; if (r.agent === item.agent && r.turnIndex === item.request.turnIndex) { opensTurn = false; before.push(...(byReq.get(r.id) || [])); } }
|
|
137
|
+
if (opensTurn) { const prev = []; for (let i = item.index - 1; i >= 0 && prev.length < 20; i--) { const r = reqs[i]; if (r.agent !== item.agent || r.turnIndex === item.request.turnIndex) continue; prev.unshift(...(byReq.get(r.id) || [])); } before.unshift(...prev.slice(-20)); }
|
|
138
|
+
before.push(...(byReq.get(item.request.id) || []));
|
|
139
|
+
for (let i = item.index + 1; i < reqs.length && after.length < 12; i++) { const r = reqs[i]; if (r.agent !== item.agent || r.turnIndex !== item.request.turnIndex) break; after.push(...(byReq.get(r.id) || [])); }
|
|
140
|
+
return { before: before.slice(-cap), after };
|
|
141
|
+
}
|
|
142
|
+
// Calls that can never be a receipt: the agent talking to the human (the sentence itself often travels in one),
|
|
143
|
+
// bookkeeping, skill and tool lookups, memory reads, and subagent results (a subagent saying "pushed" is hearsay;
|
|
144
|
+
// its own transcript is judged on its own). Their results carry words like "delivered" that the matchers look for.
|
|
145
|
+
const NOISE = new Set(['SendUserMessage', 'AskUserQuestion', 'ToolSearch', 'Skill', 'ListSkills', 'SearchSkills', 'TaskList', 'TaskGet', 'TaskCreate', 'TaskUpdate', 'TodoWrite', 'ReadNotifications', 'EnterPlanMode', 'ExitPlanMode', 'ListAgents', 'SendMessage', 'ScheduleWakeup']);
|
|
146
|
+
const usable = (c) => !NOISE.has(c.name) && c.category !== 'agent' && !/^mcp__memory__/.test(c.name || '');
|
|
147
|
+
function lastWriteAt(calls) { let t = -Infinity; for (const c of calls) { const p = (c.input && (c.input.file_path || c.input.notebook_path)) || ''; if (WRITE_TOOLS.has(c.name) && SOURCE_FILE.test(String(p)) && c.start != null && c.start > t) t = c.start; } return t; }
|
|
148
|
+
function contains(text, needle) { return needle && String(text || '').toLowerCase().includes(String(needle).toLowerCase()); }
|
|
149
|
+
|
|
150
|
+
// Judge one claim. Returns { verdict, evidence: [toolCallIds], note }.
|
|
151
|
+
function judgeOne(trace, claim, win) {
|
|
152
|
+
const all = win.before.concat(win.after).filter(usable);
|
|
153
|
+
const s = claim.text;
|
|
154
|
+
const ev = (calls, note, verdict) => ({ verdict, evidence: calls.map((c) => c.id), note });
|
|
155
|
+
if (claim.kind === 'test') {
|
|
156
|
+
const runs = all.filter((c) => (SHELL(c) && (TEST_CMD.test(cmdOf(c)) || looksLikeRun(c.resultText))) || ((SHELL(c) || c.name === 'WebFetch' || /javascript|browser_batch|get_page_text|read_page/.test(c.name) || c.name === 'Read') && CI_RESULT.test(c.resultText || '')));
|
|
157
|
+
if (!runs.length) {
|
|
158
|
+
// A count quoted from a file or note the agent read is sourced, not measured — listed, not flagged.
|
|
159
|
+
const want = claimedCount(s);
|
|
160
|
+
const src = want != null ? all.filter((c) => c.category === 'read' && contains(c.resultText, String(want))) : [];
|
|
161
|
+
if (src.length) return ev(src.slice(-1), `the count comes from something the agent read, not from a run`, 'sourced');
|
|
162
|
+
// "All green." / "All checks pass." after a round of curls is a verify claim in test clothing.
|
|
163
|
+
if (!/\b(?:tests?|suites?|specs?|e2e|audit|ci|lint(?:er)?|build|typecheck)/i.test(s)) {
|
|
164
|
+
const insp = all.filter((c) => c.category !== 'write' && c.category !== 'user' && c.category !== 'agent' && !c.isError);
|
|
165
|
+
if (insp.length) return ev(insp.slice(-2), 'no test command; backed by an inspection in the window', 'verified');
|
|
166
|
+
}
|
|
167
|
+
return ev([], 'no test, build, validate or CI result in the window', 'unverified');
|
|
168
|
+
}
|
|
169
|
+
// Which run is the sentence about? One that names the e2e, audit, lint, build or CI is judged on those runs;
|
|
170
|
+
// one that names a count is judged on the latest run whose counts agree with it, unless the same command failed
|
|
171
|
+
// later. Otherwise the latest run in the window.
|
|
172
|
+
const want = claimedCount(s);
|
|
173
|
+
const vocab = (s.match(/\b(e2e|audit|lint|build|typecheck|type-check|validat|ci)\b/i) || [])[1];
|
|
174
|
+
let pool = runs;
|
|
175
|
+
if (vocab) { const re = new RegExp('\\b' + vocab, 'i'); const v = runs.filter((c) => re.test(cmdOf(c) + ' ' + (c.resultText || '').slice(0, 200)) || (/^ci$/i.test(vocab) && CI_RESULT.test(c.resultText || ''))); if (v.length) pool = v; }
|
|
176
|
+
let last = pool[pool.length - 1];
|
|
177
|
+
if (want != null) {
|
|
178
|
+
const match = pool.filter((c) => { const k = testCounts(c.resultText); return k && (k.pass === want || k.total === want); });
|
|
179
|
+
if (match.length) { const m = match[match.length - 1]; if (!pool.some((c) => c.start != null && m.start != null && c.start > m.start && testKey(c) === testKey(m) && runFailed(c))) last = m; }
|
|
180
|
+
}
|
|
181
|
+
const ci = runs.filter((c) => CI_RESULT.test(c.resultText || ''));
|
|
182
|
+
if (runFailed(last)) return ev([last], 'the latest run in the window did not pass', 'contradicted');
|
|
183
|
+
const counts = testCounts(last.resultText);
|
|
184
|
+
if (want != null && counts && counts.pass !== want && counts.total !== want) {
|
|
185
|
+
if (ci.length && ci[ci.length - 1] !== last) return ev([last, ci[ci.length - 1]], `the run says ${counts.pass} passed, the sentence says ${want}; CI reports success`, 'verified');
|
|
186
|
+
return ev([last], `the sentence says ${want}, the last run says ${counts.pass} of ${counts.total}`, 'partial');
|
|
187
|
+
}
|
|
188
|
+
const lw = lastWriteAt(win.before.filter((c) => c.start != null && last.start != null && c.start > last.start));
|
|
189
|
+
if (lw > -Infinity && !ci.some((c) => c.start != null && c.start > lw)) return ev([last], 'the run predates a later edit to source or tests, and nothing re-ran', 'stale');
|
|
190
|
+
return ev([last], counts ? `${counts.pass} passed, ${counts.fail} failed` : (CI_RESULT.test(last.resultText || '') ? 'CI success' : 'run ok'), 'verified');
|
|
191
|
+
}
|
|
192
|
+
if (claim.kind === 'ship') {
|
|
193
|
+
const shas = (s.match(SHA_RE) || []).filter((h) => /\d/.test(h) && /[a-f]/.test(h));
|
|
194
|
+
const versions = s.match(VERSION_RE) || [];
|
|
195
|
+
const hits = [];
|
|
196
|
+
for (const c of all) {
|
|
197
|
+
const r = c.resultText || '';
|
|
198
|
+
const cmd = cmdOf(c);
|
|
199
|
+
const isShip = (SHELL(c) && SHIP_CMD.test(cmd)) || SHIP_TOOLS.has(c.name) || /\bnpm\s+view\b|\bgit\s+(?:fetch|log|ls-remote)\b|\bcurl\b/.test(cmd) || /Published|Version \d+|written|delivered|status=1|"success"|completed|dist-tags|origin\/main|\bmain\b.*->|\bOK\b|\bsaved\b|done=true|uploaded|extracted/.test(r);
|
|
200
|
+
if (!isShip) continue;
|
|
201
|
+
if (shas.length && !shas.some((h) => contains(r, h))) continue;
|
|
202
|
+
if (versions.length && !versions.some((v) => contains(r, v) || contains(inputText(c), v))) continue;
|
|
203
|
+
hits.push(c);
|
|
204
|
+
}
|
|
205
|
+
if (!hits.length && versions.length) {
|
|
206
|
+
// The version is in the sentence but not in the result: a ship-tool result in the window still counts, with the gap noted.
|
|
207
|
+
const tool = all.filter((c) => SHIP_TOOLS.has(c.name) && !c.isError);
|
|
208
|
+
if (tool.length) return ev(tool.slice(-1), `ship result in the window (the result does not carry ${versions[0]})`, 'verified');
|
|
209
|
+
const src = all.filter((c) => c.category === 'read' && versions.some((v) => contains(c.resultText, v)));
|
|
210
|
+
if (src.length) return ev(src.slice(-1), 'the version comes from something the agent read, not from a ship step', 'sourced');
|
|
211
|
+
}
|
|
212
|
+
if (!hits.length) return ev([], shas.length ? `no result carries ${shas[0]}` : versions.length ? `no result carries ${versions[0]}` : 'no push, publish, upload or commit result in the window', 'unverified');
|
|
213
|
+
const ok = hits.filter((c) => !c.isError && !/\b(?:fatal|rejected|denied|Authentication failed|ERR!)\b/i.test(c.resultText || ''));
|
|
214
|
+
// "Committed as c9b4747, but the push was refused": the failure is in the sentence, so it is not a contradiction.
|
|
215
|
+
if (!ok.length) return SHIP_DECLINED_RE.test(s) ? ev(hits.slice(-1), 'the ship step failed, as the sentence says', 'verified') : ev(hits.slice(-1), 'the ship step in the window failed', 'contradicted');
|
|
216
|
+
return ev(ok.slice(-1), 'ship result in the window', 'verified');
|
|
217
|
+
}
|
|
218
|
+
if (claim.kind === 'write') {
|
|
219
|
+
const names = [];
|
|
220
|
+
let m; PATH_RE.lastIndex = 0; while ((m = PATH_RE.exec(s))) { const n = (m[1] || m[2]).replace(/[\\/]+$/, '').split(/[\\/]/).pop(); if (n) names.push(n); }
|
|
221
|
+
// A shell call counts as a write only when its command writes (cp, mv, sed -i, a redirect, a build); `tail x` naming the file does not.
|
|
222
|
+
const writes = (c) => WRITE_TOOLS.has(c.name) || SHIP_TOOLS.has(c.name) || (SHELL(c) && (SHELL_EDIT_RE.test(cmdOf(c)) || /\b(?:mkdir|git\s+(?:rm|add|tag|worktree|commit|update-ref)|printf|zip|tar|touch|rsync|scp|curl\s+-T|npm\s+run\s+build|node\s+scripts\/build)\b/.test(cmdOf(c))));
|
|
223
|
+
const hits = all.filter((c) => !c.isError && writes(c) && (!names.length || names.some((n) => contains(inputText(c), n) || contains(c.resultText, n))));
|
|
224
|
+
if (!hits.length && names.length && !MY_WRITE_RE.test(s)) {
|
|
225
|
+
// "The repair agent wrote 10 rows into logs/x.jsonl": someone else's write, seen by reading the file.
|
|
226
|
+
const seen = all.filter((c) => !c.isError && c.category !== 'write' && names.some((n) => contains(inputText(c), n) || contains(c.resultText, n)));
|
|
227
|
+
if (seen.length) return ev(seen.slice(-1), 'the file was read in the window (the write is attributed to something else)', 'verified');
|
|
228
|
+
}
|
|
229
|
+
if (!hits.length) return ev([], names.length ? `no write touching ${names[0]}` : 'no write in the window', 'unverified');
|
|
230
|
+
return ev(hits.slice(-1), 'write in the window', 'verified');
|
|
231
|
+
}
|
|
232
|
+
if (claim.kind === 'fix') {
|
|
233
|
+
// The receipt is a run after the last edit to code or tests, or failing that any inspection after the last edit
|
|
234
|
+
// of any kind: a fix to a crash is checked by running the thing again, a fix to a label by reading the page.
|
|
235
|
+
// Shell edits (sed -i, a python heredoc, cat >) count as edits, not checks.
|
|
236
|
+
const isEdit = (c) => !c.isError && c.start != null && ((c.category === 'write') || (SHELL(c) && SHELL_EDIT_RE.test(cmdOf(c))));
|
|
237
|
+
const writes = all.filter(isEdit);
|
|
238
|
+
const codeEdits = writes.filter((c) => c.category === 'write' ? SOURCE_FILE.test(String((c.input && (c.input.file_path || c.input.notebook_path)) || '')) : (cmdOf(c).match(/[\w.\/\\-]+\.[a-z]{1,5}\b/g) || []).some((x) => SOURCE_FILE.test(x)));
|
|
239
|
+
const lwCode = codeEdits.length ? Math.max(...codeEdits.map((c) => c.start)) : -Infinity;
|
|
240
|
+
const lwAny = writes.length ? Math.max(...writes.map((c) => c.start)) : -Infinity;
|
|
241
|
+
const runs = all.filter((c) => SHELL(c) && !isEdit(c) && (TEST_CMD.test(cmdOf(c)) || /node\s+-e|node\s+bin\/|python3?\s+|curl\b/.test(cmdOf(c))) && !c.isError && c.start != null && c.start >= lwCode);
|
|
242
|
+
if (runs.length) return ev(runs.slice(-1), 'a run after the last edit succeeded', 'verified');
|
|
243
|
+
const after = all.filter((c) => !isEdit(c) && c.category !== 'user' && !c.isError && c.start != null && c.start >= lwAny);
|
|
244
|
+
if (after.length) return ev(after.slice(-1), writes.length ? 'checked after the last edit (no test or re-run)' : 'inspection in the window', 'verified');
|
|
245
|
+
if (writes.length) return ev(writes.slice(-1), 'an edit, but nothing ran or was read after it', 'partial');
|
|
246
|
+
return ev([], 'no edit, run or check in the window', 'unverified');
|
|
247
|
+
}
|
|
248
|
+
// verify / state: an inspection by this agent, in the window, that did not error. (Staleness is a test-claim
|
|
249
|
+
// notion: a report written to a scratch file does not undo a check of the code.)
|
|
250
|
+
const inspections = all.filter((c) => c.category !== 'write' && c.category !== 'user' && c.category !== 'agent' && !c.isError);
|
|
251
|
+
if (inspections.length) return ev(inspections.slice(-2), 'inspection in the window', 'verified');
|
|
252
|
+
const failedOnly = all.filter((c) => c.category !== 'write' && c.isError);
|
|
253
|
+
if (failedOnly.length) return ev(failedOnly.slice(-1), 'the only check in the window failed', 'contradicted');
|
|
254
|
+
return ev([], 'no inspection in the window', 'unverified');
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
// ---------------------------------------------------------------------------------------------
|
|
258
|
+
// The ledger
|
|
259
|
+
// ---------------------------------------------------------------------------------------------
|
|
260
|
+
export function extractClaims(trace) {
|
|
261
|
+
const texts = userFacingTexts(trace);
|
|
262
|
+
const claims = [];
|
|
263
|
+
for (const t of texts) {
|
|
264
|
+
for (const sent of sentences(t.text)) {
|
|
265
|
+
const kind = classify(sent);
|
|
266
|
+
if (!kind) continue;
|
|
267
|
+
claims.push({ id: `${t.request.id}:${claims.length}`, request: t.request.id, requestNo: t.index + 1, turn: t.request.turnIndex + 1, turnIndex: t.request.turnIndex, agent: t.agent, via: t.via, kind, text: sent, numbers: kind === 'declared' ? [] : numbersIn(sent), _item: t });
|
|
268
|
+
}
|
|
269
|
+
}
|
|
270
|
+
return { texts, claims };
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// What a claim is about: its kind, the two words before the claim words, the claim words, and its numbers.
|
|
274
|
+
function restateKey(cl) {
|
|
275
|
+
const re = (KINDS.find((k) => k[0] === cl.kind) || [])[1]; const t = cl.text.replace(/\s+/g, ' ');
|
|
276
|
+
const m = re && t.match(re); if (!m || m.index == null) return null;
|
|
277
|
+
const before = t.slice(0, m.index).toLowerCase().replace(/[^\w\s]/g, ' ').split(/\s+/).filter(Boolean).slice(-2).join(' ');
|
|
278
|
+
return `${cl.kind}|${before}|${m[0].toLowerCase()}|${(cl.numbers || []).join(',')}`;
|
|
279
|
+
}
|
|
280
|
+
export function judgeClaims(trace, opts = {}) {
|
|
281
|
+
const { texts, claims } = extractClaims(trace);
|
|
282
|
+
const allResults = opts.numbers === false ? null : trace.toolCalls.map((c) => c.resultText || '').join('\n');
|
|
283
|
+
const byReq = new Map();
|
|
284
|
+
for (const c of trace.toolCalls) { if (c.unmatched) continue; if (!byReq.has(c.requestId)) byReq.set(c.requestId, []); byReq.get(c.requestId).push(c); }
|
|
285
|
+
for (const cl of claims) {
|
|
286
|
+
if (cl.kind === 'declared') { cl.verdict = 'declared'; cl.evidence = []; cl.note = 'the agent said so'; continue; }
|
|
287
|
+
const win = windowFor(trace, cl._item, byReq, opts.window || 80);
|
|
288
|
+
const j = judgeOne(trace, cl, win);
|
|
289
|
+
cl.verdict = j.verdict; cl.evidence = j.evidence; cl.note = j.note;
|
|
290
|
+
// The same ship or write claim, verified earlier, said again in a summary: the earlier receipt stands. A commit
|
|
291
|
+
// exists and a file stays written; a test, a state or a fix claim repeated later is exactly the pattern to flag.
|
|
292
|
+
if (cl.verdict === 'unverified' && (cl.kind === 'ship' || cl.kind === 'write')) {
|
|
293
|
+
const key = restateKey(cl);
|
|
294
|
+
const earlier = key && claims.find((o) => o !== cl && o.verdict === 'verified' && o.turn <= cl.turn && restateKey(o) === key);
|
|
295
|
+
if (earlier) { cl.verdict = 'verified'; cl.evidence = earlier.evidence.slice(); cl.note = `restates turn ${earlier.turn}, verified there`; }
|
|
296
|
+
}
|
|
297
|
+
if (allResults && cl.numbers.length) { const missing = cl.numbers.filter((n) => !allResults.includes(n) && !allResults.includes(n.replace(/^\$/, ''))); if (missing.length) cl.numbersMissing = missing; }
|
|
298
|
+
// A contradicted claim the agent corrected later gets the note, and drops out of the error rule.
|
|
299
|
+
if (cl.verdict === 'contradicted') { const later = texts.find((t) => t.index > cl._item.index && t.agent === cl.agent && DECLARED_RE.test(t.text)); if (later) cl.correctedAt = later.request.turnIndex + 1; }
|
|
300
|
+
}
|
|
301
|
+
for (const cl of claims) delete cl._item;
|
|
302
|
+
const count = (v) => claims.filter((c) => c.verdict === v).length;
|
|
303
|
+
const summary = { claims: claims.length, verified: count('verified'), declared: count('declared'), sourced: count('sourced'), partial: count('partial'), stale: count('stale'), unverified: count('unverified'), contradicted: count('contradicted'), byKind: {} };
|
|
304
|
+
for (const cl of claims) summary.byKind[cl.kind] = (summary.byKind[cl.kind] || 0) + 1;
|
|
305
|
+
return { kind: 'claims', schema: 1, session: trace.meta.sessionId, generated: new Date().toISOString(), redacted: false, claims, summary };
|
|
306
|
+
}
|
|
307
|
+
|
|
308
|
+
// check rules: contradicted → error, unverified → warn, stale / partial → info.
|
|
309
|
+
export const CLAIM_RULES = { contradicted: ['contradicted-claim', 'error'], unverified: ['unverified-claim', 'warn'], stale: ['stale-claim', 'info'], partial: ['stale-claim', 'info'] };
|
|
310
|
+
const oneLine = (s, n) => { s = String(s == null ? '' : s).replace(/\s+/g, ' ').trim(); return s.length > n ? s.slice(0, n - 1) + '…' : s; };
|
|
311
|
+
// A long sentence, quoted around the words that made it a claim ("…and the tests still pass.").
|
|
312
|
+
export function focus(text, kind, n = 160) {
|
|
313
|
+
const t = String(text).replace(/\s+/g, ' ').trim();
|
|
314
|
+
if (t.length <= n) return t;
|
|
315
|
+
const re = (KINDS.find((k) => k[0] === kind) || [])[1];
|
|
316
|
+
const m = re && t.match(re);
|
|
317
|
+
if (!m || m.index == null) return t.slice(0, n - 1) + '…';
|
|
318
|
+
const start = Math.max(0, Math.min(m.index - 40, t.length - n));
|
|
319
|
+
return (start ? '…' : '') + t.slice(start, start + n) + (start + n < t.length ? '…' : '');
|
|
320
|
+
}
|
|
321
|
+
export function claimFindings(ledger) {
|
|
322
|
+
const out = [];
|
|
323
|
+
for (const cl of ledger.claims) {
|
|
324
|
+
const r = CLAIM_RULES[cl.verdict]; if (!r) continue;
|
|
325
|
+
const [id, sev] = cl.correctedAt ? ['contradicted-claim', 'info'] : r;
|
|
326
|
+
const what = cl.verdict === 'partial' ? 'only partly backed' : cl.verdict === 'stale' ? 'backed by a check that predates a later edit' : cl.verdict === 'contradicted' ? 'contradicted by the transcript' : 'with no receipt in the transcript';
|
|
327
|
+
// The sentence goes in `detail` (blanked by --redact); the title is names and numbers only.
|
|
328
|
+
out.push({ id, severity: sev, title: `Turn ${cl.turn}${cl.agent !== 'main' ? ' (subagent)' : ''}: a ${cl.kind} claim ${what}${cl.correctedAt ? ` (corrected at turn ${cl.correctedAt})` : ''}`, detail: `"${focus(cl.text, cl.kind)}" — ${cl.note}.${cl.numbersMissing ? ` Numbers not found in any tool result: ${cl.numbersMissing.join(', ')}.` : ''}`, evidence: { requestIds: [cl.request], toolCallIds: cl.evidence, turnIndex: cl.turnIndex }, metric: cl.verdict === 'contradicted' ? 3 : cl.verdict === 'unverified' ? 2 : 1, claim: { kind: cl.kind, verdict: cl.verdict, turn: cl.turn } });
|
|
329
|
+
}
|
|
330
|
+
return out;
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
// ---------------------------------------------------------------------------------------------
|
|
334
|
+
// Renderings
|
|
335
|
+
// ---------------------------------------------------------------------------------------------
|
|
336
|
+
const ORDER = { contradicted: 0, unverified: 1, partial: 2, stale: 3, sourced: 4, verified: 5, declared: 6 };
|
|
337
|
+
const blank = (s) => `«${String(s == null ? '' : s).length} chars»`;
|
|
338
|
+
// Under --redact a note must not carry pieces of the sentence either: shas, versions, the numbers it quoted.
|
|
339
|
+
const noteOf = (c, redact) => {
|
|
340
|
+
const n = redact ? String(c.note || '').replace(SHA_RE, '«sha»').replace(VERSION_RE, '«version»') : c.note;
|
|
341
|
+
const nums = c.numbersMissing ? (redact ? `«${c.numbersMissing.length} number${c.numbersMissing.length === 1 ? '' : 's'}»` : c.numbersMissing.join(', ')) : null;
|
|
342
|
+
return { note: n, nums };
|
|
343
|
+
};
|
|
344
|
+
export function headline(rep) { const s = rep.summary; return `${s.claims} claim${s.claims === 1 ? '' : 's'} · ${s.verified} verified · ${s.declared} declared · ${s.unverified} unverified · ${s.contradicted} contradicted${s.partial || s.stale || s.sourced ? ` (${s.partial} partial, ${s.stale} stale, ${s.sourced} sourced)` : ''}`; }
|
|
345
|
+
export function claimsText(rep, opts = {}) {
|
|
346
|
+
const R = (t) => opts.redact ? blank(t) : t;
|
|
347
|
+
const L = [`Glassbox claims · ${rep.session ? rep.session.slice(0, 8) : 'session'}`, ` ${headline(rep)}`, ''];
|
|
348
|
+
const rows = rep.claims.slice().sort((a, b) => ORDER[a.verdict] - ORDER[b.verdict] || a.turn - b.turn);
|
|
349
|
+
for (const c of rows) {
|
|
350
|
+
const { note, nums } = noteOf(c, opts.redact);
|
|
351
|
+
L.push(` ${c.verdict.toUpperCase().padEnd(12)} ${c.kind.padEnd(8)} t${String(c.turn).padEnd(3)} ${oneLine(R(c.text), 96)}`);
|
|
352
|
+
L.push(` ↳ ${note}${c.evidence.length ? ` (${c.evidence.length} call${c.evidence.length === 1 ? '' : 's'})` : ''}${c.correctedAt ? ` · corrected at t${c.correctedAt}` : ''}${nums ? ` · numbers not in any result: ${nums}` : ''}`);
|
|
353
|
+
}
|
|
354
|
+
if (!rows.length) L.push(' no claims found — nothing the agent said asserted a checkable result');
|
|
355
|
+
L.push('', ' A verdict is about the sentence, not the code: verified means a tool result in this transcript shows it; unverified means none does. Checks done outside the transcript are invisible here.');
|
|
356
|
+
return L.join('\n') + '\n';
|
|
357
|
+
}
|
|
358
|
+
export function claimsMarkdown(rep, trace, opts = {}) {
|
|
359
|
+
const R = (t) => opts.redact ? blank(t) : t;
|
|
360
|
+
const callLine = (id) => { const c = trace && trace.toolCalls.find((x) => x.id === id); if (!c) return `\`${id}\``; const inp = c.input || {}; const what = inp.command || inp.file_path || inp.url || inp.message || ''; return `${c.name}${c.isError ? ' (error)' : ''} · turn ${c.turnIndex + 1}${what ? ': `' + oneLine(R(what), 80).replace(/`/g, "'") + '`' : ''}`; };
|
|
361
|
+
const L = ['# Glassbox claims — said vs did', '', `Session \`${rep.session || '—'}\` · ${rep.generated.slice(0, 10)}`, '', `**${headline(rep)}**`, '', '| turn | kind | claim | evidence | verdict |', '|---|---|---|---|---|'];
|
|
362
|
+
const rows = rep.claims.slice().sort((a, b) => ORDER[a.verdict] - ORDER[b.verdict] || a.turn - b.turn);
|
|
363
|
+
for (const c of rows) L.push(`| ${c.turn}${c.agent !== 'main' ? ' (sub)' : ''} | \`${c.kind}\` | ${oneLine(R(c.text), 110).replace(/\|/g, '\\|')} | ${c.evidence.length ? c.evidence.map(callLine).join('<br>').replace(/\|/g, '\\|') : '—'} | **${c.verdict}**${c.correctedAt ? ` (corrected at turn ${c.correctedAt})` : ''} |`);
|
|
364
|
+
const bad = rows.filter((c) => c.verdict === 'contradicted' || c.verdict === 'unverified');
|
|
365
|
+
if (bad.length) { L.push('', '## Without a receipt', ''); for (const c of bad) { const { note, nums } = noteOf(c, opts.redact); L.push(`- **${c.verdict}** · turn ${c.turn} · ${oneLine(R(c.text), 160)} — ${note}${nums ? `; numbers not in any tool result: ${nums}` : ''}`); } }
|
|
366
|
+
L.push('', '## How to read this', '', 'Every claim is one sentence the agent addressed to the human. *Verified* means a tool result in this transcript shows it, after the last change that could have undone it; *declared* means the agent itself said it had not checked; *partial* means the evidence covers part of the sentence; *stale* means the check ran before a later edit; *unverified* means no result backs it; *contradicted* means a result says the opposite. Nothing here judges the code — only whether the sentence has a receipt. `glassbox claims --format json` has every row.', '');
|
|
367
|
+
return L.join('\n');
|
|
368
|
+
}
|
package/src/cli.mjs
CHANGED
|
@@ -8,6 +8,7 @@ import { createRequire } from 'node:module';
|
|
|
8
8
|
import { fileURLToPath } from 'node:url';
|
|
9
9
|
import { readTextFile } from './textfile.mjs';
|
|
10
10
|
import { guardResponse } from './guard.mjs';
|
|
11
|
+
import { judgeClaims, claimFindings, claimsText, claimsMarkdown } from './claims.mjs';
|
|
11
12
|
const require = createRequire(import.meta.url);
|
|
12
13
|
const core = require('./trace-core.js');
|
|
13
14
|
// fileURLToPath, not URL.pathname: on Windows the latter gives "/C:/…" which path.resolve turns into "\C:\…".
|
|
@@ -84,7 +85,18 @@ export function loadSessionFiles(s) {
|
|
|
84
85
|
return files;
|
|
85
86
|
}
|
|
86
87
|
|
|
87
|
-
|
|
88
|
+
// Findings are the mechanical rules from trace-core plus the claims ledger's three (contradicted / unverified /
|
|
89
|
+
// stale), in one list, so check, the hook, the Action and compare all see them.
|
|
90
|
+
const SEV_ORDER = { error: 0, warn: 1, info: 2 };
|
|
91
|
+
export function analyse(files, rates) {
|
|
92
|
+
const trace = core.parseTrace(files);
|
|
93
|
+
const findings = core.diagnose(trace, rates ? { rates } : undefined);
|
|
94
|
+
const cost = core.estimateCost(trace, rates);
|
|
95
|
+
const claims = judgeClaims(trace);
|
|
96
|
+
findings.push(...claimFindings(claims));
|
|
97
|
+
findings.sort((a, b) => SEV_ORDER[a.severity] - SEV_ORDER[b.severity] || (b.metric || 0) - (a.metric || 0));
|
|
98
|
+
return { trace, findings, cost, claims };
|
|
99
|
+
}
|
|
88
100
|
|
|
89
101
|
// A team rate card: --rates FILE or GLASSBOX_RATES. JSON keyed by model-id prefix, USD per million tokens:
|
|
90
102
|
// { "claude-opus-5": { "in": 4, "out": 20, "read": 0.4, "w5m": 5, "w1h": 8 } }
|
|
@@ -165,7 +177,7 @@ export function collapseFindings(findings) {
|
|
|
165
177
|
}
|
|
166
178
|
const findingLine = ({ f, n }) => `${f.severity.toUpperCase().padEnd(5)} ${f.id.padEnd(16)} ${f.title}${n > 1 ? ` ×${n}` : ''}`;
|
|
167
179
|
|
|
168
|
-
export function checkReport({ trace, findings, cost }, opts = {}) {
|
|
180
|
+
export function checkReport({ trace, findings, cost, claims }, opts = {}) {
|
|
169
181
|
const failOn = opts.failOn || 'error';
|
|
170
182
|
const failing = findings.filter((f) => SEV[f.severity] <= SEV[failOn]);
|
|
171
183
|
const T = trace.totals, m = trace.meta;
|
|
@@ -177,6 +189,7 @@ export function checkReport({ trace, findings, cost }, opts = {}) {
|
|
|
177
189
|
const summary = { session: m.sessionId, title: m.title ? (redact ? '«' + m.title.length + ' chars»' : m.title) : null, model: m.models, wallMs: T.wallMs, activeMs: T.activeMs, turns: T.turns, requests: T.requests, toolCalls: T.toolCalls, toolErrors: T.toolErrors, orphans: T.orphans, usage: T.usage, cacheHitRatio: T.cacheHitRatio, cost: cost.reported != null ? cost.reported : cost.total, costSource: cost.source, start: m.start != null ? new Date(m.start).toISOString() : null, end: m.end != null ? new Date(m.end).toISOString() : null };
|
|
178
190
|
// Per-file use, keyed when a legend is given; with --redact and no legend the paths would leak, so it is omitted.
|
|
179
191
|
if (legend || !redact) summary.files = core.fileStats(trace).map((r) => ({ key: legend ? legend.keyFor(r.path) : r.path, reads: r.reads, writes: r.writes, errors: r.errors, agents: r.agents, chars: r.chars }));
|
|
192
|
+
if (claims) summary.claims = claims.summary; // said vs did, in one line of numbers (additive; schema stays 2)
|
|
180
193
|
const findingJson = (f) => {
|
|
181
194
|
const ev = f.evidence || {};
|
|
182
195
|
const paths = Array.from(new Set((ev.toolCallIds || []).map((id) => { const c = trace.toolCalls.find((x) => x.id === id); return c ? core.filePathOf(c) : null; }).filter(Boolean)));
|
|
@@ -381,6 +394,10 @@ export const HELP = `glassbox — other tools show you what happened in a Claude
|
|
|
381
394
|
is my CLAUDE.md doing anything? every rule in the project's instruction files judged
|
|
382
395
|
against every session of that project: obeyed / broken per occasion, with evidence;
|
|
383
396
|
shapes it cannot check are listed as such. --fail-under 80 exits 1 below that rate
|
|
397
|
+
glassbox claims [ID|FILE] [--format text|md|json] [--out FILE] [--redact] [--fail-on contradicted|unverified|none]
|
|
398
|
+
said vs did: every claim the agent made about its own work (tests pass, committed,
|
|
399
|
+
live, verified, nothing changed) matched to the tool result behind it — or the gap.
|
|
400
|
+
exit 1 when a claim at/above --fail-on has no receipt (default: contradicted)
|
|
384
401
|
glassbox fence [ID|FILE|DIR] [--since 30d] [--format text|md|json] [--out FILE] [--fail-on error|warn|info] [--shred] [--key FILE] [--sessions-only]
|
|
385
402
|
secrets that reached a transcript: known key formats, secrets named by context,
|
|
386
403
|
credential-file reads — with a masked preview and a fingerprint, never the value.
|
|
@@ -544,6 +561,22 @@ export async function main(argv, io = {}) {
|
|
|
544
561
|
if (reply != null) out(JSON.stringify(reply));
|
|
545
562
|
return 0;
|
|
546
563
|
}
|
|
564
|
+
if (cmd === 'claims') {
|
|
565
|
+
// Said vs did: the claims ledger for one session (subagents included).
|
|
566
|
+
const s = resolveTarget(args._[1], { home });
|
|
567
|
+
const { trace } = analyse(loadSessionFiles(s), rates);
|
|
568
|
+
const rep = judgeClaims(trace);
|
|
569
|
+
const failOn = String(args.flags['fail-on'] || 'contradicted');
|
|
570
|
+
if (!['contradicted', 'unverified', 'none'].includes(failOn)) throw new Error(`--fail-on must be contradicted, unverified or none (got "${failOn}")`);
|
|
571
|
+
const redact = !!args.flags.redact;
|
|
572
|
+
if (redact) { rep.redacted = true; for (const c of rep.claims) c.text = `«${c.text.length} chars»`; }
|
|
573
|
+
const fmtOut = outputFormat(args.flags);
|
|
574
|
+
const text = fmtOut === 'json' ? JSON.stringify(rep, null, 2) : fmtOut === 'md' ? claimsMarkdown(rep, trace, { redact }) : claimsText(rep, { redact });
|
|
575
|
+
if (args.flags.out) { const f = path.resolve(args.flags.out); fs.writeFileSync(f, text); out(`${f} (${rep.summary.claims} claims, ${rep.summary.unverified} unverified, ${rep.summary.contradicted} contradicted)`); }
|
|
576
|
+
else out(text.replace(/\n$/, ''));
|
|
577
|
+
const failed = failOn === 'none' ? false : failOn === 'unverified' ? (rep.summary.unverified + rep.summary.contradicted) > 0 : rep.summary.contradicted > 0;
|
|
578
|
+
return failed ? 1 : 0;
|
|
579
|
+
}
|
|
547
580
|
if (cmd === 'fence') {
|
|
548
581
|
// Secrets that reached a transcript. No target = every session under the home; an id, a file, or a folder of .jsonl.
|
|
549
582
|
const { fence, fenceText, fenceMarkdown, failedAt } = await import('./fence.mjs');
|
package/src/trace-core.js
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
})(typeof globalThis !== 'undefined' ? globalThis : this, function () {
|
|
10
10
|
'use strict';
|
|
11
11
|
|
|
12
|
-
const VERSION = '0.
|
|
12
|
+
const VERSION = '0.10.0';
|
|
13
13
|
|
|
14
14
|
// ---------------------------------------------------------------------------
|
|
15
15
|
// Rate card (USD per million tokens). Prefix-matched against model ids so dated
|
|
@@ -934,6 +934,10 @@
|
|
|
934
934
|
'permission-denied': 'Before a call the user may not want (installs, network, deleting, publishing), say what it will do and ask; for calls they always allow, suggest a permissions allow rule so the prompt goes away.',
|
|
935
935
|
'duplicate-subagent-read': 'When several subagents need the same file, put the relevant excerpt in their briefs or have one agent summarise it once, instead of every agent reading it into its own context.',
|
|
936
936
|
'idle-cache-expiry': 'The cache expired while the session sat idle; resume sooner, use the 1-hour cache lifetime where available, or start a fresh session with a summary instead of re-sending the whole history.',
|
|
937
|
+
// claims ledger (src/claims.mjs): said vs did
|
|
938
|
+
'unverified-claim': "Say it after you've checked it: a test claim needs a test run after your last edit, a shipped claim needs the command's result in this transcript, a 'nothing changed' claim needs a git status after the thing that could have changed it. If you did not check, say 'not re-run' instead.",
|
|
939
|
+
'contradicted-claim': 'The transcript disagrees with what you told the user; read the tool result you are summarising before you summarise it, and correct the statement now.',
|
|
940
|
+
'stale-claim': 'The check you are citing ran before your last edit, or covers only part of what you said; run it again or say exactly what it covered.',
|
|
937
941
|
};
|
|
938
942
|
|
|
939
943
|
// Advice for a concrete finding: the rule's line, except where the evidence says otherwise.
|
|
@@ -982,7 +986,7 @@
|
|
|
982
986
|
}
|
|
983
987
|
|
|
984
988
|
// Rules whose detail quotes tool output or model text (everything else is names and numbers).
|
|
985
|
-
const CONTENT_DETAIL = new Set(['failed-tool', 'api-error', 'hook-error', 'duplicate-subagent-read']);
|
|
989
|
+
const CONTENT_DETAIL = new Set(['failed-tool', 'api-error', 'hook-error', 'duplicate-subagent-read', 'unverified-claim', 'contradicted-claim', 'stale-claim']);
|
|
986
990
|
function redactDetail(f) { return CONTENT_DETAIL.has(f.id) ? blank(f.detail) : f.detail; }
|
|
987
991
|
|
|
988
992
|
// ---------------------------------------------------------------------------
|