glassbox-trace 0.10.1 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -4
- package/dist/glassbox.html +1 -1
- package/package.json +4 -2
- package/src/adhere.mjs +14 -6
- package/src/claims.mjs +82 -16
- package/src/cli.mjs +70 -14
- package/src/fence.mjs +19 -0
- package/src/memory-audit.mjs +236 -0
- package/src/memory.mjs +428 -0
- package/src/trace-core.js +1 -1
package/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Glassbox
|
|
1
|
+
# Glassbox for Claude Code
|
|
2
2
|
|
|
3
3
|
[](https://github.com/aldohushi1-stack/glassbox/actions/workflows/ci.yml)
|
|
4
4
|
|
|
@@ -40,6 +40,9 @@ glassbox watch live tail: the viewer follows the newest s
|
|
|
40
40
|
glassbox collect ./reports --format md --out fleet.md
|
|
41
41
|
one report from a folder of redacted check reports, one per machine
|
|
42
42
|
glassbox claims 81c4 --format md said vs did: every claim the agent made about its own work, with the tool result behind it — or the gap
|
|
43
|
+
glassbox recall login redirect what past sessions of this project asked, wrote, ran, committed and said about it
|
|
44
|
+
glassbox memory what the next session here will be told (hook --context), and the lessons behind it
|
|
45
|
+
glassbox memory audit everything loaded before the first prompt, its token cost, and what is dead or doubled
|
|
43
46
|
glassbox clean remove the temp files `open` and the hook leave behind
|
|
44
47
|
```
|
|
45
48
|
|
|
@@ -55,7 +58,7 @@ The npm package is `glassbox-trace` (plain `glassbox` was already taken); the co
|
|
|
55
58
|
|
|
56
59
|
**Let the agent read its own recorder.** `glassbox hook install` adds a Claude Code Stop hook (the command it writes is `npx -y glassbox-trace hook …`, which resolves the package on each run; pass `--command glassbox` after `npm i -g glassbox-trace@<version>` to pin it, or use the plugin below, which runs its bundled copy): every session ends with a one-screen Glassbox summary (turns, tool calls, context peak, cost, top findings). Add `--feedback` and the findings — with their evidence and the "next time" advice — are handed back to the agent; it reads them and says, in a sentence, what it would do differently. Each finding is handed back once per session, it never loops (`stop_hook_active` is respected), never blocks a clean session, and `glassbox hook uninstall` removes it, with a `.glassbox-backup` of `settings.json` kept.
|
|
57
60
|
|
|
58
|
-
Feedback at the end of a session can only change that session's last reply. Add `--context` to carry it forward: the Stop hook keeps
|
|
61
|
+
Feedback at the end of a session can only change that session's last reply. Add `--context` to carry it forward: the Stop hook keeps a memory of the project in `<project>/.glassbox/` — git-ignored, owner-only, secrets masked — and a SessionStart hook gives the next session there the lessons that keep coming back, a line for each recent session, and what the agent said it would do differently. See [Memory across sessions](#memory-across-sessions--glassbox-recall-and-glassbox-memory) below.
|
|
59
62
|
|
|
60
63
|
**Stop a loop before it happens.** `--guard` adds a PreToolUse hook: when the agent is about to repeat a call that has already failed twice in a row, unchanged, in the current turn, it blocks the call and tells the agent why, with the last error, and to change something or ask you rather than try again. It is deliberately narrow. Anything that could change the outcome in between — an edit, another command, a success, a new prompt — resets it, so re-running the tests after a fix is never blocked; calls a person stopped don't count; it never approves anything, so your permission prompts are untouched. Replayed over 4,121 real tool calls it blocked none, so it stays out of the way until an agent is actually stuck. It runs before every tool call, so it needs an installed copy: `--guard` writes a direct `node "…/bin/glassbox.mjs" hook …` command (for the Stop hook too) and refuses to install from an `npx` cache, where each call would cost seconds. The plugin doesn't include the guard ([why](docs/PLUGIN.md#the-loop-guard-is-not-in-the-plugin)); if both are installed, the plugin sees the CLI hook and steps aside.
|
|
61
64
|
|
|
@@ -164,6 +167,36 @@ Glassbox claims · 38f81247
|
|
|
164
167
|
|
|
165
168
|
Six kinds of claim (tests pass, shipped, verified, fixed, nothing changed, file written), judged on the same agent's tool calls in that turn: a test claim needs a test run after the last edit — from `Bash`, `PowerShell` or Cowork's `device_bash` — with its counts read and compared; a shipped claim needs the commit, push, publish or upload result, carrying the sha or version the sentence names; "nothing changed" needs a `git status` after the thing that could have changed it. An agent that says "not verified" gets a *declared* row and no finding. The same ledger runs inside `check` as three rules — `contradicted-claim` (error), `unverified-claim` (warn), `stale-claim` (info) — so the Stop hook hands the sentence back to the agent that wrote it. On sixteen real sessions (367 claims; three of them checked by hand first) the ledger found one unverified sentence — "the tests still pass", 24 minutes after the last run — five fixes never re-run after their edit, and no contradictions. A verdict is about the sentence, never the code. Details in [docs/CLAIMS.md](docs/CLAIMS.md).
|
|
166
169
|
|
|
170
|
+
### Memory across sessions — `glassbox recall` and `glassbox memory`
|
|
171
|
+
|
|
172
|
+
A Claude Code session starts knowing nothing about the last one, and Claude Code deletes transcripts after 30 days by default. With `--context` (or the plugin's `context` option), the Stop hook keeps `<project>/.glassbox/memory.json`: a digest of each of the project's last 50 sessions — what was asked, the files written, the checks and pushes that ran and how they ended, the commits with their shas, the agent's last words, the findings — and **lessons**: findings counted across sessions by what they are about (`retry-loop · Bash npm test`, `failed-tool · Edit`, `unverified-claim · test`). The next session starts with this note (≤ 8,000 characters, under Claude Code's 10,000 cap):
|
|
173
|
+
|
|
174
|
+
```
|
|
175
|
+
## Recurring in this project (worst first)
|
|
176
|
+
|
|
177
|
+
- retry-loop · Bash npm test — in 3 of the last 12 sessions, last 2026-10-09: Bash called 3× with identical input, e.g. `npm test -- auth`. Next time: When a call fails, change something before calling again: …
|
|
178
|
+
|
|
179
|
+
## Recent sessions (newest first)
|
|
180
|
+
|
|
181
|
+
- 2026-10-09 14:02 · a1b2c3d4 · "fix the login redirect" — wrote src/auth.js, test/auth.test.js · ran npm test (passed, last of 3) · committed 1a2b3c4 "fix: login redirect loop" · pushed
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
A lesson is handed on until ten sessions in a row go by without it, then it retires (and comes back if it fires again). Each session that started with a lesson is counted, so `glassbox memory` can say what nobody could before: *told in 9 sessions, came back in 1*. Everything is copied or counted from the transcripts — no model call, no tokens — and whatever a `fence` detector recognises as a secret is masked before it is written.
|
|
185
|
+
|
|
186
|
+
```
|
|
187
|
+
glassbox recall login redirect sessions of this project that mention it, best match first: asked / said / wrote / ran / commit
|
|
188
|
+
glassbox recall "dark mode" --since 30d --format md a quoted phrase is one term; --format md for pasting to an agent
|
|
189
|
+
glassbox recall no words: the latest sessions, one line each
|
|
190
|
+
glassbox memory what the next session will be told, and every lesson: sessions, told, came back, quiet
|
|
191
|
+
glassbox memory forget retry-loop drop a lesson (by rule or key); it starts over if it fires again
|
|
192
|
+
glassbox memory clear delete the project's memory
|
|
193
|
+
glassbox memory audit everything loaded before the first prompt, and what it costs
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
`recall` works without the hook: it reads the project's transcripts still on disk, and the ledger supplies the sessions whose transcripts are gone. Every word must match; a word that isn't in a transcript's raw text skips parsing it. The plugin's skill tells Claude to run it before re-investigating something a past session may already have settled.
|
|
197
|
+
|
|
198
|
+
**`glassbox memory audit`** lists what Claude Code loads into every session before the first prompt — managed, user, parent-directory and project `CLAUDE.md` files, `CLAUDE.local.md`, unscoped `.claude/rules/`, `@imports` (four hops, relative to the importing file), the first 200 lines or 25 KB of the auto-memory `MEMORY.md`, and Glassbox's own note — with an estimated token cost (characters ÷ 4, HTML comments stripped as Claude Code strips them), then flags what a mechanical reader can: imports that load nothing, backticked paths and `npm run` scripts that don't exist, the same line in two places, rules that pull opposite ways (`use pnpm` here, `use npm, not pnpm` there — the same parser as `adhere`), files over the 200 lines Claude Code's docs recommend, and `MEMORY.md` lines past the cut-off that never load. `--fail-on warn` for CI. Details in [docs/MEMORY.md](docs/MEMORY.md).
|
|
199
|
+
|
|
167
200
|
## What it flags
|
|
168
201
|
|
|
169
202
|
| rule | fires when |
|
|
@@ -247,12 +280,12 @@ console.log(compare({ trace, findings, cost }, other)); // other = the same
|
|
|
247
280
|
Your context is private — by construction, not by promise.
|
|
248
281
|
|
|
249
282
|
- **Nothing leaves.** The viewer makes no network requests at all — fonts, scripts and the demo are inside the one file, and the test suite fails if a build references anything outside it. The CLI sends nothing anywhere. `glassbox watch` binds to `127.0.0.1` only and stops with the command. The one network activity is installation: `npx` fetching the package, or the plugin marketplace fetching this repo.
|
|
250
|
-
- **What it touches.** It reads `~/.claude/projects/**` and writes only what you ask for: `open` writes a self-contained HTML to your temp folder (transcript included — delete it when done, or use `--out`), `hook install` edits `~/.claude/settings.json` with a backup beside it, `--context` writes `<project>/.glassbox/last-session.md
|
|
283
|
+
- **What it touches.** It reads `~/.claude/projects/**` and writes only what you ask for: `open` writes a self-contained HTML to your temp folder (transcript included — delete it when done, or use `--out`), `hook install` edits `~/.claude/settings.json` with a backup beside it, `--context` writes `<project>/.glassbox/last-session.md` and `memory.json` (owner-only, git-ignored, secrets masked; they quote prompts and commands), `--legend` writes the legend. The full inventory — files, processes, hooks, network, uninstall — is in [docs/IT.md](docs/IT.md), written for whoever has to approve it.
|
|
251
284
|
- **Counts, not text.** Transcripts contain everything the agent saw; that's why share mode and `--redact` exist. Redacted output keeps tool names, numbers, timings and cost and drops every string — prompts, commands, results, titles. Where a finding would quote something it says `«240 chars»`.
|
|
252
285
|
- **What is already on disk.** `glassbox fence` finds credentials that reached a transcript (known key formats, secrets named by context, credential-file reads), reports them masked and fingerprinted, and `--shred` overwrites them in place. See above.
|
|
253
286
|
- **The shape, not the names.** A redacted report still needs to say "this one file was read 48 times and 6 of those failed". `check --redact --legend audit.legend.json` replaces every path with a keyed hash (`file:b94b1a35`, HMAC-SHA256 with a random salt) and writes the key → path map to the legend file, which stays on your machine. The redacted JSON carries `summary.files` (reads, writes, failures, agents, chars per key) and `evidence.files` on findings; `duplicate-subagent-read` keeps its sentence with the key in it. `glassbox reveal report.md --legend audit.legend.json` turns the keys back into paths for you. Re-using the legend keeps keys stable across runs; two machines with two legends produce different keys for the same file. Treat the legend like a password file: local, git-ignored, never attached to the same email as the report. Design notes in DESIGN.md §12.
|
|
254
287
|
|
|
255
|
-
Source: [github.com/aldohushi1-stack/glassbox](https://github.com/aldohushi1-stack/glassbox) · MIT © Aldo Hushi
|
|
288
|
+
Source: [github.com/aldohushi1-stack/glassbox](https://github.com/aldohushi1-stack/glassbox) · MIT © [Aldo Hushi](https://aldo.ltd), Adelaide · npm: [glassbox-trace](https://www.npmjs.com/package/glassbox-trace) · [Claude Code cost audit](https://aldo.ltd/audit/)
|
|
256
289
|
|
|
257
290
|
## Security
|
|
258
291
|
|
package/dist/glassbox.html
CHANGED
|
@@ -408,7 +408,7 @@ kbd{font:11px var(--mono);border:1px solid var(--line);border-bottom-width:2px;b
|
|
|
408
408
|
})(typeof globalThis !== 'undefined' ? globalThis : this, function () {
|
|
409
409
|
'use strict';
|
|
410
410
|
|
|
411
|
-
const VERSION = '0.
|
|
411
|
+
const VERSION = '0.11.1';
|
|
412
412
|
|
|
413
413
|
// ---------------------------------------------------------------------------
|
|
414
414
|
// Rate card (USD per million tokens). Prefix-matched against model ids so dated
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "glassbox-trace",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Other tools show what happened in a Claude Code session; Glassbox tells you what went wrong. Timeline, token and cost burn, auto-diagnostics with evidence and advice, session compare, live tail, a fleet view, a secrets scanner for transcripts (fence), a CLAUDE.md compliance report (adhere), a CLI and a Claude Code Stop hook that feeds the findings back to the agent.",
|
|
3
|
+
"version": "0.11.1",
|
|
4
|
+
"description": "Other tools show what happened in a Claude Code session; Glassbox tells you what went wrong. Timeline, token and cost burn, auto-diagnostics with evidence and advice, session compare, live tail, a fleet view, a secrets scanner for transcripts (fence), a CLAUDE.md compliance report (adhere), a memory the next session starts with plus a search over past sessions (recall) and an audit of what loads before the first prompt, a CLI and a Claude Code Stop hook that feeds the findings back to the agent.",
|
|
5
5
|
"main": "src/trace-core.js",
|
|
6
6
|
"scripts": {
|
|
7
7
|
"test": "node --test test/*.test.mjs",
|
|
@@ -34,6 +34,8 @@
|
|
|
34
34
|
"secret-scanning",
|
|
35
35
|
"claude-md",
|
|
36
36
|
"adherence",
|
|
37
|
+
"memory",
|
|
38
|
+
"recall",
|
|
37
39
|
"compare",
|
|
38
40
|
"live-tail",
|
|
39
41
|
"claude",
|
package/src/adhere.mjs
CHANGED
|
@@ -94,6 +94,17 @@ export function parseRules(text, file) {
|
|
|
94
94
|
return { rules, unchecked };
|
|
95
95
|
}
|
|
96
96
|
|
|
97
|
+
// Sessions of a project, newest first: the folder name Claude Code would use (every character that is not a letter
|
|
98
|
+
// or a digit becomes '-', so "my.app_v2" is "my-app-v2"; older builds only replaced separators). Failing that, a
|
|
99
|
+
// folder whose name ends with it — the same project reached through a symlinked parent (/tmp → /private/tmp). Never a
|
|
100
|
+
// substring: /home/aldo/app is not /home/aldo/app-v2, and a folder name can't tell app/v2 from app-v2.
|
|
101
|
+
export function projectSessions(project, home) {
|
|
102
|
+
const enc = [encodeProject(project), String(project).replace(/[^A-Za-z0-9]/g, '-')].map((e) => e.toLowerCase());
|
|
103
|
+
const all = findSessions({ home }), name = (s) => path.basename(s.projectDir).toLowerCase();
|
|
104
|
+
const list = all.filter((s) => enc.includes(name(s)));
|
|
105
|
+
return list.length ? list : all.filter((s) => enc.some((e) => name(s).endsWith(e)));
|
|
106
|
+
}
|
|
107
|
+
|
|
97
108
|
export function findInstructionFiles(project, home) {
|
|
98
109
|
const out = [];
|
|
99
110
|
const add = (p, label) => { if (fs.existsSync(p) && fs.statSync(p).isFile()) out.push({ file: p, label }); };
|
|
@@ -145,7 +156,7 @@ function segmentTools(cmd) {
|
|
|
145
156
|
return segments(cmd).map((seg) => { const toks = seg.split(/\s+/); let t = toks[0].replace(/^.*[\\/]/, '').toLowerCase(); if (/^python3?$/.test(t) && toks[1] === '-m' && toks[2]) t = toks[2].toLowerCase(); if (t === 'pip3') t = 'pip'; return t; });
|
|
146
157
|
}
|
|
147
158
|
const hasSeg = (cmd, re) => segments(cmd).some((seg) => re.test(seg));
|
|
148
|
-
function commitMessage(cmd) {
|
|
159
|
+
export function commitMessage(cmd) {
|
|
149
160
|
cmd = String(cmd);
|
|
150
161
|
const hd = cmd.match(/<<-?\s*['"]?(\w+)['"]?\n([\s\S]*?)\n\1/); if (hd) return hd[2].split('\n')[0].trim();
|
|
151
162
|
const q = cmd.match(/(?:^|\s)(?:-m|--message)(?:=|\s+)(?:"((?:[^"\\]|\\.)*)"|'((?:[^'\\]|\\.)*)'|(\S+))/); if (q) return (q[1] != null ? q[1] : q[2] != null ? q[2] : q[3]).split('\n')[0].trim();
|
|
@@ -165,7 +176,7 @@ const pathHits = (rulePath, filePath, project) => {
|
|
|
165
176
|
const OPTV = String.raw`(?:"[^"]*"|'[^']*'|\S)+`;
|
|
166
177
|
const GIT = String.raw`^git\s+(?:(?:-c\s+${OPTV}|-C\s+${OPTV}|--no-pager|--git-dir=${OPTV}|--work-tree=${OPTV})\s+)*`;
|
|
167
178
|
const git = (sub) => new RegExp(GIT + sub + '\\b');
|
|
168
|
-
const actionRe = { commit: git('commit'), push: git('push'), delete: new RegExp('^(?:rm|rmdir|del|Remove-Item)\\b|' + GIT + 'rm\\b'), install: /^(?:npm|pnpm|yarn|bun|pip3?|uv|poetry|cargo|gem|brew|apt(?:-get)?|choco|winget)\s+(?:install|add|i)\b/, deploy: /^(?:vercel|netlify|fly\s+deploy|wrangler\s+(?:publish|deploy)|gcloud\s+run\s+deploy|serverless\s+deploy|eb\s+deploy)\b|^\S+\s+deploy\b/i, merge: git('merge') };
|
|
179
|
+
export const actionRe = { commit: git('commit'), push: git('push'), delete: new RegExp('^(?:rm|rmdir|del|Remove-Item)\\b|' + GIT + 'rm\\b'), install: /^(?:npm|pnpm|yarn|bun|pip3?|uv|poetry|cargo|gem|brew|apt(?:-get)?|choco|winget)\s+(?:install|add|i)\b/, deploy: /^(?:vercel|netlify|fly\s+deploy|wrangler\s+(?:publish|deploy)|gcloud\s+run\s+deploy|serverless\s+deploy|eb\s+deploy)\b|^\S+\s+deploy\b/i, merge: git('merge') };
|
|
169
180
|
const askedRe = { commit: /\bcommit/i, push: /\bpush/i, delete: /\b(?:delete|remove|rm\b|clean up|get rid)/i, install: /\binstall|\badd (?:the )?(?:package|dependency|dep)/i, create: /\b(?:create|make|add|write|new)\b/i, deploy: /\bdeploy|ship|release/i, merge: /\bmerge/i };
|
|
170
181
|
|
|
171
182
|
// Judge every rule against one session. Returns per-rule { occasions:[{obeyed, session, turn, prompt, detail, at}] }.
|
|
@@ -230,10 +241,7 @@ export function adhere(opts = {}) {
|
|
|
230
241
|
if (!files.length) throw new Error(`adhere: no CLAUDE.md found for ${project}\n Looked for CLAUDE.md, .claude/CLAUDE.md, CLAUDE.local.md there and ~/.claude/CLAUDE.md.\n Point at one with --claude-md FILE, or at the project with --project DIR.`);
|
|
231
242
|
const rules = [], unchecked = [], fileRows = [];
|
|
232
243
|
for (const f of files) { if (!fs.existsSync(f.file)) throw new Error(`adhere: ${f.file} not found`); const r = parseRules(readTextFile(f.file), f.label); rules.push(...r.rules); unchecked.push(...r.unchecked); fileRows.push({ file: f.file, label: f.label, rules: r.rules.length, unchecked: r.unchecked.length }); }
|
|
233
|
-
|
|
234
|
-
const enc = encodeProject(project).toLowerCase();
|
|
235
|
-
let list = findSessions({ home }).filter((s) => path.basename(s.projectDir).toLowerCase() === enc);
|
|
236
|
-
if (!list.length) list = findSessions({ home, project: project.replace(/\\/g, '/') });
|
|
244
|
+
let list = projectSessions(project, home);
|
|
237
245
|
if (opts.since != null) list = list.filter((s) => s.mtime >= opts.since);
|
|
238
246
|
const perRule = rules.map(() => []); const sessions = [];
|
|
239
247
|
for (const s of list) { const j = judgeSession(rules, s, project); j.perRule.forEach((o, i) => perRule[i].push(...o)); sessions.push(j.session); }
|
package/src/claims.mjs
CHANGED
|
@@ -20,11 +20,12 @@ export function userFacingTexts(trace) {
|
|
|
20
20
|
const callsByReq = new Map();
|
|
21
21
|
for (const c of trace.toolCalls) { if (!callsByReq.has(c.requestId)) callsByReq.set(c.requestId, []); callsByReq.get(c.requestId).push(c); }
|
|
22
22
|
trace.requests.forEach((r, i) => {
|
|
23
|
-
for (const b of r.blocks) if (b.type === 'text' && b.text && b.text.trim()) out.push({ request: r, index: i, agent: r.agent, via: 'text', text: b.text });
|
|
23
|
+
for (const b of r.blocks) if (b.type === 'text' && b.text && b.text.trim()) out.push({ request: r, index: i, agent: r.agent, via: 'text', text: b.text, blockIndex: b.index });
|
|
24
24
|
for (const c of callsByReq.get(r.id) || []) {
|
|
25
25
|
const inp = c.input || {};
|
|
26
|
-
|
|
27
|
-
if (c.name === '
|
|
26
|
+
const blockIndex = (r.blocks.find((b) => b.id === c.id) || {}).index;
|
|
27
|
+
if (c.name === 'SendUserMessage' && typeof inp.message === 'string') out.push({ request: r, index: i, agent: r.agent, via: 'SendUserMessage', text: inp.message, call: c, blockIndex });
|
|
28
|
+
if (c.name === 'SendUserFile' && typeof inp.caption === 'string') out.push({ request: r, index: i, agent: r.agent, via: 'SendUserFile', text: inp.caption, call: c, blockIndex });
|
|
28
29
|
}
|
|
29
30
|
});
|
|
30
31
|
return out;
|
|
@@ -46,7 +47,7 @@ const DESCRIBES_RE = /\b(?:it|this|that|the\s+(?:guard|hook|rule|command|viewer|
|
|
|
46
47
|
|
|
47
48
|
const KINDS = [
|
|
48
49
|
['test', /\b(?:tests?|suites?|specs?|e2e|audit|ci|checks?|lint(?:er)?|build|typecheck|type-check|validation|coverage)\b[^.]{0,80}?\b(?:pass(?:es|ed|ing)?|green|succeed(?:s|ed)?|ok|clean|no\s+failures|validates?|valid|passed)\b|\b(?:all|\d[\d,]*)\s*(?:of\s*\d[\d,]*\s*)?(?:tests?\s+)?(?:still\s+)?(?:pass(?:es|ed|ing)?|green)\b|\b\d[\d,]*\s*(?:\/\s*\d[\d,]*\s*)?(?:tests?|specs?)\s*(?:still\s+)?(?:pass|passing|passed|green|ok)\b|\b(?:plugin|manifest|schema|json-ld)\s+validates?\b|\b(?:ci|workflow|run\s+#?\d+|publish\s+#?\d+)\s+(?:is\s+|was\s+)?(?:green|succeeded|passed|completed)\b|\bexit(?:ed| code)?\s*0\b/i],
|
|
49
|
-
['ship', /^(?:both|all|everything|it|they)\s+(?:saved|written|uploaded|committed|pushed|deployed|published|live)\b|\b(?:all|both)\s+\d+\s+(?:saved|uploaded|live)\b|\ball\s+trimmed\s+and\s+live\b|\blive\s+and\s+(?:rendering|serving)\b|(?<!\b(?:the|a|an|its|my|your|this|our|that|one|each|these|those|two|three|four|five|six|last|previously)\s)\b(?:pushed(?!\s+back)|committed|commit(?:ted)?\s+(?:it\s+)?as\b|published|deployed|uploaded|merged|released|republished|shipped|posted)\b(?!\s+(?:blog|post|page|version|package|site|copy|report|article|files?|templates?|branch|folder|build|commits?)\b)|\b(?:is\s+(?:now\s+)?(?:live|released|on\s+npm|on\s+github|up(?!\s*(?:\d|\$|by\b|to\b|from\b|and\s+running\b)))|went\s+live|now\s+live|are\s+live|on\s+npm\b|in\s+production|is\s+on\s+main\b|(?:is|are|now|already|sits?)\s+(?:up\s+)?on\s+the\s+server)\b/i],
|
|
50
|
+
['ship', /^(?:both|all|everything|it|they)\s+(?:saved|written|uploaded|committed|pushed|deployed|published|live)\b|\b(?:all|both)\s+\d+\s+(?:saved|uploaded|live)\b|\ball\s+trimmed\s+and\s+live\b|\blive\s+and\s+(?:rendering|serving)\b|(?<!\b(?:the|a|an|its|my|your|this|our|that|one|each|these|those|two|three|four|five|six|last|previously)\s)\b(?:pushed(?!\s+back)|committed|commit(?:ted)?\s+(?:it\s+)?as\b|published|deployed|uploaded|merged|released|republished|shipped|posted)\b(?!\s+(?:blog|post|page|version|package|site|copy|report|article|files?|templates?|branch|folder|build|commits?)\b)(?!\s+(?:lists?|data|guidance|rules|papers?|research|figures|prices|rates|numbers|sources?|terms|standards?|schedules?|polic(?:y|ies)|ranges)\b(?!\s+(?:to|into|onto|on|in|at)\b))|\b(?:is\s+(?:now\s+)?(?:live|released|on\s+npm|on\s+github|up(?!\s*(?:\d|\$|by\b|to\b|from\b|and\s+running\b)))|went\s+live|now\s+live|are\s+live|on\s+npm\b|in\s+production|is\s+on\s+main\b|(?:is|are|now|already|sits?)\s+(?:up\s+)?on\s+the\s+server)\b/i],
|
|
50
51
|
['state', /\b(?:nothing\s+(?:changed|was\s+changed|else\s+changed|left\s+uncommitted|was\s+left)|didn't\s+(?:disturb|touch|change)|(?:wasn't|weren't|not)\s+touched|untouched|unchanged|as\s+you\s+left\s+it|byte-identical|exactly\s+as\s+(?:you|I|we)\s+left|same\s+\d+\s+(?:staged|files)|no\s+(?:changes|diff|difference))\b/i],
|
|
51
52
|
['verify', /\b(?:all|both)\s+\d+\s+(?:done|ok|checked)\b|\b(?:verified|confirmed|double-checked|checked(?:\s+(?:that|it|the|all|every|against))?|validated|I\s+(?:ran|re-ran|tested|opened|loaded|looked|compared|hashed|checked)|renders\b|rendered\s+(?:fine|correctly|ok|as\s+expected)|loads\s+(?:fine|correctly|ok)|loaded\s+(?:fine|correctly|ok)|works?\s+(?:as\s+expected|fine|correctly|now|end-to-end)|working\s+(?:as\s+expected|fine|correctly|now)|round-?trips?|resolves?(?:d)?|matches?(?:\s+exactly)?|byte-exact|identical|(?:is|was|are|were|looks?|looked)\s+(?:fine|ok|good|correct|right)\b|has\s+(?:a\s+)?working\b|went\s+through\b|rendered(?=\s*[,.;]|\s+(?:fine|correctly|ok|properly|as\s+expected))|(?:took|measured|clocked|was|is|it's)\s+\d[\d.,]*\s*(?:ms|s|secs?|seconds?|minutes?|MB|KB|GB|bytes)\b|all\s+(?:green|\d+\s+done|\d+\s+ok)|\d+\s*\/\s*\d+,?\s*(?:zero|no)\s+errors?|tested\s+(?:it|this|that|locally|end\s+to\s+end)|no\s+errors?|zero\s+errors?|clean\b)\b/i],
|
|
52
53
|
['fix', /(?<!\b(?:the|a|its|my|your|this|our|that|one|each)\s)\bfixed\b(?!\s+(?:cost|price|fee|quote|rate|width|height|size|number|amount|set|point|term|period|version|one|copy|app))|\bfix\s+is\s+in\b|\b(?:resolved|patched|corrected|repaired)\b|\bno\s+longer\s+(?!needed|necessary|required|relevant|used|in\s+use|available|exists?|supported|part|the\s+case)\w+|\bnow\s+(?:works|handles|passes|renders|resolves|loads|decodes|parses|starts)\b|\bbug\s+(?:is\s+)?(?:fixed|gone)\b/i],
|
|
@@ -89,14 +90,14 @@ const numbersIn = (s) => Array.from(new Set((s.match(NUM_RE) || []).map((n) => n
|
|
|
89
90
|
const SHELL = (c) => c.name === 'Bash' || c.name === 'PowerShell' || /device_bash$/.test(c.name || '');
|
|
90
91
|
const cmdOf = (c) => { const i = c.input || {}; return typeof i.command === 'string' ? i.command : typeof i.script === 'string' ? i.script : ''; };
|
|
91
92
|
const inputText = (c) => { try { return JSON.stringify(c.input || {}); } catch (e) { return ''; } };
|
|
92
|
-
const TEST_CMD = /\b(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?(?:test|check|e2e|audit|lint|build|typecheck)\b|\bnode\s+--test\b|\bnode\s+(?:[\w./-]*\/)?test\/[\w.-]+\.m?js\b|\bpytest\b|\bpython3?\s+-m\s+(?:pytest|unittest)\b|\bcargo\s+test\b|\bgo\s+test\b|\bvitest\b|\bjest\b|\bmocha\b|\bplaywright\s+test\b|\btsc\b|\beslint\b|\bruff\b|\bmake\s+(?:test|check)\b|\bclaude\s+plugin\s+validate\b|\bdotnet\s+test\b|\bphpunit\b|\brspec\b|\bphp\s+-l\b|\bnode\s+(?:--check|-c)\b|\bpython3?\s+-m\s+py_compile\b|\bflake8\b|\bpylint\b|\bmypy\b|\bshellcheck\b|\bprettier\s+--check\b|\bbiome\s+(?:check|lint)\b|\bpython3?\s+(?:[\w./-]*\/)?(?:test_[\w-]+|[\w-]+_test|tests?)\.py\b|\bnode\s+(?:[\w./-]*\/)?[\w-]*test[\w-]*\.[cm]?js\b/i;
|
|
93
|
+
export const TEST_CMD = /\b(?:npm|pnpm|yarn|bun)\s+(?:run\s+)?(?:test|check|e2e|audit|lint|build|typecheck)\b|\bnode\s+--test\b|\bnode\s+(?:[\w./-]*\/)?test\/[\w.-]+\.m?js\b|\bpytest\b|\bpython3?\s+-m\s+(?:pytest|unittest)\b|\bcargo\s+test\b|\bgo\s+test\b|\bvitest\b|\bjest\b|\bmocha\b|\bplaywright\s+test\b|\btsc\b|\beslint\b|\bruff\b|\bmake\s+(?:test|check)\b|\bclaude\s+plugin\s+validate\b|\bdotnet\s+test\b|\bphpunit\b|\brspec\b|\bphp\s+-l\b|\bnode\s+(?:--check|-c)\b|\bpython3?\s+-m\s+py_compile\b|\bflake8\b|\bpylint\b|\bmypy\b|\bshellcheck\b|\bprettier\s+--check\b|\bbiome\s+(?:check|lint)\b|\bpython3?\s+(?:[\w./-]*\/)?(?:test_[\w-]+|[\w-]+_test|tests?)\.py\b|\bnode\s+(?:[\w./-]*\/)?[\w-]*test[\w-]*\.[cm]?js\b/i;
|
|
93
94
|
const CI_RESULT = /completed[:\s]+success|conclusion["']?\s*[:=]\s*["']?success|completed\s+successfully|\bci\s+#?\d+[^.\n]{0,40}\b(?:green|success)|"status":\s*"completed"[^}]{0,80}"conclusion":\s*"success"|publish\s+run[^\n]{0,40}success/i;
|
|
94
95
|
const CI_FAIL = /conclusion["']?\s*[:=]\s*["']?(?:failure|cancelled)|completed[:\s]+failure|\bfailed\b[^\n]{0,20}\bci\b/i;
|
|
95
|
-
const FAIL_MARK = /\bnot ok\b|# fail [1-9]|\b[1-9]\d* fail(?:ed|ing|ures?)?\b|\bFAIL(?:ED)?\b|AssertionError|Error:|\bTypeError|\bReferenceError|\bSyntaxError|Tests:\s+\d+\s+failed|✗|✖|\bexit(?:ed|\s*code)?\s*[=:]?\s*[1-9]\d*\b|Cannot find module|MODULE_NOT_FOUND
|
|
96
|
+
const FAIL_MARK = /\bnot ok\b|# fail [1-9]|\b[1-9]\d* fail(?:ed|ing|ures?)?\b|\bFAIL(?:ED)?\b|AssertionError|Error:|\bTypeError|\bReferenceError|\bSyntaxError|Tests:\s+\d+\s+failed|✗|✖|\bexit(?:ed|\s*code)?\s*[=:]?\s*[1-9]\d*\b|Cannot find module|MODULE_NOT_FOUND|\bERROR\s+(?:collecting\b|[\w./-]+\.py\b)|Interrupted:\s*\d+\s+errors?/;
|
|
96
97
|
// A test-like run's identity: the runner and file it names, so a failing e2e run does not contradict "58 of 58 pass"
|
|
97
98
|
// said about the unit run beside it.
|
|
98
99
|
const testKey = (c) => { const m = cmdOf(c).match(TEST_CMD); return m ? m[0].toLowerCase().replace(/\s+/g, ' ') : (c.name + ':' + (c.resultText || '').slice(0, 40)); };
|
|
99
|
-
function runFailed(c) {
|
|
100
|
+
export function runFailed(c) {
|
|
100
101
|
const r = c.resultText || ''; const k = testCounts(r);
|
|
101
102
|
return !!(c.isError || (k ? k.fail > 0 : FAIL_MARK.test(r)) || (CI_FAIL.test(r) && !CI_RESULT.test(r)));
|
|
102
103
|
}
|
|
@@ -114,7 +115,7 @@ const SOURCE_FILE = /(?:^|[\\/])(?:src|lib|test|tests|spec|specs|app|pkg|cmd|int
|
|
|
114
115
|
|
|
115
116
|
// A result that reads like a test run whatever the command was: a summary line, PASS/FAIL rows, a lint's verdict.
|
|
116
117
|
const looksLikeRun = (r) => { r = String(r || ''); return !!testCounts(r) || /audit passed|browser test passed|Validation passed|No syntax errors detected|^\s*(?:PASS|FAIL)\b.*\n\s*(?:PASS|FAIL)\b/m.test(r); };
|
|
117
|
-
function testCounts(text) {
|
|
118
|
+
export function testCounts(text) {
|
|
118
119
|
const t = String(text || '');
|
|
119
120
|
let m;
|
|
120
121
|
if ((m = t.match(/# pass (\d+)[\s\S]*?# fail (\d+)/))) return { pass: +m[1], fail: +m[2], total: +m[1] + +m[2] };
|
|
@@ -124,7 +125,33 @@ function testCounts(text) {
|
|
|
124
125
|
if ((m = t.match(/(\d+) passing(?:\s*\n\s*(\d+) failing)?/))) return { pass: +m[1], fail: +(m[2] || 0), total: +m[1] + +(m[2] || 0) };
|
|
125
126
|
return null;
|
|
126
127
|
}
|
|
127
|
-
|
|
128
|
+
// "175 of Bob's fast tests pass" names the count too: up to three words between "of" and "tests".
|
|
129
|
+
const claimedCount = (s) => { const m = s.match(/\b(?:all\s+)?(\d[\d,]*)\s*(?:of\s*\d[\d,]*\s*)?(?:tests?|specs?)?\s*(?:still\s+)?(?:pass|green)|\b(\d[\d,]*)\s*\/\s*(\d[\d,]*)|\b(\d[\d,]*)\s+of\s+(?:[\w'’-]+\s+){1,3}(?:tests?|specs?)\s+(?:still\s+)?(?:pass|green)/i); return m ? +(m[1] || m[3] || m[2] || m[4]).replace(/,/g, '') : null; };
|
|
130
|
+
// A sentence that speaks for the whole suite: with any failure in the run it is wrong, whatever else it says.
|
|
131
|
+
const WHOLE_SUITE_RE = /\b(?:all|every\s+(?:test|spec|one)|fully|entire(?:ly)?|whole\s+suite|everything|nothing\s+fail\w*|no\s+fail\w*|zero\s+fail\w*|none\s+(?:of\s+them\s+)?fail\w*|not\s+(?:a\s+single|one)\b|without\s+(?:a\s+)?fail\w*|green|(?:is|are|ran|runs|came\s+back)\s+clean)\b/i;
|
|
132
|
+
const SMALL = ['zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten', 'eleven', 'twelve'];
|
|
133
|
+
// Does the text name exactly `n` failures? "The 3 failures are…", "three failing", "fails: 3".
|
|
134
|
+
// The count may carry thousands separators ("1,200"); only test words may sit between it and the failure word, and
|
|
135
|
+
// a sentence that says the failures are fixed or gone is not owning up to them.
|
|
136
|
+
const OWNED_GONE_RE = /\b(?:fixed|gone|resolved|cleared|no\s+longer|now\s+pass\w*|went\s+away)\b/i;
|
|
137
|
+
function namesFailures(text, n) {
|
|
138
|
+
const forms = [String(n), n.toLocaleString('en-US')].concat(SMALL[n] ? [SMALL[n]] : []);
|
|
139
|
+
const num = `(?:${Array.from(new Set(forms)).map((f) => f.replace(/,/g, ',?')).join('|')})`;
|
|
140
|
+
const re = new RegExp(`(?<!\\b(?:not|no|zero|none)\\s)\\b${num}\\s+(?:(?:other|remaining|older|old|existing|known|pre-existing|flaky|unrelated|tests?|specs?|cases?)\\s+){0,2}(?:fail(?:s|ed|ing|ures?)?)\\b|\\bfail(?:s|ed|ing|ures?)?\\s*[:=]\\s*${num}\\b`, 'i');
|
|
141
|
+
return sentences(text).some((x) => re.test(x) && !OWNED_GONE_RE.test(x));
|
|
142
|
+
}
|
|
143
|
+
// A run with failures, and a sentence that gives its pass count. Returns null when the rule does not apply, true when
|
|
144
|
+
// the failures are named in the sentence or the two after it in the same message, false when they are left out.
|
|
145
|
+
function failuresDisclosed(claim, run, want) {
|
|
146
|
+
const k = testCounts(run.resultText);
|
|
147
|
+
if (!k || !k.fail || want == null || WHOLE_SUITE_RE.test(claim.text)) return null;
|
|
148
|
+
// "40/43 tests pass" names the total as well as the pass count.
|
|
149
|
+
const passNamed = k.pass === want || (k.total === want && new RegExp(`\\b${k.pass}\\s*(?:/|of)\\s*${k.total}\\b`).test(claim.text.replace(/(\d),(\d{3})/g, '$1$2')));
|
|
150
|
+
if (!passNamed) return null;
|
|
151
|
+
const msg = claim._item ? sentences(claim._item.text) : [claim.text];
|
|
152
|
+
const i = msg.indexOf(claim.text);
|
|
153
|
+
return namesFailures((i >= 0 ? msg.slice(i, i + 3) : [claim.text]).join(' '), k.fail);
|
|
154
|
+
}
|
|
128
155
|
|
|
129
156
|
// The window: this agent's calls in the same turn before the sentence (the last `cap` of them), plus the rest
|
|
130
157
|
// of the turn after it (a claim sometimes precedes its own check). A claim that opens a turn refers to the
|
|
@@ -135,8 +162,15 @@ function windowFor(trace, item, byReq, cap = 80) {
|
|
|
135
162
|
let opensTurn = true;
|
|
136
163
|
for (let i = 0; i < item.index; i++) { const r = reqs[i]; if (r.agent === item.agent && r.turnIndex === item.request.turnIndex) { opensTurn = false; before.push(...(byReq.get(r.id) || [])); } }
|
|
137
164
|
if (opensTurn) { const prev = []; for (let i = item.index - 1; i >= 0 && prev.length < 20; i--) { const r = reqs[i]; if (r.agent !== item.agent || r.turnIndex === item.request.turnIndex) continue; prev.unshift(...(byReq.get(r.id) || [])); } before.unshift(...prev.slice(-20)); }
|
|
138
|
-
before
|
|
139
|
-
|
|
165
|
+
// The sentence's own response: calls before its block came before it; calls after it (the usual "Done. Running the
|
|
166
|
+
// tests to confirm." then a tool call) come after it.
|
|
167
|
+
const own = byReq.get(item.request.id) || [];
|
|
168
|
+
const at = new Map((item.request.blocks || []).map((b) => [b.id, b.index]));
|
|
169
|
+
const later = (c) => item.blockIndex != null && at.has(c.id) && at.get(c.id) > item.blockIndex;
|
|
170
|
+
before.push(...own.filter((c) => !later(c)));
|
|
171
|
+
after.push(...own.filter(later));
|
|
172
|
+
const ownAfter = after.length;
|
|
173
|
+
for (let i = item.index + 1; i < reqs.length && after.length - ownAfter < 12; i++) { const r = reqs[i]; if (r.agent !== item.agent || r.turnIndex !== item.request.turnIndex) break; after.push(...(byReq.get(r.id) || [])); }
|
|
140
174
|
return { before: before.slice(-cap), after };
|
|
141
175
|
}
|
|
142
176
|
// Calls that can never be a receipt: the agent talking to the human (the sentence itself often travels in one),
|
|
@@ -153,7 +187,21 @@ function judgeOne(trace, claim, win) {
|
|
|
153
187
|
const s = claim.text;
|
|
154
188
|
const ev = (calls, note, verdict) => ({ verdict, evidence: calls.map((c) => c.id), note });
|
|
155
189
|
if (claim.kind === 'test') {
|
|
156
|
-
|
|
190
|
+
// A run is judged as of the sentence: runs before it when there are any; the rest of the turn only when the claim
|
|
191
|
+
// came before its own check. A later run belongs to later work (new tests written red, say). A collect-only
|
|
192
|
+
// listing (`pytest --co`) runs nothing.
|
|
193
|
+
// A run started in the background ("started") has no result yet.
|
|
194
|
+
const background = (c) => (c.input && c.input.run_in_background === true) || /(?:^|[^&])&\s*\)?\s*$/.test(cmdOf(c).replace(/\s*#[^\n'"]*$/, '').trim());
|
|
195
|
+
// A collect-only step (`pytest --co`) lists tests without running them; a real run beside it still counts.
|
|
196
|
+
const runCmd = (c) => cmdOf(c).replace(/[^;&|\n]*--co(?:llect-only)?\b[^;&|\n]*/g, ' ');
|
|
197
|
+
const isRun = (c) => !background(c) && ((SHELL(c) && (TEST_CMD.test(runCmd(c)) || (looksLikeRun(c.resultText) && runCmd(c) === cmdOf(c)))) || ((SHELL(c) || c.name === 'WebFetch' || /javascript|browser_batch|get_page_text|read_page/.test(c.name) || c.name === 'Read') && CI_RESULT.test(c.resultText || '')));
|
|
198
|
+
// After the sentence, only the check that confirms it counts: runs before the next edit. Runs after an edit
|
|
199
|
+
// belong to the next piece of work.
|
|
200
|
+
// What ends the confirming check: a write to code or tests (by extension: a changelog does not count), or any shell
|
|
201
|
+
// edit, even one that runs the tests in the same command (a heredoc that writes a test file and runs it is new work).
|
|
202
|
+
const codeEdit = (c) => (WRITE_TOOLS.has(c.name) && /\.(?:mjs|cjs|js|jsx|ts|tsx|py|rb|go|rs|java|kt|cs|php|c|cc|cpp|h|hpp|swift|sh|ps1)$/i.test(String((c.input && (c.input.file_path || c.input.notebook_path)) || ''))) || (SHELL(c) && SHELL_EDIT_RE.test(cmdOf(c)));
|
|
203
|
+
const confirming = []; for (const c of win.after) { if (codeEdit(c)) break; confirming.push(c); }
|
|
204
|
+
const runs = win.before.concat(confirming).filter(usable).filter(isRun);
|
|
157
205
|
if (!runs.length) {
|
|
158
206
|
// A count quoted from a file or note the agent read is sourced, not measured — listed, not flagged.
|
|
159
207
|
const want = claimedCount(s);
|
|
@@ -179,7 +227,14 @@ function judgeOne(trace, claim, win) {
|
|
|
179
227
|
if (match.length) { const m = match[match.length - 1]; if (!pool.some((c) => c.start != null && m.start != null && c.start > m.start && testKey(c) === testKey(m) && runFailed(c))) last = m; }
|
|
180
228
|
}
|
|
181
229
|
const ci = runs.filter((c) => CI_RESULT.test(c.resultText || ''));
|
|
182
|
-
|
|
230
|
+
let disclosed = false;
|
|
231
|
+
if (runFailed(last)) {
|
|
232
|
+
// "1,043 tests pass. The 3 failures are the same ones as before." is true; "1,043 tests pass." alone is not.
|
|
233
|
+
const d = failuresDisclosed(claim, last, want);
|
|
234
|
+
if (d === null) return ev([last], 'the latest run in the window did not pass', 'contradicted');
|
|
235
|
+
if (!d) { const k = testCounts(last.resultText); return ev([last], `${k.pass} passed as the sentence says, but ${k.fail} failed and the message does not say so`, 'contradicted'); }
|
|
236
|
+
disclosed = true;
|
|
237
|
+
}
|
|
183
238
|
const counts = testCounts(last.resultText);
|
|
184
239
|
if (want != null && counts && counts.pass !== want && counts.total !== want) {
|
|
185
240
|
if (ci.length && ci[ci.length - 1] !== last) return ev([last, ci[ci.length - 1]], `the run says ${counts.pass} passed, the sentence says ${want}; CI reports success`, 'verified');
|
|
@@ -187,7 +242,7 @@ function judgeOne(trace, claim, win) {
|
|
|
187
242
|
}
|
|
188
243
|
const lw = lastWriteAt(win.before.filter((c) => c.start != null && last.start != null && c.start > last.start));
|
|
189
244
|
if (lw > -Infinity && !ci.some((c) => c.start != null && c.start > lw)) return ev([last], 'the run predates a later edit to source or tests, and nothing re-ran', 'stale');
|
|
190
|
-
return ev([last], counts ? `${counts.pass} passed, ${counts.fail} failed` : (CI_RESULT.test(last.resultText || '') ? 'CI success' : 'run ok'), 'verified');
|
|
245
|
+
return ev([last], counts ? `${counts.pass} passed, ${counts.fail} failed${disclosed ? '; the failures are named in the same message' : ''}` : (CI_RESULT.test(last.resultText || '') ? 'CI success' : 'run ok'), 'verified');
|
|
191
246
|
}
|
|
192
247
|
if (claim.kind === 'ship') {
|
|
193
248
|
const shas = (s.match(SHA_RE) || []).filter((h) => /\d/.test(h) && /[a-f]/.test(h));
|
|
@@ -210,9 +265,20 @@ function judgeOne(trace, claim, win) {
|
|
|
210
265
|
if (src.length) return ev(src.slice(-1), 'the version comes from something the agent read, not from a ship step', 'sourced');
|
|
211
266
|
}
|
|
212
267
|
if (!hits.length) return ev([], shas.length ? `no result carries ${shas[0]}` : versions.length ? `no result carries ${versions[0]}` : 'no push, publish, upload or commit result in the window', 'unverified');
|
|
213
|
-
|
|
268
|
+
// `"rejected":[]` is a device write's way of saying nothing was rejected.
|
|
269
|
+
// A device write with nothing in "written" saved nothing.
|
|
270
|
+
const ok = hits.filter((c) => !c.isError && !/"written"\s*:\s*\[\s*\]/.test(c.resultText || '') && !/\b(?:fatal|rejected|denied|Authentication failed|ERR!)\b/i.test(String(c.resultText || '').replace(/"rejected"\s*:\s*\[\s*\]/g, '')));
|
|
214
271
|
// "Committed as c9b4747, but the push was refused": the failure is in the sentence, so it is not a contradiction.
|
|
215
|
-
if (!ok.length
|
|
272
|
+
if (!ok.length && SHIP_DECLINED_RE.test(s)) return ev(hits.slice(-1), 'the ship step failed, as the sentence says', 'verified');
|
|
273
|
+
if (!ok.length) {
|
|
274
|
+
// A failed click (upload, push) the agent tried again with the same tool: the retry's result may not say what
|
|
275
|
+
// happened, so the failure is no longer the last word. No receipt either way.
|
|
276
|
+
const failed = hits[hits.length - 1];
|
|
277
|
+
const shipVerb = (c) => (cmdOf(c).match(SHIP_CMD) || [''])[0].toLowerCase();
|
|
278
|
+
const retry = win.before.filter(usable).filter((c) => c.name === failed.name && c !== failed && !c.isError && c.start != null && failed.start != null && c.start > failed.start && (!SHELL(c) || (shipVerb(c) && shipVerb(c) === shipVerb(failed))));
|
|
279
|
+
if (retry.length) return ev([failed, retry[retry.length - 1]], 'the ship step failed and was retried with the same tool; the retry\'s result does not show the ship', 'unverified');
|
|
280
|
+
return ev(hits.slice(-1), 'the ship step in the window failed', 'contradicted');
|
|
281
|
+
}
|
|
216
282
|
return ev(ok.slice(-1), 'ship result in the window', 'verified');
|
|
217
283
|
}
|
|
218
284
|
if (claim.kind === 'write') {
|
package/src/cli.mjs
CHANGED
|
@@ -9,6 +9,8 @@ import { fileURLToPath } from 'node:url';
|
|
|
9
9
|
import { readTextFile } from './textfile.mjs';
|
|
10
10
|
import { guardResponse } from './guard.mjs';
|
|
11
11
|
import { judgeClaims, claimFindings, claimsText, claimsMarkdown } from './claims.mjs';
|
|
12
|
+
import { NOTES_FILE, ensureMemoryDir, recordSession, memoryNote, noteTold, maskedTrace, recall, recallText, recallMarkdown, memoryReport, memoryText, memoryMarkdown, forget, clearMemory } from './memory.mjs';
|
|
13
|
+
import { maskSecrets } from './fence.mjs';
|
|
12
14
|
const require = createRequire(import.meta.url);
|
|
13
15
|
const core = require('./trace-core.js');
|
|
14
16
|
// fileURLToPath, not URL.pathname: on Windows the latter gives "/C:/…" which path.resolve turns into "\C:\…".
|
|
@@ -228,23 +230,29 @@ export function sessionFromTranscriptPath(p) {
|
|
|
228
230
|
return { id: path.basename(file).replace(/\.jsonl$/, ''), file, project: null, projectDir: path.dirname(file), size: fs.existsSync(file) ? fs.statSync(file).size : 0, mtime: fs.existsSync(file) ? fs.statSync(file).mtimeMs : 0 };
|
|
229
231
|
}
|
|
230
232
|
|
|
231
|
-
// Persisted feedback (--context): the Stop hook leaves <project>/.glassbox/last-session.md behind and
|
|
232
|
-
// the SessionStart hook hands
|
|
233
|
-
|
|
233
|
+
// Persisted feedback (--context): the Stop hook leaves <project>/.glassbox/last-session.md behind and folds the
|
|
234
|
+
// session into <project>/.glassbox/memory.json (src/memory.mjs); the SessionStart hook hands the next session
|
|
235
|
+
// the lessons still recurring, a line per recent session and the last session's notes, so feedback reaches the
|
|
236
|
+
// session that can use it. Anything a fence detector recognises as a secret is masked before it is written.
|
|
237
|
+
export { NOTES_FILE };
|
|
234
238
|
const NOTES_MAX = 6000;
|
|
235
239
|
function notesPath(cwd) { return cwd && fs.existsSync(cwd) && fs.statSync(cwd).isDirectory() ? path.join(cwd, NOTES_FILE) : null; }
|
|
236
240
|
function writeNotes(cwd, text) {
|
|
237
241
|
const file = notesPath(cwd); if (!file) return null;
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
fs.writeFileSync(file, text.length > NOTES_MAX ? text.slice(0, NOTES_MAX) + '\n\n… (truncated; run `glassbox check --format md` for the full report)\n' : text);
|
|
242
|
+
ensureMemoryDir(cwd);
|
|
243
|
+
text = maskSecrets(text);
|
|
244
|
+
fs.writeFileSync(file, text.length > NOTES_MAX ? text.slice(0, NOTES_MAX) + '\n\n… (truncated; run `glassbox check --format md` for the full report)\n' : text, { mode: 0o600 });
|
|
245
|
+
try { fs.chmodSync(file, 0o600); } catch (e) { } // it quotes the session; a file left by 0.10 was 0644
|
|
241
246
|
return file;
|
|
242
247
|
}
|
|
243
248
|
export function sessionStartResponse(input) {
|
|
244
|
-
const
|
|
245
|
-
if (!
|
|
246
|
-
const
|
|
247
|
-
return { suppressOutput: true
|
|
249
|
+
const cwd = input && input.cwd;
|
|
250
|
+
if (!notesPath(cwd) || (input.source && !['startup', 'clear', 'compact'].includes(input.source))) return { suppressOutput: true };
|
|
251
|
+
const note = memoryNote(cwd, { exclude: input.session_id });
|
|
252
|
+
if (!note) return { suppressOutput: true };
|
|
253
|
+
// Told counts at the start only: compaction hands the same note over again mid-session. The count is a nicety; the note is the job.
|
|
254
|
+
if (input.source !== 'compact') { try { noteTold(cwd, input.session_id, note.told); } catch (e) { } }
|
|
255
|
+
return { suppressOutput: true, hookSpecificOutput: { hookEventName: 'SessionStart', additionalContext: note.text } };
|
|
248
256
|
}
|
|
249
257
|
|
|
250
258
|
// Findings already handed back in this session, keyed by rule + title, kept in the temp dir.
|
|
@@ -267,7 +275,7 @@ export function hookResponse(input, opts = {}) {
|
|
|
267
275
|
// Second Stop after a feedback block: record what the agent said it would do differently, then let it stop.
|
|
268
276
|
if (opts.context && input && input.hook_event_name === 'Stop' && input.stop_hook_active) {
|
|
269
277
|
const file = notesPath(input.cwd);
|
|
270
|
-
if (file && fs.existsSync(file) && input.last_assistant_message && fs.readFileSync(file, 'utf8').includes(`session ${String(input.session_id || '').slice(0, 8)}`)) fs.appendFileSync(file, `\n## What the agent said it would do differently\n\n${String(input.last_assistant_message).trim().slice(0, 1500)}\n`);
|
|
278
|
+
if (file && fs.existsSync(file) && input.last_assistant_message && fs.readFileSync(file, 'utf8').includes(`session ${String(input.session_id || '').slice(0, 8)}`)) fs.appendFileSync(file, `\n## What the agent said it would do differently\n\n${maskSecrets(String(input.last_assistant_message).trim()).slice(0, 1500)}\n`);
|
|
271
279
|
return { suppressOutput: true };
|
|
272
280
|
}
|
|
273
281
|
if (!input || !input.transcript_path || !fs.existsSync(input.transcript_path)) return { systemMessage: 'Glassbox: no transcript_path in hook input.', suppressOutput: true };
|
|
@@ -286,9 +294,11 @@ export function hookResponse(input, opts = {}) {
|
|
|
286
294
|
const file = notesPath(input.cwd);
|
|
287
295
|
if (top.length) {
|
|
288
296
|
const when = new Date().toISOString().slice(0, 16).replace('T', ' ');
|
|
289
|
-
const body = core.reportMarkdown(res.trace, top.slice(0, 8), null, { maxEvidence: 2, instruction: false, title: 'last session' }).replace(/^# [^\n]*\n\n/, '').replace(/\n## Tools[\s\S]*?(?=\n_Generated)/, '\n').replace(/\n_Generated[^\n]*\n?$/, '');
|
|
297
|
+
const body = core.reportMarkdown(maskedTrace(res.trace), top.slice(0, 8).map((f) => Object.assign({}, f, { title: maskSecrets(f.title), detail: maskSecrets(f.detail) })), null, { maxEvidence: 2, instruction: false, title: 'last session' }).replace(/^# [^\n]*\n\n/, '').replace(/\n## Tools[\s\S]*?(?=\n_Generated)/, '\n').replace(/\n_Generated[^\n]*\n?$/, '');
|
|
290
298
|
writeNotes(input.cwd, `# Glassbox: last session in this project\n\n${when} UTC · session ${String(input.session_id || s.id).slice(0, 8)} · ${head.replace(/^Glassbox · /, '')}\n\n${body.trim()}\n`);
|
|
291
299
|
} else if (file && fs.existsSync(file)) fs.unlinkSync(file); // a clean session: don't keep old advice around
|
|
300
|
+
// The memory ledger: this session's digest and lessons. A failure here must not cost the summary.
|
|
301
|
+
try { recordSession(input.cwd, res, { sessionId: input.session_id || s.id, failOn }); } catch (e) { out.systemMessage += `\nGlassbox memory: ${e.message}`; }
|
|
292
302
|
}
|
|
293
303
|
// Feedback loop: only on Stop, never while continuing from our own block (stop_hook_active), and each
|
|
294
304
|
// finding only once per session. stop_hook_active resets every turn, so without the delivered set a
|
|
@@ -351,7 +361,7 @@ export function parseArgs(argv) {
|
|
|
351
361
|
const args = { _: [], flags: {} };
|
|
352
362
|
for (let i = 0; i < argv.length; i++) {
|
|
353
363
|
const a = argv[i];
|
|
354
|
-
if (a.startsWith('--')) { const [k, v] = a.slice(2).split('='); if (v !== undefined) args.flags[k] = v; else if (i + 1 < argv.length && !argv[i + 1].startsWith('-') && ['last', 'grep', 'project', 'out', 'fail-on', 'home', 'command', 'events', 'format', 'label-a', 'label-b', 'port', 'rates', 'since', 'legend', 'claude-md', 'fail-under', 'key'].includes(k)) args.flags[k] = argv[++i]; else args.flags[k] = true; }
|
|
364
|
+
if (a.startsWith('--')) { const [k, v] = a.slice(2).split('='); if (v !== undefined) args.flags[k] = v; else if (i + 1 < argv.length && !argv[i + 1].startsWith('-') && ['last', 'grep', 'project', 'out', 'fail-on', 'home', 'command', 'events', 'format', 'label-a', 'label-b', 'port', 'rates', 'since', 'legend', 'claude-md', 'fail-under', 'key', 'limit'].includes(k)) args.flags[k] = argv[++i]; else args.flags[k] = true; }
|
|
355
365
|
else args._.push(a);
|
|
356
366
|
}
|
|
357
367
|
return args;
|
|
@@ -382,7 +392,8 @@ export const HELP = `glassbox — other tools show you what happened in a Claude
|
|
|
382
392
|
glassbox hook install [--feedback] [--context] [--guard] [--fail-on warn]
|
|
383
393
|
add a Claude Code Stop hook so every session ends with a Glassbox summary;
|
|
384
394
|
--feedback also hands the findings back to the agent once, so it can learn from them;
|
|
385
|
-
--context keeps
|
|
395
|
+
--context keeps a memory in <project>/.glassbox/ (lessons that keep coming back, a digest
|
|
396
|
+
per session, the last session's notes) and gives the next session there what it needs;
|
|
386
397
|
--guard blocks a tool call that already failed twice in a row, unchanged, and says why
|
|
387
398
|
glassbox hook uninstall remove it (a .glassbox-backup of settings.json is kept)
|
|
388
399
|
glassbox collect DIR [--since 30d] [--format text|md|json] [--out FILE] [--top N]
|
|
@@ -403,6 +414,16 @@ export const HELP = `glassbox — other tools show you what happened in a Claude
|
|
|
403
414
|
credential-file reads — with a masked preview and a fingerprint, never the value.
|
|
404
415
|
no target = every session under the home; exit 1 when anything at/above --fail-on was found
|
|
405
416
|
--shred overwrites each value in place with [FENCED:<rule>:<fingerprint>] (no backup)
|
|
417
|
+
glassbox recall [WORDS…] [--project DIR] [--since 30d] [--limit N] [--format text|md|json] [--out FILE]
|
|
418
|
+
search this project's past sessions: what was asked, written, run, committed and said,
|
|
419
|
+
from the transcripts still on disk and .glassbox/memory.json; no words = the latest ones
|
|
420
|
+
glassbox memory [--project DIR] [--format text|md|json]
|
|
421
|
+
what the next session in this project will be told (hook --context) and the lessons behind it
|
|
422
|
+
glassbox memory forget KEY|RULE · glassbox memory clear
|
|
423
|
+
drop a lesson (it starts over if it fires again) · delete the project's memory
|
|
424
|
+
glassbox memory audit [--project DIR] [--format text|md|json] [--out FILE] [--fail-on error|warn|info|none]
|
|
425
|
+
everything loaded before the first prompt (CLAUDE.md files, rules, @imports, auto memory,
|
|
426
|
+
Glassbox's own note) with its token cost, dead imports and paths, duplicates and conflicting rules
|
|
406
427
|
glassbox hook (what Claude Code runs: reads the hook JSON on stdin, replies on stdout)
|
|
407
428
|
|
|
408
429
|
Options --home DIR use DIR instead of ~/.claude (or set GLASSBOX_HOME)
|
|
@@ -602,6 +623,41 @@ export async function main(argv, io = {}) {
|
|
|
602
623
|
else out(text.replace(/\n$/, ''));
|
|
603
624
|
return failUnder != null && rep.summary.rate != null && rep.summary.rate * 100 < failUnder ? 1 : 0;
|
|
604
625
|
}
|
|
626
|
+
const writeOut = (text, summary) => { if (args.flags.out) { const f = path.resolve(args.flags.out); fs.writeFileSync(f, text); out(`${f} (${summary})`); } else out(text.replace(/\n$/, '')); };
|
|
627
|
+
if (cmd === 'recall') {
|
|
628
|
+
// Search a project's past sessions. Each argument is a word, or a phrase when the shell kept it quoted.
|
|
629
|
+
const since = args.flags.since ? Date.now() - parseSince(args.flags.since) : null;
|
|
630
|
+
let limit = 10; if (args.flags.limit !== undefined) { limit = Number(args.flags.limit); if (!Number.isInteger(limit) || limit < 1) throw new Error(`--limit must be a whole number above 0 (got "${args.flags.limit}")`); }
|
|
631
|
+
const query = args._.slice(1).map((a) => /\s/.test(a) ? `"${a.replace(/"/g, '')}"` : a).join(' ');
|
|
632
|
+
const fmtOut = outputFormat(args.flags);
|
|
633
|
+
const rep = recall({ home, project: args.flags.project, query, since, limit });
|
|
634
|
+
writeOut(fmtOut === 'json' ? JSON.stringify(rep, null, 2) : fmtOut === 'md' ? recallMarkdown(rep) : recallText(rep), `${rep.total} of ${rep.searched.sessions} sessions`);
|
|
635
|
+
return 0;
|
|
636
|
+
}
|
|
637
|
+
if (cmd === 'memory') {
|
|
638
|
+
const sub = args._[1] || 'show';
|
|
639
|
+
const project = path.resolve(args.flags.project || process.cwd());
|
|
640
|
+
const fmtOut = outputFormat(args.flags);
|
|
641
|
+
if (sub === 'audit') {
|
|
642
|
+
const { auditMemory, auditText, auditMarkdown, auditFailed } = await import('./memory-audit.mjs');
|
|
643
|
+
const failOn = String(args.flags['fail-on'] || 'error');
|
|
644
|
+
const rep = auditMemory({ home, project });
|
|
645
|
+
const failed = auditFailed(rep, failOn);
|
|
646
|
+
writeOut(fmtOut === 'json' ? JSON.stringify(rep, null, 2) : fmtOut === 'md' ? auditMarkdown(rep) : auditText(rep), `${rep.summary.startupFiles} files at startup, ≈ ${core.fmtInt(rep.summary.startupTokens)} tokens, ${rep.findings.length} findings`);
|
|
647
|
+
return failed ? 1 : 0;
|
|
648
|
+
}
|
|
649
|
+
if (sub === 'forget') {
|
|
650
|
+
const what = args._.slice(2).join(' ');
|
|
651
|
+
const keys = forget(project, what);
|
|
652
|
+
out(keys.length ? `Forgot ${keys.length} lesson${keys.length === 1 ? '' : 's'} in ${project}:\n ${keys.join('\n ')}` : `No lesson matches "${what}" in ${project} — glassbox memory lists them`);
|
|
653
|
+
return keys.length ? 0 : 1;
|
|
654
|
+
}
|
|
655
|
+
if (sub === 'clear') { const removed = clearMemory(project); out(removed.length ? `Removed:\n ${removed.join('\n ')}` : `No Glassbox memory in ${project}`); return 0; }
|
|
656
|
+
if (sub !== 'show') throw new Error(`memory: unknown subcommand "${sub}" (show, forget, clear, audit)`);
|
|
657
|
+
const rep = memoryReport({ project });
|
|
658
|
+
writeOut(fmtOut === 'json' ? JSON.stringify(rep, null, 2) : fmtOut === 'md' ? memoryMarkdown(rep) : memoryText(rep), `${rep.sessions} sessions, ${rep.summary.active} active lessons`);
|
|
659
|
+
return 0;
|
|
660
|
+
}
|
|
605
661
|
err('Unknown command: ' + cmd + '\n'); out(HELP); return 2;
|
|
606
662
|
} catch (e) { err('glassbox: ' + e.message); return 2; }
|
|
607
663
|
}
|
package/src/fence.mjs
CHANGED
|
@@ -167,6 +167,25 @@ export function scanText(text, opts = {}) {
|
|
|
167
167
|
return hits.concat(fileReads).sort((a, b) => a.start - b.start || SEV[a.severity] - SEV[b.severity]);
|
|
168
168
|
}
|
|
169
169
|
|
|
170
|
+
// Text Glassbox keeps for later (memory notes, session digests): every value a detector recognises becomes
|
|
171
|
+
// [FENCED:<rule>], with no fingerprint — the note is for the next session to read, not for matching keys.
|
|
172
|
+
export function maskSecrets(text) {
|
|
173
|
+
const s = String(text == null ? '' : text); const spans = [];
|
|
174
|
+
for (const d of DETECTORS) {
|
|
175
|
+
d.re.lastIndex = 0; let m;
|
|
176
|
+
while ((m = d.re.exec(s))) {
|
|
177
|
+
const value = d.value ? d.value(m) : m[0];
|
|
178
|
+
const start = d.value ? m.index + m[0].length - value.length : m.index, end = start + value.length;
|
|
179
|
+
if ((d.ok && !d.ok(value)) || spans.some(([a, b]) => start < b && end > a)) continue;
|
|
180
|
+
spans.push([start, end, d.id]);
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
if (!spans.length) return s;
|
|
184
|
+
spans.sort((a, b) => a[0] - b[0]);
|
|
185
|
+
let out = '', at = 0; for (const [a, b, id] of spans) { out += s.slice(at, a) + `[FENCED:${id}]`; at = b; }
|
|
186
|
+
return out + s.slice(at);
|
|
187
|
+
}
|
|
188
|
+
|
|
170
189
|
// Same value in the same place collapses to one finding with a count. Values are kept on the object (for shred)
|
|
171
190
|
// but never serialised: `toJSON` drops them.
|
|
172
191
|
function rows(hits) {
|