@ngockhoale/ukit 3.5.13 → 3.5.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/package.json +1 -1
- package/scripts/audit/field-replay.mjs +194 -0
- package/scripts/audit/field-telemetry.mjs +341 -0
- package/scripts/release/candidate-state.mjs +4 -1
- package/template_project/.claude/hooks/context-hardcap-gate.sh +40 -230
- package/template_project/.claude/hooks/context-window-guard.sh +1 -1
- package/template_project/.claude/hooks/sensitive-data-guard.mjs +297 -13
- package/template_project/.claude/ukit/index/post-edit-verify.mjs +3 -7
- package/template_project/.claude/ukit/index/pre-edit-backup.mjs +1 -0
- package/template_project/.claude/ukit/runtime/compact-threshold.mjs +3 -3
- package/template_project/.claude/ukit/runtime/execution-ledger.mjs +92 -3
- package/template_project/.claude/ukit/runtime/output-compression.mjs +28 -1
- package/template_project/.claude/ukit/runtime/safe-patch-core.mjs +22 -0
- package/template_project/.claude/ukit/runtime/stop-coordinator.mjs +36 -0
- package/template_project/.omp/config.yml +1 -1
- package/template_project/AGENTS.md +1 -1
- package/template_project/CLAUDE.md +1 -1
- package/template_project/docs/AI_HANDOFF/RULES.md +1 -1
- package/template_project/instructions/core.md +1 -1
- package/template_project/ukit/storage/config.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,39 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to UKit are documented here.
|
|
4
4
|
|
|
5
|
+
## 3.5.14 - 2026-11-20
|
|
6
|
+
|
|
7
|
+
Field-telemetry fix round 1. Source: Claude Code transcripts and `.ukit/storage/` of every consumer project (BAGuide, CKit, 4vn, XoonCube, UnicDB, …). New offline tools measure each class before and after a fix: `scripts/audit/field-telemetry.mjs` clusters hook messages and latency, and `scripts/audit/field-replay.mjs` re-runs historical hook inputs through the current hooks.
|
|
8
|
+
|
|
9
|
+
- **Context pressure no longer stops the run to ask for /compact.** The soft and hard context advisories (`compact-threshold.mjs`, `context-window-guard.sh`) told the model to make "gõ /compact ngay" the first line of its reply. The model then ended its turn and waited for the user. They now say: keep working, persist progress to docs/STATUS.md, delegate broad reads, and let the host auto-compact. The hard-cap gate, which really refuses tools, still tells the user to compact.
|
|
10
|
+
- **`release:verify` no longer crashes on an untracked nested repo.** `git ls-files --others` lists a test's temp project with its own `.git` as `dir/`, and `candidateFingerprint` threw `Unsupported candidate entry`. That made the release gate flaky while other suites were running. Such entries are now dropped before hashing, so a temp repo appearing mid-run neither crashes the gate nor drifts the candidate.
|
|
11
|
+
- **Sensitive-data guard false positives, round 2 (#4b).** Count-only and list-only greps are allowed: `grep -c/-q/-l` on `.env`, `env | grep -c`, and grep pattern operands inside `$()`. A `jq .env.X` path is labelled as a JSON credential read and blocks only credential-named fields. Placeholder fixture keys (`sk-claude-secret-…`, canary or sequential bodies) no longer trip the token scan; random-looking keys still block. Replay since 11-14: 29 → 22 blocks, and the 22 left are real dumps. Ref: gitleaks allowlist stopwords.
|
|
12
|
+
- **Stop gate waits for background subagents (#1).** 149 of 487 Stop blocks fired while a Claude Code background agent (`Agent` → `async_launched`) was still running. The gate demanded edits or verification the parent could not produce yet. The agent is now read from the host transcript and gets the same bounded WAIT as a background shell job:
|
|
13
|
+
- one release per agent id;
|
|
14
|
+
- no continuation budget spent and no edit demand;
|
|
15
|
+
- after its `<task-notification>` arrives, the normal gate applies again.
|
|
16
|
+
- On omp and Codex the transcript has no such shape, so nothing changes there.
|
|
17
|
+
- **Re-entered stop no longer re-blocks on an evaluator failure (#2).** The first evaluator failure, timeout or hook-deadline overrun still blocks fail-closed. A repeat on a re-entered Stop (`stop_hook_active`) is released with a visible "released WITHOUT verification" advisory, with outcome blocked and never complete. Claude Code's default for a failed hook is non-blocking; a second block only wastes a turn.
|
|
18
|
+
- **Printed runner status is accepted as exit proof (#3).** These shapes were the main source of "unproven" storms:
|
|
19
|
+
- `runner > log 2>&1; echo "exit=$?"`
|
|
20
|
+
- `runner | tail; echo "exit=${PIPESTATUS[0]}"` (bash), or `${pipestatus[1]}` (zsh)
|
|
21
|
+
|
|
22
|
+
They now count only when all of the following hold:
|
|
23
|
+
- the echo comes immediately after the only runner in the command;
|
|
24
|
+
- the status variable names the runner's own stage;
|
|
25
|
+
- the last stdout line exactly matches the echo with a number.
|
|
26
|
+
|
|
27
|
+
Anything else stays unproven, and a printed non-zero status is a real failure. The approach follows OpenHands and SWE-ReX, which echo `$?` after every command.
|
|
28
|
+
- **sensitive-data-guard false positives (#4).** These were blocked before and are allowed now:
|
|
29
|
+
- grep/rg pattern operands (`grep "process\.env"`) are no longer treated as files;
|
|
30
|
+
- quoted separators (`grep -E "release|history"`) no longer split the command;
|
|
31
|
+
- quoted-heredoc bodies are treated as literal text.
|
|
32
|
+
|
|
33
|
+
Real secret reads are still denied: `-f`/`--file=`, unquoted heredoc substitutions, and `.env` operands.
|
|
34
|
+
- **No "ownership UNKNOWN" advisory beside a closed run cursor (#5).** About 2,500 advisories came from `Phase: done` cursors.
|
|
35
|
+
- **Safe Patch stops talking on success (#16, #10).** The `[ukit-safe-patch] OK: …` systemMessage appeared after every clean edit (798 field messages) and is now silent, since a hook that succeeds should exit 0 with no output. Edits to OS temp scratch files (`/tmp`, `/private/tmp`, `os.tmpdir()`, `$TMPDIR`) no longer raise "out-of-project edit target" (157 messages). Other out-of-project targets are still reported, and a project that itself lives under a temp dir gets no exemption.
|
|
36
|
+
- **Output compression keeps ranked tables intact (#11).** A count-sorted table is now passed through raw, up to 6k tokens. Before, the line picker reordered such a table and dropped its middle. rtk-ai/rtk#582 reports the same lesson.
|
|
37
|
+
|
|
5
38
|
## 3.5.13 - 2026-11-20
|
|
6
39
|
|
|
7
40
|
Memory-safe test design and execution. A field Python/Sanic backend (~790 files, ~10k tests) grew one pytest process to ~5 GB RSS and swapped for hours; adding pytest-xdist workers made it worse. The cause was a per-test leak in app fixtures, not runner speed. UKit now helps agents write tests that do not leak and run large suites without swapping. No new command: the skills route to the runner. This release also ships the 3.5.12 Stop-gate fix, which reached git and GitHub but not npm.
|
package/package.json
CHANGED
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* field-replay.mjs — debug-round replay harness
|
|
4
|
+
* (docs/plans/2026-11-20-field-telemetry-fix-program.md, "Debug-round protocol").
|
|
5
|
+
*
|
|
6
|
+
* Extracts historical PreToolUse Bash commands (assistant tool_use blocks,
|
|
7
|
+
* name "Bash", input.command) from Claude Code transcripts and replays each
|
|
8
|
+
* one through the CURRENT template hook, in a temp project with an empty
|
|
9
|
+
* .ukit/storage/config.json. Reports block counts, block-label classes and
|
|
10
|
+
* sample commands per class (redacted + truncated; values never printed).
|
|
11
|
+
*
|
|
12
|
+
* Usage: node scripts/audit/field-replay.mjs [--projects-dir DIR] [--match SUBSTR]...
|
|
13
|
+
* [--since YYYY-MM-DD] [--limit N] [--hook sensitive] [--mode inproc|cli] [--json]
|
|
14
|
+
*
|
|
15
|
+
* Hooks are registered in HOOKS below; add block-dangerous / execution-ledger
|
|
16
|
+
* (PostToolUse) by adding an entry with its own payload builder + verdict parser.
|
|
17
|
+
*/
|
|
18
|
+
import fs from 'node:fs';
|
|
19
|
+
import fsp from 'node:fs/promises';
|
|
20
|
+
import os from 'node:os';
|
|
21
|
+
import path from 'node:path';
|
|
22
|
+
import readline from 'node:readline';
|
|
23
|
+
import { spawn } from 'node:child_process';
|
|
24
|
+
import { fileURLToPath, pathToFileURL } from 'node:url';
|
|
25
|
+
import { parseArgs, listTranscripts, normalizeMessage, redact, truncate } from './field-telemetry.mjs';
|
|
26
|
+
|
|
27
|
+
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
28
|
+
const HOOK_DIR = path.join(REPO, 'template_project', '.claude', 'hooks');
|
|
29
|
+
|
|
30
|
+
const bashPreToolUse = (command) => ({ hook_event_name: 'PreToolUse', tool_name: 'Bash', tool_input: { command } });
|
|
31
|
+
const stderrLabels = (stderr) =>
|
|
32
|
+
String(stderr || '')
|
|
33
|
+
.split('\n')
|
|
34
|
+
.filter((l) => l.startsWith(' - '))
|
|
35
|
+
.map((l) => l.slice(4));
|
|
36
|
+
|
|
37
|
+
export const HOOKS = {
|
|
38
|
+
sensitive: {
|
|
39
|
+
file: path.join(HOOK_DIR, 'sensitive-data-guard.mjs'),
|
|
40
|
+
event: 'PreToolUse',
|
|
41
|
+
buildPayload: bashPreToolUse,
|
|
42
|
+
labels: stderrLabels,
|
|
43
|
+
},
|
|
44
|
+
// Future: 'dangerous' → block-dangerous.mjs (same payload shape, labels from stderr/stdout);
|
|
45
|
+
// 'ledger' → execution-ledger PostToolUse (needs tool_response from the transcript).
|
|
46
|
+
};
|
|
47
|
+
|
|
48
|
+
// --- command extraction --------------------------------------------------------
|
|
49
|
+
export function extractBashCommands(line, out, { since } = {}) {
|
|
50
|
+
if (!line.includes('"tool_use"') || !line.includes('"Bash"')) return;
|
|
51
|
+
let o;
|
|
52
|
+
try {
|
|
53
|
+
o = JSON.parse(line);
|
|
54
|
+
} catch {
|
|
55
|
+
return;
|
|
56
|
+
}
|
|
57
|
+
if (o?.type !== 'assistant' || !Array.isArray(o.message?.content)) return;
|
|
58
|
+
if (since && typeof o.timestamp === 'string' && o.timestamp.slice(0, 10) < since) return;
|
|
59
|
+
for (const c of o.message.content) {
|
|
60
|
+
if (c && c.type === 'tool_use' && c.name === 'Bash' && typeof c.input?.command === 'string' && c.input.command.trim()) {
|
|
61
|
+
out.set(c.input.command, (out.get(c.input.command) || 0) + 1);
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
async function collectCommands(opts) {
|
|
67
|
+
const files = await listTranscripts(opts);
|
|
68
|
+
const cmds = new Map();
|
|
69
|
+
for (const f of files) {
|
|
70
|
+
const rl = readline.createInterface({ input: fs.createReadStream(f, { encoding: 'utf8' }), crlfDelay: Infinity });
|
|
71
|
+
for await (const line of rl) extractBashCommands(line, cmds, opts);
|
|
72
|
+
}
|
|
73
|
+
return { files: files.length, commands: [...cmds.keys()].slice(-opts.limit) };
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
// --- runners ---------------------------------------------------------------------
|
|
77
|
+
async function makeTempProject() {
|
|
78
|
+
const root = await fsp.mkdtemp(path.join(os.tmpdir(), 'ukit-replay-'));
|
|
79
|
+
await fsp.mkdir(path.join(root, '.ukit', 'storage'), { recursive: true });
|
|
80
|
+
await fsp.writeFile(path.join(root, '.ukit', 'storage', 'config.json'), '{}\n');
|
|
81
|
+
return root;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function hookEnv(projectRoot) {
|
|
85
|
+
// Minimal env: never forward the caller's secrets into the replayed hook.
|
|
86
|
+
return { PATH: process.env.PATH || '', HOME: projectRoot, PROJECT_ROOT: projectRoot, UKIT_HOOK_DEADLINE_MS: '10000' };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
async function runInproc(hook, mod, payload, projectRoot) {
|
|
90
|
+
const v = await mod.runHook({ payload, payloadText: JSON.stringify(payload), projectRoot, env: hookEnv(projectRoot), deadlineMs: 10000 });
|
|
91
|
+
return { code: v?.code ?? 0, stdout: v?.stdout || '', stderr: v?.stderr || '' };
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
function runCli(hook, payload, projectRoot, tmpDir, idx) {
|
|
95
|
+
return new Promise((resolve) => {
|
|
96
|
+
const inputFile = path.join(tmpDir, `in-${idx}.json`);
|
|
97
|
+
fs.writeFileSync(inputFile, JSON.stringify(payload));
|
|
98
|
+
const child = spawn(process.execPath, [hook.file], { env: { ...hookEnv(projectRoot), INPUT_FILE: inputFile }, cwd: projectRoot, stdio: ['ignore', 'pipe', 'pipe'] });
|
|
99
|
+
let stdout = '';
|
|
100
|
+
let stderr = '';
|
|
101
|
+
child.stdout.on('data', (d) => (stdout += d));
|
|
102
|
+
child.stderr.on('data', (d) => (stderr += d));
|
|
103
|
+
child.on('close', (code) => {
|
|
104
|
+
fs.rmSync(inputFile, { force: true });
|
|
105
|
+
resolve({ code: code ?? 1, stdout, stderr });
|
|
106
|
+
});
|
|
107
|
+
});
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
// --- aggregation -------------------------------------------------------------------
|
|
111
|
+
export function classifyResults(results, hook, { samples = 40 } = {}) {
|
|
112
|
+
const classes = new Map();
|
|
113
|
+
let blocked = 0;
|
|
114
|
+
for (const { command, verdict } of results) {
|
|
115
|
+
if (verdict.code !== 2) continue;
|
|
116
|
+
blocked += 1;
|
|
117
|
+
const labels = hook.labels(verdict.stderr);
|
|
118
|
+
const keys = labels.length ? [...new Set(labels.map((l) => normalizeMessage(l.replace(/:\s.*$/, ''))))] : ['(no label) ' + normalizeMessage(verdict.stderr.split('\n')[0] || 'exit 2')];
|
|
119
|
+
for (const k of keys) {
|
|
120
|
+
let c = classes.get(k);
|
|
121
|
+
if (!c) classes.set(k, (c = { cls: k, count: 0, samples: [] }));
|
|
122
|
+
c.count += 1;
|
|
123
|
+
if (c.samples.length < samples) c.samples.push(truncate(redact(command), 200));
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
return { blocked, classes: [...classes.values()].sort((a, b) => b.count - a.count || a.cls.localeCompare(b.cls)) };
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
export async function main(argv = process.argv.slice(2)) {
|
|
130
|
+
const opts = parseArgs(argv);
|
|
131
|
+
if (opts.help) {
|
|
132
|
+
process.stdout.write('Usage: field-replay.mjs [--projects-dir DIR] [--match SUBSTR]... [--since YYYY-MM-DD] [--limit N] [--hook sensitive] [--mode inproc|cli] [--json]\n');
|
|
133
|
+
return 0;
|
|
134
|
+
}
|
|
135
|
+
const hook = HOOKS[opts.hook];
|
|
136
|
+
if (!hook) throw new Error(`unknown --hook ${opts.hook}; known: ${Object.keys(HOOKS).join(', ')}`);
|
|
137
|
+
const t0 = Date.now();
|
|
138
|
+
const { files, commands } = await collectCommands(opts);
|
|
139
|
+
const projectRoot = await makeTempProject();
|
|
140
|
+
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'ukit-replay-in-'));
|
|
141
|
+
const results = [];
|
|
142
|
+
let errors = 0;
|
|
143
|
+
try {
|
|
144
|
+
const mod = opts.mode === 'cli' ? null : await import(pathToFileURL(hook.file).href);
|
|
145
|
+
if (mod && typeof mod.runHook !== 'function') throw new Error(`${hook.file} has no exported runHook; use --mode cli`);
|
|
146
|
+
const concurrency = opts.mode === 'cli' ? 8 : 1;
|
|
147
|
+
for (let i = 0; i < commands.length; i += concurrency) {
|
|
148
|
+
const batch = commands.slice(i, i + concurrency);
|
|
149
|
+
const verdicts = await Promise.all(
|
|
150
|
+
batch.map((command, j) => {
|
|
151
|
+
const payload = hook.buildPayload(command);
|
|
152
|
+
return (mod ? runInproc(hook, mod, payload, projectRoot) : runCli(hook, payload, projectRoot, tmpDir, i + j)).catch((e) => {
|
|
153
|
+
errors += 1;
|
|
154
|
+
return { code: -1, stdout: '', stderr: String(e && e.message) };
|
|
155
|
+
});
|
|
156
|
+
}),
|
|
157
|
+
);
|
|
158
|
+
batch.forEach((command, j) => results.push({ command, verdict: verdicts[j] }));
|
|
159
|
+
}
|
|
160
|
+
} finally {
|
|
161
|
+
await fsp.rm(projectRoot, { recursive: true, force: true });
|
|
162
|
+
await fsp.rm(tmpDir, { recursive: true, force: true });
|
|
163
|
+
}
|
|
164
|
+
const rep = classifyResults(results, hook);
|
|
165
|
+
const meta = { hook: opts.hook, mode: opts.mode, files, since: opts.since, replayed: commands.length, blocked: rep.blocked, errors, ms: Date.now() - t0 };
|
|
166
|
+
if (opts.json) {
|
|
167
|
+
process.stdout.write(`${JSON.stringify({ meta, classes: rep.classes }, null, 2)}\n`);
|
|
168
|
+
} else {
|
|
169
|
+
const L = [`field-replay [${meta.hook}/${meta.mode}]: ${files} transcript(s), since ${opts.since || 'all'}, replayed ${meta.replayed} unique command(s), blocked ${meta.blocked}, errors ${errors}, ${meta.ms} ms`];
|
|
170
|
+
for (const c of rep.classes) {
|
|
171
|
+
L.push('', `## [${c.count}] ${c.cls}`);
|
|
172
|
+
for (const s of c.samples) L.push(` - ${s}`);
|
|
173
|
+
}
|
|
174
|
+
process.stdout.write(`${L.join('\n')}\n`);
|
|
175
|
+
}
|
|
176
|
+
return 0;
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
const isDirect = (() => {
|
|
180
|
+
try {
|
|
181
|
+
return process.argv[1] && fs.realpathSync(process.argv[1]) === fs.realpathSync(fileURLToPath(import.meta.url));
|
|
182
|
+
} catch {
|
|
183
|
+
return false;
|
|
184
|
+
}
|
|
185
|
+
})();
|
|
186
|
+
if (isDirect) {
|
|
187
|
+
main().then(
|
|
188
|
+
(code) => process.exit(code),
|
|
189
|
+
(err) => {
|
|
190
|
+
process.stderr.write(`field-replay: ${err && err.message ? err.message : err}\n`);
|
|
191
|
+
process.exit(1);
|
|
192
|
+
},
|
|
193
|
+
);
|
|
194
|
+
}
|
|
@@ -0,0 +1,341 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* field-telemetry.mjs — T0 of docs/plans/2026-11-20-field-telemetry-fix-program.md
|
|
4
|
+
*
|
|
5
|
+
* Offline measurement of UKit hook behaviour in Claude Code transcripts
|
|
6
|
+
* (~/.claude/projects/<project>/**.jsonl). Streams every file line by line.
|
|
7
|
+
*
|
|
8
|
+
* Collected (shapes verified against Claude Code 2.1.x transcripts):
|
|
9
|
+
* - attachment.type "hook_blocking_error" → attachment.blockingError.blockingError
|
|
10
|
+
* - attachment.type "hook_system_message" → attachment.content
|
|
11
|
+
* - top-level / attachment `systemMessage` strings (older shapes)
|
|
12
|
+
* - tool_result with permissionDecision.source === "hook" (PreToolUse denial;
|
|
13
|
+
* its " - <label>" lines are the block labels)
|
|
14
|
+
* - system/stop_hook_summary → hookInfos[].durationMs + hookErrors[]
|
|
15
|
+
* - attachment.type "hook_success" durationMs → per-hookName latency
|
|
16
|
+
* The Bash command preceding a PreToolUse block is resolved via toolUseID →
|
|
17
|
+
* the assistant tool_use block with the same id.
|
|
18
|
+
*
|
|
19
|
+
* Usage: node scripts/audit/field-telemetry.mjs [--projects-dir DIR]
|
|
20
|
+
* [--match SUBSTR]... [--since YYYY-MM-DD] [--json]
|
|
21
|
+
*/
|
|
22
|
+
import fs from 'node:fs';
|
|
23
|
+
import fsp from 'node:fs/promises';
|
|
24
|
+
import os from 'node:os';
|
|
25
|
+
import path from 'node:path';
|
|
26
|
+
import readline from 'node:readline';
|
|
27
|
+
import { fileURLToPath } from 'node:url';
|
|
28
|
+
|
|
29
|
+
// --- normalization / redaction ----------------------------------------------
|
|
30
|
+
const UUID_RE = /\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi;
|
|
31
|
+
const TOOLU_RE = /\btoolu_[A-Za-z0-9]+\b/g;
|
|
32
|
+
const HEX_RE = /\b(?=[0-9a-f]*[0-9])(?=[0-9a-f]*[a-f])[0-9a-f]{7,}\b/gi;
|
|
33
|
+
const PATH_RE = /(?<![\w.-])(?:~|\$[A-Z_]+|\.{1,2})?(?:\/[^\s"'`:,;()<>\]\[{}]+)+\/?|\b[\w.-]+(?:\/[\w.-]+){2,}\b|\b[\w.-]+(?:\/[\w.-]+)*\/[\w-]+\.[A-Za-z]\w{0,5}\b/g;
|
|
34
|
+
// ^ relative paths need >=2 slashes or a file extension, so "dotenv/secret" stays readable.
|
|
35
|
+
const NUM_RE = /\b\d+(?:\.\d+)?(?:ms|s|m|h|d|%|x|MB|KB|GB)?\b/g;
|
|
36
|
+
|
|
37
|
+
export function normalizeMessage(text, max = 140) {
|
|
38
|
+
let s = String(text ?? '');
|
|
39
|
+
s = s.replace(/<\/?system-reminder>/g, ' ');
|
|
40
|
+
s = s.replace(UUID_RE, '#').replace(TOOLU_RE, '#');
|
|
41
|
+
s = s.replace(PATH_RE, '<p>');
|
|
42
|
+
s = s.replace(HEX_RE, '#');
|
|
43
|
+
s = s.replace(NUM_RE, 'N');
|
|
44
|
+
s = s.replace(/\s+/g, ' ').trim();
|
|
45
|
+
return s.length > max ? `${s.slice(0, max - 1)}…` : s;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
// Masks values that look like credentials before anything is printed.
|
|
49
|
+
export function redact(text) {
|
|
50
|
+
return String(text ?? '')
|
|
51
|
+
.replace(
|
|
52
|
+
/\b([A-Za-z_][A-Za-z0-9_]*(?:KEY|TOKEN|SECRET|PASSWORD|PASSWD|PWD|CREDENTIAL|AUTH)[A-Za-z0-9_]*)\s*([=:])\s*("[^"]*"|'[^']*'|[^\s;&|]+)/gi,
|
|
53
|
+
'$1$2<redacted>',
|
|
54
|
+
)
|
|
55
|
+
// vendor-formatted keys
|
|
56
|
+
.replace(/\b(?:sk|pk|rk)-[A-Za-z0-9_-]{8,}/g, '<redacted-key>')
|
|
57
|
+
.replace(/\b(?:gh[pousr]_[A-Za-z0-9]{10,}|github_pat_[A-Za-z0-9_]{10,}|glpat-[A-Za-z0-9_-]{10,}|xox[abposr]-[A-Za-z0-9-]{10,}|AKIA[0-9A-Z]{12,}|AIza[0-9A-Za-z_-]{20,})/g, '<redacted-key>')
|
|
58
|
+
.replace(/\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g, '<redacted-jwt>')
|
|
59
|
+
.replace(/\b(Bearer|Basic)\s+[A-Za-z0-9._~+/=-]{8,}/g, '$1 <redacted>')
|
|
60
|
+
.replace(/-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?(-----END [A-Z ]*PRIVATE KEY-----|$)/g, '<private-key>')
|
|
61
|
+
// high-entropy standalone tokens (not path segments)
|
|
62
|
+
.replace(/(?<![\w/.-])(?=[A-Za-z0-9_\-+]*[0-9])(?=[A-Za-z0-9_\-+]*[A-Z])(?=[A-Za-z0-9_\-+]*[a-z])[A-Za-z0-9_\-+]{32,}={0,2}(?![\w/])/g, '<redacted>');
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
export function truncate(text, max) {
|
|
66
|
+
const s = String(text ?? '').replace(/\s+/g, ' ').trim();
|
|
67
|
+
return s.length > max ? `${s.slice(0, max - 1)}…` : s;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
// --- transcript parsing --------------------------------------------------------
|
|
71
|
+
const INTERESTING = ['hook_', 'stop_hook_summary', 'systemMessage', '"tool_use"', 'permissionDecision'];
|
|
72
|
+
|
|
73
|
+
function isoDay(ts) {
|
|
74
|
+
return typeof ts === 'string' && ts.length >= 10 ? ts.slice(0, 10) : 'unknown';
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
function toolResultText(content) {
|
|
78
|
+
if (typeof content === 'string') return content;
|
|
79
|
+
if (Array.isArray(content)) return content.map((c) => (c && typeof c.text === 'string' ? c.text : '')).join('\n');
|
|
80
|
+
return '';
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/**
|
|
84
|
+
* Parse one transcript's lines into events. `state.commands` maps tool_use id
|
|
85
|
+
* → Bash command (shared across files so subagent transcripts resolve too).
|
|
86
|
+
* Returns { events, stopDurations, hookLatency } appended into `acc`.
|
|
87
|
+
*/
|
|
88
|
+
export function parseTranscriptLine(line, acc, { since } = {}) {
|
|
89
|
+
if (!INTERESTING.some((k) => line.includes(k))) return;
|
|
90
|
+
let o;
|
|
91
|
+
try {
|
|
92
|
+
o = JSON.parse(line);
|
|
93
|
+
} catch {
|
|
94
|
+
return;
|
|
95
|
+
}
|
|
96
|
+
if (!o || typeof o !== 'object') return;
|
|
97
|
+
const ts = o.timestamp;
|
|
98
|
+
if (since && typeof ts === 'string' && ts.slice(0, 10) < since) {
|
|
99
|
+
// still learn tool_use ids so later blocks can resolve their command
|
|
100
|
+
if (o.type !== 'assistant') return;
|
|
101
|
+
}
|
|
102
|
+
const day = isoDay(ts);
|
|
103
|
+
const inWindow = !since || day === 'unknown' || day >= since;
|
|
104
|
+
|
|
105
|
+
// assistant tool_use → remember Bash commands by id
|
|
106
|
+
if (o.type === 'assistant' && Array.isArray(o.message?.content)) {
|
|
107
|
+
for (const c of o.message.content) {
|
|
108
|
+
if (c && c.type === 'tool_use' && c.name === 'Bash' && c.input && typeof c.input.command === 'string') {
|
|
109
|
+
acc.commands.set(c.id, c.input.command);
|
|
110
|
+
acc.lastBash = c.input.command;
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
return;
|
|
114
|
+
}
|
|
115
|
+
if (!inWindow) return;
|
|
116
|
+
|
|
117
|
+
const push = (kind, hookEvent, hookName, text, toolUseID) => {
|
|
118
|
+
if (!text || !String(text).trim()) return;
|
|
119
|
+
acc.events.push({
|
|
120
|
+
kind,
|
|
121
|
+
hookEvent: hookEvent || 'unknown',
|
|
122
|
+
hookName: hookName || '',
|
|
123
|
+
day,
|
|
124
|
+
cls: normalizeMessage(text),
|
|
125
|
+
toolUseID: toolUseID || null,
|
|
126
|
+
command: toolUseID && acc.commands.has(toolUseID) ? acc.commands.get(toolUseID) : null,
|
|
127
|
+
});
|
|
128
|
+
};
|
|
129
|
+
|
|
130
|
+
const a = o.attachment;
|
|
131
|
+
if (a && typeof a === 'object') {
|
|
132
|
+
if (a.type === 'hook_blocking_error') {
|
|
133
|
+
const be = a.blockingError;
|
|
134
|
+
const text = typeof be === 'string' ? be : be && be.blockingError;
|
|
135
|
+
push('hook_blocking_error', a.hookEvent, a.hookName, text, a.toolUseID);
|
|
136
|
+
} else if (a.type === 'hook_system_message') {
|
|
137
|
+
push('hook_system_message', a.hookEvent, a.hookName, a.content, a.toolUseID);
|
|
138
|
+
} else if (a.type === 'hook_success' && Number.isFinite(a.durationMs)) {
|
|
139
|
+
const key = a.hookName || a.hookEvent || 'unknown';
|
|
140
|
+
(acc.hookLatency[key] ||= []).push(a.durationMs);
|
|
141
|
+
}
|
|
142
|
+
if (typeof a.systemMessage === 'string') push('systemMessage', a.hookEvent, a.hookName, a.systemMessage, a.toolUseID);
|
|
143
|
+
return;
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
if (typeof o.systemMessage === 'string') push('systemMessage', o.hookEvent, '', o.systemMessage, o.toolUseID);
|
|
147
|
+
|
|
148
|
+
if (o.type === 'system' && o.subtype === 'stop_hook_summary') {
|
|
149
|
+
const infos = Array.isArray(o.hookInfos) ? o.hookInfos : [];
|
|
150
|
+
let total = 0;
|
|
151
|
+
let seen = false;
|
|
152
|
+
for (const h of infos) {
|
|
153
|
+
if (h && Number.isFinite(h.durationMs)) {
|
|
154
|
+
total += h.durationMs;
|
|
155
|
+
seen = true;
|
|
156
|
+
acc.stopHookDurations.push(h.durationMs);
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
if (seen) acc.stopDurations.push(total);
|
|
160
|
+
return;
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
// PreToolUse denial surfaced as an is_error tool_result
|
|
164
|
+
if (o.type === 'user' && o.permissionDecision && o.permissionDecision.source === 'hook') {
|
|
165
|
+
const tr = Array.isArray(o.message?.content) ? o.message.content.find((c) => c && c.type === 'tool_result') : null;
|
|
166
|
+
if (!tr) return;
|
|
167
|
+
const text = toolResultText(tr.content);
|
|
168
|
+
const labels = text.split('\n').filter((l) => l.startsWith(' - ')).map((l) => l.slice(4));
|
|
169
|
+
const reason = o.permissionDecision.reasonCode || 'hook';
|
|
170
|
+
const event = /PreToolUse/.test(reason) ? 'PreToolUse' : reason.replace(/^hook:\s*/, '');
|
|
171
|
+
const headline = (text.split(']: ')[1] || text).split('\n')[0];
|
|
172
|
+
for (const label of labels.length ? labels : [headline]) {
|
|
173
|
+
push('tool_denial', event, '', label, tr.tool_use_id);
|
|
174
|
+
}
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
export function newAccumulator() {
|
|
179
|
+
return { events: [], commands: new Map(), lastBash: null, stopDurations: [], stopHookDurations: [], hookLatency: {} };
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
export async function parseTranscriptFile(file, acc, opts) {
|
|
183
|
+
const rl = readline.createInterface({ input: fs.createReadStream(file, { encoding: 'utf8' }), crlfDelay: Infinity });
|
|
184
|
+
for await (const line of rl) parseTranscriptLine(line, acc, opts);
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
// --- aggregation ---------------------------------------------------------------
|
|
188
|
+
export function percentile(values, p) {
|
|
189
|
+
if (!values.length) return null;
|
|
190
|
+
const sorted = [...values].sort((x, y) => x - y);
|
|
191
|
+
const idx = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1));
|
|
192
|
+
return sorted[idx];
|
|
193
|
+
}
|
|
194
|
+
|
|
195
|
+
const stats = (v) => ({ n: v.length, p50: percentile(v, 50), p95: percentile(v, 95), max: v.length ? Math.max(...v) : null });
|
|
196
|
+
|
|
197
|
+
export function aggregate(acc, { top = 15, cmdTop = 20 } = {}) {
|
|
198
|
+
const byKey = new Map();
|
|
199
|
+
for (const e of acc.events) {
|
|
200
|
+
const key = `${e.hookEvent} | ${e.kind} | ${e.cls}`;
|
|
201
|
+
let r = byKey.get(key);
|
|
202
|
+
if (!r) {
|
|
203
|
+
r = { key, hookEvent: e.hookEvent, kind: e.kind, cls: e.cls, count: 0, days: {}, commands: new Map() };
|
|
204
|
+
byKey.set(key, r);
|
|
205
|
+
}
|
|
206
|
+
r.count += 1;
|
|
207
|
+
r.days[e.day] = (r.days[e.day] || 0) + 1;
|
|
208
|
+
if (e.hookEvent === 'PreToolUse' && e.command) {
|
|
209
|
+
const c = truncate(redact(e.command), 160);
|
|
210
|
+
r.commands.set(c, (r.commands.get(c) || 0) + 1);
|
|
211
|
+
}
|
|
212
|
+
}
|
|
213
|
+
const classes = [...byKey.values()].sort((x, y) => y.count - x.count || x.key.localeCompare(y.key));
|
|
214
|
+
const latency = Object.fromEntries(
|
|
215
|
+
Object.entries(acc.hookLatency)
|
|
216
|
+
.map(([k, v]) => [k, stats(v)])
|
|
217
|
+
.sort((x, y) => (y[1].p95 ?? 0) - (x[1].p95 ?? 0)),
|
|
218
|
+
);
|
|
219
|
+
return {
|
|
220
|
+
totalEvents: acc.events.length,
|
|
221
|
+
classes: classes.map((r, i) => ({
|
|
222
|
+
hookEvent: r.hookEvent,
|
|
223
|
+
kind: r.kind,
|
|
224
|
+
cls: r.cls,
|
|
225
|
+
count: r.count,
|
|
226
|
+
perDay: i < top ? Object.fromEntries(Object.entries(r.days).sort()) : undefined,
|
|
227
|
+
commands:
|
|
228
|
+
r.hookEvent === 'PreToolUse'
|
|
229
|
+
? [...r.commands.entries()].sort((x, y) => y[1] - x[1]).slice(0, cmdTop).map(([command, count]) => ({ command, count }))
|
|
230
|
+
: undefined,
|
|
231
|
+
})),
|
|
232
|
+
stop: { perStop: stats(acc.stopDurations), perHook: stats(acc.stopHookDurations) },
|
|
233
|
+
hookLatency: latency,
|
|
234
|
+
};
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
// --- file discovery ---------------------------------------------------------------
|
|
238
|
+
export function parseArgs(argv) {
|
|
239
|
+
const opts = { projectsDir: path.join(os.homedir(), '.claude', 'projects'), match: [], since: null, json: false, limit: 20000, hook: 'sensitive', mode: 'inproc' };
|
|
240
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
241
|
+
const a = argv[i];
|
|
242
|
+
const next = () => argv[++i];
|
|
243
|
+
if (a === '--projects-dir') opts.projectsDir = next();
|
|
244
|
+
else if (a === '--match') opts.match.push(next());
|
|
245
|
+
else if (a === '--since') opts.since = next();
|
|
246
|
+
else if (a === '--json') opts.json = true;
|
|
247
|
+
else if (a === '--limit') opts.limit = Number.parseInt(next(), 10) || opts.limit;
|
|
248
|
+
else if (a === '--hook') opts.hook = next();
|
|
249
|
+
else if (a === '--mode') opts.mode = next();
|
|
250
|
+
else if (a === '--help' || a === '-h') opts.help = true;
|
|
251
|
+
}
|
|
252
|
+
if (!opts.match.length) opts.match = ['KHOA-EXTENAL'];
|
|
253
|
+
if (opts.since && !/^\d{4}-\d{2}-\d{2}$/.test(opts.since)) throw new Error(`--since must be YYYY-MM-DD, got ${opts.since}`);
|
|
254
|
+
return opts;
|
|
255
|
+
}
|
|
256
|
+
|
|
257
|
+
export async function listTranscripts({ projectsDir, match, since }) {
|
|
258
|
+
const sinceMs = since ? Date.parse(`${since}T00:00:00Z`) : 0;
|
|
259
|
+
const out = [];
|
|
260
|
+
const walk = async (dir) => {
|
|
261
|
+
let entries;
|
|
262
|
+
try {
|
|
263
|
+
entries = await fsp.readdir(dir, { withFileTypes: true });
|
|
264
|
+
} catch {
|
|
265
|
+
return;
|
|
266
|
+
}
|
|
267
|
+
for (const ent of entries) {
|
|
268
|
+
const p = path.join(dir, ent.name);
|
|
269
|
+
if (ent.isDirectory()) await walk(p);
|
|
270
|
+
else if (ent.isFile() && ent.name.endsWith('.jsonl')) {
|
|
271
|
+
if (sinceMs) {
|
|
272
|
+
try {
|
|
273
|
+
if ((await fsp.stat(p)).mtimeMs < sinceMs) continue; // nothing appended after `since`
|
|
274
|
+
} catch {
|
|
275
|
+
continue;
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
out.push(p);
|
|
279
|
+
}
|
|
280
|
+
}
|
|
281
|
+
};
|
|
282
|
+
const top = await fsp.readdir(projectsDir, { withFileTypes: true });
|
|
283
|
+
for (const ent of top) {
|
|
284
|
+
if (ent.isDirectory() && match.some((m) => ent.name.includes(m))) await walk(path.join(projectsDir, ent.name));
|
|
285
|
+
}
|
|
286
|
+
// Older first so tool_use ids are learned before later blocks reference them.
|
|
287
|
+
return out.sort();
|
|
288
|
+
}
|
|
289
|
+
|
|
290
|
+
// --- report ---------------------------------------------------------------------------
|
|
291
|
+
function renderText(rep, meta) {
|
|
292
|
+
const L = [];
|
|
293
|
+
L.push(`field-telemetry: ${meta.files} transcript(s), since ${meta.since || 'all'}, ${rep.totalEvents} hook message(s), ${meta.ms} ms`);
|
|
294
|
+
L.push('', '## Top classes (hookEvent | kind | class)');
|
|
295
|
+
rep.classes.slice(0, 30).forEach((c, i) => L.push(`${String(i + 1).padStart(2)}. ${String(c.count).padStart(5)} ${c.hookEvent} | ${c.kind} | ${c.cls}`));
|
|
296
|
+
L.push('', '## Per-day counts (top 15)');
|
|
297
|
+
rep.classes.slice(0, 15).forEach((c, i) => L.push(`${String(i + 1).padStart(2)}. ${Object.entries(c.perDay || {}).map(([d, n]) => `${d.slice(5)}:${n}`).join(' ')}`));
|
|
298
|
+
L.push('', '## Bash command before each PreToolUse block (top 20 per class)');
|
|
299
|
+
for (const c of rep.classes.filter((x) => x.commands && x.commands.length)) {
|
|
300
|
+
L.push(`- [${c.count}] ${c.cls}`);
|
|
301
|
+
for (const cmd of c.commands) L.push(` ${String(cmd.count).padStart(3)}x ${cmd.command}`);
|
|
302
|
+
}
|
|
303
|
+
const f = (s) => `n=${s.n} p50=${s.p50 ?? '-'}ms p95=${s.p95 ?? '-'}ms max=${s.max ?? '-'}ms`;
|
|
304
|
+
L.push('', '## Stop hook duration', `per Stop (sum of hooks): ${f(rep.stop.perStop)}`, `per hook command: ${f(rep.stop.perHook)}`);
|
|
305
|
+
L.push('', '## Hook latency (hook_success durationMs, top 15 by p95)');
|
|
306
|
+
Object.entries(rep.hookLatency).slice(0, 15).forEach(([k, s]) => L.push(` ${k.padEnd(34)} ${f(s)}`));
|
|
307
|
+
return L.join('\n');
|
|
308
|
+
}
|
|
309
|
+
|
|
310
|
+
export async function main(argv = process.argv.slice(2)) {
|
|
311
|
+
const opts = parseArgs(argv);
|
|
312
|
+
if (opts.help) {
|
|
313
|
+
process.stdout.write('Usage: field-telemetry.mjs [--projects-dir DIR] [--match SUBSTR]... [--since YYYY-MM-DD] [--json]\n');
|
|
314
|
+
return 0;
|
|
315
|
+
}
|
|
316
|
+
const t0 = Date.now();
|
|
317
|
+
const files = await listTranscripts(opts);
|
|
318
|
+
const acc = newAccumulator();
|
|
319
|
+
for (const f of files) await parseTranscriptFile(f, acc, { since: opts.since });
|
|
320
|
+
const rep = aggregate(acc);
|
|
321
|
+
const meta = { files: files.length, since: opts.since, ms: Date.now() - t0 };
|
|
322
|
+
process.stdout.write(opts.json ? `${JSON.stringify({ meta, ...rep }, null, 2)}\n` : `${renderText(rep, meta)}\n`);
|
|
323
|
+
return 0;
|
|
324
|
+
}
|
|
325
|
+
|
|
326
|
+
const isDirect = (() => {
|
|
327
|
+
try {
|
|
328
|
+
return process.argv[1] && fs.realpathSync(process.argv[1]) === fs.realpathSync(fileURLToPath(import.meta.url));
|
|
329
|
+
} catch {
|
|
330
|
+
return false;
|
|
331
|
+
}
|
|
332
|
+
})();
|
|
333
|
+
if (isDirect) {
|
|
334
|
+
main().then(
|
|
335
|
+
(code) => process.exit(code),
|
|
336
|
+
(err) => {
|
|
337
|
+
process.stderr.write(`field-telemetry: ${err && err.message ? err.message : err}\n`);
|
|
338
|
+
process.exit(1);
|
|
339
|
+
},
|
|
340
|
+
);
|
|
341
|
+
}
|
|
@@ -8,9 +8,12 @@ import path from 'node:path';
|
|
|
8
8
|
const mutable = /^(?:\.ukit\/storage\/|\.claude\/ukit\/runtime\/\.ukit\/storage\/|\.cache\/|node_modules\/)/;
|
|
9
9
|
|
|
10
10
|
export function candidateFingerprint(root) {
|
|
11
|
+
// `git ls-files --others` lists an untracked nested repository (a running test's temp project
|
|
12
|
+
// with its own `.git`) as one `dir/` entry. It is not a candidate input; it is dropped before
|
|
13
|
+
// hashing, so its appearing/vanishing mid-run neither crashes nor drifts the fingerprint.
|
|
11
14
|
const files = execFileSync('git', ['ls-files', '-z', '--cached', '--others', '--exclude-standard'], {
|
|
12
15
|
cwd: root, encoding: 'utf8', maxBuffer: 32 * 1024 * 1024, timeout: 30_000,
|
|
13
|
-
}).split('\0').filter((file) => file && !mutable.test(file));
|
|
16
|
+
}).split('\0').filter((file) => file && !file.endsWith('/') && !mutable.test(file));
|
|
14
17
|
const hash = createHash('sha256');
|
|
15
18
|
for (const file of [...new Set(files)].sort()) {
|
|
16
19
|
hash.update(JSON.stringify(file));
|