@ngockhoale/ukit 3.5.11 → 3.5.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +62 -0
- package/manifests/documentation.yaml +11 -0
- package/manifests/platform.full.yaml +39 -2
- package/package.json +1 -1
- package/scripts/audit/field-replay.mjs +194 -0
- package/scripts/audit/field-telemetry.mjs +341 -0
- package/scripts/release/candidate-state.mjs +4 -1
- package/template_project/.claude/agents/code-reviewer.md +1 -0
- package/template_project/.claude/agents/feature-implementer.md +1 -0
- package/template_project/.claude/hooks/completion-gate.sh +3 -3
- package/template_project/.claude/hooks/context-hardcap-gate.sh +40 -230
- package/template_project/.claude/hooks/context-window-guard.sh +1 -1
- package/template_project/.claude/hooks/sensitive-data-guard.mjs +297 -13
- package/template_project/.claude/settings.json +2 -2
- package/template_project/.claude/skills/code-review/SKILL.md +14 -0
- package/template_project/.claude/skills/testing-quality/SKILL.md +31 -1
- package/template_project/.claude/skills/testing-quality/memory-safe-tests.md +133 -0
- package/template_project/.claude/ukit/index/post-edit-verify.mjs +3 -7
- package/template_project/.claude/ukit/index/pre-edit-backup.mjs +1 -0
- package/template_project/.claude/ukit/index/test-batches-adapters.mjs +261 -0
- package/template_project/.claude/ukit/index/test-batches.mjs +829 -0
- package/template_project/.claude/ukit/runtime/compact-threshold.mjs +3 -3
- package/template_project/.claude/ukit/runtime/execution-ledger.mjs +92 -3
- package/template_project/.claude/ukit/runtime/output-compression.mjs +28 -1
- package/template_project/.claude/ukit/runtime/safe-patch-core.mjs +22 -0
- package/template_project/.claude/ukit/runtime/stop-coordinator.mjs +40 -4
- package/template_project/.omp/agents/code-reviewer.md +1 -0
- package/template_project/.omp/agents/feature-implementer.md +1 -0
- package/template_project/.omp/config.yml +1 -1
- package/template_project/AGENTS.md +1 -1
- package/template_project/CLAUDE.md +1 -1
- package/template_project/docs/AI_HANDOFF/RULES.md +1 -1
- package/template_project/docs/memory/testing.md +10 -0
- package/template_project/instructions/core.md +1 -1
- package/template_project/ukit/storage/config.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,68 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to UKit are documented here.
|
|
4
4
|
|
|
5
|
+
## 3.5.14 - 2026-11-20
|
|
6
|
+
|
|
7
|
+
Field-telemetry fix round 1. Source: Claude Code transcripts and `.ukit/storage/` of every consumer project (BAGuide, CKit, 4vn, XoonCube, UnicDB, …). New offline tools measure each class before and after a fix: `scripts/audit/field-telemetry.mjs` clusters hook messages and latency, and `scripts/audit/field-replay.mjs` re-runs historical hook inputs through the current hooks.
|
|
8
|
+
|
|
9
|
+
- **Context pressure no longer stops the run to ask for /compact.** The soft and hard context advisories (`compact-threshold.mjs`, `context-window-guard.sh`) told the model to make "gõ /compact ngay" the first line of its reply. The model then ended its turn and waited for the user. They now say: keep working, persist progress to docs/STATUS.md, delegate broad reads, and let the host auto-compact. The hard-cap gate, which really refuses tools, still tells the user to compact.
|
|
10
|
+
- **`release:verify` no longer crashes on an untracked nested repo.** `git ls-files --others` lists a test's temp project with its own `.git` as `dir/`, and `candidateFingerprint` threw `Unsupported candidate entry`. That made the release gate flaky while other suites were running. Such entries are now dropped before hashing, so a temp repo appearing mid-run neither crashes the gate nor drifts the candidate.
|
|
11
|
+
- **Sensitive-data guard false positives, round 2 (#4b).** Count-only and list-only greps are allowed: `grep -c/-q/-l` on `.env`, `env | grep -c`, and grep pattern operands inside `$()`. A `jq .env.X` path is labelled as a JSON credential read and blocks only credential-named fields. Placeholder fixture keys (`sk-claude-secret-…`, canary or sequential bodies) no longer trip the token scan; random-looking keys still block. Replay since 11-14: 29 → 22 blocks, and the 22 left are real dumps. Ref: gitleaks allowlist stopwords.
|
|
12
|
+
- **Stop gate waits for background subagents (#1).** 149 of 487 Stop blocks fired while a Claude Code background agent (`Agent` → `async_launched`) was still running. The gate demanded edits or verification the parent could not produce yet. The agent is now read from the host transcript and gets the same bounded WAIT as a background shell job:
|
|
13
|
+
- one release per agent id;
|
|
14
|
+
- no continuation budget spent and no edit demand;
|
|
15
|
+
- after its `<task-notification>` arrives, the normal gate applies again.
|
|
16
|
+
- On omp and Codex the transcript has no such shape, so nothing changes there.
|
|
17
|
+
- **Re-entered stop no longer re-blocks on an evaluator failure (#2).** The first evaluator failure, timeout or hook-deadline overrun still blocks fail-closed. A repeat on a re-entered Stop (`stop_hook_active`) is released with a visible "released WITHOUT verification" advisory, with outcome blocked and never complete. Claude Code's default for a failed hook is non-blocking; a second block only wastes a turn.
|
|
18
|
+
- **Printed runner status is accepted as exit proof (#3).** These shapes were the main source of "unproven" storms:
|
|
19
|
+
- `runner > log 2>&1; echo "exit=$?"`
|
|
20
|
+
- `runner | tail; echo "exit=${PIPESTATUS[0]}"` (bash), or `${pipestatus[1]}` (zsh)
|
|
21
|
+
|
|
22
|
+
They now count only when all of the following hold:
|
|
23
|
+
- the echo comes immediately after the only runner in the command;
|
|
24
|
+
- the status variable names the runner's own stage;
|
|
25
|
+
- the last stdout line exactly matches the echo with a number.
|
|
26
|
+
|
|
27
|
+
Anything else stays unproven, and a printed non-zero status is a real failure. The approach follows OpenHands and SWE-ReX, which echo `$?` after every command.
|
|
28
|
+
- **sensitive-data-guard false positives (#4).** These were blocked before and are allowed now:
|
|
29
|
+
- grep/rg pattern operands (`grep "process\.env"`) are no longer treated as files;
|
|
30
|
+
- quoted separators (`grep -E "release|history"`) no longer split the command;
|
|
31
|
+
- quoted-heredoc bodies are treated as literal text.
|
|
32
|
+
|
|
33
|
+
Real secret reads are still denied: `-f`/`--file=`, unquoted heredoc substitutions, and `.env` operands.
|
|
34
|
+
- **No "ownership UNKNOWN" advisory beside a closed run cursor (#5).** About 2,500 advisories came from `Phase: done` cursors.
|
|
35
|
+
- **Safe Patch stops talking on success (#16, #10).** The `[ukit-safe-patch] OK: …` systemMessage appeared after every clean edit (798 field messages) and is now silent, since a hook that succeeds should exit 0 with no output. Edits to OS temp scratch files (`/tmp`, `/private/tmp`, `os.tmpdir()`, `$TMPDIR`) no longer raise "out-of-project edit target" (157 messages). Other out-of-project targets are still reported, and a project that itself lives under a temp dir gets no exemption.
|
|
36
|
+
- **Output compression keeps ranked tables intact (#11).** A count-sorted table is now passed through raw, up to 6k tokens. Before, the line picker reordered such a table and dropped its middle. rtk-ai/rtk#582 reports the same lesson.
|
|
37
|
+
|
|
38
|
+
## 3.5.13 - 2026-11-20
|
|
39
|
+
|
|
40
|
+
Memory-safe test design and execution. A field Python/Sanic backend (~790 files, ~10k tests) grew one pytest process to ~5 GB RSS and swapped for hours; adding pytest-xdist workers made it worse. The cause was a per-test leak in app fixtures, not runner speed. UKit now helps agents write tests that do not leak and run large suites without swapping. No new command: the skills route to the runner. This release also ships the 3.5.12 Stop-gate fix, which reached git and GitHub but not npm.
|
|
41
|
+
|
|
42
|
+
- **Full-suite runner `.claude/ukit/index/test-batches.mjs`** (Node builtins only; pytest, vitest and jest adapters in `test-batches-adapters.mjs`, with a documented adapter contract for other frameworks).
|
|
43
|
+
- `run` discovers tests natively (pytest `--collect-only`, `vitest list`, `jest --listTests`) and writes a deterministic manifest. It then runs whole-file batches one at a time, each in a fresh process (`--batch-files`, `--batch-size` cases).
|
|
44
|
+
- Every file runs. RAM thresholds only flag heavy batches. When free memory is critically low, it waits a bounded time and then stops as INCOMPLETE.
|
|
45
|
+
- Verdicts: PASS 0, FAIL 1, INCOMPLETE 2 (crash, timeout, collection error, unaccounted ids, deferred batches), UNSAFE 3 (a `--guard` real-data path changed). Only PASS satisfies a release gate.
|
|
46
|
+
- Each batch records a log, duration, ms/test and peak RSS (`/usr/bin/time`: bytes on macOS, KiB on Linux; pytest `ru_maxrss` fallback; unknown is never 0). A multi-file batch peak is never attributed to one file; measured-heavy files run solo in later plans.
|
|
47
|
+
- Leftover child processes are reaped. Temp dirs and the injected plugin are removed, and only the newest 5 runs are kept.
|
|
48
|
+
- **Other modes.**
|
|
49
|
+
- `plan` shows the inventory and batches without running anything.
|
|
50
|
+
- `audit [--changed]` checks case budgets, including parametrize expansion (unit 40 · integration 20 · app-heavy 10 · e2e 10 per file, configurable in `.ukit/storage/test-batches.json`). It also flags fixtures without teardown, app-per-test registries, real data paths and timer leaks. These are errors only on new files; legacy files are reported, never rewritten.
|
|
51
|
+
- `detect-leaks --file` reports in-process RSS/heap growth, and `--resource-warnings` turns on pytest `-W error::ResourceWarning`.
|
|
52
|
+
- **Skills and agents.**
|
|
53
|
+
- testing-quality gains "Writing Memory-Safe Tests" and "Memory-Safe Full Suite", plus a `memory-safe-tests.md` reference covering layout planning, fixture lifecycle, test data safety, leak diagnosis and cleanup. The skill now installs as a directory.
|
|
54
|
+
- code-review gains test-diff rules: missing fixture teardown or per-test app registries are P1; reachable real data is P0/P1; skipping tests to save RAM is P1.
|
|
55
|
+
- The Claude and omp `code-reviewer` and `feature-implementer` agents carry the same rules.
|
|
56
|
+
- **Memory.** A seed-once `docs/memory/testing.md` (never overwritten; added to existing projects by `ukit update`) lets `ukit memory recall` surface these patterns on test tasks.
|
|
57
|
+
- Discovery failures name a missing local framework install instead of a bare npx error.
|
|
58
|
+
|
|
59
|
+
## 3.5.12 - 2026-11-20
|
|
60
|
+
|
|
61
|
+
The Stop gate blocked every stop on a heavily loaded machine. On a field host with load average 22-40, starting any process took about 500ms. The completion evaluator's 6 git calls therefore took about 6s, against a fixed 3s slice. Every Stop was reported as "evaluator could not finish" until the breaker released it after 7 tries. No new command.
|
|
62
|
+
|
|
63
|
+
- **Evaluator slice scales with the deadline.** `resolveLedgerBudgetMs` now gives the evaluator `deadline - 3000ms` (at least 3000ms, at most 8000ms), so the shipped deadline gives 7000ms. Short deadlines keep their old slices.
|
|
64
|
+
- **Longer Stop wrapper deadline.** `completion-gate.sh` arms 10000ms (was 5000ms); the chain-runner slot is `:16` (was `:12`), and the host Stop timeout is 26s (was 22s) so it keeps the 10s margin over the chain. omp is unaffected: its bridge runs the evaluator in-process with no time slice.
|
|
65
|
+
- Still fail-closed: a real overrun still blocks and still feeds the same breaker.
|
|
66
|
+
|
|
5
67
|
## 3.5.11 - 2026-11-19
|
|
6
68
|
|
|
7
69
|
Hotfix for 3.5.10 output compression. Since 3.5.10 the summary replaces the Bash output on Claude Code, and the replacement also hit commands whose output *is* the content: `sed -n 25,120p file` returned ten mangled lines instead of the code, and chained status commands (`node -p …; npm view …; git status`) lost whole sections.
|
|
@@ -914,6 +914,17 @@ entries:
|
|
|
914
914
|
load_policy: on-demand
|
|
915
915
|
validation: [manual]
|
|
916
916
|
archive_policy: never
|
|
917
|
+
- id: tpl-docs-memory-testing
|
|
918
|
+
path: template_project/docs/memory/testing.md
|
|
919
|
+
class: canonical
|
|
920
|
+
audience: [agent, maintainer]
|
|
921
|
+
owner: product
|
|
922
|
+
source_of_truth: template_project/docs/memory/testing.md
|
|
923
|
+
merge_strategy: skip
|
|
924
|
+
load_policy: on-demand
|
|
925
|
+
validation: [manual]
|
|
926
|
+
archive_policy: never
|
|
927
|
+
notes: '3.5.13: seed-once default testing patterns (memory-safe tests) so memory recall surfaces them on test tasks; user-owned after install'
|
|
917
928
|
- id: tpl-docs-worklog
|
|
918
929
|
path: template_project/docs/WORKLOG.md
|
|
919
930
|
class: canonical
|
|
@@ -44,6 +44,19 @@ items:
|
|
|
44
44
|
packs:
|
|
45
45
|
- core
|
|
46
46
|
|
|
47
|
+
# Default testing patterns for memory recall (docs/memory/*.md is a recall source). Seed-once:
|
|
48
|
+
# created only when absent (new installs AND updates of existing projects), never overwritten.
|
|
49
|
+
- id: docs-memory-testing
|
|
50
|
+
type: config
|
|
51
|
+
sourceTemplate: docs/memory/testing.md
|
|
52
|
+
targetPath: docs/memory/testing.md
|
|
53
|
+
requires: []
|
|
54
|
+
mergeStrategy: skip
|
|
55
|
+
variables: []
|
|
56
|
+
enabledByDefault: true
|
|
57
|
+
packs:
|
|
58
|
+
- core
|
|
59
|
+
|
|
47
60
|
- id: runtime-readme
|
|
48
61
|
type: config
|
|
49
62
|
sourceTemplate: ukit/README.md
|
|
@@ -474,8 +487,8 @@ items:
|
|
|
474
487
|
|
|
475
488
|
- id: testing-quality-skill
|
|
476
489
|
type: skill
|
|
477
|
-
sourceTemplate: .claude/skills/testing-quality
|
|
478
|
-
targetPath: .claude/skills/testing-quality
|
|
490
|
+
sourceTemplate: .claude/skills/testing-quality
|
|
491
|
+
targetPath: .claude/skills/testing-quality
|
|
479
492
|
requires:
|
|
480
493
|
- core-skill-delivery
|
|
481
494
|
mergeStrategy: overwrite_with_backup
|
|
@@ -1701,6 +1714,30 @@ items:
|
|
|
1701
1714
|
packs:
|
|
1702
1715
|
- core
|
|
1703
1716
|
|
|
1717
|
+
# Memory-safe test execution + test-design audit (testing-quality skill § Memory-Safe Full Suite).
|
|
1718
|
+
- id: ukit-test-batches-script
|
|
1719
|
+
type: config
|
|
1720
|
+
sourceTemplate: .claude/ukit/index/test-batches.mjs
|
|
1721
|
+
targetPath: .claude/ukit/index/test-batches.mjs
|
|
1722
|
+
requires:
|
|
1723
|
+
- ukit-test-batches-adapters-script
|
|
1724
|
+
mergeStrategy: overwrite_with_backup
|
|
1725
|
+
variables: []
|
|
1726
|
+
enabledByDefault: true
|
|
1727
|
+
packs:
|
|
1728
|
+
- core
|
|
1729
|
+
|
|
1730
|
+
- id: ukit-test-batches-adapters-script
|
|
1731
|
+
type: config
|
|
1732
|
+
sourceTemplate: .claude/ukit/index/test-batches-adapters.mjs
|
|
1733
|
+
targetPath: .claude/ukit/index/test-batches-adapters.mjs
|
|
1734
|
+
requires: []
|
|
1735
|
+
mergeStrategy: overwrite_with_backup
|
|
1736
|
+
variables: []
|
|
1737
|
+
enabledByDefault: true
|
|
1738
|
+
packs:
|
|
1739
|
+
- core
|
|
1740
|
+
|
|
1704
1741
|
- id: ukit-ci-check-script
|
|
1705
1742
|
type: config
|
|
1706
1743
|
sourceTemplate: .claude/ukit/index/ci-check.mjs
|
package/package.json
CHANGED
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* field-replay.mjs — debug-round replay harness
|
|
4
|
+
* (docs/plans/2026-11-20-field-telemetry-fix-program.md, "Debug-round protocol").
|
|
5
|
+
*
|
|
6
|
+
* Extracts historical PreToolUse Bash commands (assistant tool_use blocks,
|
|
7
|
+
* name "Bash", input.command) from Claude Code transcripts and replays each
|
|
8
|
+
* one through the CURRENT template hook, in a temp project with an empty
|
|
9
|
+
* .ukit/storage/config.json. Reports block counts, block-label classes and
|
|
10
|
+
* sample commands per class (redacted + truncated; values never printed).
|
|
11
|
+
*
|
|
12
|
+
* Usage: node scripts/audit/field-replay.mjs [--projects-dir DIR] [--match SUBSTR]...
|
|
13
|
+
* [--since YYYY-MM-DD] [--limit N] [--hook sensitive] [--mode inproc|cli] [--json]
|
|
14
|
+
*
|
|
15
|
+
* Hooks are registered in HOOKS below; add block-dangerous / execution-ledger
|
|
16
|
+
* (PostToolUse) by adding an entry with its own payload builder + verdict parser.
|
|
17
|
+
*/
|
|
18
|
+
import fs from 'node:fs';
|
|
19
|
+
import fsp from 'node:fs/promises';
|
|
20
|
+
import os from 'node:os';
|
|
21
|
+
import path from 'node:path';
|
|
22
|
+
import readline from 'node:readline';
|
|
23
|
+
import { spawn } from 'node:child_process';
|
|
24
|
+
import { fileURLToPath, pathToFileURL } from 'node:url';
|
|
25
|
+
import { parseArgs, listTranscripts, normalizeMessage, redact, truncate } from './field-telemetry.mjs';
|
|
26
|
+
|
|
27
|
+
const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
28
|
+
const HOOK_DIR = path.join(REPO, 'template_project', '.claude', 'hooks');
|
|
29
|
+
|
|
30
|
+
const bashPreToolUse = (command) => ({ hook_event_name: 'PreToolUse', tool_name: 'Bash', tool_input: { command } });
|
|
31
|
+
const stderrLabels = (stderr) =>
|
|
32
|
+
String(stderr || '')
|
|
33
|
+
.split('\n')
|
|
34
|
+
.filter((l) => l.startsWith(' - '))
|
|
35
|
+
.map((l) => l.slice(4));
|
|
36
|
+
|
|
37
|
+
export const HOOKS = {
|
|
38
|
+
sensitive: {
|
|
39
|
+
file: path.join(HOOK_DIR, 'sensitive-data-guard.mjs'),
|
|
40
|
+
event: 'PreToolUse',
|
|
41
|
+
buildPayload: bashPreToolUse,
|
|
42
|
+
labels: stderrLabels,
|
|
43
|
+
},
|
|
44
|
+
// Future: 'dangerous' → block-dangerous.mjs (same payload shape, labels from stderr/stdout);
|
|
45
|
+
// 'ledger' → execution-ledger PostToolUse (needs tool_response from the transcript).
|
|
46
|
+
};
|
|
47
|
+
|
|
48
|
+
// --- command extraction --------------------------------------------------------
|
|
49
|
+
export function extractBashCommands(line, out, { since } = {}) {
|
|
50
|
+
if (!line.includes('"tool_use"') || !line.includes('"Bash"')) return;
|
|
51
|
+
let o;
|
|
52
|
+
try {
|
|
53
|
+
o = JSON.parse(line);
|
|
54
|
+
} catch {
|
|
55
|
+
return;
|
|
56
|
+
}
|
|
57
|
+
if (o?.type !== 'assistant' || !Array.isArray(o.message?.content)) return;
|
|
58
|
+
if (since && typeof o.timestamp === 'string' && o.timestamp.slice(0, 10) < since) return;
|
|
59
|
+
for (const c of o.message.content) {
|
|
60
|
+
if (c && c.type === 'tool_use' && c.name === 'Bash' && typeof c.input?.command === 'string' && c.input.command.trim()) {
|
|
61
|
+
out.set(c.input.command, (out.get(c.input.command) || 0) + 1);
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
async function collectCommands(opts) {
|
|
67
|
+
const files = await listTranscripts(opts);
|
|
68
|
+
const cmds = new Map();
|
|
69
|
+
for (const f of files) {
|
|
70
|
+
const rl = readline.createInterface({ input: fs.createReadStream(f, { encoding: 'utf8' }), crlfDelay: Infinity });
|
|
71
|
+
for await (const line of rl) extractBashCommands(line, cmds, opts);
|
|
72
|
+
}
|
|
73
|
+
return { files: files.length, commands: [...cmds.keys()].slice(-opts.limit) };
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
// --- runners ---------------------------------------------------------------------
|
|
77
|
+
async function makeTempProject() {
|
|
78
|
+
const root = await fsp.mkdtemp(path.join(os.tmpdir(), 'ukit-replay-'));
|
|
79
|
+
await fsp.mkdir(path.join(root, '.ukit', 'storage'), { recursive: true });
|
|
80
|
+
await fsp.writeFile(path.join(root, '.ukit', 'storage', 'config.json'), '{}\n');
|
|
81
|
+
return root;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function hookEnv(projectRoot) {
|
|
85
|
+
// Minimal env: never forward the caller's secrets into the replayed hook.
|
|
86
|
+
return { PATH: process.env.PATH || '', HOME: projectRoot, PROJECT_ROOT: projectRoot, UKIT_HOOK_DEADLINE_MS: '10000' };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
async function runInproc(hook, mod, payload, projectRoot) {
|
|
90
|
+
const v = await mod.runHook({ payload, payloadText: JSON.stringify(payload), projectRoot, env: hookEnv(projectRoot), deadlineMs: 10000 });
|
|
91
|
+
return { code: v?.code ?? 0, stdout: v?.stdout || '', stderr: v?.stderr || '' };
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
function runCli(hook, payload, projectRoot, tmpDir, idx) {
|
|
95
|
+
return new Promise((resolve) => {
|
|
96
|
+
const inputFile = path.join(tmpDir, `in-${idx}.json`);
|
|
97
|
+
fs.writeFileSync(inputFile, JSON.stringify(payload));
|
|
98
|
+
const child = spawn(process.execPath, [hook.file], { env: { ...hookEnv(projectRoot), INPUT_FILE: inputFile }, cwd: projectRoot, stdio: ['ignore', 'pipe', 'pipe'] });
|
|
99
|
+
let stdout = '';
|
|
100
|
+
let stderr = '';
|
|
101
|
+
child.stdout.on('data', (d) => (stdout += d));
|
|
102
|
+
child.stderr.on('data', (d) => (stderr += d));
|
|
103
|
+
child.on('close', (code) => {
|
|
104
|
+
fs.rmSync(inputFile, { force: true });
|
|
105
|
+
resolve({ code: code ?? 1, stdout, stderr });
|
|
106
|
+
});
|
|
107
|
+
});
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
// --- aggregation -------------------------------------------------------------------
|
|
111
|
+
export function classifyResults(results, hook, { samples = 40 } = {}) {
|
|
112
|
+
const classes = new Map();
|
|
113
|
+
let blocked = 0;
|
|
114
|
+
for (const { command, verdict } of results) {
|
|
115
|
+
if (verdict.code !== 2) continue;
|
|
116
|
+
blocked += 1;
|
|
117
|
+
const labels = hook.labels(verdict.stderr);
|
|
118
|
+
const keys = labels.length ? [...new Set(labels.map((l) => normalizeMessage(l.replace(/:\s.*$/, ''))))] : ['(no label) ' + normalizeMessage(verdict.stderr.split('\n')[0] || 'exit 2')];
|
|
119
|
+
for (const k of keys) {
|
|
120
|
+
let c = classes.get(k);
|
|
121
|
+
if (!c) classes.set(k, (c = { cls: k, count: 0, samples: [] }));
|
|
122
|
+
c.count += 1;
|
|
123
|
+
if (c.samples.length < samples) c.samples.push(truncate(redact(command), 200));
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
return { blocked, classes: [...classes.values()].sort((a, b) => b.count - a.count || a.cls.localeCompare(b.cls)) };
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
export async function main(argv = process.argv.slice(2)) {
|
|
130
|
+
const opts = parseArgs(argv);
|
|
131
|
+
if (opts.help) {
|
|
132
|
+
process.stdout.write('Usage: field-replay.mjs [--projects-dir DIR] [--match SUBSTR]... [--since YYYY-MM-DD] [--limit N] [--hook sensitive] [--mode inproc|cli] [--json]\n');
|
|
133
|
+
return 0;
|
|
134
|
+
}
|
|
135
|
+
const hook = HOOKS[opts.hook];
|
|
136
|
+
if (!hook) throw new Error(`unknown --hook ${opts.hook}; known: ${Object.keys(HOOKS).join(', ')}`);
|
|
137
|
+
const t0 = Date.now();
|
|
138
|
+
const { files, commands } = await collectCommands(opts);
|
|
139
|
+
const projectRoot = await makeTempProject();
|
|
140
|
+
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'ukit-replay-in-'));
|
|
141
|
+
const results = [];
|
|
142
|
+
let errors = 0;
|
|
143
|
+
try {
|
|
144
|
+
const mod = opts.mode === 'cli' ? null : await import(pathToFileURL(hook.file).href);
|
|
145
|
+
if (mod && typeof mod.runHook !== 'function') throw new Error(`${hook.file} has no exported runHook; use --mode cli`);
|
|
146
|
+
const concurrency = opts.mode === 'cli' ? 8 : 1;
|
|
147
|
+
for (let i = 0; i < commands.length; i += concurrency) {
|
|
148
|
+
const batch = commands.slice(i, i + concurrency);
|
|
149
|
+
const verdicts = await Promise.all(
|
|
150
|
+
batch.map((command, j) => {
|
|
151
|
+
const payload = hook.buildPayload(command);
|
|
152
|
+
return (mod ? runInproc(hook, mod, payload, projectRoot) : runCli(hook, payload, projectRoot, tmpDir, i + j)).catch((e) => {
|
|
153
|
+
errors += 1;
|
|
154
|
+
return { code: -1, stdout: '', stderr: String(e && e.message) };
|
|
155
|
+
});
|
|
156
|
+
}),
|
|
157
|
+
);
|
|
158
|
+
batch.forEach((command, j) => results.push({ command, verdict: verdicts[j] }));
|
|
159
|
+
}
|
|
160
|
+
} finally {
|
|
161
|
+
await fsp.rm(projectRoot, { recursive: true, force: true });
|
|
162
|
+
await fsp.rm(tmpDir, { recursive: true, force: true });
|
|
163
|
+
}
|
|
164
|
+
const rep = classifyResults(results, hook);
|
|
165
|
+
const meta = { hook: opts.hook, mode: opts.mode, files, since: opts.since, replayed: commands.length, blocked: rep.blocked, errors, ms: Date.now() - t0 };
|
|
166
|
+
if (opts.json) {
|
|
167
|
+
process.stdout.write(`${JSON.stringify({ meta, classes: rep.classes }, null, 2)}\n`);
|
|
168
|
+
} else {
|
|
169
|
+
const L = [`field-replay [${meta.hook}/${meta.mode}]: ${files} transcript(s), since ${opts.since || 'all'}, replayed ${meta.replayed} unique command(s), blocked ${meta.blocked}, errors ${errors}, ${meta.ms} ms`];
|
|
170
|
+
for (const c of rep.classes) {
|
|
171
|
+
L.push('', `## [${c.count}] ${c.cls}`);
|
|
172
|
+
for (const s of c.samples) L.push(` - ${s}`);
|
|
173
|
+
}
|
|
174
|
+
process.stdout.write(`${L.join('\n')}\n`);
|
|
175
|
+
}
|
|
176
|
+
return 0;
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
const isDirect = (() => {
|
|
180
|
+
try {
|
|
181
|
+
return process.argv[1] && fs.realpathSync(process.argv[1]) === fs.realpathSync(fileURLToPath(import.meta.url));
|
|
182
|
+
} catch {
|
|
183
|
+
return false;
|
|
184
|
+
}
|
|
185
|
+
})();
|
|
186
|
+
if (isDirect) {
|
|
187
|
+
main().then(
|
|
188
|
+
(code) => process.exit(code),
|
|
189
|
+
(err) => {
|
|
190
|
+
process.stderr.write(`field-replay: ${err && err.message ? err.message : err}\n`);
|
|
191
|
+
process.exit(1);
|
|
192
|
+
},
|
|
193
|
+
);
|
|
194
|
+
}
|