@feigi/fleet-ctl 3.19.3 → 3.19.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/commands/review-and-fix.md +2 -2
- package/package.json +1 -1
- package/scripts/dispatch-block-golden-prose.test.mjs +3 -2
- package/scripts/fix-applier-result-file-prose.test.mjs +42 -9
- package/scripts/recipe-prove.mjs +24 -5
- package/scripts/recipe-prove.test.mjs +67 -7
- package/scripts/review-core-cwd-isolation.test.mjs +1 -2
- package/scripts/review-core-shared-test-run.test.mjs +44 -7
- package/scripts/review-core-unrun.test.mjs +48 -39
- package/scripts/review-core.mjs +2 -17
- package/scripts/review-host-fixture.mjs +1 -3
- package/skills/run-team/SKILL.md +10 -9
|
@@ -44,7 +44,7 @@ Three claims, none substituting for another: **run-binding** (this green belongs
|
|
|
44
44
|
```bash
|
|
45
45
|
R=<scratch>/review-<pr>.json
|
|
46
46
|
jq '{pr, head, resume, testEnvironment, dimensionsRun, dimensionsUnrun, cwdAudit, counts}' "$R" # the digest
|
|
47
|
-
jq -r '.testEnvironment' "$R" # what
|
|
47
|
+
jq -r '.testEnvironment' "$R" # what the shared test run is evidence about
|
|
48
48
|
jq '.survived' "$R" # apply candidates, every one
|
|
49
49
|
jq '[.unverified[] | select(.refutersDispatched > 0)]' "$R" # refuters crashed, twice: defer, naming the crash
|
|
50
50
|
jq '[.unverified[] | select(.refutersDispatched == 0)]' "$R" # suggestions, 0-refuter budget: step 2's scope split
|
|
@@ -53,7 +53,7 @@ jq '.refuted' "$R" # refuted: your
|
|
|
53
53
|
|
|
54
54
|
**Every finding in the file is yours to rule on, not the ones that read sharpest.** The file is the only complete copy a member can reach — the specialists' transcripts exist, but nothing you hold addresses one — so a finding you never read is one nobody rules on. `counts` is the check that you read them all: your apply/defer split covers every `survived` and `unverified` finding it counts, and your disposition record (step 2) carries an entry for each `refuted` one you reversed, with the evidence in its `reason`, which the controller copies to the ledger's `ruled` line.
|
|
55
55
|
|
|
56
|
-
**Read `testEnvironment` and `cwdAudit` before you act on any
|
|
56
|
+
**Read `testEnvironment` and `cwdAudit` before you act on any result of the shared test run.** `testEnvironment` says whether the snapshot the specialists ran in was verified as the reviewed tree: verified, a red there is a fact about the tree; UNVERIFIED, tests that ask git what ships decline or fail for the environment, so a red or a short count there is not a finding until you reproduce it in the worktree, and not a reason to rule a dimension unrun either. `cwdAudit` is each specialist's audit of the checkout it inherited, `{dimension, state, line}`: `dirty` and `missing` are equally loud, because either names a checkout a specialist may have written to (#1433, #1673). Check it with `git -C <that path> status --porcelain -uall` — the path is in a `dirty` entry's `line`, and a `missing` entry's is the main checkout, the parent of `git rev-parse --path-format=absolute --git-common-dir` — and report what it printed; never revert content you did not write. A `dimensionsUnrun` entry is a dimension that did not cover its ground, its `reason` naming which failure: name it unrun in your report, because an absence of findings is not coverage, and re-run nothing — the review already re-dispatched every crashed specialist once, in-run.
|
|
57
57
|
|
|
58
58
|
**Scan the findings against each other for MUTUAL EXCLUSION, and rule before the conflicting change reaches the tree.** Refuters are blind to their siblings, so two can approve changes that cannot both land — one preserving a comment block another's change deletes, one keeping a binding another collapses. Two runs produced it: PR #332, and PR #404, where a refuter proved a variable collapse behavior-equivalent over 844 cases while its own blast-radius scan named the two comment blocks a *different* refuter had just established as guarding distinct pinned hazards. You hold every finding, so only you can see the pair — and the scan's two inputs arrive at different times, so run it twice. **Before you apply anything**, group the findings by `file:line` and by whether one's fix undoes another's; that is in the findings' own text. **Then as each refuter you dispatch reports**, read its blast radius against what you have already ruled on. **A conflicting pair goes to a SINGLE refuter, briefed with both claims** — that is what settled #332's, one refuter ruling *"B wins: A's premise is false"* where two independent refuters would each have approved their own side. Defer only when no single refuter can settle it, with the equivalence evidence recorded in the filed issue so the work is not redone. Never let both reach the tree and hope the diff coheres.
|
|
59
59
|
|
package/package.json
CHANGED
|
@@ -522,8 +522,9 @@ const REVIEWER_BLOCKS = [
|
|
|
522
522
|
"the rest of the recipes and every ruling on the findings: both",
|
|
523
523
|
"mutual-exclusion scans, the suggested-fix re-derivation, `refuted=false` ≠",
|
|
524
524
|
"apply, per-site measurement for a sibling-site extension, and `testEnvironment`",
|
|
525
|
-
"and `cwdAudit` before any
|
|
526
|
-
"nobody upstream has made one, and the controller never
|
|
525
|
+
"and `cwdAudit` before any result of the shared test run. Every one of those",
|
|
526
|
+
"rulings is yours — nobody upstream has made one, and the controller never",
|
|
527
|
+
"reads the findings.",
|
|
527
528
|
"",
|
|
528
529
|
"**Every factual claim your diff restates needs a settling command run",
|
|
529
530
|
"against the tree first** — the issue body is a lead, never a citation.",
|
|
@@ -3,12 +3,13 @@
|
|
|
3
3
|
// and owns every per-PR ruling the controller used to make before dispatch —
|
|
4
4
|
// both mutual-exclusion scans, the suggested-fix re-derivation, `refuted=false`
|
|
5
5
|
// ≠ apply, per-site measurement for a sibling-site extension, and reading
|
|
6
|
-
// `testEnvironment`/`cwdAudit` before acting on a
|
|
7
|
-
// review-and-fix.md's `## The review result file`, which
|
|
8
|
-
// fix-applier reads. The controller-side copies in
|
|
9
|
-
// be retired once the controller stops making
|
|
10
|
-
// retirement that loses the destination copy too
|
|
11
|
-
// by nobody, with every pin that read SKILL.md
|
|
6
|
+
// `testEnvironment`/`cwdAudit` before acting on a result of the shared test
|
|
7
|
+
// run. Their new home is review-and-fix.md's `## The review result file`, which
|
|
8
|
+
// is the copy the fix-applier reads. The controller-side copies in
|
|
9
|
+
// run-team/SKILL.md are due to be retired once the controller stops making
|
|
10
|
+
// these rulings (§ 8), and a retirement that loses the destination copy too
|
|
11
|
+
// would leave the rulings made by nobody, with every pin that read SKILL.md
|
|
12
|
+
// simply deleted alongside it.
|
|
12
13
|
//
|
|
13
14
|
// Two halves. The recipes half is not prose: a `jq` path naming a field the
|
|
14
15
|
// result does not carry prints `null` at exit 0, which reads exactly like an
|
|
@@ -19,11 +20,12 @@ import { test } from "node:test";
|
|
|
19
20
|
import assert from "node:assert/strict";
|
|
20
21
|
import { readFileSync } from "node:fs";
|
|
21
22
|
import { join } from "node:path";
|
|
22
|
-
import { between, phrase } from "./prose-pin.mjs";
|
|
23
|
+
import { between, paragraph, phrase } from "./prose-pin.mjs";
|
|
23
24
|
import { DIGEST_KEYS } from "./review-core.mjs";
|
|
24
25
|
|
|
25
26
|
const REPO = join(import.meta.dirname, "..");
|
|
26
27
|
const DOC = readFileSync(join(REPO, "commands", "review-and-fix.md"), "utf8");
|
|
28
|
+
const RUN_TEAM = readFileSync(join(REPO, "skills", "run-team", "SKILL.md"), "utf8");
|
|
27
29
|
const RESULT_KEYS = new Set([...DIGEST_KEYS, "snapshot", "survived", "refuted", "unverified"]);
|
|
28
30
|
|
|
29
31
|
const section = () => between(DOC, "\n## The review result file\n", "\n## Specialists\n", "review-and-fix.md's result-file section");
|
|
@@ -92,9 +94,40 @@ test("the fix-applier owns refuted=false ≠ apply, and per-site measurement for
|
|
|
92
94
|
);
|
|
93
95
|
});
|
|
94
96
|
|
|
95
|
-
test("the fix-applier reads testEnvironment and cwdAudit before acting on any
|
|
96
|
-
const p = ruling("**Read `testEnvironment` and `cwdAudit` before you act on any
|
|
97
|
+
test("the fix-applier reads testEnvironment and cwdAudit before acting on any result of the shared test run, and re-runs nothing", () => {
|
|
98
|
+
const p = ruling("**Read `testEnvironment` and `cwdAudit` before you act on any result of the shared test run.**");
|
|
97
99
|
assert.match(p, phrase("status --porcelain -uall"), "the cwdAudit check lost its explicit untracked mode");
|
|
98
100
|
assert.match(p, phrase("never revert content you did not write"));
|
|
99
101
|
assert.match(p, phrase("re-run nothing"), "a dimensionsUnrun entry now reads as an instruction to re-run the review");
|
|
100
102
|
});
|
|
103
|
+
|
|
104
|
+
// The two run-team/SKILL.md sentences that hand the fix-applier the same ruling
|
|
105
|
+
// were reworded off the specialists' retired `test_run` field (#2374), and
|
|
106
|
+
// nothing read either one. They go with the rest of the SKILL.md copies (§ 8).
|
|
107
|
+
test("run-team's digest paragraph tells the controller the fix-applier reads testEnvironment before acting on any count from the shared test run", () => {
|
|
108
|
+
const p = paragraph(
|
|
109
|
+
RUN_TEAM,
|
|
110
|
+
"`testEnvironment` says what the review's shared test run is evidence about",
|
|
111
|
+
"run-team/SKILL.md's testEnvironment digest paragraph",
|
|
112
|
+
{ emphasisTolerant: true },
|
|
113
|
+
);
|
|
114
|
+
assert.match(
|
|
115
|
+
p,
|
|
116
|
+
phrase("fix-applier reads it before acting on any count from the shared test run"),
|
|
117
|
+
"the digest paragraph no longer says the fix-applier reads testEnvironment before acting on a count from the shared test run",
|
|
118
|
+
);
|
|
119
|
+
});
|
|
120
|
+
|
|
121
|
+
test("run-team's fix-applier dispatch hands it testEnvironment and cwdAudit before any result of the shared test run", () => {
|
|
122
|
+
const p = between(
|
|
123
|
+
RUN_TEAM,
|
|
124
|
+
"\n**Then dispatch a fix-applier** — on `DISPATCH fix-pr PR#<M>`",
|
|
125
|
+
"\n**A `DISPATCH fix-pr PR#<M>` on a conflict hold**",
|
|
126
|
+
"run-team/SKILL.md's fix-applier dispatch paragraph",
|
|
127
|
+
);
|
|
128
|
+
assert.match(
|
|
129
|
+
p,
|
|
130
|
+
phrase("`testEnvironment`/`cwdAudit` before any result of the shared test run."),
|
|
131
|
+
"the fix-applier dispatch paragraph no longer says to read testEnvironment/cwdAudit before a result of the shared test run",
|
|
132
|
+
);
|
|
133
|
+
});
|
package/scripts/recipe-prove.mjs
CHANGED
|
@@ -45,8 +45,9 @@
|
|
|
45
45
|
// Exit 1: NOT PROVEN — no cache written; the reason on stderr.
|
|
46
46
|
// Exit 2: no verdict on the proof — it could not be attempted (usage, not a
|
|
47
47
|
// repository, origin/main missing, the worktree could not be made), or
|
|
48
|
-
// a filesystem fault stopped it before the cache was settled
|
|
49
|
-
// cache or its temp file
|
|
48
|
+
// a git or filesystem fault stopped it before the cache was settled
|
|
49
|
+
// (a git command could not be started; the cache or its temp file
|
|
50
|
+
// could not be written).
|
|
50
51
|
|
|
51
52
|
import { spawnSync } from "node:child_process";
|
|
52
53
|
import { closeSync, existsSync, mkdirSync, mkdtempSync, openSync, readFileSync, renameSync, rmSync, unlinkSync, writeFileSync } from "node:fs";
|
|
@@ -108,8 +109,16 @@ function parseArgs(argv) {
|
|
|
108
109
|
};
|
|
109
110
|
}
|
|
110
111
|
|
|
112
|
+
// A git that cannot be started is no verdict, as for `sh` below: read as a
|
|
113
|
+
// failed git command, it would surface as whatever that command's failure
|
|
114
|
+
// means to its caller — a missing repository, an unfetched origin/main, a
|
|
115
|
+
// proof that did not hold. Only a spawn that produced no process counts:
|
|
116
|
+
// spawnSync also sets `error` for a git that ran and was killed — ENOBUFS,
|
|
117
|
+
// when its output outgrows the default maxBuffer — and that is a failed git
|
|
118
|
+
// command like any other, never a git that could not be started.
|
|
111
119
|
function git(args, cwd) {
|
|
112
120
|
const r = spawnSync("git", args, { cwd, env: ENV, encoding: "utf8" });
|
|
121
|
+
if (r.error && !r.pid) throw cannot(`could not start git: ${r.error.message}`);
|
|
113
122
|
return { ok: r.status === 0, out: (r.stdout ?? "").replace(/\n$/, ""), err: (r.stderr ?? "").trim() };
|
|
114
123
|
}
|
|
115
124
|
|
|
@@ -231,9 +240,19 @@ function writeCache(cache, recipe, repo) {
|
|
|
231
240
|
// is the usual one; the repo ignores them) makes both git's removal and a
|
|
232
241
|
// plain recursive delete fail, so the fallback makes the tree writable first.
|
|
233
242
|
// The prune always runs: it is what drops the registration of a worktree
|
|
234
|
-
// whose directory is gone.
|
|
243
|
+
// whose directory is gone. A git that cannot be started is reported and
|
|
244
|
+
// skipped here, since this runs in the proof's `finally`.
|
|
235
245
|
function removeWorktree(wt, repo) {
|
|
236
|
-
|
|
246
|
+
const tryGit = (args) => {
|
|
247
|
+
try {
|
|
248
|
+
return git(args, repo).ok;
|
|
249
|
+
} catch (e) {
|
|
250
|
+
if (!(e instanceof Refusal)) throw e;
|
|
251
|
+
process.stderr.write(`${NAME}: skipped git ${args.join(" ")}: ${e.message}\n`);
|
|
252
|
+
return false;
|
|
253
|
+
}
|
|
254
|
+
};
|
|
255
|
+
if (!tryGit(["worktree", "remove", "--force", wt])) {
|
|
237
256
|
spawnSync("chmod", ["-R", "u+w", wt], { stdio: "ignore" });
|
|
238
257
|
try {
|
|
239
258
|
rmSync(wt, { recursive: true, force: true });
|
|
@@ -241,7 +260,7 @@ function removeWorktree(wt, repo) {
|
|
|
241
260
|
process.stderr.write(`${NAME}: could not remove the throwaway worktree ${wt}: ${e.message}\n`);
|
|
242
261
|
}
|
|
243
262
|
}
|
|
244
|
-
|
|
263
|
+
tryGit(["worktree", "prune"]);
|
|
245
264
|
}
|
|
246
265
|
|
|
247
266
|
function main(argv) {
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { test } from "node:test";
|
|
2
2
|
import assert from "node:assert/strict";
|
|
3
3
|
import { execFileSync, spawnSync } from "node:child_process";
|
|
4
|
-
import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
|
|
4
|
+
import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
|
5
5
|
import { join } from "node:path";
|
|
6
6
|
import { tempDir } from "./temp-dir.mjs";
|
|
7
7
|
import { writeExecStub } from "./exec-stub.mjs";
|
|
@@ -371,13 +371,75 @@ test("a proof that cannot be attempted exits 2, distinct from a Recipe that is n
|
|
|
371
371
|
"a refusal at the argument boundary creates no worktree");
|
|
372
372
|
});
|
|
373
373
|
|
|
374
|
+
// Every temp dir the proof made, scanned for the throwaway worktree it names `wt`.
|
|
375
|
+
function assertNoWorktreeLeft(tmp) {
|
|
376
|
+
for (const d of readdirSync(tmp)) {
|
|
377
|
+
assert.deepEqual(readdirSync(join(tmp, d)).filter((n) => n === "wt"), [], `${d} still holds the throwaway worktree`);
|
|
378
|
+
}
|
|
379
|
+
}
|
|
380
|
+
|
|
374
381
|
test("the proof leaves no worktree directory behind in its temp dir", () => {
|
|
375
382
|
const { dir } = repo(MAVEN_FILES);
|
|
376
383
|
const r = prove(dir, MAVEN_PROOF);
|
|
377
384
|
assert.equal(r.status, 0, r.err);
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
385
|
+
assertNoWorktreeLeft(r.tmp);
|
|
386
|
+
});
|
|
387
|
+
|
|
388
|
+
test("a git that cannot be started is named as such, not reported as a missing repository", () => {
|
|
389
|
+
const { dir } = repo(MAVEN_FILES);
|
|
390
|
+
const r = prove(dir, MAVEN_PROOF, { env: { PATH: "/nonexistent" } });
|
|
391
|
+
assert.equal(r.status, 2, r.err);
|
|
392
|
+
assert.match(r.err, /^recipe-prove: could not start git: .*ENOENT/);
|
|
393
|
+
assert.doesNotMatch(r.err, /not a git repository/);
|
|
394
|
+
});
|
|
395
|
+
|
|
396
|
+
// A PATH holding only `sh` and a `git` that execs the real one, so the Install
|
|
397
|
+
// step can take git away by deleting it: every later git call then
|
|
398
|
+
// fails to start, in the proof and in the worktree cleanup alike.
|
|
399
|
+
test("a git that stops starting during the Install step is no verdict, never NOT PROVEN — and the worktree is still removed", () => {
|
|
400
|
+
const realGit = execFileSync("sh", ["-c", "command -v git"], { encoding: "utf8" }).trim();
|
|
401
|
+
const { dir } = repo(MAVEN_FILES);
|
|
402
|
+
const bin = tempDir("recipe-prove-path-");
|
|
403
|
+
writeExecStub(join(bin, "git"), `#!/bin/sh\nexec '${realGit}' "$@"\n`);
|
|
404
|
+
symlinkSync("/bin/sh", join(bin, "sh"));
|
|
405
|
+
const install = `'${process.execPath}' -e 'require("fs").rmSync(process.argv[1])' '${join(bin, "git")}'`;
|
|
406
|
+
const r = prove(dir, ["--install", install, "--test", "true", "--count-line", "x 1", "--test-count", "1"], { env: { PATH: bin } });
|
|
407
|
+
assert.equal(r.status, 2, r.err);
|
|
408
|
+
assert.match(r.err, /could not start git: .*ENOENT/);
|
|
409
|
+
assert.match(r.err, /skipped git worktree remove/);
|
|
410
|
+
assert.doesNotMatch(r.err, /NOT PROVEN/);
|
|
411
|
+
assertNoWorktreeLeft(r.tmp);
|
|
412
|
+
});
|
|
413
|
+
|
|
414
|
+
// A git that started and ran is never "could not start": here its status
|
|
415
|
+
// output outgrows spawnSync's default 1 MiB buffer, which kills it and sets
|
|
416
|
+
// `error` (ENOBUFS) on a process that did run. The tree the Install step left
|
|
417
|
+
// is unreadable, so the proof does not hold — NOT PROVEN, exit 1.
|
|
418
|
+
test("a git whose output outgrows the spawn buffer is a failed command, not a git that could not be started", () => {
|
|
419
|
+
const { dir } = repo(MAVEN_FILES);
|
|
420
|
+
const flood = 'const fs = require("fs"); for (let i = 0; i < 7000; i++) fs.writeFileSync("f".repeat(200) + i, "")';
|
|
421
|
+
const install = `'${process.execPath}' -e '${flood}'`;
|
|
422
|
+
const r = prove(dir, ["--install", install, "--test", "true", "--count-line", "x 1", "--test-count", "1"]);
|
|
423
|
+
assert.equal(r.status, 1, r.err);
|
|
424
|
+
assert.match(r.err, /NOT PROVEN — could not read the tree state in the throwaway worktree/);
|
|
425
|
+
assert.doesNotMatch(r.err, /could not start git/);
|
|
426
|
+
});
|
|
427
|
+
|
|
428
|
+
// A git that starts for the worktree removal and not for the prune after it:
|
|
429
|
+
// the stub deletes itself once `worktree remove` has run. The proof's own
|
|
430
|
+
// outcome is NOT PROVEN (the Install step fails), and the prune that cannot
|
|
431
|
+
// start must not replace it.
|
|
432
|
+
test("a git that stops starting during cleanup does not replace the proof's own NOT PROVEN", () => {
|
|
433
|
+
const realGit = execFileSync("sh", ["-c", "command -v git"], { encoding: "utf8" }).trim();
|
|
434
|
+
const { dir } = repo(MAVEN_FILES);
|
|
435
|
+
const bin = tempDir("recipe-prove-path-");
|
|
436
|
+
const stub = join(bin, "git");
|
|
437
|
+
writeExecStub(stub, `#!/bin/sh\n'${realGit}' "$@"\nrc=$?\nif [ "$2" = remove ]; then /bin/rm -f '${stub}'; fi\nexit $rc\n`);
|
|
438
|
+
symlinkSync("/bin/sh", join(bin, "sh"));
|
|
439
|
+
const r = prove(dir, ["--install", "false", "--test", "true", "--count-line", "x 1", "--test-count", "1"], { env: { PATH: bin } });
|
|
440
|
+
assert.equal(r.status, 1, r.err);
|
|
441
|
+
assert.match(r.err, /NOT PROVEN/);
|
|
442
|
+
assert.match(r.err, /skipped git worktree prune: could not start git/);
|
|
381
443
|
});
|
|
382
444
|
|
|
383
445
|
// Whether the process ignores file modes: root writes into a 0555 directory.
|
|
@@ -393,9 +455,7 @@ test("read-only install output cannot turn a proof that held into a crash", { sk
|
|
|
393
455
|
const r = prove(dir, ["--install", READ_ONLY_INSTALL, ...MAVEN_PROOF.slice(2)]);
|
|
394
456
|
assert.equal(r.status, 0, r.err);
|
|
395
457
|
assert.equal(JSON.parse(readFileSync(cachePath(dir), "utf8")).derivedAt, head);
|
|
396
|
-
|
|
397
|
-
assert.deepEqual(readdirSync(join(r.tmp, d)).filter((n) => n === "wt"), [], `${d} still holds the throwaway worktree`);
|
|
398
|
-
}
|
|
458
|
+
assertNoWorktreeLeft(r.tmp);
|
|
399
459
|
assert.equal(git(dir, "worktree", "list", "--porcelain").split("\n").filter((l) => l.startsWith("worktree ")).length, 1);
|
|
400
460
|
});
|
|
401
461
|
|
|
@@ -25,7 +25,7 @@
|
|
|
25
25
|
// agent must not write to, `pwd` fixes which directory that is, and the
|
|
26
26
|
// `CWD-AUDIT:` line is what makes a CLEAN run say so. An audit reported only
|
|
27
27
|
// when it finds something is indistinguishable from one never run — the same
|
|
28
|
-
// reading `unrunReason` already applies to a
|
|
28
|
+
// reading `unrunReason` already applies to a shared test run reporting nothing, and
|
|
29
29
|
// the reason "four files dirty" was found by chance rather than by a report.
|
|
30
30
|
//
|
|
31
31
|
// WHY THESE PINS RENDER RATHER THAN GREP, and why the slice is the pin: same
|
|
@@ -335,7 +335,6 @@ function fakeReviewHost(scopeSearched) {
|
|
|
335
335
|
dimension: "correctness",
|
|
336
336
|
scope_searched: scopeSearched,
|
|
337
337
|
findings: [],
|
|
338
|
-
test_run: { command: "node --test", tests: 5, pass: 5, fail: 0 },
|
|
339
338
|
};
|
|
340
339
|
throw new Error(`fakeReviewHost: unexpected dispatch ${opts.label}`);
|
|
341
340
|
},
|
|
@@ -18,7 +18,7 @@ import { execFileSync } from "node:child_process";
|
|
|
18
18
|
import { mkdtempSync, mkdirSync, readFileSync, existsSync, rmSync } from "node:fs";
|
|
19
19
|
import { tmpdir } from "node:os";
|
|
20
20
|
import { join } from "node:path";
|
|
21
|
-
import { runReview, DEFAULT_DIMENSIONS,
|
|
21
|
+
import { runReview, DEFAULT_DIMENSIONS, CRASHED_REASON } from "./review-core.mjs";
|
|
22
22
|
import { pipeline, parallel, ARGS, SNAP, review, finding, vote, scriptedHost } from "./review-host-fixture.mjs";
|
|
23
23
|
|
|
24
24
|
const ALL = DEFAULT_DIMENSIONS.map((d) => d.key);
|
|
@@ -229,10 +229,47 @@ test("a crashed dimension is listed once, with its crash reason, beside the shar
|
|
|
229
229
|
assert.equal(result.dimensionsUnrun.find((u) => u.dimension === "types").reason, CRASHED_REASON);
|
|
230
230
|
});
|
|
231
231
|
|
|
232
|
-
//
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
232
|
+
// The shared run alone decides. A specialist whose payload still carries its
|
|
233
|
+
// own `test_run` — from a prompt that predates the field's removal — is
|
|
234
|
+
// ignored: a count that disagrees with the shared run changes no verdict, and
|
|
235
|
+
// leaves no trace in the result. The baseline is the same script with no stray
|
|
236
|
+
// field, so the comparison is the whole result, not a chosen key.
|
|
237
|
+
test("a specialist's stray test_run never changes the result, and the shared run alone decides", async () => {
|
|
238
|
+
const stray = (r, testRun) => ({ ...r, test_run: testRun });
|
|
239
|
+
const cases = [
|
|
240
|
+
// The shared run counted nothing; a stray copy claims a full suite ran.
|
|
241
|
+
["shared tests 0, stray 9999", { exitCode: 0, tests: 0, pass: 0, fail: 0 }, { command: "MARKER-CMD", tests: 9999, pass: 9999, fail: 0 }, {}],
|
|
242
|
+
// The shared run is clean; a stray copy claims nothing ran. Findings on
|
|
243
|
+
// three dimensions put an entry in survived, refuted and unverified.
|
|
244
|
+
[
|
|
245
|
+
"shared clean, stray 0",
|
|
246
|
+
{ exitCode: 0, tests: 5, pass: 5, fail: 0 },
|
|
247
|
+
{ command: "MARKER-CMD", tests: 0 },
|
|
248
|
+
{
|
|
249
|
+
"review:correctness": [review([finding("critical")])],
|
|
250
|
+
"verify:correctness": [vote(false)],
|
|
251
|
+
"review:types": [review([finding("important")])],
|
|
252
|
+
"verify:types": [vote(true)],
|
|
253
|
+
"review:tests": [review([finding("suggestion")])],
|
|
254
|
+
},
|
|
255
|
+
],
|
|
256
|
+
];
|
|
257
|
+
for (const [name, shared, testRun, extra] of cases) {
|
|
258
|
+
const base = script({ "test-run": [shared], ...extra });
|
|
259
|
+
const withStray = Object.fromEntries(
|
|
260
|
+
Object.entries(base).map(([label, seq]) => [label, label.startsWith("review:") ? seq.map((r) => stray(r, testRun)) : seq]),
|
|
261
|
+
);
|
|
262
|
+
const expected = await run(scriptedHost(base).host);
|
|
263
|
+
const actual = await run(scriptedHost(withStray).host);
|
|
264
|
+
assert.deepEqual(actual, expected, `${name}: a stray test_run changed the result`);
|
|
265
|
+
const text = JSON.stringify(actual);
|
|
266
|
+
assert.doesNotMatch(text, /MARKER-CMD|test_run/, `${name}: the stray test_run reached the result`);
|
|
267
|
+
if (shared.tests === 0) {
|
|
268
|
+
assert.deepEqual(actual.dimensionsUnrun.map((u) => u.dimension), ALL, `${name}: not every live dimension is unrun`);
|
|
269
|
+
assert.doesNotMatch(text, /9999/, `${name}: the stray count reached the result`);
|
|
270
|
+
} else {
|
|
271
|
+
assert.deepEqual(actual.dimensionsUnrun, [], `${name}: a stray count of 0 made a dimension unrun`);
|
|
272
|
+
for (const key of ["survived", "refuted", "unverified"]) assert.equal(actual[key].length, 1, `${name}: ${key} holds no finding — this case tests nothing`);
|
|
273
|
+
}
|
|
274
|
+
}
|
|
238
275
|
});
|
|
@@ -292,27 +292,22 @@ test("scope_searched is in the required array, not only in its own description",
|
|
|
292
292
|
const { required } = FINDINGS_SCHEMA;
|
|
293
293
|
assert.ok(Array.isArray(required), "FINDINGS_SCHEMA no longer has a top-level required array — update this test");
|
|
294
294
|
assert.ok(required.includes("scope_searched"), "scope_searched is documented Required and is not in the required array (#139)");
|
|
295
|
-
assert.ok(required.includes("test_run"), "test_run is not required, so an unrun dimension can still return a clean-looking object (#137)");
|
|
296
295
|
});
|
|
297
296
|
|
|
298
|
-
//
|
|
299
|
-
//
|
|
300
|
-
//
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
// and a guessed count is worse than an absent one.
|
|
313
|
-
for (const optional of ["pass", "fail"]) {
|
|
314
|
-
assert.ok(!testRun.required.includes(optional), `${optional} must stay optional — see the comment above this assertion`);
|
|
315
|
-
}
|
|
297
|
+
// The review's ONE shared run is the single record of the test run: a
|
|
298
|
+
// specialist is not asked to copy it, so the payload neither requires nor
|
|
299
|
+
// declares a field for the copy. Both halves pinned together, because under
|
|
300
|
+
// `additionalProperties: false` either one re-added alone is a contract the
|
|
301
|
+
// specialist cannot meet — a required field the schema rejects, or a declared
|
|
302
|
+
// one nothing reads.
|
|
303
|
+
test("FINDINGS_SCHEMA neither requires nor declares a specialist copy of the test run", () => {
|
|
304
|
+
assert.deepEqual(
|
|
305
|
+
[...FINDINGS_SCHEMA.required].sort(),
|
|
306
|
+
["dimension", "findings", "scope_searched"],
|
|
307
|
+
"FINDINGS_SCHEMA's required list changed — a specialist must return exactly these",
|
|
308
|
+
);
|
|
309
|
+
assert.ok(!Object.hasOwn(FINDINGS_SCHEMA.properties, "test_run"), "FINDINGS_SCHEMA declares test_run again — nothing reads a specialist's copy of the shared run");
|
|
310
|
+
assert.equal(FINDINGS_SCHEMA.additionalProperties, false);
|
|
316
311
|
});
|
|
317
312
|
|
|
318
313
|
// The entry, not the call site. `unrunEntries` returns zero entries or one per
|
|
@@ -403,20 +398,34 @@ test("the returned object carries dimensionsUnrun alongside dimensionsRun", () =
|
|
|
403
398
|
assert.match(tail, /dimensionsUnrun/, "the return never surfaces dimensionsUnrun — the classification dies in the script (#137, #138)");
|
|
404
399
|
});
|
|
405
400
|
|
|
406
|
-
// The
|
|
407
|
-
//
|
|
408
|
-
//
|
|
409
|
-
//
|
|
410
|
-
//
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
401
|
+
// The note hands every specialist the shared run and tells it not to run the
|
|
402
|
+
// full command itself — and nothing more about the run: the shared run is the
|
|
403
|
+
// single record of it, so no specialist is told to report or copy it anywhere.
|
|
404
|
+
// Every rendering the note has — usable, no counts (NOT usable), and failing
|
|
405
|
+
// tests for the owner and for a non-owner — is checked, because each appends a
|
|
406
|
+
// different paragraph to the same opening one.
|
|
407
|
+
test("the shared-run note hands over the run and forbids a rerun, and asks for no copy of it in any rendering", () => {
|
|
408
|
+
const usable = { command: "node --test", logPath: "/r/test-run.log", exitCode: 0, tests: 5, pass: 5, fail: 0 };
|
|
409
|
+
const noCounts = { command: "node --test", logPath: "/r/test-run.log", error: "no summary" };
|
|
410
|
+
const failing = { command: "node --test", logPath: "/r/test-run.log", exitCode: 1, tests: 5, pass: 3, fail: 2 };
|
|
411
|
+
for (const [name, run, key, paragraph] of [
|
|
412
|
+
["usable", usable, "tests", null],
|
|
413
|
+
["no counts", noCounts, "tests", /This run is NOT usable: /],
|
|
414
|
+
["failing, owner", failing, "correctness", /filing them is YOUR job in this review/],
|
|
415
|
+
["failing, non-owner", failing, "tests", /the correctness dimension files the finding/],
|
|
415
416
|
]) {
|
|
416
|
-
const note = sharedRunNote(run, "correctness",
|
|
417
|
-
assert.
|
|
418
|
-
assert.match(note,
|
|
419
|
-
assert.match(note,
|
|
417
|
+
const note = sharedRunNote(run, "correctness", key);
|
|
418
|
+
assert.doesNotMatch(note, /test_run/, `${name}: the note still names test_run`);
|
|
419
|
+
assert.match(note, /Do NOT run that full command yourself/, `${name}: the note no longer forbids a rerun`);
|
|
420
|
+
assert.match(note, /\n {2}command: node --test\n/, `${name}: the note lacks the command line`);
|
|
421
|
+
assert.match(note, /\n {2}counts: {2}\S/, `${name}: the note lacks the counts line`);
|
|
422
|
+
assert.match(note, /\n {2}exit: {4}\S/, `${name}: the note lacks the exit line`);
|
|
423
|
+
assert.match(note, /\n {2}log: {5}\/r\/test-run\.log\n/, `${name}: the note lacks the log line`);
|
|
424
|
+
// The opening paragraph ends on the log instruction: nothing after it
|
|
425
|
+
// asks the specialist to report the run.
|
|
426
|
+
const first = note.split(/\n(?=This run is NOT usable|It has )/)[0];
|
|
427
|
+
assert.match(first, /Read the log for anything the counts do not say\.$/, `${name}: the opening paragraph says more after the log instruction`);
|
|
428
|
+
if (paragraph) assert.match(note, paragraph, `${name}: the paragraph this rendering appends is missing`);
|
|
420
429
|
}
|
|
421
430
|
});
|
|
422
431
|
|
|
@@ -437,13 +446,13 @@ test("run-team's Reviewers section documents dimensionsUnrun, not only dimension
|
|
|
437
446
|
});
|
|
438
447
|
|
|
439
448
|
// What the absence of an unrun classification ESTABLISHES is that a suite ran —
|
|
440
|
-
// never that the dimension is covered. `unrunReason` above reads
|
|
441
|
-
// counts and quotes `command` into its message; it never
|
|
442
|
-
//
|
|
443
|
-
//
|
|
444
|
-
//
|
|
445
|
-
//
|
|
446
|
-
//
|
|
449
|
+
// never that the dimension is covered. `unrunReason` above reads the shared
|
|
450
|
+
// run's counts and quotes its `command` into its message; it never judges what
|
|
451
|
+
// that command covers, so a narrower runner reports a non-zero count, is not
|
|
452
|
+
// classified unrun, and reads as covered having validated a fraction of the
|
|
453
|
+
// suite. Comparing that command against the dispatched one was refuted 2-0 and
|
|
454
|
+
// stays refuted (#535) — the doc claiming only what the classifier can see is
|
|
455
|
+
// the remedy, and nothing pinned the word it turns on.
|
|
447
456
|
//
|
|
448
457
|
// The positive pin carries the claim: sliced to the paragraph so a failure
|
|
449
458
|
// prints it rather than the whole 2000-line doc, matched through `phrase` so a
|
package/scripts/review-core.mjs
CHANGED
|
@@ -40,7 +40,7 @@ import { isDigits } from "./arg.mjs";
|
|
|
40
40
|
export const FINDINGS_SCHEMA = {
|
|
41
41
|
type: "object",
|
|
42
42
|
additionalProperties: false,
|
|
43
|
-
required: ["dimension", "scope_searched", "findings"
|
|
43
|
+
required: ["dimension", "scope_searched", "findings"],
|
|
44
44
|
properties: {
|
|
45
45
|
dimension: { type: "string" },
|
|
46
46
|
scope_searched: {
|
|
@@ -48,19 +48,6 @@ export const FINDINGS_SCHEMA = {
|
|
|
48
48
|
description:
|
|
49
49
|
"The exact commands/paths this pass covered. Required so a negative claim is bounded: a grep that found nothing looks identical to a grep never run.",
|
|
50
50
|
},
|
|
51
|
-
test_run: {
|
|
52
|
-
type: "object",
|
|
53
|
-
additionalProperties: false,
|
|
54
|
-
required: ["command", "tests"],
|
|
55
|
-
description:
|
|
56
|
-
"The review's ONE shared test run, copied from your prompt — you do not run the full suite yourself (#2315). Report it even when it failed or produced nothing.",
|
|
57
|
-
properties: {
|
|
58
|
-
command: { type: "string", description: "The shared run's command, verbatim as your prompt states it." },
|
|
59
|
-
tests: { type: "integer", description: "The shared run's test count as your prompt states it; 0 when it states none." },
|
|
60
|
-
pass: { type: "integer" },
|
|
61
|
-
fail: { type: "integer" },
|
|
62
|
-
},
|
|
63
|
-
},
|
|
64
51
|
findings: {
|
|
65
52
|
type: "array",
|
|
66
53
|
items: {
|
|
@@ -536,9 +523,7 @@ still yours to run.
|
|
|
536
523
|
counts: ${countsOf(run) || "none — the run produced no counts"}
|
|
537
524
|
exit: ${run.exitCode ?? "(not reported)"}
|
|
538
525
|
log: ${run.logPath}
|
|
539
|
-
Read the log for anything the counts do not say
|
|
540
|
-
\`test_run\` — its command verbatim and these counts, \`tests: 0\` when it states
|
|
541
|
-
none — never a run of your own.`,
|
|
526
|
+
Read the log for anything the counts do not say.`,
|
|
542
527
|
];
|
|
543
528
|
const why = testRunReason(run);
|
|
544
529
|
if (why)
|
|
@@ -38,16 +38,14 @@ export const SNAP = {
|
|
|
38
38
|
repoVerified: true,
|
|
39
39
|
testCmd: "node --test",
|
|
40
40
|
};
|
|
41
|
-
export const RAN = { command: "node --test", tests: 5, pass: 5, fail: 0 };
|
|
42
41
|
// #2315. The shared test run's agent report — the `test-run` dispatch every
|
|
43
42
|
// review makes once, before any specialist. Clean, so a script that is about
|
|
44
43
|
// something else need not answer it; one that is about it names its own.
|
|
45
44
|
export const SHARED = { exitCode: 0, tests: 5, pass: 5, fail: 0 };
|
|
46
|
-
export const review = (findings
|
|
45
|
+
export const review = (findings) => ({
|
|
47
46
|
dimension: "correctness",
|
|
48
47
|
scope_searched: "CWD-AUDIT: clean /repo",
|
|
49
48
|
findings,
|
|
50
|
-
test_run: testRun,
|
|
51
49
|
});
|
|
52
50
|
export const finding = (severity) => ({ severity, claim: `a ${severity} claim`, file: "a.js", line: 3, evidence: "line 3 has no else" });
|
|
53
51
|
export const vote = (refuted) => ({ refuted, reason: "measured. CWD-AUDIT: clean /repo" });
|
package/skills/run-team/SKILL.md
CHANGED
|
@@ -2024,7 +2024,7 @@ up to ~10k characters, in every fix-applier prompt you wrote. The fix-applier
|
|
|
2024
2024
|
reads every one off the file and makes every ruling on them
|
|
2025
2025
|
(`review-and-fix.md`'s **The review result file**). What the digest tells you:
|
|
2026
2026
|
|
|
2027
|
-
**`testEnvironment` says what
|
|
2027
|
+
**`testEnvironment` says what the review's shared test run is evidence about**,
|
|
2028
2028
|
and it is present on a healthy run as well as a degraded one, so there is
|
|
2029
2029
|
nothing to notice by its absence. The snapshot is `git archive`d and then
|
|
2030
2030
|
`git init`ed with one commit, and the block that cuts it compares that commit's
|
|
@@ -2032,8 +2032,8 @@ tree hash against the reviewed commit's (#1056): verified, a suite run in the
|
|
|
2032
2032
|
snapshot collects and runs what a checkout does, and a red is a fact about the
|
|
2033
2033
|
tree. UNVERIFIED, it is not — tests that ask git what ships decline or fail for
|
|
2034
2034
|
the environment, and this field carries which line of the cut failed. The
|
|
2035
|
-
fix-applier reads it before acting on any
|
|
2036
|
-
degraded review never passes for a clean one.
|
|
2035
|
+
fix-applier reads it before acting on any count from the shared test run; you
|
|
2036
|
+
read it so a degraded review never passes for a clean one.
|
|
2037
2037
|
`unverified` is *not* "checked and cleared" — a `suggestion` skips the pass by
|
|
2038
2038
|
policy, and a finding whose refuters all crashed lands there too. Both go to the
|
|
2039
2039
|
fix-applier with the rest, in the file; never rule on them yourself.
|
|
@@ -2104,10 +2104,10 @@ the same `testCmd` you passed the review, and the **path** `<scratch>/review-<pr
|
|
|
2104
2104
|
— never the findings. The fix-applier owns every per-PR ruling you once made
|
|
2105
2105
|
before dispatch: both mutual-exclusion scans, the suggested-fix re-derivation,
|
|
2106
2106
|
`refuted=false` ≠ apply, per-site measurement for a sibling-site extension, and
|
|
2107
|
-
`testEnvironment`/`cwdAudit` before any
|
|
2108
|
-
you hold none, so there is nothing for you to rank,
|
|
2109
|
-
report from a refuter it spawns that surfaces to you
|
|
2110
|
-
yours to scan or pass on.
|
|
2107
|
+
`testEnvironment`/`cwdAudit` before any result of the shared test run. It
|
|
2108
|
+
holds every finding and you hold none, so there is nothing for you to rank,
|
|
2109
|
+
relay or pre-rule — and a report from a refuter it spawns that surfaces to you
|
|
2110
|
+
is its to retrieve, never yours to scan or pass on.
|
|
2111
2111
|
|
|
2112
2112
|
**A `DISPATCH fix-pr PR#<M>` on a conflict hold** — the row carries the merge
|
|
2113
2113
|
bot's unresolved `conflict-hold:#<M>` (`run-merge-bot.md` step 1's fallback) —
|
|
@@ -2225,8 +2225,9 @@ all.
|
|
|
2225
2225
|
> the rest of the recipes and every ruling on the findings: both
|
|
2226
2226
|
> mutual-exclusion scans, the suggested-fix re-derivation, `refuted=false` ≠
|
|
2227
2227
|
> apply, per-site measurement for a sibling-site extension, and `testEnvironment`
|
|
2228
|
-
> and `cwdAudit` before any
|
|
2229
|
-
> nobody upstream has made one, and the controller never
|
|
2228
|
+
> and `cwdAudit` before any result of the shared test run. Every one of those
|
|
2229
|
+
> rulings is yours — nobody upstream has made one, and the controller never
|
|
2230
|
+
> reads the findings.
|
|
2230
2231
|
>
|
|
2231
2232
|
> **Every factual claim your diff restates needs a settling command run
|
|
2232
2233
|
> against the tree first** — the issue body is a lead, never a citation.
|