@feigi/fleet-ctl 3.47.4 → 3.47.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -28,8 +28,15 @@ and confirm the named test stays green. Vary the syntactic form of the
|
|
|
28
28
|
mutation — a guard written to catch `// whole-line` comments may let `code; //
|
|
29
29
|
trailing` through, and a test that only exercises one form is not proven to
|
|
30
30
|
discriminate the class. `<testCmd>` is the `command` your dispatch prompt's
|
|
31
|
-
Tests paragraph names, run only in your own copy of the snapshot.
|
|
32
|
-
|
|
31
|
+
Tests paragraph names, run only in your own copy of the snapshot. Make that
|
|
32
|
+
copy from the snapshot's tracked files only —
|
|
33
|
+
`git -C <snapshot> archive HEAD | tar -x -C <copy>`, or a `git clone` of the
|
|
34
|
+
snapshot repository — then `git init`, `git add -A -f` and commit in it, then
|
|
35
|
+
run the Install step your dispatch prompt names from the copy's root before
|
|
36
|
+
you measure anything there. Never `cp -R` the snapshot, and never symlink or
|
|
37
|
+
clone its ignored paths: a copy sharing the snapshot's installed output can
|
|
38
|
+
run the snapshot's source, and your mutation never runs. A kill is scored by
|
|
39
|
+
these rules and nothing else:
|
|
33
40
|
|
|
34
41
|
**Valid red:** a run that completed — the command returned on its own, with
|
|
35
42
|
no signal and no timeout, deadline or kill having fired — in which the test
|
|
@@ -82,8 +82,8 @@ It cannot ask the user anything and cannot wait on CI — rebasing, watching che
|
|
|
82
82
|
- **Say read-only.** No `fleet-review-*` definition's frontmatter restricts tool access — a `.agent.md` prompt saying "report only" is instruction, not enforcement — so an agent hand-dispatched here, named or general-purpose, has write tools it must not use: one committed and pushed during a report-only dispatch. No commits, no pushes, no worktree edits.
|
|
83
83
|
- **Cut TWO trees, not one — a shared snapshot is not enough.** Mutation probing is a *write*, so a mutating specialist and read-only specialists cannot share a copy: the readers then observe the mutant exactly as if it were real code. Give them `snap-ro`, pristine and **never written**, and one private copy **per mutating specialist**. Keep every one out of the worktree. Observed with a single shared snapshot: three specialists read two *different* in-flight mutations, one reporting the PR's own bug as still present; elsewhere three watched their file go clean → `M` mid-analysis; twice a reviewer came one step from filing a false finding. Reverting a probe does not help — it leaves a window where every concurrent reader sees a lie, and serializing does not close it because readers are concurrent with the *mutator*.
|
|
84
84
|
- **Do not trust the obvious contamination check.** `diff -rq` and `md5` against the mutating tree both returned *clean* — because the probe had been reverted between the two reads. A clean diff against a live tree is not evidence in either direction; only `snap-ro` and the object store settle it.
|
|
85
|
-
- **`git archive` carries tracked files only.** No installed dependencies, no gitignored runner or config. Provision in the same step — run the Install step in the copy (read it off the worktree or main checkout, `derive-testcmd.sh <worktree or main checkout> install`; pointed at the copy it refuses, since a copy holds no Recipe cache)
|
|
86
|
-
- **Initialize the copy before anyone measures in it — a bare `git archive` tree cannot validly run every suite.** It is not a git repository, so anything that asks git about the checkout breaks: `derive_workspace_id` shells out to `git rev-parse`, falls back to the directory basename and the hook tests fail on the *name* — naming the dir after the repo makes those pass **by coincidence, not correctness** — while every sweep that asks git what ships declines instead of running, which is the silent half (measured on this repo at `ffa9026`: 2401 tests, 2382 pass, 19 skipped in an extraction against 2401/2401/0 in a worktree, the totals identical). `git init && git add -A -f && git commit` in the extracted copy is what makes a count mean what it looks like; the review's snapshot agent does exactly that
|
|
85
|
+
- **`git archive` carries tracked files only.** No installed dependencies, no gitignored runner or config. Provision in the same step — run the Install step in the copy (read it off the worktree or main checkout, `derive-testcmd.sh <worktree or main checkout> install`; pointed at the copy it refuses, since a copy holds no Recipe cache), from the copy's root, before anyone measures in it — or specialists silently have no runnable suite and reason from source instead of measuring. That failure is invisible: you get confident prose where you asked for a measurement. Never `cp -R` an installed tree, and never symlink or clone its ignored paths into a copy: an install can record absolute paths back into the tree it ran in, so the copy runs that tree's source and a mutation made in the copy never runs.
|
|
86
|
+
- **Initialize the copy before anyone measures in it — a bare `git archive` tree cannot validly run every suite.** It is not a git repository, so anything that asks git about the checkout breaks: `derive_workspace_id` shells out to `git rev-parse`, falls back to the directory basename and the hook tests fail on the *name* — naming the dir after the repo makes those pass **by coincidence, not correctness** — while every sweep that asks git what ships declines instead of running, which is the silent half (measured on this repo at `ffa9026`: 2401 tests, 2382 pass, 19 skipped in an extraction against 2401/2401/0 in a worktree, the totals identical). `git init && git add -A -f && git commit` in the extracted copy is what makes a count mean what it looks like; the review's snapshot agent does exactly that, verifies it by comparing the new commit's tree hash against the reviewed commit's, and then runs the Install step in it, and a hand-cut tree needs the same before you read a count off it. **Every agent you send to work in its own copy of the snapshot makes that copy the same way:** copy the tracked files only (`git -C <snapshot> archive HEAD | tar -x -C <copy>`, or a `git clone` of the snapshot repository), `git init`, `git add -A -f` and commit, then run the Install step in the copy before measuring anything in it — never `cp -R` the snapshot. Suites that need the real worktree for some *other* reason — the stack below — still do.
|
|
87
87
|
- **Give specialists a stack-free test command, not the worktree's runner, and their own scratch dir** — `<scratch>/pr<N>/fleet-review-<key>/`, resolved to an absolute path that you write into that specialist's prompt; a specialist never derives its own. Specialists inherit no environment, and one falling back to the default config runs a `globalSetup` that brings the shared compose stack up and tears it down, recreating the DB mid-run for every sibling. **The snapshot does not cover this** — the compose project name comes from the environment, not the working directory, so three agents on three copies still collide. But `./agent-test` does not fix it either: it exports **one** `TEST_COMPOSE_PROJECT`/port triple per *worktree* (`ab-<issue>`), so N concurrent specialists sharing it tear down each other's postgres mid-run. Observed: a probe returned `No test files found / No such container` and would have read as a test failure. So tell specialists the repo's test command directly — the Test entrypoint, read off the Recipe cache the way step 3 reads it (`derive-testcmd.sh <worktree or main checkout> test` prints it; the snapshot carries no `.fleet/` cache, so pointing it at the snapshot refuses) — when it brings up no shared stack, and the repo's own stack-free config when it does; never name a config the repo does not have, which sends every specialist into a missing-file error that reads as a broken tree rather than a bad instruction. Keep `./agent-test` for your own verification. **Hand that command with its reading rule: `tests 0` is a failed run, not a pass.** A command is not self-checking: a path glob or filter that matches nothing can exit 0 having run nothing. Measured with `node --test` over a glob matching no file: bash passes the pattern through literally, node globs it itself and finds no files, and you get `ℹ tests 0` / exit 0 — and `# tests 0` to a pipe on older node, so a rule that greps only for `ℹ` finds nothing in a CI log and cannot tell a zero-test run from a full one. Do not count on the shell to catch it: zsh errors (`no matches found`, exit 1) and bash does not, so the guard has to be the reading rule. **And zero is not the only no-work count.** **0 passes with no failures** is everything skipped, and a count **materially below the full suite** is a partial copy of the tree — which is the *documented* behavior, since the prompt tells specialists to run their probe work inside their own copy of the snapshot and nothing anchors an expected count for them. `review-core.mjs` classifies the first itself, in `unrunReason`; the second it structurally cannot, because that function is kept pure and so never learns how many tests the tree has. Handing the rule over with the command is the only guard the partial-copy case has. Carry the rule, not the command: wherever a repo has a stack, N concurrent specialists sharing one runner collide, and the escape is whatever that repo's stack-free config is.
|
|
88
88
|
- **Completion is not delivery — but you can reach a specialist, so never wait idle.** A specialist you dispatched reports when it settles; if a report is missing, `hub send` it by name and ask for the report. Judge delivery by whether a report came out. **Do not settle a dimension you hold no report for**: ask the specialist, then the controller by name; only when neither answers, rule and name that dimension **unrun** — a killed specialist never reports, so waiting on one is unbounded. Never ship "dispatched, never delivered" as settled; a ruling citing a report you do not hold gets verified from source, never applied on trust.
|
|
89
89
|
- **Apply only once the fan-out is settled — every report either in hand or accounted for.** Collection is not guaranteed; a report that never arrived is an open dimension, not a smaller set. And editing while they read is the same defect as probing — one specialist reviewed uncommitted code that was never in the PR diff.
|
package/package.json
CHANGED
package/scripts/review-core.mjs
CHANGED
|
@@ -3,8 +3,10 @@
|
|
|
3
3
|
// This file holds every host-independent declaration PR review needs: the
|
|
4
4
|
// JSON schemas, `usableDiff`, `readRules`, `resolveTestCmd`,
|
|
5
5
|
// `decodeArgs`, `selectDimensions`, `resolveDimensions`, `snapshotMissing`,
|
|
6
|
-
// `unrunReason`/`unrunEntries`/`unrunCrashed`,
|
|
7
|
-
// `
|
|
6
|
+
// `installReason`/`installFinding`, `unrunReason`/`unrunEntries`/`unrunCrashed`,
|
|
7
|
+
// `sharedRunNote`, `copyRule`,
|
|
8
|
+
// `verdictFor`, `resumeFor`, and `runReview(host, args)` — the orchestration
|
|
9
|
+
// function itself.
|
|
8
10
|
//
|
|
9
11
|
// review-eval.mjs imports this file directly, a RELATIVE specifier: both
|
|
10
12
|
// ship together under the same Install root, so the Resolver is only needed
|
|
@@ -120,6 +122,23 @@ export const SNAPSHOT_SCHEMA = {
|
|
|
120
122
|
prHead: { type: "string" },
|
|
121
123
|
testCmd: { type: "string" },
|
|
122
124
|
testCmdError: { type: "string" },
|
|
125
|
+
// The Recipe's Install step as the snapshot agent read it, and its exit
|
|
126
|
+
// status in the snapshot. Optional, so an absent `installExit` is read
|
|
127
|
+
// as an install that did not succeed, never as one that did:
|
|
128
|
+
// `installError` carries derive-testcmd.sh's refusal when the step could
|
|
129
|
+
// not be read at all.
|
|
130
|
+
installCmd: { type: "string" },
|
|
131
|
+
installExit: { type: "integer" },
|
|
132
|
+
installError: { type: "string" },
|
|
133
|
+
// What `git status --porcelain --untracked-files=all` printed in the
|
|
134
|
+
// snapshot after the Install step, one entry per line, the first twenty
|
|
135
|
+
// at most; `untrackedCount` is how many it printed in all. The
|
|
136
|
+
// repository's own ignore rules decide it, so it is reported, never
|
|
137
|
+
// excluded. `statusError` says the read itself failed, which is not an
|
|
138
|
+
// empty answer.
|
|
139
|
+
untracked: { type: "array", items: { type: "string" } },
|
|
140
|
+
untrackedCount: { type: "integer" },
|
|
141
|
+
statusError: { type: "string" },
|
|
123
142
|
},
|
|
124
143
|
};
|
|
125
144
|
|
|
@@ -138,7 +157,7 @@ export const TEST_RUN_SCHEMA = {
|
|
|
138
157
|
tests: { type: "integer" },
|
|
139
158
|
pass: { type: "integer" },
|
|
140
159
|
fail: { type: "integer" },
|
|
141
|
-
expectedFail: { type: "integer", description: "Tests the runner reports as failing by design (vitest
|
|
160
|
+
expectedFail: { type: "integer", description: "Tests the runner reports as failing by design (vitest `expected fail`, pytest `xfailed`), never folded into `fail`." },
|
|
142
161
|
cancelled: { type: "integer" },
|
|
143
162
|
skipped: { type: "integer" },
|
|
144
163
|
error: { type: "string", description: "Only when no summary was printed: what happened instead." },
|
|
@@ -458,7 +477,32 @@ export function snapshotMissing(snap, runRootPrefix) {
|
|
|
458
477
|
// commit, no ancestry, no remotes, so `git log`/`git diff` now return
|
|
459
478
|
// plausible output about the snapshot in place of the `fatal:` that used to
|
|
460
479
|
// stop that question being asked.
|
|
480
|
+
//
|
|
481
|
+
// What the snapshot reads untracked after the Install step rides on the end
|
|
482
|
+
// as one line: the repository's own ignore rules left it, so it is a fact
|
|
483
|
+
// about the repository, and the review names it rather than hiding it. The
|
|
484
|
+
// line is interpolated into every specialist and refuter prompt and the
|
|
485
|
+
// payload, so it carries the first `LISTED_UNTRACKED` entries and a count of
|
|
486
|
+
// the rest — a PR that drops its dependency directory from `.gitignore` would
|
|
487
|
+
// otherwise put one entry per installed file in each of them; the whole
|
|
488
|
+
// status is in the run root's `status.txt`. A status the block could not read
|
|
489
|
+
// is said so, never left to read as an empty one.
|
|
490
|
+
const LISTED_UNTRACKED = 20;
|
|
491
|
+
|
|
461
492
|
export function environmentNote(snap) {
|
|
493
|
+
const untracked = snap && Array.isArray(snap.untracked) ? snap.untracked.filter((l) => typeof l === "string" && l.trim()) : [];
|
|
494
|
+
const total = Math.max(untracked.length, snap && Number.isInteger(snap.untrackedCount) ? snap.untrackedCount : 0);
|
|
495
|
+
const shown = untracked.slice(0, LISTED_UNTRACKED);
|
|
496
|
+
const more = total - shown.length;
|
|
497
|
+
let note = repoNote(snap);
|
|
498
|
+
if (snap && typeof snap.statusError === "string" && snap.statusError.trim())
|
|
499
|
+
note += `\nAfter the Install step the snapshot's status could not be read (${snap.statusError.trim()}), so what it left untracked is not known.`;
|
|
500
|
+
if (shown.length)
|
|
501
|
+
note += `\nAfter the Install step the snapshot reads untracked under the repository's own ignore rules: ${shown.join(", ")}${more > 0 ? `, and ${more} more (the whole status is in ${snap.runRoot ?? "the run root"}/status.txt)` : ""}.`;
|
|
502
|
+
return note;
|
|
503
|
+
}
|
|
504
|
+
|
|
505
|
+
function repoNote(snap) {
|
|
462
506
|
if (snap && snap.repoVerified === true)
|
|
463
507
|
return `Test environment: the snapshot is a git repository whose single commit holds
|
|
464
508
|
the reviewed tree — its tree hash was compared against the commit under review
|
|
@@ -482,10 +526,52 @@ validation of the tree: its counts are snapshot-measured, and a failure in it
|
|
|
482
526
|
cannot be told apart from a regression.`;
|
|
483
527
|
}
|
|
484
528
|
|
|
529
|
+
// The Install step's verdict on the snapshot: null when it ran there and
|
|
530
|
+
// exited 0, otherwise why no test of this review can run in it. An absent
|
|
531
|
+
// `installExit` is never a success — the step was unreadable (`installError`
|
|
532
|
+
// says why) or the snapshot agent dropped the line — because a snapshot
|
|
533
|
+
// nothing was installed into measures its own missing dependencies, not the
|
|
534
|
+
// tree.
|
|
535
|
+
export function installReason(snap) {
|
|
536
|
+
if (snap && snap.installExit === 0) return null;
|
|
537
|
+
if (snap && Number.isInteger(snap.installExit))
|
|
538
|
+
return `the Install step${snap.installCmd ? ` \`${snap.installCmd}\`` : ""} exited ${snap.installExit} in the snapshot (log ${snap.runRoot}/install.log) — nothing it installs is there, so the shared test run was skipped`;
|
|
539
|
+
const why = (snap && typeof snap.installError === "string" && snap.installError.trim()) || "the snapshot agent reported no exit status for it";
|
|
540
|
+
return `the Install step did not run in the snapshot — ${why} — so the shared test run was skipped`;
|
|
541
|
+
}
|
|
542
|
+
|
|
543
|
+
// The one finding an Install step that exits non-zero in the snapshot files
|
|
544
|
+
// on the PR, when the snapshot is the reviewed tree. It is the review's own
|
|
545
|
+
// measurement, an exit status, not a specialist's claim, so it carries no
|
|
546
|
+
// refuter vote — and it claims only what was measured: which tree the step
|
|
547
|
+
// ran on and how it exited. Why it exited is not known here, and the same
|
|
548
|
+
// status can come from the PR (a script it deleted exits 127) or from the host
|
|
549
|
+
// (a kill, a timeout), so the claim says the cause is not established rather
|
|
550
|
+
// than naming the PR's defect. The Recipe cache, proven on the base, stays as
|
|
551
|
+
// it is. A snapshot that is NOT the reviewed tree (`repoVerified` not true)
|
|
552
|
+
// files nothing: an exit there is a fact about a different tree, which
|
|
553
|
+
// `environmentNote` already says. An Install step that could not be READ files
|
|
554
|
+
// nothing either: that is the Recipe's state, not the PR's.
|
|
555
|
+
export function installFinding(snap) {
|
|
556
|
+
if (!snap || snap.repoVerified !== true || !Number.isInteger(snap.installExit) || snap.installExit === 0) return null;
|
|
557
|
+
const cmd = snap.installCmd ? `\`${snap.installCmd}\`` : "the Recipe's Install step";
|
|
558
|
+
return {
|
|
559
|
+
severity: "critical",
|
|
560
|
+
claim: `The Install step exited ${snap.installExit} on this PR's tree: ${cmd}, run from the root of a snapshot of ${snap.head}, so no test of this review ran. The review did not establish why it exited — the PR's tree or this host.`,
|
|
561
|
+
evidence: `The review ran ${cmd} once from the root of ${snap.path} and it exited ${snap.installExit}; its output is in ${snap.runRoot}/install.log.`,
|
|
562
|
+
dimension: "install",
|
|
563
|
+
verdict: "survived",
|
|
564
|
+
votes: [],
|
|
565
|
+
refutersDispatched: 0,
|
|
566
|
+
refutersInconclusive: 0,
|
|
567
|
+
};
|
|
568
|
+
}
|
|
569
|
+
|
|
485
570
|
// Every run-quality verdict below reads the review's ONE shared test
|
|
486
571
|
// run — `{command, logPath, exitCode?, tests?, pass?, fail?, expectedFail?,
|
|
487
572
|
// cancelled?, skipped?, error?}`, `command`/`logPath` the caller's own and the
|
|
488
|
-
// rest the test-run agent's report
|
|
573
|
+
// rest the test-run agent's report, or `{command, logPath, unprovisioned}`
|
|
574
|
+
// when `installReason` skipped the run — never a specialist's own run: no
|
|
489
575
|
// specialist runs the full suite any more, so a verdict on one would judge a
|
|
490
576
|
// copy of this.
|
|
491
577
|
// `run` null is a caller holding no run at all — runReview always hands one
|
|
@@ -498,6 +584,7 @@ cannot be told apart from a regression.`;
|
|
|
498
584
|
// TEST_RUN_EXIT line would otherwise read as green.
|
|
499
585
|
function testRunReason(run) {
|
|
500
586
|
if (!run) return "the review's shared test run returned nothing — no counts exist, so no dimension's tests ran";
|
|
587
|
+
if (run.unprovisioned) return run.unprovisioned;
|
|
501
588
|
// The wrapper's wait cap expired: it launched nothing, so there are no
|
|
502
589
|
// counts to judge and the exit status is the wrapper's, not the suite's. A
|
|
503
590
|
// run that printed counts and happened to exit 75 is the suite's own answer
|
|
@@ -567,13 +654,42 @@ function countsOf(run) {
|
|
|
567
654
|
.join(", ");
|
|
568
655
|
}
|
|
569
656
|
|
|
657
|
+
// How a specialist or refuter makes "your own copy of the snapshot", in
|
|
658
|
+
// every prompt that sends one to work there. Tracked files only, then the
|
|
659
|
+
// Install step in the copy: what an install writes can record absolute paths
|
|
660
|
+
// back into the tree it ran in, so a copy that shares the snapshot's
|
|
661
|
+
// installed output imports the snapshot's source, and a mutation made in the
|
|
662
|
+
// copy never runs.
|
|
663
|
+
export function copyRule(snap) {
|
|
664
|
+
const step = snap && snap.installCmd
|
|
665
|
+
? `the Install step, \`${snap.installCmd}\`,`
|
|
666
|
+
: "the Install step (this review read none, so a copy has nothing installed and a suite run in it does not measure the tree)";
|
|
667
|
+
return `Make your own copy of the snapshot from its tracked files only —
|
|
668
|
+
\`git -C ${snap && snap.path} archive HEAD | tar -x -C <copy>\`, or a \`git clone\`
|
|
669
|
+
of the snapshot repository — then make it a repository with \`git init\`,
|
|
670
|
+
\`git add -A -f\` and a commit (a clone already is one), then run
|
|
671
|
+
${step} from the copy's root before you measure anything in it.
|
|
672
|
+
Never \`cp -R\` the snapshot, and never symlink or clone its ignored paths: an
|
|
673
|
+
install can record absolute paths back into the tree it ran in, so a copy
|
|
674
|
+
sharing the snapshot's installed output runs the snapshot's source, and a
|
|
675
|
+
mutation in the copy never runs.`;
|
|
676
|
+
}
|
|
677
|
+
|
|
570
678
|
// The Tests paragraph of every specialist prompt: the shared run's
|
|
571
679
|
// command, counts, exit status and log path, in place of the instruction to
|
|
572
680
|
// run the full suite that each specialist used to follow — one review was up
|
|
573
681
|
// to six full sweeps of one immutable snapshot. `owner` is the one dimension
|
|
574
682
|
// told to file the finding about failing tests, so the others are told not
|
|
575
|
-
// to duplicate it rather than left to race each other to it.
|
|
683
|
+
// to duplicate it rather than left to race each other to it. A run the
|
|
684
|
+
// Install step skipped has no counts and no log to describe, so it gets a
|
|
685
|
+
// paragraph of its own.
|
|
576
686
|
export function sharedRunNote(run, owner, key) {
|
|
687
|
+
if (run.unprovisioned)
|
|
688
|
+
return `Tests: this review did NOT run its test command: ${run.unprovisioned}.
|
|
689
|
+
Every dimension of this review is reported unrun for it — do not run the full
|
|
690
|
+
command yourself to replace it, and file nothing about the install: the review
|
|
691
|
+
reports it itself.
|
|
692
|
+
command: ${run.command}`;
|
|
577
693
|
const lines = [
|
|
578
694
|
`Tests: this review ran its test command ONCE, from the snapshot's root, before
|
|
579
695
|
dispatching you. Do NOT run that full command yourself — every dimension reads
|
|
@@ -873,15 +989,37 @@ export async function runReview(host, args) {
|
|
|
873
989
|
( cd "$SNAP" && git init -q && git add -A -f && git -c user.name=fleet -c user.email=fleet@invalid commit -q --no-verify -m "review snapshot of $SHA" ) || echo SNAPSHOT_INIT_FAILED
|
|
874
990
|
SNAPTREE=$(git -C "$SNAP" rev-parse 'HEAD^{tree}' 2>/dev/null); SRCTREE=$(git -C ${worktree} rev-parse 'HEAD^{tree}')
|
|
875
991
|
{ [ -d "$SNAP/.git" ] && [ -n "$SNAPTREE" ] && [ "$SNAPTREE" = "$SRCTREE" ]; } && echo SNAPSHOT_TREE_MATCH || echo SNAPSHOT_TREE_MISMATCH="snapshot $SNAPTREE vs commit $SRCTREE"
|
|
876
|
-
|
|
877
|
-
|
|
992
|
+
INSTALL=$(~/.fleet/bin/fleet-run derive-testcmd.sh ${worktree} install) && echo SNAPSHOT_INSTALL_STEP="$INSTALL" || { INSTALL=; echo SNAPSHOT_INSTALL_UNREAD; }
|
|
993
|
+
[ -z "$INSTALL" ] || { ( cd "$SNAP" && sh -c "$INSTALL" ) >"$RUN/install.log" 2>&1; echo SNAPSHOT_INSTALL_EXIT=$?; }
|
|
994
|
+
if [ -d "$SNAP/.git" ] && git -C "$SNAP" status --porcelain --untracked-files=all >"$RUN/status.txt"; then head -n 20 "$RUN/status.txt" | sed 's/^/SNAPSHOT_UNTRACKED=/'; echo SNAPSHOT_UNTRACKED_COUNT=$(wc -l <"$RUN/status.txt" | tr -d ' '); else echo SNAPSHOT_STATUS_FAILED; fi
|
|
995
|
+
|
|
996
|
+
The Install step can take minutes: run the block as ONE blocking foreground
|
|
997
|
+
command with a command deadline of 3600 seconds, never backgrounded and never
|
|
998
|
+
polled. The Install step takes no suite slot.
|
|
999
|
+
|
|
1000
|
+
Report \`installCmd\` = the SNAPSHOT_INSTALL_STEP value and \`installExit\` = the
|
|
1001
|
+
SNAPSHOT_INSTALL_EXIT value, both copied as printed. If the block printed
|
|
1002
|
+
SNAPSHOT_INSTALL_UNREAD instead, omit both and report \`installError\` = what
|
|
1003
|
+
derive-testcmd.sh printed on stderr. The Install step runs once, in the
|
|
1004
|
+
snapshot, from its root: never rerun it, never repair a failure, and never run
|
|
1005
|
+
it anywhere else. A non-zero exit is reported as printed, and the caller files
|
|
1006
|
+
it as a finding on the PR.
|
|
1007
|
+
|
|
1008
|
+
Report \`untracked\` = every SNAPSHOT_UNTRACKED= value, copied as printed, in
|
|
1009
|
+
order, and omit it when the block printed none; the block prints twenty at most,
|
|
1010
|
+
so report \`untrackedCount\` = the SNAPSHOT_UNTRACKED_COUNT value, copied as
|
|
1011
|
+
printed. If the block printed SNAPSHOT_STATUS_FAILED, report \`statusError\` =
|
|
1012
|
+
SNAPSHOT_STATUS_FAILED: the status could not be read, which is not an empty
|
|
1013
|
+
status. The repository's own ignore rules decide what reads untracked after the
|
|
1014
|
+
Install step: list it, and add nothing to the snapshot's \`.git/info/exclude\` or
|
|
1015
|
+
anywhere else to hide it.
|
|
878
1016
|
|
|
879
1017
|
Report \`pathVerified\` = true ONLY if the 'ls -A' line printed SNAPSHOT_NONEMPTY,
|
|
880
|
-
run in that order — BEFORE the 'git init'
|
|
881
|
-
|
|
882
|
-
|
|
1018
|
+
run in that order — BEFORE the 'git init', never after it. The init creates an
|
|
1019
|
+
entry of its own, so a probe below it prints SNAPSHOT_NONEMPTY on a
|
|
1020
|
+
'git archive' that extracted nothing at all.
|
|
883
1021
|
|
|
884
|
-
Report \`repoVerified\` = true ONLY if the
|
|
1022
|
+
Report \`repoVerified\` = true ONLY if the tree-compare line above printed
|
|
885
1023
|
SNAPSHOT_TREE_MATCH; otherwise report it false and put the line it printed
|
|
886
1024
|
instead — SNAPSHOT_INIT_FAILED, or the whole SNAPSHOT_TREE_MISMATCH= value with
|
|
887
1025
|
both hashes — in \`repoError\`. The init is what makes the snapshot MEASURABLE:
|
|
@@ -891,8 +1029,9 @@ extraction than in a checkout at the same commit, with nothing in the payload
|
|
|
891
1029
|
saying the measurement happened somewhere else. The tree-hash compare is
|
|
892
1030
|
the verification: two commits whose trees hash the same hold byte-identical
|
|
893
1031
|
content, so a match settles that the snapshot IS the reviewed tree — a stronger
|
|
894
|
-
check than reading a few files, and the reason the commit
|
|
895
|
-
|
|
1032
|
+
check than reading a few files, and the reason the commit and the compare come
|
|
1033
|
+
BEFORE the Install step: 'git add -A -f' stages ignored files too, so whatever
|
|
1034
|
+
the Install step writes would enter the index and change the hash.
|
|
896
1035
|
|
|
897
1036
|
Report \`head\` = the full sha 'git -C ${worktree} rev-parse HEAD' prints — the
|
|
898
1037
|
destination path carries only the short form. The byte-identity spot check this
|
|
@@ -960,8 +1099,8 @@ specialists into another run's tree. Report the HEAD sha, and — in
|
|
|
960
1099
|
\`diffStats\` — the SINGLE-LINE JSON object diff-stats.mjs prints to
|
|
961
1100
|
STDOUT, copied verbatim. Only runRoot, path, head, pathVerified and repoVerified
|
|
962
1101
|
are ever required — diffStats, diffPath, diffLines, refHead and prHead are each
|
|
963
|
-
omitted independently when their command failed,
|
|
964
|
-
a false repoVerified.`,
|
|
1102
|
+
omitted independently when their command failed, repoError only accompanies
|
|
1103
|
+
a false repoVerified, and installError only replaces installCmd and installExit.`,
|
|
965
1104
|
{ label: `snapshot${forPr}`, phase: "Snapshot", agentType: SNAPSHOT_AGENT_TYPE, schema: SNAPSHOT_SCHEMA },
|
|
966
1105
|
);
|
|
967
1106
|
|
|
@@ -979,6 +1118,11 @@ a false repoVerified.`,
|
|
|
979
1118
|
|
|
980
1119
|
const testCmd = resolveTestCmd(A.testCmd, snap);
|
|
981
1120
|
log(`testCmd ${A.testCmd ? "(caller override)" : "(derived)"} ${testCmd}`);
|
|
1121
|
+
// An Install step that failed or never ran leaves a snapshot that cannot
|
|
1122
|
+
// measure the tree: the shared run is skipped, every dimension is reported
|
|
1123
|
+
// unrun through `testRunReason`, and a non-zero exit is filed as a finding.
|
|
1124
|
+
const notInstalled = installReason(snap);
|
|
1125
|
+
log(notInstalled ? `install: ${notInstalled}` : `install: ${snap.installCmd ?? "the Install step"} exited 0 in the snapshot`);
|
|
982
1126
|
|
|
983
1127
|
const usable = usableDiff(snap);
|
|
984
1128
|
log(
|
|
@@ -1012,7 +1156,7 @@ a false repoVerified.`,
|
|
|
1012
1156
|
// every dimension, never a pass.
|
|
1013
1157
|
const logPath = `${snap.runRoot}/test-run.log`;
|
|
1014
1158
|
phase("Test run");
|
|
1015
|
-
const ran = await agent(
|
|
1159
|
+
const ran = notInstalled ? null : await agent(
|
|
1016
1160
|
`Run this repository's test command ONCE, for the review of PR #${pr}, and
|
|
1017
1161
|
report what it printed. Every review specialist reads your counts instead of
|
|
1018
1162
|
running the suite itself, so this is the review's only full run.
|
|
@@ -1057,7 +1201,7 @@ Then read the summary the runner printed at the end of the log —
|
|
|
1057
1201
|
\`exitCode\` = the TEST_RUN_EXIT value, and \`tests\`, \`pass\`, \`fail\`,
|
|
1058
1202
|
\`expectedFail\`, \`cancelled\`, \`skipped\` = the counts that summary states, each
|
|
1059
1203
|
copied as printed; omit any the runner does not print. A test the runner reports
|
|
1060
|
-
as failing by design (vitest
|
|
1204
|
+
as failing by design (vitest \`expected fail\`, pytest \`xfailed\`) goes in \`expectedFail\`, never in
|
|
1061
1205
|
\`fail\`. Copy the counts, never judge them: 'tests 0' is a FAILED run, not a
|
|
1062
1206
|
pass, and 0 passes with no failures is everything skipped — report both
|
|
1063
1207
|
exactly as printed, and the caller reports every dimension unrun for them.
|
|
@@ -1068,8 +1212,10 @@ write 0 for a count the log does not state: an absent count is how the caller
|
|
|
1068
1212
|
learns the run produced none.`,
|
|
1069
1213
|
{ label: `test-run${forPr}`, phase: "Test run", agentType: TEST_RUN_AGENT_TYPE, schema: TEST_RUN_SCHEMA },
|
|
1070
1214
|
).catch((e) => ({ error: `the test-run dispatch threw: ${e?.message ?? e}` }));
|
|
1071
|
-
const sharedRun =
|
|
1072
|
-
|
|
1215
|
+
const sharedRun = notInstalled
|
|
1216
|
+
? { command: testCmd, logPath, unprovisioned: notInstalled }
|
|
1217
|
+
: { ...(ran || { error: "the test-run agent returned nothing" }), command: testCmd, logPath };
|
|
1218
|
+
log(notInstalled ? `shared test run: skipped — ${notInstalled}` : `shared test run: ${countsOf(sharedRun) || "no counts"} — exit ${sharedRun.exitCode ?? "(absent)"} — log ${logPath}`);
|
|
1073
1219
|
// Failing tests are the review's to report, not each dimension's: the first
|
|
1074
1220
|
// selected dimension is told to file them, and every other one not to.
|
|
1075
1221
|
const failureOwner = dimensions[0].key;
|
|
@@ -1110,6 +1256,7 @@ READ ONLY FROM THE SNAPSHOT: ${snap.path} (HEAD ${snap.head}) — plus the diff
|
|
|
1110
1256
|
file named below, if one is given.
|
|
1111
1257
|
Never read or write ${worktree} — other agents are using it.
|
|
1112
1258
|
Run any mutation or probe work inside your own copy of the snapshot.
|
|
1259
|
+
${copyRule(snap)}
|
|
1113
1260
|
|
|
1114
1261
|
Your shell starts in NEITHER of those directories, and what it does start in is
|
|
1115
1262
|
a tree you must not write to: this dispatch carries no working directory of its
|
|
@@ -1206,6 +1353,8 @@ decide it and nothing else:
|
|
|
1206
1353
|
|
|
1207
1354
|
${KILL_RULES}
|
|
1208
1355
|
|
|
1356
|
+
${copyRule(snap)}
|
|
1357
|
+
|
|
1209
1358
|
Abstain when your own check could produce neither a valid red nor a valid
|
|
1210
1359
|
green: set \`inconclusive\` to true and name the runs in \`reason\`, each with
|
|
1211
1360
|
its evidence line. \`refuted\` is still required, and the tally ignores it on
|
|
@@ -1291,7 +1440,8 @@ how three reviews from one cell left four files modified in that checkout.`,
|
|
|
1291
1440
|
},
|
|
1292
1441
|
);
|
|
1293
1442
|
|
|
1294
|
-
const
|
|
1443
|
+
const installed = installFinding(snap);
|
|
1444
|
+
const all = [...(installed ? [installed] : []), ...reviewed.flat().filter(Boolean)];
|
|
1295
1445
|
const survived = all.filter((f) => f.verdict === "survived");
|
|
1296
1446
|
const refuted = all.filter((f) => f.verdict === "refuted");
|
|
1297
1447
|
const unverified = all.filter((f) => f.verdict === "unverified");
|