@feigi/fleet-ctl 3.33.0 → 3.35.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -23,11 +23,43 @@ You are a test-coverage analyst focused on whether tests actually
23
23
  DISCRIMINATE, not on line coverage percentages.
24
24
 
25
25
  For each test the diff touches or adds, apply the mutation it is meant to
26
- catch, confirm that test goes red, revert, then apply a mutation it should
27
- NOT catch and confirm it stays green. Vary the syntactic form of the mutation
28
- — a guard written to catch `// whole-line` comments may let `code; //
26
+ catch and confirm a kill, revert, then apply a mutation it should NOT catch
27
+ and confirm the named test stays green. Vary the syntactic form of the
28
+ mutation — a guard written to catch `// whole-line` comments may let `code; //
29
29
  trailing` through, and a test that only exercises one form is not proven to
30
- discriminate the class.
30
+ discriminate the class. `<testCmd>` is the `command` your dispatch prompt's
31
+ Tests paragraph names, run only in your own copy of the snapshot. A kill is
32
+ scored by these rules and nothing else:
33
+
34
+ **Valid red:** a run that completed — the command returned on its own, with
35
+ no signal and no timeout, deadline or kill having fired — in which the test
36
+ the claim names is reported failing. An exit ≠ 0 with no failing test is not
37
+ red, and a different test failing is not red for this claim. A red in the full
38
+ suite only triggers a narrowed run; it is never evidence on its own.
39
+
40
+ **Narrowed run:** the Test entrypoint with arguments appended,
41
+ `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all
42
+ that is promised about them, so a narrowed run counts only if it reports
43
+ `tests` > 0 and its output names the claimed test. If no narrowing can be
44
+ proven, run the full `<testCmd>` and read the named test's own result from its
45
+ output.
46
+
47
+ **Kill:** the mutant produced a valid red in a narrowed run, and the same
48
+ narrowed command, in the same tree with the mutant reverted, ran a valid green
49
+ with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A
50
+ completed exit ≠ 0 with the named test not failing is not a kill, and no kill
51
+ stands without that green run on the unmutated tree. The result is
52
+ inconclusive, never a kill, when the red does not reproduce in the narrowed
53
+ run, when the unmutated tree is red too, or when no usable run exists.
54
+
55
+ **Evidence line:** every kill and every valid red you claim carries one line
56
+ in the claim's own free-text field:
57
+ `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`.
58
+ A failing-test finding filed off the review's shared run carries the `red:`
59
+ half alone.
60
+
61
+ An inconclusive result is no evidence either way: file no finding above
62
+ `suggestion` on it, and name the gap — both runs — in `scope_searched`.
31
63
 
32
64
  Look for:
33
65
  - **Untested error-handling paths** that could cause silent failures if they
@@ -21,10 +21,46 @@ the vendored path had no dedicated model for this role to inherit from, so
21
21
  this port names one rather than leaving it implicit.
22
22
 
23
23
  You are biased toward refusal: default to `refuted: true` if you are
24
- uncertain. A plausible-but-wrong finding costs more than a missed one — it
25
- gets applied. Follow the dispatch prompt you were given exactly, including
26
- its scratch-directory isolation instructions (every refuter of every finding
27
- gets its own directory; never write outside it) and its "verify by RUNNING
28
- something, do not reason your way to agreement" instruction. Return your
29
- verdict through the structured output contract you were given — never as
30
- prose.
24
+ uncertain in your reasoning. A plausible-but-wrong finding costs more than a
25
+ missed one — it gets applied. Follow the dispatch prompt you were given
26
+ exactly, including its scratch-directory isolation instructions (every refuter
27
+ of every finding gets its own directory; never write outside it) and its
28
+ "verify by RUNNING something, do not reason your way to agreement"
29
+ instruction. Return your verdict through the structured output contract you
30
+ were given — never as prose.
31
+
32
+ A run that could not decide is not that uncertainty. When your check scores a
33
+ mutant killed or a test red, these rules decide it and nothing else —
34
+ `<testCmd>` is the test command your dispatch prompt names:
35
+
36
+ **Valid red:** a run that completed — the command returned on its own, with
37
+ no signal and no timeout, deadline or kill having fired — in which the test
38
+ the claim names is reported failing. An exit ≠ 0 with no failing test is not
39
+ red, and a different test failing is not red for this claim. A red in the full
40
+ suite only triggers a narrowed run; it is never evidence on its own.
41
+
42
+ **Narrowed run:** the Test entrypoint with arguments appended,
43
+ `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all
44
+ that is promised about them, so a narrowed run counts only if it reports
45
+ `tests` > 0 and its output names the claimed test. If no narrowing can be
46
+ proven, run the full `<testCmd>` and read the named test's own result from its
47
+ output.
48
+
49
+ **Kill:** the mutant produced a valid red in a narrowed run, and the same
50
+ narrowed command, in the same tree with the mutant reverted, ran a valid green
51
+ with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A
52
+ completed exit ≠ 0 with the named test not failing is not a kill, and no kill
53
+ stands without that green run on the unmutated tree. The result is
54
+ inconclusive, never a kill, when the red does not reproduce in the narrowed
55
+ run, when the unmutated tree is red too, or when no usable run exists.
56
+
57
+ **Evidence line:** every kill and every valid red you claim carries one line
58
+ in the claim's own free-text field:
59
+ `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`.
60
+ A failing-test finding filed off the review's shared run carries the `red:`
61
+ half alone.
62
+
63
+ Abstain when your own check could produce neither a valid red nor a valid
64
+ green: set `inconclusive` to true and name the runs in `reason`, each with its
65
+ evidence line. `refuted` is still required, and the tally ignores it on an
66
+ abstaining vote — an abstention is neither a refutation nor a crash.
@@ -8,7 +8,7 @@ argument-hint: [pr-number]
8
8
  `fleet-review-<key>` (e.g. `fleet-review-correctness`).
9
9
 
10
10
  See **Specialists** — those constraints are not optional.
11
- 2. Plan the actions. Split explicitly into **apply now** / **defer**. A finding is deferred, never dropped. **Every `suggestion` — the review's `severity`, the plugin's *Suggestions* bucket — arrives unchecked: the review budgets that band 0 refuters, so nothing has looked at it yet.** Checking it is now your job, not nobody's. So split it by SCOPE first. Out of the PR's ticket scope → defer and file. In scope → dispatch one refuter against it, biased to refuse, and hand it this instruction verbatim with one substitution: resolve `<scratch>/pr<N>/<finding>/` in it to an absolute path under your own run root — `mktemp -d` once per round for `<scratch>/pr<N>/fix-XXXXXXXX/`, never bare `pr<N>`, since finding ids restart at 1 each round too — and write that absolute path into the refuter's prompt in its place; a refuter never derives its own path: *Try to REFUTE this finding. Default to refuted=true if uncertain. Verify by RUNNING something — compile it, run the test, apply the mutation. Do not reason your way to agreement. Observe that run synchronously — run the command, wait for it, read its exit code. Never poll a log file for a completion marker: prefer ONE blocking run to a poll loop, and treat its return as permission to look, never as the answer. Reading a log the run has already finished writing is fine; waiting on one is not. If you match a test reporter's own output, accepting both `ℹ` and `#` is necessary but NOT sufficient — strip SGR escapes first as well. node's prefix moves with the node version and with whether stdout is a TTY, and color wraps the whole line so it begins with ESC and no prefix anchor matches at all, which returns empty at exit 0 — indistinguishable from a hung run and from a run of zero tests. For an uncolored baseline use `env -u FORCE_COLOR`; `FORCE_COLOR=` empty still enables color, so it is not a control. State your search scope AND what your pattern would have missed. A grep over one ref does not support a claim about history; a pattern built from the token a diff removed does not support a claim that the category is empty. A failure injection with no positive control has produced NO result, never a negative one. Before you read an injected fault — an env var, an argv word, a mutant — as having had no effect, prove the injection reached the child: one run whose output differs with it present versus absent, or the child echoing the injected value back. Uncontrolled, the cell is unrun — say so in your verdict instead of reporting a no-effect result. Build such an invocation as an array expanded braced and quoted — `cfg=(SETB=1 BADJ=1); env "${cfg[@]}" sh ./probe.sh` — or inline the assignments literally — `env SETB=1 BADJ=1 sh ./probe.sh`; NEVER from an unquoted scalar — `cfg="SETB=1 BADJ=1"; env $cfg sh ./probe.sh` — which under zsh passes ONE argument, sets a variable literally named `SETB` to `1 BADJ=1`, never sets `BADJ` at all, and still exits 0. `env $cfg[@]` is not the portable spelling either: measured, bash word-splits it into `SETB=1` and `BADJ=1[@]`, so the injection variable is set to a corrupted value, while zsh behaves exactly as with the bare `$cfg` — `BADJ` never set, exit 0. Everything you write — mutants, fixtures, scratch repos — goes under `<scratch>/pr<N>/<finding>/` and nowhere else; the checkout and any worktree are never write targets, though `git show`/`git archive` at a pinned ref read fine anywhere. Chain the directory change into the command, `cd "$D" && git …`, never `cd "$D"; git …`, so a failed `cd` cannot leave a `git` command running in the checkout — and bracket a fixture's own git with `git rev-parse --show-toplevel`: before `git init` it must NOT resolve to the repository, and a fresh scratch dir's `fatal: not a git repository` (exit 128) is the pass, not a failure; before any `git commit` it must resolve to your scratch path — compare resolved forms (`realpath`), since `--show-toplevel` can report `/private/tmp/…` for a `/tmp` scratch dir on macOS.* That last clause is the whole mechanism — a refuter left to reason its way to agreement rubber-stamps, and "apply only what survives" silently degrades into "apply everything in scope". Apply only what survives; a refuted one defers with the refutation in its body. "In scope" means the ticket's scope, not merely the same files — and applying one still never licenses editing a file the ticket has no business in. **A finding in `unverified` whose refuters ran and crashed always defers, whatever its severity** — and which of the two it is, you read off the finding's **`refutersDispatched`**, never off severity and never off an empty vote list: above zero with nothing surviving means every refuter dispatched against it died, and at `critical` that is every one of them. Only a `survived` finding has been checked; severity says how much a finding would matter if true, never whether anything looked. **Deferring a crash is your move, and re-running it is not** — the review already re-dispatched every crashed refuter pair once, in-run, so a crash that reaches you crashed twice, and the result's `resume` field names the one move left, which is not yours: nothing replays a review that ran in a runner's kernel — it is reported, not acted on. So say *in the deferral* that its refuters crashed rather than that it went unchecked, and re-run nothing for it. **A `suggestion` is also in `unverified`, for a different reason — `refutersDispatched` of zero, the 0-refuter budget the workflow gives that band by policy, so nothing looked *there* — and the scope split above, not this rule, covers it.**
11
+ 2. Plan the actions. Split explicitly into **apply now** / **defer**. A finding is deferred, never dropped. **Every `suggestion` — the review's `severity`, the plugin's *Suggestions* bucket — arrives unchecked: the review budgets that band 0 refuters, so nothing has looked at it yet.** Checking it is now your job, not nobody's. So split it by SCOPE first. Out of the PR's ticket scope → defer and file. In scope → dispatch one refuter against it, biased to refuse, and hand it this instruction verbatim with one substitution: resolve `<scratch>/pr<N>/<finding>/` in it to an absolute path under your own run root — `mktemp -d` once per round for `<scratch>/pr<N>/fix-XXXXXXXX/`, never bare `pr<N>`, since finding ids restart at 1 each round too — and write that absolute path into the refuter's prompt in its place; a refuter never derives its own path, and tell it your step-3 `testCmd`, which its rules call `<testCmd>`: *Try to REFUTE this finding. Default to refuted=true if uncertain. Verify by RUNNING something — compile it, run the test, apply the mutation. Do not reason your way to agreement. Observe that run synchronously — run the command, wait for it, read its exit code. Never poll a log file for a completion marker: prefer ONE blocking run to a poll loop, and treat its return as permission to look, never as the answer. Reading a log the run has already finished writing is fine; waiting on one is not. If you match a test reporter's own output, accepting both `ℹ` and `#` is necessary but NOT sufficient — strip SGR escapes first as well. node's prefix moves with the node version and with whether stdout is a TTY, and color wraps the whole line so it begins with ESC and no prefix anchor matches at all, which returns empty at exit 0 — indistinguishable from a hung run and from a run of zero tests. For an uncolored baseline use `env -u FORCE_COLOR`; `FORCE_COLOR=` empty still enables color, so it is not a control. State your search scope AND what your pattern would have missed. A grep over one ref does not support a claim about history; a pattern built from the token a diff removed does not support a claim that the category is empty. When your check scores a mutant killed or a test red, run only in your own copy of the tree, and these rules decide it and nothing else: Valid red: a run that completed — the command returned on its own, with no signal and no timeout, deadline or kill having fired — in which the test the claim names is reported failing. An exit ≠ 0 with no failing test is not red, and a different test failing is not red for this claim. A red in the full suite only triggers a narrowed run; it is never evidence on its own. Narrowed run: the Test entrypoint with arguments appended, `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all that is promised about them, so a narrowed run counts only if it reports `tests` > 0 and its output names the claimed test. If no narrowing can be proven, run the full `<testCmd>` and read the named test's own result from its output. Kill: the mutant produced a valid red in a narrowed run, and the same narrowed command, in the same tree with the mutant reverted, ran a valid green with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A completed exit ≠ 0 with the named test not failing is not a kill, and no kill stands without that green run on the unmutated tree. The result is inconclusive, never a kill, when the red does not reproduce in the narrowed run, when the unmutated tree is red too, or when no usable run exists. Evidence line: every kill and every valid red you claim carries one line in the claim's own free-text field: `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`. A failing-test finding filed off the review's shared run carries the `red:` half alone. This refuter has no abstain: a run that could not decide — neither a valid red nor a valid green — is no kill and no refutation, so name the runs in `reason`, each with its evidence line, and leave `refuted` true, so the suggestion defers on the record instead of applying on a check that decided nothing. A failure injection with no positive control has produced NO result, never a negative one. Before you read an injected fault — an env var, an argv word, a mutant — as having had no effect, prove the injection reached the child: one run whose output differs with it present versus absent, or the child echoing the injected value back. Uncontrolled, the cell is unrun — say so in your verdict instead of reporting a no-effect result. Build such an invocation as an array expanded braced and quoted — `cfg=(SETB=1 BADJ=1); env "${cfg[@]}" sh ./probe.sh` — or inline the assignments literally — `env SETB=1 BADJ=1 sh ./probe.sh`; NEVER from an unquoted scalar — `cfg="SETB=1 BADJ=1"; env $cfg sh ./probe.sh` — which under zsh passes ONE argument, sets a variable literally named `SETB` to `1 BADJ=1`, never sets `BADJ` at all, and still exits 0. `env $cfg[@]` is not the portable spelling either: measured, bash word-splits it into `SETB=1` and `BADJ=1[@]`, so the injection variable is set to a corrupted value, while zsh behaves exactly as with the bare `$cfg` — `BADJ` never set, exit 0. Everything you write — mutants, fixtures, scratch repos — goes under `<scratch>/pr<N>/<finding>/` and nowhere else; the checkout and any worktree are never write targets, though `git show`/`git archive` at a pinned ref read fine anywhere. Chain the directory change into the command, `cd "$D" && git …`, never `cd "$D"; git …`, so a failed `cd` cannot leave a `git` command running in the checkout — and bracket a fixture's own git with `git rev-parse --show-toplevel`: before `git init` it must NOT resolve to the repository, and a fresh scratch dir's `fatal: not a git repository` (exit 128) is the pass, not a failure; before any `git commit` it must resolve to your scratch path — compare resolved forms (`realpath`), since `--show-toplevel` can report `/private/tmp/…` for a `/tmp` scratch dir on macOS.* That last clause is the whole mechanism — a refuter left to reason its way to agreement rubber-stamps, and "apply only what survives" silently degrades into "apply everything in scope". Apply only what survives; a refuted one defers with the refutation in its body. "In scope" means the ticket's scope, not merely the same files — and applying one still never licenses editing a file the ticket has no business in. **A finding in `unverified` whose refuters ran and crashed or abstained always defers, whatever its severity** — and which it is, you read off the finding's **`refutersDispatched`** and **`refutersInconclusive`**, never off severity and never off an empty vote list: `refutersDispatched` above zero with nothing decided means every refuter dispatched against it died or abstained, and `refutersInconclusive` counts the ones that abstained. Only a `survived` finding has been checked; severity says how much a finding would matter if true, never whether anything looked. **Deferring a crash is your move, and re-running it is not** — the review already re-dispatched every crashed refuter pair once, in-run, so a crash that reaches you crashed twice, and the result's `resume` field names the one move left, which is not yours: nothing replays a review that ran in a runner's kernel — it is reported, not acted on. So say *in the deferral* that its refuters crashed rather than that it went unchecked, and re-run nothing for it. **An abstention is not a crash** — a refuter that abstained ran and could not decide: defer and file the finding, citing the inconclusive runs each abstaining refuter named in its `reason`, and never describe its refuters as crashed. **A `suggestion` is also in `unverified`, for a different reason — `refutersDispatched` of zero, the 0-refuter budget the workflow gives that band by policy, so nothing looked *there* — and the scope split above, not this rule, covers it.**
12
12
  **Write every ruling to `<scratch>/dispositions-<your name>.json`, `<your name>` the `fix-pr-<pr>[-x]` you were dispatched as (standalone, `fix-pr-<pr>`) — the disposition record `dispositions-check.mjs` judges against the review result file before any finisher is dispatched.** It is `{"head": "<the review file's head>", "run": "<the run root's name in the review path>", "entries": [ … ]}` — `run` the `<run>` of the review path your prompt names, which the check holds against the one the PR's latest `reviewed=` names, so a record written against an earlier review of the same PR, a same-head one included, is a mismatch (standalone, with no ledger naming a run, leave it out): one entry per `survived` finding and per `unverified` finding, plus one per `refuted` finding you reverse. Each entry carries `bucket` (`survived|unverified|refuted`), `index` (its position in that bucket of the review file — stable, since a round's file never changes), `scope` (`in|out`), `claimKind` (`behavior|shape`) and `disposition` (`apply|defer`), and where they apply `reason`, `issue` (the number you filed it to or commented on), `verdictPath` (an in-scope suggestion's refuter verdict file) and `remedyFiles` (the files its remedy names). A `refuted` entry carries `scope`, `claimKind` and `disposition` like every other entry, and a non-empty `reason` holding the evidence that reverses it; a missing or blank `reason` is a mismatch. The check reads scope off the diff before it reads your entry: a finding with no `line`, or on a line `git diff` touches from the PR's merge-base with `origin/main` to the review's `head`, is in scope whatever `scope` says, and only otherwise does the declared one stand. An in-scope `survived` finding deferred passes only with `reason` `false-rationale`, `mutual-exclusion`, `remedy-worse` or `remedy-outside-diff`. `remedy-outside-diff` — the remedy would edit a file the PR's diff does not touch — also needs `remedyFiles` to name at least one file absent from `git diff` from that merge-base to the review's `head`: none named, or every one in that diff, is a mismatch. A `critical` or `important` finding deferred for it goes to a human instead of passing: the script exits 1 with verdict `escalate`, `ledger.mjs dispatch` refuses the PR's finisher, and nothing retries it; a `suggestion` deferred for it passes. Any other reason, none, or a `survived` or `unverified` finding with no entry is a mismatch: the script exits 1 naming each violating entry's bucket, index and rule, and `ledger.mjs dispatch` refuses the PR's finisher. `disposition` has no `dismiss`: an in-scope `survived` finding verified correct, needing no change, is deferred with `reason` `false-rationale` and filed on the closed suggestion-band record — step 5's `PR #<pr> review: the suggestion band, checked`.
13
13
  3. **Run `testCmd` before you commit**, from the worktree, copied verbatim — not a runner you picked. `testCmd` is the command your dispatcher handed you; **standalone, with no dispatcher to hand you one, it is the repo's own test command**, the Test entrypoint in the repo's Recipe cache (`derive-testcmd.sh <repo> test` prints it). Read it off the repo, never invent one: "not a runner you picked" is what this step forbids, and on the standalone path nothing else defines the referent. With no usable Recipe cache (`derive-testcmd.sh` refuses), run the Recipe derivation step first — dispatch a `task` member, agent `fleet-recipe-deriver`, its prompt naming the main checkout's absolute path as `<repo>` and a scratch directory of its own; it proves both commands in a throwaway worktree and writes the cache only on proof, and its `RECIPE NOT PROVEN` is a stop, reported with its reason. Never fall back to a runner of your own choosing. `tests 0` is a FAILED run, not a pass: a glob matching nothing still exits 0. **Zero is not the only no-work count.** A run reporting **0 passes with no failures** skipped everything it collected, and a count **materially below the whole suite** means you ran a partial tree — this suite reported `tests 937` when this line was written, so compare what you get against a full run rather than against zero. The number drifts; that is the safe direction, because a stale one produces a loud false red while its absence produces a silent false green. Red, zero-test, or either no-work count → fix it or move the finding to defer; never commit over it. Then commit and push — `git push origin HEAD:<branch>`, the explicit refspec: this step also runs from a `--detach` worktree (`run-team/SKILL.md`'s worktree-recreation form), where a bare `git push` exits 128 (`fatal: You are not currently on a branch`) and any push whose destination is `HEAD` rather than a full or branch refname (`-u` or not, forced or not) exits 1 (`error: The destination you provided is not a full refname`) — and report the test result alongside the SHA. **Then reconcile the PR body with what you actually did.** It merges into the permanent record and nothing downstream re-derives it: your commit messages are yours to fix and usually get fixed, the body is neither. Removing a change, or reversing one, leaves the body asserting work the diff no longer contains. Measured twice in one run — one body kept asserting a calibration its own fix had just deleted as false, another kept claiming a hunk the fix had removed as "this diff owns it" — both caught by a finisher's reading rather than by any rule, and both would otherwise have merged intact. `gh pr edit <pr> --body-file <path>`, and re-read the body against the final diff rather than against your memory of it. **Never `--no-verify`.** The fleet runs in repos it knows nothing about: it must not assume a pre-commit hook exists, and must never bypass one that does — a hook that fails is a finding to report, not an obstacle. Nothing re-reviews what you commit here, so this run is the only gate between your edit and a merge.
14
14
  4. **Under the fleet, do not hold this wait.** The controller owns a persistent CI Monitor and dispatches a finisher once your report has arrived and CI is green — so push, then report to the controller a **final verdict** (every specialist you dispatched + its outcome + your apply/defer adjudication + the pushed SHA) and stop. A specialist that has neither reported nor been relayed to you is not a verdict line — ask the controller for it by name before ruling; see **Specialists**. The controller keys "review complete" on that verdict, **never on you merely going idle** — a slow specialist can still be running when your turn ends, and finishing off idle-alone ships a premature label (observed: a PR labelled while its contract-test specialist was still running and about to surface a vacuous pin). A turn-based reviewer re-reading `gh pr checks` each idle cycle rebuilds its whole context for nothing the Monitor lacks. **Standalone (no controller): watch** the run until the diff-validating jobs are green (bind via `ci-state.mjs` below); fix a **genuine** failure and repeat from 3. **Staleness is not a failure:** a `rebase-check` red or heavy jobs `skipped` **solely from a non-zero behind-count** never clears by waiting and must not be rebased away here — see **`skipped` is not `passed`** and **Do NOT rebase to label** below for what it looks like and why. Stop watching and go to 6. A turn-based reviewer that "waits for green" on a behind PR stalls forever and ends its turn without labelling. **`ci-state.mjs` returns `verdict: "no-ci"`** → there is no run to watch, so don't: go straight to 6 and let that step's `--declare-no-ci` branch decide.
@@ -48,7 +48,8 @@ R=<scratch>/pr<pr>/<run>/review.json
48
48
  jq '{pr, head, resume, testEnvironment, dimensionsRun, dimensionsUnrun, cwdAudit, counts}' "$R" # the digest
49
49
  jq -r '.testEnvironment' "$R" # what the shared test run is evidence about
50
50
  jq '.survived' "$R" # apply candidates, every one
51
- jq '[.unverified[] | select(.refutersDispatched > 0)]' "$R" # refuters crashed, twice: defer, naming the crash
51
+ jq '[.unverified[] | select(.refutersDispatched > (.refutersInconclusive // 0))]' "$R" # a refuter crashed: defer, naming the crash
52
+ jq '[.unverified[] | select(.refutersDispatched > 0 and .refutersDispatched == .refutersInconclusive)]' "$R" # every refuter abstained: defer and file, citing the inconclusive runs in its votes' `reason`, never as crashed
52
53
  jq '[.unverified[] | select(.refutersDispatched == 0)]' "$R" # suggestions, 0-refuter budget: step 2's scope split
53
54
  jq '.refuted' "$R" # refuted: yours to reverse on new evidence
54
55
  ```
@@ -97,5 +98,5 @@ Tell readers **`git show <sha>:<path>` is the source of truth** — the object s
97
98
 
98
99
  - **Verify, don't reason.** "Removing X makes this compile/fail" → compile it in an isolated copy. One command. Four successive confident claims about one file were all wrong; the compiler settled it.
99
100
  - **Distrust negative claims hardest.** "Nothing else references this", "the sweep is clean" — most likely false, least likely checked: a grep that found nothing looks like a grep never run. Make the specialist state its search scope. Negative claim vs specific finding with paths → paths win.
100
- - **Mutation-verify any "this test pins X".** Apply the mutation, confirm *that* test fails, revert. Then one it should **not** catch, staying green — else you proved it fails, not that it discriminates. **One syntactic form is not the class:** a stripper killing `// whole-line` passed a guard that `code; // trailing` walked through.
101
+ - **Mutation-verify any "this test pins X".** Apply the mutation, confirm *that* test fails — a valid red, and a kill only against the green run on the unmutated tree, as step 2's refuter rules define them — revert. Then one it should **not** catch, staying green — else you proved it fails, not that it discriminates. **One syntactic form is not the class:** a stripper killing `// whole-line` passed a guard that `code; // trailing` walked through.
101
102
  - **Comments are findings**, and the most common one. A comment asserting what the code does not do is a defect. One run shipped five: an invented citation, a mis-stated failure mode, a test file named as covering what it had zero coverage of, an overstated cast fix, a claim its own dependency's source contradicts. Check every added assertion against the tree — **including comments in files this diff does not touch** but whose claims it falsifies (test-name references, "N of 3" counts, tracking-issue pointers).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@feigi/fleet-ctl",
3
- "version": "3.33.0",
3
+ "version": "3.35.0",
4
4
  "description": "Agent fleet: run-team controller, merge bot, PR reviewer, and the ticket pipeline they share",
5
5
  "license": "Apache-2.0",
6
6
  "repository": {
@@ -85,7 +85,8 @@
85
85
  // run root, `<scratch>/pr<N>/fix-XXXXXXXX/<finding>/`, holding a JSON
86
86
  // object that validates against the refuter verdict schema
87
87
  // `{refuted, reason}`. No `verdictPath`, a path outside that root, a file
88
- // missing, or one failing the schema is a mismatch. `refuted: true` puts
88
+ // missing, or one failing the schema — or abstaining, `inconclusive:
89
+ // true` — is a mismatch. `refuted: true` puts
89
90
  // the finding on filing row 4, where applying it is a mismatch too;
90
91
  // `refuted: false` on row 5.
91
92
  // - Filing: every deferral names in `issue` the issue it was filed to, and
@@ -98,6 +99,7 @@
98
99
  // reason other than false-rationale
99
100
  // 2 survived, in scope, deferred false-rationale closed suggestion-band record
100
101
  // 3 unverified, its refuters dispatched and crashed open, needs-triage
102
+ // or abstained
101
103
  // 4 in-scope suggestion, its refuter refuted it closed suggestion-band record
102
104
  // 5 in-scope suggestion, its refuter let it survive applied, or deferred as row 1
103
105
  // 6 out-of-scope suggestion alleging wrong behavior open, needs-triage
@@ -181,7 +183,7 @@ const ENUMS = { scope: ["in", "out"], claimKind: ["behavior", "shape"], disposit
181
183
  export const FILING_ROWS = Object.freeze({
182
184
  1: { finding: "an in-scope survived finding deferred for an allowed reason other than false-rationale", open: "ready-for-agent" },
183
185
  2: { finding: "an in-scope survived finding deferred false-rationale", record: true },
184
- 3: { finding: "an unverified finding whose refuters crashed", open: "needs-triage" },
186
+ 3: { finding: "an unverified finding whose refuters crashed or abstained", open: "needs-triage" },
185
187
  4: { finding: "an in-scope suggestion its refuter refuted", record: true },
186
188
  5: { finding: "an in-scope suggestion its refuter let survive", open: "ready-for-agent" },
187
189
  6: { finding: "an out-of-scope suggestion alleging wrong behavior", open: "needs-triage" },
@@ -224,8 +226,13 @@ function describeIssue({ state, title, labels }) {
224
226
  return `${state.toLowerCase()}, titled ${JSON.stringify(title)}, labelled ${labels.length === 0 ? "nothing" : labels.join(", ")}`;
225
227
  }
226
228
 
229
+ const ABSTAINED = "is inconclusive — its refuter abstained, so it decided nothing about the finding";
230
+
227
231
  // Why `value` is not a refuter verdict, or null when it is: the shape
228
- // VERDICT_SCHEMA declares, checked key by key.
232
+ // VERDICT_SCHEMA declares, checked key by key. An abstaining verdict
233
+ // (`inconclusive: true`) is refused on top: the in-scope suggestion refuter
234
+ // this reads is never offered an abstain, and one that abstained anyway
235
+ // decided nothing a filing row can be keyed on.
229
236
  export function verdictProblem(value) {
230
237
  if (value === null || typeof value !== "object" || Array.isArray(value)) return "is not a JSON object";
231
238
  for (const key of VERDICT_SCHEMA.required) {
@@ -236,6 +243,7 @@ export function verdictProblem(value) {
236
243
  if (declared === undefined) return `carries ${key}, which a refuter verdict does not`;
237
244
  if (typeof v !== declared.type) return `has a ${key} that is not a ${declared.type}`;
238
245
  }
246
+ if (value.inconclusive === true) return ABSTAINED;
239
247
  return null;
240
248
  }
241
249
 
@@ -672,7 +680,9 @@ function verdictReader(scratch, pr) {
672
680
  return { problem: `cannot be read as JSON — ${e.message}` };
673
681
  }
674
682
  const why = verdictProblem(value);
675
- return why === null ? { refuted: value.refuted } : { problem: `fails the refuter verdict schema — it ${why}` };
683
+ if (why === null) return { refuted: value.refuted };
684
+ // An abstention validates against the schema; it is refused as evidence, not as a malformed file.
685
+ return { problem: why === ABSTAINED ? why : `fails the refuter verdict schema — it ${why}` };
676
686
  };
677
687
  }
678
688
 
@@ -0,0 +1,188 @@
1
+ #!/usr/bin/env node
2
+ // The consumer-repo pre-flight: what the fleet needs from this machine and
3
+ // this repository before a run-team run may shortlist or dispatch anything.
4
+ // run-team calls it at the start of Phase 0, so the checks run without the
5
+ // maintainer copying a block by hand.
6
+ //
7
+ // THE CHECKS are the CHECKS table below, one probe per row, each run with the
8
+ // main checkout as its cwd. A row passes when its argv exits 0 and, where the
9
+ // row says `want`, its stdout carries a line equal to that string — or, for
10
+ // `want: NONEMPTY`, any non-blank line. A row marked `warn` reports a miss as a
11
+ // WARN and passes: the CI-workflow row, because a repo without a workflow
12
+ // named CI still runs, its workflow named per call (`ci-state.mjs
13
+ // --workflow`) or declared absent (`--declare-no-ci`). Every other miss is a
14
+ // FAIL. Every row runs, so one run names every failure, not the first.
15
+ //
16
+ // No row names a manifest, a lockfile or a test runner: the consumer's Install
17
+ // step and Test entrypoint come from its Recipe, and the fleet keeps no table
18
+ // of supported technologies.
19
+ //
20
+ // THE MARKER. A run where nothing FAILed writes `.fleet/preflight.json`
21
+ // (resolved through fleet-dir.mjs, so a worktree and the main checkout name
22
+ // the same file) holding a sha256 of the check set: every row's name, argv,
23
+ // `want` and `warn`. A later run whose check set hashes the same prints
24
+ // PREFLIGHT SKIPPED and runs nothing — the rows cost several `gh` calls, and
25
+ // every controller tick already spends its own share of the secondary rate
26
+ // limit. A row added, removed or changed moves the hash, so the next run
27
+ // checks again. A failed run writes no marker. A marker that cannot be read
28
+ // is treated as absent. Deleting the file makes the next run check again.
29
+ //
30
+ // CLI (no arguments):
31
+ // preflight.mjs one `ok|WARN|FAIL <name>[: why]` line per row, then
32
+ // `PREFLIGHT OK`, `PREFLIGHT SKIPPED` or
33
+ // `PREFLIGHT FAILED: <names>`.
34
+ // Exit 0 passed or skipped, 1 a row failed, 2 could not run
35
+ // (an argument given, no repository) or could not write the
36
+ // marker after every row passed, the row lines printed first.
37
+ //
38
+ // Zero deps: node builtins and sibling scripts only.
39
+
40
+ import { spawnSync } from "node:child_process";
41
+ import { createHash } from "node:crypto";
42
+ import { mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
43
+ import { dirname, join } from "node:path";
44
+ import { isCLI } from "./is-cli.mjs";
45
+ import { makeDie } from "./arg.mjs";
46
+ import { fleetFile } from "./fleet-dir.mjs";
47
+ import { gitEnv } from "./git-env.mjs";
48
+
49
+ export const MARKER_FILE = "preflight.json";
50
+ export const NONEMPTY = { nonEmpty: true };
51
+ const PROBE_TIMEOUT_MS = 60_000;
52
+ const NAME = "preflight";
53
+ const die = makeDie(NAME);
54
+
55
+ const label = (l) => ({ name: `label:${l}`, argv: ["gh", "label", "list", "--search", l, "--json", "name", "--jq", ".[].name"], want: l });
56
+
57
+ // A workflow file whose top-level `name:` is CI, matched the way ci-state.mjs
58
+ // reads a workflow's name: either quote style, and a trailing ` # comment` is
59
+ // not part of the name. A plain `^name: *CI *$` would WARN for a repo
60
+ // `--workflow CI` resolves without trouble.
61
+ const CI_NAMED = String.raw`grep -lE "^name:[[:space:]]*[\"']?CI[\"']?([[:space:]]+#.*)?[[:space:]]*\$" .github/workflows/*.y*ml`;
62
+
63
+ // A leading `~/` in an argv element is the user's home, expanded at run time
64
+ // and hashed as written, so the hash does not change with the box.
65
+ export const CHECKS = [
66
+ { name: "binary:node", argv: ["node", "-v"] },
67
+ { name: "binary:git", argv: ["git", "--version"] },
68
+ { name: "binary:gh", argv: ["gh", "--version"] },
69
+ { name: "binary:jq", argv: ["jq", "--version"] },
70
+ { name: "binary:python3", argv: ["python3", "--version"] },
71
+ { name: "binary:shasum", argv: ["shasum", "--version"] },
72
+ { name: "gh-auth", argv: ["gh", "auth", "status"] },
73
+ { name: "origin-remote", argv: ["git", "remote", "get-url", "origin"] },
74
+ { name: "origin-main", argv: ["git", "rev-parse", "--verify", "-q", "origin/main"] },
75
+ { name: "worktrees-ignored", argv: ["git", "check-ignore", "-q", ".worktrees/probe"] },
76
+ { name: "fleet-ignored", argv: ["git", "check-ignore", "-q", ".fleet/probe"] },
77
+ label("ready-for-agent"),
78
+ label("in-progress"),
79
+ label("ready-to-merge"),
80
+ { name: "allow-merge-commit", argv: ["gh", "api", "repos/{owner}/{repo}", "--jq", ".allow_merge_commit"], want: "true" },
81
+ { name: "ruleset", argv: ["gh", "api", "repos/{owner}/{repo}/rulesets", "--jq", ".[].name"], want: NONEMPTY },
82
+ { name: "ci-workflow", argv: ["sh", "-c", CI_NAMED], warn: true },
83
+ { name: "resolver", argv: ["~/.fleet/bin/fleet-run", "--root"] },
84
+ ];
85
+
86
+ /** sha256 over every row's name, argv, want and warn, in table order. */
87
+ export function checkSetHash(checks) {
88
+ const rows = checks.map(({ name, argv, want, warn }) => [name, argv, want, warn === true]);
89
+ return createHash("sha256").update(JSON.stringify(rows)).digest("hex");
90
+ }
91
+
92
+ const wanted = (want) => (want === NONEMPTY ? "a non-blank line" : `a line "${want}"`);
93
+
94
+ // One row: null when it passed, else why it did not.
95
+ function probe({ argv, want }, cwd, env, timeoutMs) {
96
+ const shown = argv.join(" ");
97
+ if (argv.some((a) => a.startsWith("~/")) && !env.HOME) return `${shown} did not run: HOME is not set`;
98
+ const [cmd, ...args] = argv.map((a) => (a.startsWith("~/") ? join(env.HOME, a.slice(2)) : a));
99
+ const r = spawnSync(cmd, args, { cwd, env, encoding: "utf8", timeout: timeoutMs });
100
+ // ENOENT and EACCES mean it never started; ETIMEDOUT and ENOBUFS mean it ran
101
+ // and was killed, so a hung `gh` is not reported as a missing binary.
102
+ if (r.error?.code === "ETIMEDOUT") return `${shown} timed out after ${timeoutMs}ms`;
103
+ if (r.error?.code === "ENOBUFS") return `${shown} output exceeded maxBuffer`;
104
+ if (r.error) return `${shown} did not run: ${r.error.code ?? r.error.message}`;
105
+ if (r.status !== 0) {
106
+ const last = r.stderr.trim().split("\n").at(-1);
107
+ return `${shown} ${r.signal ? `killed by ${r.signal}` : `exited ${r.status}`}${last ? `: ${last}` : ""}`;
108
+ }
109
+ if (want === undefined) return null;
110
+ const lines = r.stdout.split("\n").map((l) => l.trim());
111
+ const said = lines.filter(Boolean);
112
+ const hit = want === NONEMPTY ? said.length > 0 : lines.includes(want);
113
+ return hit ? null : `${shown} printed ${said.length ? JSON.stringify(said.join(" ")) : "nothing"}, want ${wanted(want)}`;
114
+ }
115
+
116
+ function readMarker(path) {
117
+ try {
118
+ return JSON.parse(readFileSync(path, "utf8"))?.checks;
119
+ } catch {
120
+ return null;
121
+ }
122
+ }
123
+
124
+ /**
125
+ * Run `checks` from the main checkout `cwd` belongs to, unless the marker
126
+ * records this exact check set as passed. Writes the marker when no row
127
+ * failed. Throws FleetDirUnresolvable when `cwd` is in no repository.
128
+ *
129
+ * Returns `{ skipped, marker, results: [{name, warn, ok, why}], failed,
130
+ * writeError }`; `writeError` is null, or says why the marker could not be
131
+ * written — the rows have run by then, so they come back with it.
132
+ */
133
+ export function runPreflight({ checks = CHECKS, cwd = process.cwd(), env = process.env, timeoutMs = PROBE_TIMEOUT_MS } = {}) {
134
+ const marker = fleetFile(MARKER_FILE, { cwd });
135
+ const workspace = dirname(dirname(marker));
136
+ const hash = checkSetHash(checks);
137
+ if (readMarker(marker) === hash) return { skipped: true, marker, results: [], failed: [], writeError: null };
138
+
139
+ const childEnv = gitEnv({}, env);
140
+ const results = checks.map((c) => {
141
+ const why = probe(c, workspace, childEnv, timeoutMs);
142
+ return { name: c.name, warn: c.warn === true, ok: why === null, why };
143
+ });
144
+ const failed = results.filter((r) => !r.ok && !r.warn).map((r) => r.name);
145
+ let writeError = null;
146
+ if (failed.length === 0) {
147
+ const tmp = `${marker}.${process.pid}.tmp`;
148
+ try {
149
+ mkdirSync(dirname(marker), { recursive: true });
150
+ writeFileSync(tmp, `${JSON.stringify({ checks: hash, passed: new Date().toISOString() })}\n`);
151
+ try {
152
+ renameSync(tmp, marker);
153
+ } catch (e) {
154
+ rmSync(tmp, { force: true });
155
+ throw e;
156
+ }
157
+ } catch (e) {
158
+ writeError = `cannot write ${marker}: ${e?.code ?? e?.message}`;
159
+ }
160
+ }
161
+ return { skipped: false, marker, results, failed, writeError };
162
+ }
163
+
164
+ function main() {
165
+ // No flags at all, so not defineFlags: its unknown-flag refusal lists the
166
+ // accepted flags, and this script has none to list.
167
+ if (process.argv.length > 2) die(`takes no arguments, got: ${process.argv.slice(2).join(" ")}`);
168
+ let r;
169
+ try {
170
+ r = runPreflight();
171
+ } catch (e) {
172
+ die(e.message);
173
+ }
174
+ if (r.skipped) {
175
+ console.log(`PREFLIGHT SKIPPED: this check set already passed (${r.marker}; delete it to check again)`);
176
+ return;
177
+ }
178
+ for (const x of r.results) console.log(x.ok ? `ok ${x.name}` : `${x.warn ? "WARN" : "FAIL"} ${x.name}: ${x.why}`);
179
+ if (r.failed.length) {
180
+ console.log(`PREFLIGHT FAILED: ${r.failed.join(", ")}`);
181
+ process.exitCode = 1;
182
+ return;
183
+ }
184
+ if (r.writeError) die(r.writeError);
185
+ console.log(`PREFLIGHT OK (${r.marker})`);
186
+ }
187
+
188
+ if (isCLI(import.meta.url)) main();
@@ -79,6 +79,11 @@ export const VERDICT_SCHEMA = {
79
79
  refuted: { type: "boolean" },
80
80
  reason: { type: "string" },
81
81
  counter_evidence: { type: "string", description: "Command + output, if any." },
82
+ inconclusive: {
83
+ type: "boolean",
84
+ description:
85
+ "True only when your own check could produce neither a valid red nor a valid green: you abstain, and `reason` names the runs. Left out of the tally, never counted as a crash.",
86
+ },
82
87
  },
83
88
  };
84
89
 
@@ -133,6 +138,24 @@ export const TEST_RUN_SCHEMA = {
133
138
  },
134
139
  };
135
140
 
141
+ // --- Test verdict rules -------------------------------------------------
142
+ // The one statement of what a red and a kill are. Every site that scores a
143
+ // mutant killed, or files a failing test, carries these paragraphs word for
144
+ // word — the prompts below interpolate them, and the Markdown sites (the
145
+ // `tests` and verifier agent definitions, run-team's fix-applier and finisher
146
+ // briefs) quote them — so a reword lands everywhere or reds a pin. An exit
147
+ // status is never a kill on its own: a runner can exit 1 with nothing failing,
148
+ // and a load-sensitive test can fail once in a full suite and pass on rerun.
149
+ export const VALID_RED_RULE =
150
+ "**Valid red:** a run that completed — the command returned on its own, with no signal and no timeout, deadline or kill having fired — in which the test the claim names is reported failing. An exit ≠ 0 with no failing test is not red, and a different test failing is not red for this claim. A red in the full suite only triggers a narrowed run; it is never evidence on its own.";
151
+ export const NARROWED_RUN_RULE =
152
+ "**Narrowed run:** the Test entrypoint with arguments appended, `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all that is promised about them, so a narrowed run counts only if it reports `tests` > 0 and its output names the claimed test. If no narrowing can be proven, run the full `<testCmd>` and read the named test's own result from its output.";
153
+ export const KILL_RULE =
154
+ "**Kill:** the mutant produced a valid red in a narrowed run, and the same narrowed command, in the same tree with the mutant reverted, ran a valid green with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A completed exit ≠ 0 with the named test not failing is not a kill, and no kill stands without that green run on the unmutated tree. The result is inconclusive, never a kill, when the red does not reproduce in the narrowed run, when the unmutated tree is red too, or when no usable run exists.";
155
+ export const EVIDENCE_LINE_RULE =
156
+ "**Evidence line:** every kill and every valid red you claim carries one line in the claim's own free-text field: `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`. A failing-test finding filed off the review's shared run carries the `red:` half alone.";
157
+ const KILL_RULES = [VALID_RED_RULE, NARROWED_RUN_RULE, KILL_RULE, EVIDENCE_LINE_RULE].join("\n\n");
158
+
136
159
  // --- Dimension catalog --------------------------------------------------
137
160
  // No dimension carries a `model`/`effort` field —
138
161
  // dispatch tier lives ONLY in each fleet-owned agent definition's own
@@ -153,8 +176,11 @@ export const DEFAULT_DIMENSIONS = [
153
176
  {
154
177
  key: "tests",
155
178
  agentType: "fleet-review-tests",
156
- prompt:
157
- "whether each test DISCRIMINATES: apply the mutation it should catch, confirm that test goes red, revert, then apply one it should NOT catch and confirm green. Vary the syntactic form — a guard catching `// whole-line` may let `code; // trailing` through",
179
+ prompt: `whether each test DISCRIMINATES: apply the mutation it should catch and confirm a kill, revert, then apply one it should NOT catch and confirm the named test stays green. Vary the syntactic form — a guard catching \`// whole-line\` may let \`code; // trailing\` through. \`<testCmd>\` is the \`command\` your Tests paragraph names, run only in your own copy of the snapshot. A kill is scored by these rules and nothing else:
180
+
181
+ ${KILL_RULES}
182
+
183
+ An inconclusive result is no evidence either way: file no finding above \`suggestion\` on it, and name the gap — both runs — in \`scope_searched\`.`,
158
184
  },
159
185
  {
160
186
  key: "comments",
@@ -525,7 +551,8 @@ export function sharedRunNote(run, owner, key) {
525
551
  `Tests: this review ran its test command ONCE, from the snapshot's root, before
526
552
  dispatching you. Do NOT run that full command yourself — every dimension reads
527
553
  this one run. Targeted probes and mutations in your own copy of the snapshot are
528
- still yours to run.
554
+ still yours to run, and so is the full command in your own copy of the snapshot,
555
+ against your own mutant or to reproduce a failure, when no narrowed run can be proven.
529
556
  command: ${run.command}
530
557
  counts: ${countsOf(run) || "none — the run produced no counts"}
531
558
  exit: ${run.exitCode ?? "(not reported)"}
@@ -541,10 +568,20 @@ unrun for it — do not run the full command yourself to replace it.`);
541
568
  key === owner
542
569
  ? `It has ${failingDesc(run)}, and filing them is YOUR job in this review:
543
570
  file at least one finding about them, naming each failing test from the log. A
544
- failure you cannot separate from the environment is still filed — say so in the
545
- finding, so it is reproduced in the worktree before anyone acts on it. If none is
546
- filed, or this review's refuters reject every finding filed, every dimension of
547
- this review is reported unrun.`
571
+ failing test is filed off this run only on a valid red: rerun it first in a
572
+ narrowed run in your own copy of the snapshot, where \`<testCmd>\` is the
573
+ command above, and when that run does not show it failing, file no finding
574
+ above \`suggestion\` on it and name both runs. A failure you cannot separate
575
+ from the environment is still filed — say so in the finding, so it is
576
+ reproduced in the worktree before anyone acts on it. If none is filed, or this
577
+ review's refuters reject every finding filed, every dimension of this review
578
+ is reported unrun. A red is read by these rules and nothing else:
579
+
580
+ ${VALID_RED_RULE}
581
+
582
+ ${NARROWED_RUN_RULE}
583
+
584
+ ${EVIDENCE_LINE_RULE}`
548
585
  : `It has ${failingDesc(run)}, and the ${owner} dimension files the finding
549
586
  about them. Do not file a duplicate — cite a failure as evidence only where it
550
587
  bears on your own lens.`,
@@ -570,13 +607,19 @@ export function cwdAuditFrom(text) {
570
607
  return m ? { state: m[1], line: m[0].trim() } : { state: "missing", line: null };
571
608
  }
572
609
 
610
+ // An abstaining vote (`inconclusive: true`) is left out of the tally exactly
611
+ // as a crashed (`null`) one is, so a run that could not decide never breaks a
612
+ // tie toward `refuted`. It stays in `votes`, since its `reason` names the runs
613
+ // the fix-applier cites, and `refutersInconclusive` counts it — the one field
614
+ // that tells an all-abstained finding from an all-crashed one.
573
615
  export function verdictFor(dispatched, votes) {
574
616
  const live = votes.filter(Boolean);
575
- const refuted = live.filter((v) => v.refuted).length;
617
+ const decided = live.filter((v) => v.inconclusive !== true);
618
+ const refuted = decided.filter((v) => v.refuted).length;
576
619
  let verdict;
577
- if (live.length === 0) verdict = "unverified";
578
- else verdict = refuted * 2 >= live.length ? "refuted" : "survived";
579
- return { verdict, votes: live, refutersDispatched: dispatched };
620
+ if (decided.length === 0) verdict = "unverified";
621
+ else verdict = refuted * 2 >= decided.length ? "refuted" : "survived";
622
+ return { verdict, votes: live, refutersDispatched: dispatched, refutersInconclusive: live.length - decided.length };
580
623
  }
581
624
 
582
625
  // One in-run re-dispatch for a crashed dispatch, before the result is
@@ -597,12 +640,14 @@ export function retryCrashed(dispatch, crashed) {
597
640
  // omp has no cached-replay mechanism: a fresh
598
641
  // review re-dispatches every agent() live rather than only the crashed
599
642
  // legs, and the one re-dispatch this run gets was already spent in-run
600
- // — a crashed finding is reported and deferred, never resumed.
643
+ // — a crashed finding is reported and deferred, never resumed. Crashed
644
+ // means a `null` refuter: an `unverified` finding whose dispatched refuters
645
+ // all abstained is no crash, and stays out of this bucket.
601
646
  export function resumeFor(unverified) {
602
- const crashed = unverified.filter((f) => f.refutersDispatched > 0);
647
+ const crashed = unverified.filter((f) => f.refutersDispatched > (f.refutersInconclusive ?? 0));
603
648
  if (!crashed.length) return { crashed, resume: null };
604
649
  const claim =
605
- "Findings in `unverified` with `refutersDispatched` above zero and no surviving vote had every refuter die, and die again on the in-run retry — nothing looked at them. ";
650
+ "Findings in `unverified` with `refutersDispatched` above `refutersInconclusive` lost a refuter to a crash — a pair whose every vote died was re-dispatched once in-run and died again — and no refuter that ran decided them. ";
606
651
  const verb =
607
652
  "Defer them as crashed — reported, not acted on: omp's eval has no cached-replay mechanism, so a fresh review re-dispatches every agent() live rather than only the crashed legs, and the in-run retry was this review's one re-dispatch. Re-run nothing for them.";
608
653
  return { crashed, resume: claim + verb };
@@ -1051,15 +1096,17 @@ to name.`,
1051
1096
  (review && review.findings ? review.findings : []).map((f, fi) => () => {
1052
1097
  const n = verifiersForRun(f.severity);
1053
1098
  if (n === 0) return Promise.resolve({ ...f, dimension: d.key, ...verdictFor(0, []) });
1054
- // A pair whose EVERY vote died (or whose dispatch threw) is
1055
- // re-dispatched once, as a pair, before `verdictFor` reads it — one
1056
- // live vote means the pair did not crash and is ruled on that vote.
1099
+ // A pair in which no refuter decided and at least one died (or whose
1100
+ // dispatch threw) is re-dispatched once, as a pair, before
1101
+ // `verdictFor` reads it — one deciding vote means the pair did not
1102
+ // crash and is ruled on that vote, and a pair that ALL abstained ran
1103
+ // and could not decide, so a second dispatch would only repeat it.
1057
1104
  return retryCrashed(
1058
1105
  () =>
1059
1106
  parallel(
1060
1107
  Array.from({ length: n }, (_, i) => () =>
1061
1108
  agent(
1062
- `Try to REFUTE this finding from PR #${pr}. Default to refuted=true if uncertain.
1109
+ `Try to REFUTE this finding from PR #${pr}. Default to refuted=true if uncertain in your reasoning — a run that could not decide is not that uncertainty, and the abstain rule below covers it.
1063
1110
 
1064
1111
  claim: ${f.claim}
1065
1112
  where: ${f.file || "?"}:${f.line || "?"}
@@ -1082,6 +1129,17 @@ search scope AND what your pattern would have missed. A grep over one ref
1082
1129
  does not support a claim about history; a pattern built from the token a diff
1083
1130
  removed does not support a claim that the category is empty.
1084
1131
 
1132
+ When your check scores a mutant killed or a test red, \`<testCmd>\` is
1133
+ \`${testCmd}\`, run only in your own copy of the snapshot, and these rules
1134
+ decide it and nothing else:
1135
+
1136
+ ${KILL_RULES}
1137
+
1138
+ Abstain when your own check could produce neither a valid red nor a valid
1139
+ green: set \`inconclusive\` to true and name the runs in \`reason\`, each with
1140
+ its evidence line. \`refuted\` is still required, and the tally ignores it on
1141
+ an abstaining vote — an undecided run never defaults to refuted=true.
1142
+
1085
1143
  Chain the directory change into the command, \`cd "$D" && git …\`, never
1086
1144
  \`cd "$D"; git …\`, so a failed \`cd\` cannot leave a \`git\` command running in the
1087
1145
  checkout — and bracket a fixture's own git with \`git rev-parse --show-toplevel\`:
@@ -1133,7 +1191,11 @@ how three reviews from one cell left four files modified in that checkout.`,
1133
1191
  ),
1134
1192
  ),
1135
1193
  ),
1136
- (votes) => !(votes && votes.some(Boolean)),
1194
+ // Retry unless a refuter decided or every one abstained.
1195
+ (votes) =>
1196
+ !votes ||
1197
+ !votes.length ||
1198
+ (!votes.some((v) => v && v.inconclusive !== true) && !votes.every((v) => v && v.inconclusive === true)),
1137
1199
  ).then(
1138
1200
  (votes) => ({ ...f, dimension: d.key, ...verdictFor(n, votes) }),
1139
1201
  // A rejection here means retryCrashed's OWN final attempt
@@ -1183,8 +1245,9 @@ how three reviews from one cell left four files modified in that checkout.`,
1183
1245
  // Refuted findings are RETURNED, not dropped. A refutation is itself a claim,
1184
1246
  // and the controller has reversed a refutation on new evidence before.
1185
1247
  // `unverified` are findings the adversarial pass did not settle — a suggestion
1186
- // that skipped it by policy, or one whose refuters all crashed — surfaced
1187
- // separately so the caller never mistakes "not checked" for "survived".
1248
+ // that skipped it by policy, or one whose refuters crashed or abstained and
1249
+ // decided nothing — surfaced separately so the caller never mistakes "not
1250
+ // checked" for "survived".
1188
1251
  // `dimensionsRun` names what was DISPATCHED after the size trim: a trimmed
1189
1252
  // fan-out must say so, never read as full coverage. It is not a coverage claim
1190
1253
  // on its own and never was — a specialist can be dispatched and die, or the
@@ -288,6 +288,22 @@ phase, or in any later one, asks the maintainer which tickets to take.
288
288
  read that fails. `main-checkout.mjs --check` prints the same answer
289
289
  without a tick.
290
290
 
291
+ **Run the pre-flight, once the pin and the baseline are recorded.**
292
+ `~/.fleet/bin/fleet-run preflight.mjs` checks what this machine and this
293
+ repository must give the fleet — the binaries, `gh` auth, `origin/main`,
294
+ `.worktrees/` and `.fleet/` ignored, the three labels, merge commits on, a
295
+ ruleset, the Resolver — and prints one `ok`, `WARN` or `FAIL` line per
296
+ check, each naming its check. **Exit 1 stops the run:** no shortlist, no
297
+ dispatch. Report every `FAIL` line to the maintainer by its check's name;
298
+ the run starts again once they are fixed. Exit 2 means it could not run at
299
+ all, or every check passed and the marker could not be written, and stops
300
+ the run the same way. `WARN ci-workflow` stops nothing:
301
+ a repo with no workflow named CI still runs, its workflow named per call
302
+ with `--workflow` or declared absent with `--declare-no-ci`. A pass writes
303
+ `.fleet/preflight.json`, keyed to a hash of the check set, and every later
304
+ run prints `PREFLIGHT SKIPPED` without a single `gh` call until a check is
305
+ added or changed.
306
+
291
307
  **Check the merge gate against its spec, once the fast-forward has put the
292
308
  ratified spec in front of you.** Where the workspace checks a ruleset spec
293
309
  in, `.github/scripts/apply-ruleset.sh --check` compares it to the live gate
@@ -2198,11 +2214,13 @@ the environment, and this field carries which line of the cut failed. The
2198
2214
  fix-applier reads it before acting on any count from the shared test run; you
2199
2215
  read it so a degraded review never passes for a clean one.
2200
2216
  `unverified` is *not* "checked and cleared" — a `suggestion` skips the pass by
2201
- policy, and a finding whose refuters all crashed lands there too. Both go to the
2202
- fix-applier with the rest, in the file; never rule on them yourself.
2203
- **`refutersDispatched`, carried on every finding, is what tells those two
2204
- apart** — zero is the policy skip, above zero with no surviving vote is the
2205
- crash — so the population is read off that field rather than off severity,
2217
+ policy, and a finding whose refuters all crashed or abstained lands there too.
2218
+ All go to the fix-applier with the rest, in the file; never rule on them yourself.
2219
+ **`refutersDispatched`, carried on every finding, is what tells the skip
2220
+ apart** — zero is the policy skip, above zero with no deciding vote is a
2221
+ refuter pass that decided nothing — and `refutersInconclusive` splits that
2222
+ one: the refuters it counts abstained, every other one dispatched crashed. So
2223
+ the population is read off those fields rather than off severity,
2206
2224
  which records only how much a finding would matter if true. `resume` is
2207
2225
  non-null exactly when a specialist or a refuter pair crashed again after the
2208
2226
  review's own in-run retry, and **resume beats handing a crash-heavy review
@@ -2406,14 +2424,18 @@ all.
2406
2424
  > its zero counts only after the same command finds a line known to match.**
2407
2425
  >
2408
2426
  > **Apply `survived` findings. A finding in `unverified` whose refuters ran and
2409
- > crashed always defers** — and which of the two it is, you read off
2410
- > `refutersDispatched`, never off severity and never off an empty vote list:
2411
- > above zero with nothing surviving means every refuter dispatched against it
2412
- > died, and at `critical` that is every one of them. Severity records how much a
2427
+ > crashed or abstained always defers** — and which it is, you read off
2428
+ > `refutersDispatched` and `refutersInconclusive`, never off severity and never
2429
+ > off an empty vote list: `refutersDispatched` above zero with nothing decided
2430
+ > means every refuter dispatched against it died or abstained, and
2431
+ > `refutersInconclusive` counts the ones that abstained. Severity records how much a
2413
2432
  > finding would matter if true, never whether anything looked. Say *in the
2414
- > deferral* that its refuters crashed rather than that it went unchecked — the
2415
- > review re-dispatched each crash once already and reports what died again, so
2416
- > a deferral that names the crash can still be re-verified. **A `suggestion` is also in `unverified`,
2433
+ > deferral* which it was rather than that it went unchecked. Name a crash as
2434
+ > crashed refuters — the review re-dispatched each crash once already and
2435
+ > reports what died again, so a deferral that names the crash can still be
2436
+ > re-verified. An abstention is not a crash: cite the inconclusive runs each
2437
+ > abstaining refuter named in its `reason`, and never describe its refuters as
2438
+ > crashed. **A `suggestion` is also in `unverified`,
2417
2439
  > for a different reason — `refutersDispatched` of zero, the 0-refuter budget
2418
2440
  > the review gives that band by policy — and the rule below, not this one,
2419
2441
  > covers it.**
@@ -2445,7 +2467,43 @@ all.
2445
2467
  > > empty still enables color, so it is not a control. State your search scope
2446
2468
  > > AND what your pattern would have missed. A grep over one ref does not
2447
2469
  > > support a claim about history; a pattern built from the token a diff removed
2448
- > > does not support a claim that the category is empty. A failure injection with
2470
+ > > does not support a claim that the category is empty.
2471
+ > >
2472
+ > > When your check scores a mutant killed or a test red, run only in your own
2473
+ > > copy of the tree, and these rules decide it and nothing else:
2474
+ > >
2475
+ > > **Valid red:** a run that completed — the command returned on its own, with no
2476
+ > > signal and no timeout, deadline or kill having fired — in which the test the
2477
+ > > claim names is reported failing. An exit ≠ 0 with no failing test is not red,
2478
+ > > and a different test failing is not red for this claim. A red in the full
2479
+ > > suite only triggers a narrowed run; it is never evidence on its own.
2480
+ > >
2481
+ > > **Narrowed run:** the Test entrypoint with arguments appended, `<testCmd>
2482
+ > > <args>`. That appended arguments reach the Test entrypoint is all that is
2483
+ > > promised about them, so a narrowed run counts only if it reports `tests` > 0
2484
+ > > and its output names the claimed test. If no narrowing can be proven, run the
2485
+ > > full `<testCmd>` and read the named test's own result from its output.
2486
+ > >
2487
+ > > **Kill:** the mutant produced a valid red in a narrowed run, and the same
2488
+ > > narrowed command, in the same tree with the mutant reverted, ran a valid green
2489
+ > > with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A
2490
+ > > completed exit ≠ 0 with the named test not failing is not a kill, and no kill
2491
+ > > stands without that green run on the unmutated tree. The result is
2492
+ > > inconclusive, never a kill, when the red does not reproduce in the narrowed
2493
+ > > run, when the unmutated tree is red too, or when no usable run exists.
2494
+ > >
2495
+ > > **Evidence line:** every kill and every valid red you claim carries one line
2496
+ > > in the claim's own free-text field: `red: <cmd> → exit N, fail K incl <test>;
2497
+ > > baseline: <cmd> → exit 0, <test> pass`. A failing-test finding filed off the
2498
+ > > review's shared run carries the `red:` half alone.
2499
+ > >
2500
+ > > This refuter has no abstain: a run that could not decide — neither a valid red
2501
+ > > nor a valid green — is no kill and no refutation, so name the runs in
2502
+ > > `reason`, each with its evidence line, and leave `refuted` true, so the
2503
+ > > suggestion defers on the record instead of applying on a check that decided
2504
+ > > nothing.
2505
+ > >
2506
+ > > A failure injection with
2449
2507
  > > no positive control has produced NO result, never a negative one. Before you
2450
2508
  > > read an injected fault — an env var, an argv word, a mutant — as having had
2451
2509
  > > no effect, prove the injection reached the child: one run whose output
@@ -2536,11 +2594,42 @@ all.
2536
2594
  > claim, never your patch.
2537
2595
  >
2538
2596
  > **A test you ADD must kill its own mutant** — break what it pins, confirm it and
2539
- > only it goes red, restore. **Then a change it should *not* catch, staying
2540
- > green** — else you proved it fails, not that it discriminates: a pin asserting
2541
- > whole-file text clears "it and only it goes red" and still reddens on any edit
2542
- > (measured). A green suite says nothing about a new test: one pin this run
2543
- > survived the exact mutation it was named for.
2597
+ > only it goes red and that the red is a kill by the rules below, restore. **Then
2598
+ > a change it should *not* catch, staying green** — else you proved it fails, not
2599
+ > that it discriminates: a pin asserting whole-file text clears "it and only it
2600
+ > goes red" and still reddens on any edit (measured). A green suite says nothing
2601
+ > about a new test: one pin this run survived the exact mutation it was named for.
2602
+ >
2603
+ > **Valid red:** a run that completed — the command returned on its own, with
2604
+ > no signal and no timeout, deadline or kill having fired — in which the test
2605
+ > the claim names is reported failing. An exit ≠ 0 with no failing test is not
2606
+ > red, and a different test failing is not red for this claim. A red in the full
2607
+ > suite only triggers a narrowed run; it is never evidence on its own.
2608
+ >
2609
+ > **Narrowed run:** the Test entrypoint with arguments appended,
2610
+ > `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all
2611
+ > that is promised about them, so a narrowed run counts only if it reports
2612
+ > `tests` > 0 and its output names the claimed test. If no narrowing can be
2613
+ > proven, run the full `<testCmd>` and read the named test's own result from its
2614
+ > output.
2615
+ >
2616
+ > **Kill:** the mutant produced a valid red in a narrowed run, and the same
2617
+ > narrowed command, in the same tree with the mutant reverted, ran a valid green
2618
+ > with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A
2619
+ > completed exit ≠ 0 with the named test not failing is not a kill, and no kill
2620
+ > stands without that green run on the unmutated tree. The result is
2621
+ > inconclusive, never a kill, when the red does not reproduce in the narrowed
2622
+ > run, when the unmutated tree is red too, or when no usable run exists.
2623
+ >
2624
+ > **Evidence line:** every kill and every valid red you claim carries one line
2625
+ > in the claim's own free-text field:
2626
+ > `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`.
2627
+ > A failing-test finding filed off the review's shared run carries the `red:`
2628
+ > half alone.
2629
+ >
2630
+ > An inconclusive result proves nothing about the test: do not report the test
2631
+ > as proven, and measure again in a clean state. Put each kill's evidence line in
2632
+ > your report, beside the test result.
2544
2633
  >
2545
2634
  > **A mutation that never landed is not a green — it is a cell that did not
2546
2635
  > run.** Same rule for any fault you inject to see what breaks: prove the
@@ -2796,7 +2885,7 @@ a minute apart showed *different* mutants, so a member's report and any single
2796
2885
  **Whatever you verify by RUNNING, run in a tree nobody else owns.**
2797
2886
  Independently re-running the ticket's acceptance mutation — loosen the
2798
2887
  regex, delete the guard clause, flip the per-entry reset, run `<testCmd>`,
2799
- watch it redden, discard — is the half of this duty reading a diff cannot
2888
+ confirm a kill by the test-run verdict rule below, discard — is the half of this duty reading a diff cannot
2800
2889
  do, and it stays. But it **writes**, and the PR's worktree belongs to
2801
2890
  another member. Measured once: a finisher was inside that worktree at the
2802
2891
  instant the fix-applier announced one more clause to land, and neither
@@ -2924,11 +3013,43 @@ it reached, so reading that summary alone reads a partial suite as a whole one:
2924
3013
  > `other` and the exit code plus the summary line as evidence; label nothing.
2925
3014
  > The same holds for the mutation gate: a killed run is not a mutant that
2926
3015
  > reddened, and the files it cancelled are not failures — only a run that
2927
- > completed can kill a mutant. A completed run that exits 1 is a plain failure,
2928
- > not a killed run, whatever its `cancelled` count, and exit 0 with `tests 0`
2929
- > stays FAILED.
3016
+ > completed can kill a mutant. A completed run that exits 1 is not a killed run
3017
+ > either, whatever its `cancelled` count, and not a killed mutant on its exit
3018
+ > status alone: the acceptance mutation scores a kill only by the rules below.
3019
+ > Exit 0 with `tests 0` stays FAILED.
2930
3020
  > Do not wrap `<testCmd>` in an external `timeout`. Run a long suite in one
2931
3021
  > `eval` cell with `timeout: 0`, as the review runner does, and wait for it.
3022
+ >
3023
+ > **Valid red:** a run that completed — the command returned on its own, with
3024
+ > no signal and no timeout, deadline or kill having fired — in which the test
3025
+ > the claim names is reported failing. An exit ≠ 0 with no failing test is not
3026
+ > red, and a different test failing is not red for this claim. A red in the full
3027
+ > suite only triggers a narrowed run; it is never evidence on its own.
3028
+ >
3029
+ > **Narrowed run:** the Test entrypoint with arguments appended,
3030
+ > `<testCmd> <args>`. That appended arguments reach the Test entrypoint is all
3031
+ > that is promised about them, so a narrowed run counts only if it reports
3032
+ > `tests` > 0 and its output names the claimed test. If no narrowing can be
3033
+ > proven, run the full `<testCmd>` and read the named test's own result from its
3034
+ > output.
3035
+ >
3036
+ > **Kill:** the mutant produced a valid red in a narrowed run, and the same
3037
+ > narrowed command, in the same tree with the mutant reverted, ran a valid green
3038
+ > with the named test passing — completed, exit 0, `tests` > 0, `cancelled` 0. A
3039
+ > completed exit ≠ 0 with the named test not failing is not a kill, and no kill
3040
+ > stands without that green run on the unmutated tree. The result is
3041
+ > inconclusive, never a kill, when the red does not reproduce in the narrowed
3042
+ > run, when the unmutated tree is red too, or when no usable run exists.
3043
+ >
3044
+ > **Evidence line:** every kill and every valid red you claim carries one line
3045
+ > in the claim's own free-text field:
3046
+ > `red: <cmd> → exit N, fail K incl <test>; baseline: <cmd> → exit 0, <test> pass`.
3047
+ > A failing-test finding filed off the review's shared run carries the `red:`
3048
+ > half alone.
3049
+ >
3050
+ > An inconclusive acceptance mutation is neither a kill nor a pass: halt with
3051
+ > cause `other`, with both runs — the mutant's and the unmutated tree's — as
3052
+ > evidence; label nothing. A kill's evidence line goes in your report.
2932
3053
 
2933
3054
  **Give the finisher duty 3 verbatim as well, as its two steps.** The finisher's
2934
3055
  agent definition carries no duty text, so this brief is the whole of duty 3 it