@feigi/fleet-ctl 3.18.14 → 3.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agents/fleet-recipe-deriver.agent.md +109 -0
- package/commands/review-and-fix.md +1 -1
- package/package.json +1 -1
- package/scripts/ambient-git-vars-mjs-prose.test.mjs +12 -0
- package/scripts/derive-testcmd.sh +2 -1
- package/scripts/recipe-derivation-prose.test.mjs +99 -0
- package/scripts/recipe-prove.mjs +288 -0
- package/scripts/recipe-prove.test.mjs +537 -0
- package/skills/next-ticket/SKILL.md +2 -2
- package/skills/run-team/SKILL.md +62 -12
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fleet-recipe-deriver
|
|
3
|
+
description: Derives a repository's Recipe — its Install step and Test entrypoint — by reading the repository, proves both with recipe-prove.mjs, and reports. Dispatched by the run-team controller in phase 0, or by a standalone next-ticket or review-and-fix session, when the repository has no usable Recipe cache. Never invoked directly.
|
|
4
|
+
model: "@task:medium"
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
You derive one repository's **Recipe**: the **Install step** (the shell command
|
|
8
|
+
that materialises its dependencies in a fresh checkout) and the **Test
|
|
9
|
+
entrypoint** (the shell command that runs its suite). The fleet keeps no table
|
|
10
|
+
of technologies — no lockfile map, no test-file pattern, no list of build
|
|
11
|
+
tools — so you are the only part of it that reads a repository and decides
|
|
12
|
+
how it is built and tested. You decide by reading; a script proves what you
|
|
13
|
+
decided; nothing is cached on a guess.
|
|
14
|
+
|
|
15
|
+
Your dispatch prompt gives you `<repo>` (the repository's main checkout, an
|
|
16
|
+
absolute path) and `<scratch>` (a directory that is yours alone). Read files
|
|
17
|
+
under `<repo>`; write only under `<scratch>`. **Never edit, create or delete a
|
|
18
|
+
file under `<repo>`, and never write `.fleet/recipe.json` yourself** — the
|
|
19
|
+
proof below is the cache's only writer, and it writes only on proof.
|
|
20
|
+
|
|
21
|
+
## 1. Read the repository
|
|
22
|
+
|
|
23
|
+
`git -C <repo> fetch origin`, then read what a new engineer would read, at
|
|
24
|
+
`origin/main` (`git -C <repo> show origin/main:<path>`, `git -C <repo>
|
|
25
|
+
ls-tree -r --name-only origin/main`):
|
|
26
|
+
|
|
27
|
+
- the README and any CONTRIBUTING or development guide;
|
|
28
|
+
- the build and manifest files at the root and one level down — whatever this
|
|
29
|
+
repository uses to declare dependencies and tasks;
|
|
30
|
+
- `.github/workflows/*` (or the repository's other CI configuration): the
|
|
31
|
+
step that runs the suite in CI is the repository's own statement of what
|
|
32
|
+
its suite is, and the step before it is usually its install;
|
|
33
|
+
- scripts the repository tracks for running tests (a `Makefile` target, a
|
|
34
|
+
`scripts/test`, a tracked `agent-test`).
|
|
35
|
+
|
|
36
|
+
Propose one Install step and one Test entrypoint, both shell commands run
|
|
37
|
+
from the repository root by `sh -c`:
|
|
38
|
+
|
|
39
|
+
- **Install step** — the form that installs exactly what the repository pins
|
|
40
|
+
and rewrites nothing: the frozen or locked mode where the tool has one.
|
|
41
|
+
`true` when nothing needs installing. It must leave every file exactly as
|
|
42
|
+
checked out; build output the repository ignores is fine.
|
|
43
|
+
- **Test entrypoint** — the whole suite, the way CI runs it. Members run it
|
|
44
|
+
with their own arguments appended (`sh -c '<test> "$@"'`), so prefer a form
|
|
45
|
+
where an appended test file or test name lands as an argument to the test
|
|
46
|
+
runner. The cache reader refuses a command whose first word does not
|
|
47
|
+
resolve from the repository root (a builtin, a `PATH` entry, or an
|
|
48
|
+
executable path), one ending in `;`, `&` or a newline, and one containing a
|
|
49
|
+
word that starts with `#`.
|
|
50
|
+
|
|
51
|
+
## 2. Prove it — this procedure exactly
|
|
52
|
+
|
|
53
|
+
Every run below happens in a throwaway worktree of `origin/main` that the
|
|
54
|
+
script creates and removes; `<repo>`'s own tree is never touched.
|
|
55
|
+
|
|
56
|
+
**First run, to see the output** — no proof flags:
|
|
57
|
+
|
|
58
|
+
~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>'
|
|
59
|
+
|
|
60
|
+
It exits 1 with `NOT PROVEN — no proof given` once both commands ran, and
|
|
61
|
+
names the Test entrypoint's output file (`test output: <path>`). Any other
|
|
62
|
+
`NOT PROVEN` reason is about your commands — read it, and the log it names,
|
|
63
|
+
and revise.
|
|
64
|
+
|
|
65
|
+
**Then prove it, with ONE of:**
|
|
66
|
+
|
|
67
|
+
- **The count proof** — when the runner prints how many tests it ran. Copy
|
|
68
|
+
that summary line's text, as printed, into `--count-line`, and the number it
|
|
69
|
+
carries into `--test-count`:
|
|
70
|
+
|
|
71
|
+
~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' \
|
|
72
|
+
--count-line '<literal text of the count line>' --test-count <n>
|
|
73
|
+
|
|
74
|
+
Copy only text that is the same on every run — the count, not a duration.
|
|
75
|
+
A count of 0 is a failed proof, never a pass.
|
|
76
|
+
- **The mutation proof** — when the runner prints no count, or prints one
|
|
77
|
+
you cannot pin. It needs the unmutated run green. Choose one test and write
|
|
78
|
+
a shell command that breaks it deliberately: change an expected value in
|
|
79
|
+
the test, or the code that test checks, so the test must fail. The command
|
|
80
|
+
must change a tracked file; the script runs it in the throwaway worktree,
|
|
81
|
+
requires the suite to go red, and discards the worktree after:
|
|
82
|
+
|
|
83
|
+
~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' \
|
|
84
|
+
--mutate '<shell command that breaks one test>' --mutation '<one line: what it breaks>'
|
|
85
|
+
|
|
86
|
+
Exit 0 is the proof holding: the script wrote the Recipe cache and printed
|
|
87
|
+
`PROVEN` and the cache's contents. Exit 1 is `NOT PROVEN`, with the reason;
|
|
88
|
+
no cache was written. Exit 2 means no verdict on the proof: it could not be
|
|
89
|
+
attempted (no `origin/main`, not a repository) or the cache could not be
|
|
90
|
+
written — report it, do not work around it.
|
|
91
|
+
|
|
92
|
+
You may revise and re-run after a `NOT PROVEN` you can act on — a wrong
|
|
93
|
+
command, a count line you mis-copied, a mutation that changed nothing — **at
|
|
94
|
+
most four `recipe-prove.mjs` runs in all**, the first one included. Never
|
|
95
|
+
weaken the proof to get past it: a suite that stays green under a real
|
|
96
|
+
mutation, or reports zero tests, is the finding, not an obstacle.
|
|
97
|
+
|
|
98
|
+
## 3. Report
|
|
99
|
+
|
|
100
|
+
Your last message is exactly one of:
|
|
101
|
+
|
|
102
|
+
- `RECIPE PROVEN` followed by the JSON line `recipe-prove.mjs` printed.
|
|
103
|
+
- `RECIPE NOT PROVEN` followed by the last run's `NOT PROVEN` reason
|
|
104
|
+
verbatim, the commands you tried, and one line naming the cause:
|
|
105
|
+
**vacuous** (the suite ran no tests, or none a mutation could reach),
|
|
106
|
+
**did not run** (a command not found or not executable — a toolchain this
|
|
107
|
+
machine lacks), **dirty install** (the Install step changes the tree), or
|
|
108
|
+
**unreadable** (you could not tell from the repository how it is built or
|
|
109
|
+
tested).
|
|
@@ -10,7 +10,7 @@ argument-hint: [pr-number]
|
|
|
10
10
|
See **Specialists** — those constraints are not optional.
|
|
11
11
|
2. Plan the actions. Split explicitly into **apply now** / **defer**. A finding is deferred, never dropped. **Every `suggestion` — the review's `severity`, the plugin's *Suggestions* bucket — arrives unchecked: the review budgets that band 0 refuters, so nothing has looked at it yet.** Checking it is now your job, not nobody's. So split it by SCOPE first. Out of the PR's ticket scope → defer and file. In scope → dispatch one refuter against it, biased to refuse, and hand it this instruction verbatim with one substitution: resolve `<scratch>/pr<N>/<finding>/` in it to an absolute path under your own run root — `mktemp -d` once per round for `<scratch>/pr<N>/fix-XXXXXXXX/`, never bare `pr<N>`, since finding ids restart at 1 each round too — and write that absolute path into the refuter's prompt in its place; a refuter never derives its own path: *Try to REFUTE this finding. Default to refuted=true if uncertain. Verify by RUNNING something — compile it, run the test, apply the mutation. Do not reason your way to agreement. Observe that run synchronously — run the command, wait for it, read its exit code. Never poll a log file for a completion marker: prefer ONE blocking run to a poll loop, and treat its return as permission to look, never as the answer. Reading a log the run has already finished writing is fine; waiting on one is not. If you match a test reporter's own output, accepting both `ℹ` and `#` is necessary but NOT sufficient — strip SGR escapes first as well. node's prefix moves with the node version and with whether stdout is a TTY, and color wraps the whole line so it begins with ESC and no prefix anchor matches at all, which returns empty at exit 0 — indistinguishable from a hung run and from a run of zero tests. For an uncolored baseline use `env -u FORCE_COLOR`; `FORCE_COLOR=` empty still enables color, so it is not a control. State your search scope AND what your pattern would have missed. A grep over one ref does not support a claim about history; a pattern built from the token a diff removed does not support a claim that the category is empty. A failure injection with no positive control has produced NO result, never a negative one. Before you read an injected fault — an env var, an argv word, a mutant — as having had no effect, prove the injection reached the child: one run whose output differs with it present versus absent, or the child echoing the injected value back. Uncontrolled, the cell is unrun — say so in your verdict instead of reporting a no-effect result. Build such an invocation as an array expanded braced and quoted — `cfg=(SETB=1 BADJ=1); env "${cfg[@]}" sh ./probe.sh` — or inline the assignments literally — `env SETB=1 BADJ=1 sh ./probe.sh`; NEVER from an unquoted scalar — `cfg="SETB=1 BADJ=1"; env $cfg sh ./probe.sh` — which under zsh passes ONE argument, sets a variable literally named `SETB` to `1 BADJ=1`, never sets `BADJ` at all, and still exits 0. `env $cfg[@]` is not the portable spelling either: measured, bash word-splits it into `SETB=1` and `BADJ=1[@]`, so the injection variable is set to a corrupted value, while zsh behaves exactly as with the bare `$cfg` — `BADJ` never set, exit 0. Everything you write — mutants, fixtures, scratch repos — goes under `<scratch>/pr<N>/<finding>/` and nowhere else; the checkout and any worktree are never write targets, though `git show`/`git archive` at a pinned ref read fine anywhere. Chain the directory change into the command, `cd "$D" && git …`, never `cd "$D"; git …`, so a failed `cd` cannot leave a `git` command running in the checkout — and bracket a fixture's own git with `git rev-parse --show-toplevel`: before `git init` it must NOT resolve to the repository, and a fresh scratch dir's `fatal: not a git repository` (exit 128) is the pass, not a failure; before any `git commit` it must resolve to your scratch path — compare resolved forms (`realpath`), since `--show-toplevel` can report `/private/tmp/…` for a `/tmp` scratch dir on macOS.* That last clause is the whole mechanism — a refuter left to reason its way to agreement rubber-stamps, and "apply only what survives" silently degrades into "apply everything in scope". Apply only what survives; a refuted one defers with the refutation in its body. "In scope" means the ticket's scope, not merely the same files — and applying one still never licenses editing a file the ticket has no business in. **A finding in `unverified` whose refuters ran and crashed always defers, whatever its severity** — and which of the two it is, you read off the finding's **`refutersDispatched`**, never off severity and never off an empty vote list: above zero with nothing surviving means every refuter dispatched against it died, and at `critical` that is every one of them. Only a `survived` finding has been checked; severity says how much a finding would matter if true, never whether anything looked. **Deferring a crash is your move, and re-running it is not** — the review already re-dispatched every crashed refuter pair once, in-run, so a crash that reaches you crashed twice, and the result's `resume` field names the one move left, which is not yours: nothing replays a review that ran in a runner's kernel — it is reported, not acted on. So say *in the deferral* that its refuters crashed rather than that it went unchecked, and re-run nothing for it. **A `suggestion` is also in `unverified`, for a different reason — `refutersDispatched` of zero, the 0-refuter budget the workflow gives that band by policy, so nothing looked *there* — and the scope split above, not this rule, covers it.**
|
|
12
12
|
**Write every ruling to `<scratch>/dispositions-<pr>.json`, beside the review result file — the disposition record `dispositions-check.mjs` judges against it before any finisher is dispatched.** It is `{"head": "<the review file's head>", "entries": [ … ]}`: one entry per `survived` finding and per `unverified` finding, plus one per `refuted` finding you reverse, its evidence in `reason`. Each entry carries `bucket` (`survived|unverified|refuted`), `index` (its position in that bucket of the review file — stable, since a round's file never changes), `scope` (`in|out`), `claimKind` (`behavior|shape`) and `disposition` (`apply|defer`), and where they apply `reason`, `issue` (the number you filed it to or commented on), `verdictPath` (an in-scope suggestion's refuter verdict file) and `remedyFiles` (the files its remedy names). The check reads scope off the diff before it reads your entry: a finding with no `line`, or on a line `git diff` touches from the PR's merge-base with `origin/main` to the review's `head`, is in scope whatever `scope` says, and only otherwise does the declared one stand. An in-scope `survived` finding deferred passes only with `reason` `false-rationale`, `mutual-exclusion`, `remedy-worse` or `remedy-outside-diff`. `remedy-outside-diff` — the remedy would edit a file the PR's diff does not touch — also needs `remedyFiles` to name at least one file absent from `git diff` from that merge-base to the review's `head`: none named, or every one in that diff, is a mismatch. A `critical` or `important` finding deferred for it goes to a human instead of passing: the script exits 1 with verdict `escalate`, `ledger.mjs dispatch` refuses the PR's finisher, and nothing retries it; a `suggestion` deferred for it passes. Any other reason, none, or a `survived` or `unverified` finding with no entry is a mismatch: the script exits 1 naming each violating entry's bucket, index and rule, and `ledger.mjs dispatch` refuses the PR's finisher.
|
|
13
|
-
3. **Run `testCmd` before you commit**, from the worktree, copied verbatim — not a runner you picked. `testCmd` is the command your dispatcher handed you; **standalone, with no dispatcher to hand you one, it is the repo's own test command**, the Test entrypoint in the repo's Recipe cache (`derive-testcmd.sh <repo> test` prints it) — here `node --test plugin/scripts/*.test.mjs`, which is this repo's example and not a value any derivation guarantees. Read it off the repo, never invent one: "not a runner you picked" is what this step forbids, and on the standalone path nothing else defines the referent. With no Recipe cache, run the Recipe derivation step;
|
|
13
|
+
3. **Run `testCmd` before you commit**, from the worktree, copied verbatim — not a runner you picked. `testCmd` is the command your dispatcher handed you; **standalone, with no dispatcher to hand you one, it is the repo's own test command**, the Test entrypoint in the repo's Recipe cache (`derive-testcmd.sh <repo> test` prints it) — here `node --test plugin/scripts/*.test.mjs`, which is this repo's example and not a value any derivation guarantees. Read it off the repo, never invent one: "not a runner you picked" is what this step forbids, and on the standalone path nothing else defines the referent. With no usable Recipe cache (`derive-testcmd.sh` refuses), run the Recipe derivation step first — dispatch a `task` member, agent `fleet-recipe-deriver`, its prompt naming the main checkout's absolute path as `<repo>` and a scratch directory of its own; it proves both commands in a throwaway worktree and writes the cache only on proof, and its `RECIPE NOT PROVEN` is a stop, reported with its reason. Never fall back to a runner of your own choosing. `tests 0` is a FAILED run, not a pass: a glob matching nothing still exits 0. **Zero is not the only no-work count.** A run reporting **0 passes with no failures** skipped everything it collected, and a count **materially below the whole suite** means you ran a partial tree — this suite reported `tests 937` when this line was written, so compare what you get against a full run rather than against zero. The number drifts; that is the safe direction, because a stale one produces a loud false red while its absence produces a silent false green. Red, zero-test, or either no-work count → fix it or move the finding to defer; never commit over it. Then commit and push — `git push origin HEAD:<branch>`, the explicit refspec: this step also runs from a `--detach` worktree (`run-team/SKILL.md`'s worktree-recreation form), where a bare `git push` exits 128 (`fatal: You are not currently on a branch`) and any push whose destination is `HEAD` rather than a full or branch refname (`-u` or not, forced or not) exits 1 (`error: The destination you provided is not a full refname`) — and report the test result alongside the SHA. **Then reconcile the PR body with what you actually did.** It merges into the permanent record and nothing downstream re-derives it: your commit messages are yours to fix and usually get fixed, the body is neither. Removing a change, or reversing one, leaves the body asserting work the diff no longer contains. Measured twice in one run — one body kept asserting a calibration its own fix had just deleted as false, another kept claiming a hunk the fix had removed as "this diff owns it" — both caught by a finisher's reading rather than by any rule, and both would otherwise have merged intact. `gh pr edit <pr> --body-file <path>`, and re-read the body against the final diff rather than against your memory of it. **Never `--no-verify`.** The fleet runs in repos it knows nothing about: it must not assume a pre-commit hook exists, and must never bypass one that does — a hook that fails is a finding to report, not an obstacle. Nothing re-reviews what you commit here, so this run is the only gate between your edit and a merge.
|
|
14
14
|
4. **Under the fleet, do not hold this wait.** The controller owns a persistent CI Monitor and dispatches a finisher once your report has arrived and CI is green — so push, then report to the controller a **final verdict** (every specialist you dispatched + its outcome + your apply/defer adjudication + the pushed SHA) and stop. A specialist that has neither reported nor been relayed to you is not a verdict line — ask the controller for it by name before ruling; see **Specialists**. The controller keys "review complete" on that verdict, **never on you merely going idle** — a slow specialist can still be running when your turn ends, and finishing off idle-alone ships a premature label (observed: a PR labelled while its contract-test specialist was still running and about to surface a vacuous pin). A turn-based reviewer re-reading `gh pr checks` each idle cycle rebuilds its whole context for nothing the Monitor lacks. **Standalone (no controller): watch** the run until the diff-validating jobs are green (bind via `ci-state.mjs` below); fix a **genuine** failure and repeat from 3. **Staleness is not a failure:** a `rebase-check` red or heavy jobs `skipped` **solely from a non-zero behind-count** never clears by waiting and must not be rebased away here — see **`skipped` is not `passed`** and **Do NOT rebase to label** below for what it looks like and why. Stop watching and go to 6. A turn-based reviewer that "waits for green" on a behind PR stalls forever and ends its turn without labelling. **`ci-state.mjs` returns `verdict: "no-ci"`** → there is no run to watch, so don't: go straight to 6 and let that step's `--declare-no-ci` branch decide.
|
|
15
15
|
5. File each deferred finding: `gh issue create --label <ready-for-agent|needs-triage>` with the finding, `file:line`, why deferred, `Deferred from PR #<pr> review`, and **the finding's `dimension` in the issue body** — the review returns it on every finding, and a backlog that does not carry it cannot be attributed to the specialist that generated it. **The label follows the finding's state, not this step** — but it is never omitted; an unlabelled issue is invisible to both `candidates.mjs --require-label` and the cockpit's pool query, reachable only by a human. **The bar here is whether the DEFECT is confirmed — never whether the remedy is settled.** Confirmed defect with the remedy still open → `ready-for-agent`, and say in the body that the remedy is open. Torn → `needs-triage`, narrowed: torn means **unsure anything is broken**, not unsure how to fix it. **A finding a refuter CHECKED and did not survive is neither** — it is not a confirmed defect, and you are not unsure whether anything is broken: you measured that nothing is. Refutations go as **one issue per review, titled `PR #<pr> review: the suggestion band, checked`, labelled `wontfix`, and CLOSED** — #483, #542, #571 and #934 are the precedent, and a run that files them open instead has to be corrected afterwards, as one was on 2026-08-27. Left open they read as queued agent work and rot into filing artifacts; a measured triage pass found a quarter of its `needs-triage` backlog was refuted records, bulk-closable by title shape. Closed is still a durable home — `check` reads the tracker at `--state all`, so a future run re-deriving the finding lands on the evidence rather than re-litigating it. Split the band only where an entry is **not** a refutation: a residual the refuter surfaced, or a finding refused as out of scope and never checked, is real deferred work — and real deferred work earns its own open issue only past a second bar. **Worth a claim: a finding whose own claim is that correct code could be shaped better — style, naming, layout, redundancy, a micro-simplification — never gets its own open issue.** Its only open question is "is this worth doing?", which answers itself under the triage bar #239 repriced: measured 2026-08-15→31, 133 issues were triage-closed `wontfix` in sixteen days, 121 of them review deferrals, mostly this shape (`docs/adr/0002-filing-second-bar-worth-a-claim.md`). Append it to the same closed record issue under a `Below the claim bar` heading — `file:line`, dimension, the claim, and a re-open trigger per entry. Closed is a durable home for these too: `check` reads `--state all`, so a run re-deriving the finding lands on the record instead of re-filing — and a later review returning the same finding as `survived`, or a maintainer hitting it, is the promotion signal: file it open then, citing the record. **An in-scope rediscovery step 2 applied is that same promotion signal, and it files nothing** — so the record carries no mark unless you write one: run `check` over each finding step 2 applied as well, and where its tracker rows put one on a `Below the claim bar` entry, comment on that record issue, opening `Promoted — applied, not filed`, and name the entry, that its refuter let it through, and the commit carrying the fix. That comment is the mark ADR 0002's Trigger A counts for this path, and it is **not** a cross-reference — measured on this tracker, a comment mints `cross-referenced` on the issue its own text *names*, never on the issue it sits under (#1033's comment citing #1453 landed on #1453's timeline and nowhere else), so the fixed opening is what carries the signal instead. **The bar reads the finding's claim, never its review band** — a finding that alleges wrong behavior, whatever band it sat in, `unverified`-with-crashed-refuters included, is above this bar and files open under the confirmed/torn split; #591's conflation is exactly why a band cannot carry this decision. Do **not** reach for phase 0's **decided?** test here. That is a *consumption* gate, priced for a seat where implementer divergence costs a claim, a worktree and a dispatch; at filing time divergence costs one PR review the maintainer already runs. Same test, two seats, opposite correct answers — and reading it in this seat took `needs-triage` from 11 to 135 before it was repriced (#239, ruled 2026-08-18). **Dedupe through `~/.fleet/bin/fleet-run ledger.mjs check "<subject>"`, never a bare `gh issue list --search`** — the duplicate class that guard exists to catch was filed from *here*: the ledger row for the finding rediscovered five times in one measured run carries a reviewer's own source tag, `(review-pr-108)`. `check` reads this run's filed list *and* the tracker at `--state all`, so a closed duplicate still counts. Exit **1** is already filed in this run; exit **3** is a clean ledger with matching tracker rows that score against the subject — read the ones it prints and comment on the existing follow-up rather than filing a second issue. **Exit 0 is not automatically "safe to file"**: it covers `clean`, `soft-hit` and `unverified` alike, and an unreachable `gh` degrades to a ledger-only answer that says `TRACKER NOT CHECKED`, so read the stdout `verdict` to tell a tracker that was searched and clean from one nobody read — under `unverified`, search it yourself before filing — or, for the applied-promotion comment above, before treating a tracker nobody could read as a reason to skip it. **`soft-hit` is the answer that costs most to misread**: it means related rows were found and none of them was established as a duplicate — a near-miss above the floor, or tracker rows the scorer rates 0.00 — so read the rows it printed and decide, and never treat that exit 0 as permission. Exit **2** is a usage error, meaning nothing was checked at all: fix the invocation and re-run, never read it as a duplicate. Full exit-code and `verdict` semantics live in `$(~/.fleet/bin/fleet-run --root)/skills/run-team/SKILL.md`'s **Run ledger** section — read them there rather than restating them here. **The guard dedupes, it never drops:** a duplicate becomes a comment on the issue that already covers it, never a dropped finding. **Record every issue you create, at once:** immediately after each `gh issue create`, run `~/.fleet/bin/fleet-run ledger.mjs filed <N> "<subject>"` — `<N>` the number `gh issue create` printed, and **the same subject string you passed to `check`**, so the next `check` for that finding meets it in the filed list instead of scoring a rewording. Without it the guard's ledger half never holds a row, and its exit **1** can never fire. If `filed` fails, retry it once; if it fails again, leave the issue alone — it exists and is correct — and report it as `unrecorded: #N <subject>`, so the controller runs `filed` for it. Report `filed: #N <subject>` for every issue you created — under the fleet, in your report to the controller. **The one window the guard cannot see:** `check` → `gh issue create` → `filed` is three commands, and the ledger's lock covers each alone, so two filers can both read `clean` in the seconds before either records. That gap is accepted, never a reason to skip `filed`. Post the numbers as one PR comment.
|
|
16
16
|
6. Diff-check green (run-bound per `ci-state.mjs`; under staleness see **`skipped` is not `passed`** for what the label then rests on) **and** deferrals filed **and** the PR carrying exactly one release label (`patch`/`minor`/`major`, counting only those three; **required on every PR** — `release-label.yml` auto-adds `patch` when none is present and hard-fails on more than one, so this never needs to check whether the repo defines the gate at all, and `gh label list` settles nothing here any more) **and** `dispositions-check.mjs --member fix-pr-<pr> --scratch <scratch>` run over your own record exiting 0 (standalone that is yours to run, from the worktree, after step 2 writes `<scratch>/dispositions-<pr>.json`; it reads the review file beside it, which you write yourself from the findings step 1 collected, in the shape **The review result file** describes — `pr`, `head`, `snapshot`, `counts` and the `survived`, `refuted` and `unverified` buckets; standalone you pass `--no-ledger`, so it writes no token and the exit status is the verdict — a `.fleet/ledger.md` the repository already holds is a previous run's, and without the flag it exits 2 for a ledger with no row for you; under the fleet `ledger.mjs dispatch` has already refused a finisher without it) → `gh pr edit <pr> --add-label ready-to-merge`. Do not merge. A non-zero dispositions exit halts you before the label, and you report the violations it printed. Zero or more than one release label halts you before the label, naming which you found — a timed-out `gh pr create` opens the PR with its `--label` unapplied and no exit status to react to (#375), and nothing after this step re-derives the release label. **Under the fleet a finisher does this** — a fresh small agent the controller dispatches (after your final verdict, never on your bare idle). Gate it on the diff-validating **`check`** job, **not** on `ci-state --quiet` exit 0, which a behind PR — the normal case once any sibling merges — never reaches: a `rebase-check` red or heavy jobs `skipped` off the behind-count hold it short of full green, so an exit-0 gate strands every behind PR unlabelled. Label when `check` is present and green **and no heavy job (the diff-validating suites — *not* the `rebase-check` currency gate) is in `failure`** — a `skipped` heavy job is behind-count staleness and fine — **solely** off that count, which is a condition to establish rather than infer, see the five-condition note in `$(~/.fleet/bin/fleet-run --root)/skills/run-team/SKILL.md`'s fix-applier prompt block. A `failure` is the diff's own fault and blocks the label, else a genuinely broken PR gets labelled off a green `check` and burns a merge-bot rebase+CI cycle before the failure resurfaces. Read per-job state for this: run `ci-state.mjs` **without** `--quiet`, or read its `jobs` — the `--quiet` payload drops the per-job `jobs`/`missing` and the `dropped` list, keeping `reasons` (which still distinguishes `is failure` from `is skipped`). You reach step 6 yourself only when green is already in hand at push time.
|
package/package.json
CHANGED
|
@@ -243,6 +243,18 @@ const COVERED_MJS = {
|
|
|
243
243
|
// another repository does not change the answer" (the diff and merge-base)
|
|
244
244
|
// and "… does not change which ledger is found" (the common dir).
|
|
245
245
|
"dispositions-check.mjs": 1,
|
|
246
|
+
// ONE scrubbed env, built once and handed to every child — the `git(args,
|
|
247
|
+
// cwd)` primitive behind the common-dir, origin/main, worktree, status and
|
|
248
|
+
// diff calls, AND the one `sh -c` runner of the Install step, the Test
|
|
249
|
+
// entrypoint and the mutation. An ambient GIT_DIR would answer the first for
|
|
250
|
+
// another repository and run the Recipe against that repository's tree while
|
|
251
|
+
// the cache landed here. Measured in recipe-prove.test.mjs, "an ambient
|
|
252
|
+
// GIT_DIR naming another repository does not change the answer" (the git
|
|
253
|
+
// primitive) and "the Install step and the Test entrypoint run against their
|
|
254
|
+
// own worktree whatever GIT_DIR the caller exports" (the `sh -c` runner). The
|
|
255
|
+
// cache reader child gets the same env but unsets both itself, so no
|
|
256
|
+
// fixture can tell its env apart.
|
|
257
|
+
"recipe-prove.mjs": 1,
|
|
246
258
|
};
|
|
247
259
|
|
|
248
260
|
// The two-name-only exemption list #1599's second design question answers
|
|
@@ -30,7 +30,8 @@
|
|
|
30
30
|
#
|
|
31
31
|
# At least one of testCount/mutation is the proof; a cache carrying neither,
|
|
32
32
|
# or `installClean` anything but `true`, is an UNPROVEN Recipe and refused like
|
|
33
|
-
# an absent one. Written only by the
|
|
33
|
+
# an absent one. Written only by recipe-prove.mjs, the derivation step's proof,
|
|
34
|
+
# and only once that proof holds; this script never writes it.
|
|
34
35
|
#
|
|
35
36
|
# INVALID on failure to RUN, never on failing tests: the command this script
|
|
36
37
|
# is asked for must name something that resolves from <repo> (a builtin, a
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
import { test } from "node:test";
|
|
2
|
+
import assert from "node:assert/strict";
|
|
3
|
+
import { readFileSync } from "node:fs";
|
|
4
|
+
import { join } from "node:path";
|
|
5
|
+
import { between, paragraph, phrase } from "./prose-pin.mjs";
|
|
6
|
+
|
|
7
|
+
// The Recipe derivation step's prose: where it lives in run-team's phase 0,
|
|
8
|
+
// what reuses a cache instead of deriving, when a run re-derives and when it
|
|
9
|
+
// stalls, and the deriver definition that carries the proof procedure. The
|
|
10
|
+
// proof's own behaviour is recipe-prove.test.mjs's; this file pins that the
|
|
11
|
+
// runbook routes to it, and that every refusal naming "the Recipe derivation
|
|
12
|
+
// step" names a step that exists.
|
|
13
|
+
const PLUGIN = join(import.meta.dirname, "..");
|
|
14
|
+
const read = (rel) => readFileSync(join(PLUGIN, rel), "utf8");
|
|
15
|
+
const RUN_TEAM = read("skills/run-team/SKILL.md");
|
|
16
|
+
const DERIVER = read("agents/fleet-recipe-deriver.agent.md");
|
|
17
|
+
const PROVER = read("scripts/recipe-prove.mjs");
|
|
18
|
+
|
|
19
|
+
const phase0 = () => between(RUN_TEAM, "## Phase 0 — shortlist", "## Phase 1 — Pull", "run-team/SKILL.md phase 0");
|
|
20
|
+
// The step spans its quoted dispatch prompt, so it is bounded by the next
|
|
21
|
+
// rule's opener rather than by a blank line.
|
|
22
|
+
const step = () => between(phase0(), "**Derive the Recipe — the Recipe derivation step — before the fold-in below", "**Mid-run, a cache that stops running", "the phase-0 Recipe derivation step");
|
|
23
|
+
const stall = () => paragraph(phase0(), "**Mid-run, a cache that stops running is re-derived once", "the re-derive-once and stall rule");
|
|
24
|
+
|
|
25
|
+
test("phase 0 derives the Recipe before the fold-in and before the first claim", () => {
|
|
26
|
+
const p0 = phase0();
|
|
27
|
+
const at = p0.indexOf("**Derive the Recipe");
|
|
28
|
+
assert.ok(at !== -1 && at < p0.indexOf("**Fold in every PR a prior run left open"),
|
|
29
|
+
"the Recipe derivation step must come before the fold-in: folded-in PRs are reviewed, and a review reads the Test entrypoint from the cache");
|
|
30
|
+
assert.match(step(), phrase("before the first claim."));
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
test("a present, valid cache is used as it stands — derivation is the controller's, once per run start", () => {
|
|
34
|
+
const s = step();
|
|
35
|
+
assert.match(s, phrase("Both exit 0 → the cache is present and valid: use it as it stands and derive nothing."));
|
|
36
|
+
assert.match(s, phrase("one derivation per run start at most, never one per member, and a member never derives."));
|
|
37
|
+
assert.match(s, phrase("agent `fleet-recipe-deriver`"));
|
|
38
|
+
});
|
|
39
|
+
|
|
40
|
+
test("the dispatch names the proof: a clean install, and a non-zero count or a mutation turning the run red", () => {
|
|
41
|
+
const s = step();
|
|
42
|
+
assert.match(s, phrase("the Install step must leave the tree clean"));
|
|
43
|
+
assert.match(s, phrase("the runner's own non-zero count, or a deliberate mutation of one test that turns the run red."));
|
|
44
|
+
assert.match(s, phrase("Only a proof writes the cache"));
|
|
45
|
+
});
|
|
46
|
+
|
|
47
|
+
test("an invalid cache is re-derived once; a failed derivation stalls the run instead of looping", () => {
|
|
48
|
+
const s = stall();
|
|
49
|
+
assert.match(s, phrase("dispatch `derive-recipe` again exactly as above, then retry what refused."));
|
|
50
|
+
assert.match(s, phrase("A red suite never triggers it: a failing test is a finding, not a stale Recipe."));
|
|
51
|
+
assert.match(s, phrase("`RECIPE NOT PROVEN` from any derivation, or a second invalid-cache refusal after this run's one re-derivation, halts the run"));
|
|
52
|
+
assert.match(s, phrase("It is never a loop: no third derivation, and never a cache written by hand."));
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
test("the claim paragraph sends an absent or invalid cache to the derivation step, and infers no Install step", () => {
|
|
56
|
+
const p = paragraph(RUN_TEAM, "**The Install step comes from the Recipe cache", "run-team/SKILL.md claim Install paragraph");
|
|
57
|
+
assert.match(p, phrase("never pass or infer one."));
|
|
58
|
+
assert.match(p, phrase("No cache is phase 0's Recipe derivation step not having run"));
|
|
59
|
+
});
|
|
60
|
+
|
|
61
|
+
// Every refusal that sends its reader to re-derive names the step by the same
|
|
62
|
+
// words. If phase 0 stopped carrying a step by that name, each refusal would
|
|
63
|
+
// point at nothing.
|
|
64
|
+
test("every refusal naming the Recipe derivation step names one that phase 0 carries", () => {
|
|
65
|
+
const sources = {
|
|
66
|
+
"derive-testcmd.sh": read("scripts/derive-testcmd.sh").match(/^derive="([^"]*)"/m),
|
|
67
|
+
"claim-ticket.sh": read("scripts/claim-ticket.sh").match(/^rederive="([^"]*)"/m),
|
|
68
|
+
"review-core.mjs": read("scripts/review-core.mjs").match(/DERIVATION_STEP =\s*"([^"]*)"/),
|
|
69
|
+
};
|
|
70
|
+
for (const [file, m] of Object.entries(sources)) {
|
|
71
|
+
assert.ok(m, `${file}: could not find its derivation-step message — update this test`);
|
|
72
|
+
assert.match(m[1], /run the Recipe derivation step \(run-team phase 0, before the first claim/, `${file} no longer names the step`);
|
|
73
|
+
}
|
|
74
|
+
assert.match(step(), /the Recipe derivation step/);
|
|
75
|
+
});
|
|
76
|
+
|
|
77
|
+
test("the deriver is told never to write the cache itself, and to prove through recipe-prove.mjs", () => {
|
|
78
|
+
assert.match(DERIVER, phrase("never write `.fleet/recipe.json` yourself"));
|
|
79
|
+
assert.match(DERIVER, phrase("~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>'"));
|
|
80
|
+
assert.match(DERIVER, phrase("A count of 0 is a failed proof, never a pass."));
|
|
81
|
+
assert.match(DERIVER, phrase("requires the suite to go red"));
|
|
82
|
+
});
|
|
83
|
+
|
|
84
|
+
test("every flag the deriver's procedure passes is one recipe-prove.mjs accepts", () => {
|
|
85
|
+
// The invocations are the indented command blocks naming recipe-prove.mjs;
|
|
86
|
+
// flags elsewhere in the definition (its `git` reads) are not the proof's.
|
|
87
|
+
const blocks = DERIVER.split(/\n[ \t]*\n/).filter((b) => /^ {4,}/.test(b) && b.includes("recipe-prove.mjs"));
|
|
88
|
+
assert.equal(blocks.length, 3, "expected the observe, count-proof and mutation-proof invocations — update this test");
|
|
89
|
+
const used = new Set(blocks.join("\n").match(/--[a-z][a-z-]+/g));
|
|
90
|
+
const accepted = new Set(PROVER.match(/new Set\(\[([^\]]*)\]\)/)[1].match(/--[a-z-]+/g));
|
|
91
|
+
assert.ok(used.size >= 6, `found only ${[...used].join(", ")} in the deriver — the procedure lost its proof flags`);
|
|
92
|
+
for (const flag of used) assert.ok(accepted.has(flag), `the deriver passes ${flag}, which recipe-prove.mjs does not accept`);
|
|
93
|
+
});
|
|
94
|
+
|
|
95
|
+
test("the deriver reports one of two lines, naming vacuity as a cause", () => {
|
|
96
|
+
assert.match(DERIVER, /`RECIPE PROVEN` followed by the JSON line/);
|
|
97
|
+
assert.match(DERIVER, /`RECIPE NOT PROVEN` followed by the last run's `NOT PROVEN` reason/);
|
|
98
|
+
assert.match(DERIVER, /\*\*vacuous\*\*/);
|
|
99
|
+
});
|
|
@@ -0,0 +1,288 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Prove a repository's Recipe — its Install step and Test entrypoint — and
|
|
3
|
+
// write the Recipe cache only if the proof holds. This is the Recipe
|
|
4
|
+
// derivation step's proof, and the ONE writer of the cache derive-testcmd.sh
|
|
5
|
+
// reads: the deriving agent chooses both commands by reading the repository,
|
|
6
|
+
// and this script is what turns that choice into a cache, or refuses to.
|
|
7
|
+
//
|
|
8
|
+
// recipe-prove.mjs <repo> --install <cmd> --test <cmd>
|
|
9
|
+
// [--count-line <literal> --test-count <n>]
|
|
10
|
+
// [--mutate <cmd> --mutation <description>]
|
|
11
|
+
//
|
|
12
|
+
// Every check runs in a throwaway worktree of `origin/main`, detached and
|
|
13
|
+
// removed afterwards, so neither the caller's checkout nor its uncommitted
|
|
14
|
+
// state takes part — `derivedAt` is the commit the proof ran against:
|
|
15
|
+
//
|
|
16
|
+
// 1. The Install step runs (`sh -c`, from the worktree root) and must exit
|
|
17
|
+
// 0, then `git status --porcelain -uall` must print nothing: an install
|
|
18
|
+
// that rewrites a tracked file, or creates one the tree neither tracks
|
|
19
|
+
// nor ignores, corrupts every worktree it later runs in. That is the
|
|
20
|
+
// same check, in the same form, a claim re-asserts on every Install.
|
|
21
|
+
// 2. The Test entrypoint runs and must RUN — 126/127 is the shell's "cannot
|
|
22
|
+
// execute"/"not found". A red suite is not a failed proof by itself.
|
|
23
|
+
// 3. Real tests must be shown to have executed, by at least one of:
|
|
24
|
+
// - the count proof: `--count-line` is a literal the run's own output
|
|
25
|
+
// must contain (the runner's summary line, read off a previous run),
|
|
26
|
+
// carrying `--test-count`, a positive integer. `tests 0` is a failed
|
|
27
|
+
// proof however honestly the runner reports it.
|
|
28
|
+
// - the mutation proof: the unmutated run must be green; `--mutate` is a
|
|
29
|
+
// shell command that must change a tracked file (the deliberate
|
|
30
|
+
// failing mutation of one test, or of the code one test covers) — run
|
|
31
|
+
// against the tree restored to HEAD, so a file the unmutated run itself
|
|
32
|
+
// rewrote is not credited to it; the run must then go red. A mutation
|
|
33
|
+
// that leaves the run green is a suite that runs nothing that matters.
|
|
34
|
+
// With neither, nothing is proven: the refusal still names where the test
|
|
35
|
+
// run's output is, so the caller can read the count line off it.
|
|
36
|
+
//
|
|
37
|
+
// On proof, the cache is written atomically to <workspace>/.fleet/recipe.json
|
|
38
|
+
// — <workspace> being the directory holding the repository's common git dir,
|
|
39
|
+
// the place derive-testcmd.sh reads it from — and then read back through
|
|
40
|
+
// derive-testcmd.sh for both fields. A cache the reader refuses is rolled
|
|
41
|
+
// back to whatever stood before, so the reader stays the one authority on
|
|
42
|
+
// what a usable cache is. A failed proof never touches an existing cache.
|
|
43
|
+
//
|
|
44
|
+
// Exit 0: proven, cache written (its path and contents on stdout).
|
|
45
|
+
// Exit 1: NOT PROVEN — no cache written; the reason on stderr.
|
|
46
|
+
// Exit 2: no verdict on the proof — it could not be attempted (usage, not a
|
|
47
|
+
// repository, origin/main missing, the worktree could not be made), or
|
|
48
|
+
// a filesystem fault stopped it before the cache was settled (the
|
|
49
|
+
// cache or its temp file could not be written).
|
|
50
|
+
|
|
51
|
+
import { spawnSync } from "node:child_process";
|
|
52
|
+
import { closeSync, existsSync, mkdirSync, mkdtempSync, openSync, readFileSync, renameSync, rmSync, unlinkSync, writeFileSync } from "node:fs";
|
|
53
|
+
import { tmpdir } from "node:os";
|
|
54
|
+
import { dirname, join } from "node:path";
|
|
55
|
+
import { fileURLToPath } from "node:url";
|
|
56
|
+
import { gitEnv, workspaceDirFromGitCommonDir } from "./git-env.mjs";
|
|
57
|
+
import { isDigits } from "./arg.mjs";
|
|
58
|
+
|
|
59
|
+
const NAME = "recipe-prove";
|
|
60
|
+
const USAGE =
|
|
61
|
+
"usage: recipe-prove.mjs <repo> --install <cmd> --test <cmd> [--count-line <literal> --test-count <n>] [--mutate <cmd> --mutation <description>]";
|
|
62
|
+
const READER = join(dirname(fileURLToPath(import.meta.url)), "derive-testcmd.sh");
|
|
63
|
+
|
|
64
|
+
// Scrubbed once, for every child: an ambient GIT_DIR or GIT_WORK_TREE — set
|
|
65
|
+
// by a git hook, `rebase --exec` or `bisect run` — would retarget the git
|
|
66
|
+
// calls below at another repository, and the Install step and Test entrypoint
|
|
67
|
+
// at that repository's tree, while the cache still landed here.
|
|
68
|
+
const ENV = gitEnv();
|
|
69
|
+
|
|
70
|
+
class Refusal extends Error {
|
|
71
|
+
constructor(code, message) {
|
|
72
|
+
super(message);
|
|
73
|
+
this.code = code;
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
const cannot = (message) => new Refusal(2, message);
|
|
77
|
+
const notProven = (message) => new Refusal(1, message);
|
|
78
|
+
|
|
79
|
+
function parseArgs(argv) {
|
|
80
|
+
const [repo, ...rest] = argv;
|
|
81
|
+
if (!repo || repo.startsWith("--")) throw cannot(USAGE);
|
|
82
|
+
const flags = new Set(["--install", "--test", "--count-line", "--test-count", "--mutate", "--mutation"]);
|
|
83
|
+
const a = {};
|
|
84
|
+
for (let i = 0; i < rest.length; i += 2) {
|
|
85
|
+
const [flag, value] = [rest[i], rest[i + 1]];
|
|
86
|
+
// A value that is itself one of these flags is a missing value — `--install
|
|
87
|
+
// --test x` would otherwise run `--test` as the Install step.
|
|
88
|
+
if (!flags.has(flag) || value === undefined || flags.has(value)) throw cannot(USAGE);
|
|
89
|
+
if (flag in a) throw cannot(`${flag} given twice`);
|
|
90
|
+
a[flag] = value;
|
|
91
|
+
}
|
|
92
|
+
for (const f of ["--install", "--test"]) {
|
|
93
|
+
if (!a[f] || !a[f].trim()) throw cannot(`${f} needs a non-empty command — ${USAGE}`);
|
|
94
|
+
}
|
|
95
|
+
if (("--count-line" in a) !== ("--test-count" in a)) throw cannot("--count-line and --test-count go together");
|
|
96
|
+
if ("--test-count" in a && !isDigits(a["--test-count"])) {
|
|
97
|
+
throw cannot(`--test-count must be a whole number, got '${a["--test-count"]}'`);
|
|
98
|
+
}
|
|
99
|
+
if (("--mutate" in a) !== ("--mutation" in a)) throw cannot("--mutate and --mutation go together");
|
|
100
|
+
return {
|
|
101
|
+
repo,
|
|
102
|
+
install: a["--install"],
|
|
103
|
+
test: a["--test"],
|
|
104
|
+
countLine: a["--count-line"],
|
|
105
|
+
testCount: a["--test-count"],
|
|
106
|
+
mutate: a["--mutate"],
|
|
107
|
+
mutation: a["--mutation"],
|
|
108
|
+
};
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
function git(args, cwd) {
|
|
112
|
+
const r = spawnSync("git", args, { cwd, env: ENV, encoding: "utf8" });
|
|
113
|
+
return { ok: r.status === 0, out: (r.stdout ?? "").replace(/\n$/, ""), err: (r.stderr ?? "").trim() };
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
// Run a Recipe command through `sh -c` from the worktree root, its output to
|
|
117
|
+
// a log file rather than a pipe: a real suite's output has no useful size
|
|
118
|
+
// bound, and the log is what the caller reads a count line off. A command
|
|
119
|
+
// killed by a signal (an OOM kill, a timeout wrapper) has no exit status, so
|
|
120
|
+
// it is refused here and never reported as one: a crashed run must not count
|
|
121
|
+
// as a red suite.
|
|
122
|
+
function sh(cmd, cwd, log) {
|
|
123
|
+
const fd = openSync(log, "a");
|
|
124
|
+
try {
|
|
125
|
+
const r = spawnSync("sh", ["-c", cmd], { cwd, env: ENV, stdio: ["ignore", fd, fd] });
|
|
126
|
+
if (r.error) throw cannot(`could not start sh: ${r.error.message}`);
|
|
127
|
+
if (r.signal) throw notProven(`'${cmd}' was killed by ${r.signal}, so it has no exit status to read; output: ${log}`);
|
|
128
|
+
return r.status;
|
|
129
|
+
} finally {
|
|
130
|
+
closeSync(fd);
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
const didNotRun = (rc) => rc === 126 || rc === 127;
|
|
135
|
+
|
|
136
|
+
// `-uall` for the reason the header gives: an untracked, unignored file an
|
|
137
|
+
// install creates dirties the tree as surely as a rewrite, and the untracked
|
|
138
|
+
// mode is config — under status.showUntrackedFiles=no it is invisible unless
|
|
139
|
+
// pinned.
|
|
140
|
+
function treeChanges(wt) {
|
|
141
|
+
const s = git(["status", "--porcelain", "-uall"], wt);
|
|
142
|
+
if (!s.ok) throw notProven(`could not read the tree state in the throwaway worktree: ${s.err}`);
|
|
143
|
+
return s.out;
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
function prove(o, wt, logs) {
|
|
147
|
+
const installLog = join(logs, "install.log");
|
|
148
|
+
const irc = sh(o.install, wt, installLog);
|
|
149
|
+
if (didNotRun(irc)) {
|
|
150
|
+
throw notProven(`the Install step '${o.install}' did not run (exit ${irc}: not executable or not found); install output: ${installLog}`);
|
|
151
|
+
}
|
|
152
|
+
if (irc !== 0) throw notProven(`the Install step '${o.install}' failed (exit ${irc}); install output: ${installLog}`);
|
|
153
|
+
const dirty = treeChanges(wt);
|
|
154
|
+
if (dirty) throw notProven(`the Install step changed the tree (first: ${dirty.split("\n")[0]}); it must leave every file as checked out`);
|
|
155
|
+
|
|
156
|
+
const testLog = join(logs, "test.log");
|
|
157
|
+
const trc = sh(o.test, wt, testLog);
|
|
158
|
+
const where = `test output: ${testLog}`;
|
|
159
|
+
if (didNotRun(trc)) {
|
|
160
|
+
throw notProven(`the Test entrypoint '${o.test}' did not run (exit ${trc}: not executable or not found); ${where}`);
|
|
161
|
+
}
|
|
162
|
+
if (o.countLine === undefined && o.mutate === undefined) {
|
|
163
|
+
throw notProven(`no proof given — pass --count-line/--test-count or --mutate/--mutation (the run exited ${trc}); ${where}`);
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
const proof = {};
|
|
167
|
+
if (o.countLine !== undefined) {
|
|
168
|
+
const n = Number(o.testCount);
|
|
169
|
+
if (n === 0) throw notProven(`vacuous: a test count of 0 is a run that executed no tests; ${where}`);
|
|
170
|
+
if (!new RegExp(`(^|[^0-9])0*${n}([^0-9]|$)`).test(o.countLine)) {
|
|
171
|
+
throw notProven(`the count line '${o.countLine}' does not carry the test count ${n}`);
|
|
172
|
+
}
|
|
173
|
+
if (!readFileSync(testLog, "utf8").includes(o.countLine)) {
|
|
174
|
+
throw notProven(`vacuous: the Test entrypoint's output does not contain the count line '${o.countLine}' — no evidence any test ran; ${where}`);
|
|
175
|
+
}
|
|
176
|
+
proof.testCount = n;
|
|
177
|
+
}
|
|
178
|
+
if (o.mutate !== undefined) {
|
|
179
|
+
if (trc !== 0) {
|
|
180
|
+
throw notProven(`the unmutated run is already red (exit ${trc}), so a mutation turning it red proves nothing — use the count proof; ${where}`);
|
|
181
|
+
}
|
|
182
|
+
// What the unmutated Test run rewrote is not the mutation's work: restore
|
|
183
|
+
// every tracked file first, so the change read below is the mutation's alone.
|
|
184
|
+
const reset = git(["reset", "--hard", "-q", "HEAD"], wt);
|
|
185
|
+
if (!reset.ok) throw notProven(`could not restore the tree before the mutation: ${reset.err}`);
|
|
186
|
+
const mutateLog = join(logs, "mutate.log");
|
|
187
|
+
const mrc = sh(o.mutate, wt, mutateLog);
|
|
188
|
+
if (mrc !== 0) throw notProven(`the mutation command '${o.mutate}' failed (exit ${mrc}); its output: ${mutateLog}`);
|
|
189
|
+
// Against HEAD, not the index: a mutation that stages its edit (`git mv`,
|
|
190
|
+
// `git add`) changed a tracked file just as surely.
|
|
191
|
+
const changed = git(["diff", "HEAD", "--name-only"], wt);
|
|
192
|
+
if (!changed.ok) throw notProven(`could not read what the mutation changed: ${changed.err}`);
|
|
193
|
+
if (!changed.out) throw notProven(`the mutation (${o.mutation}) changed no tracked file — mutate a test, or the code a test covers`);
|
|
194
|
+
const mutatedLog = join(logs, "test-mutated.log");
|
|
195
|
+
const red = sh(o.test, wt, mutatedLog);
|
|
196
|
+
if (didNotRun(red)) throw notProven(`the Test entrypoint did not run on the mutated tree (exit ${red}); test output: ${mutatedLog}`);
|
|
197
|
+
if (red === 0) {
|
|
198
|
+
throw notProven(`vacuous: the mutation (${o.mutation}) did not turn the run red — the suite ran nothing it changed; test output: ${mutatedLog}`);
|
|
199
|
+
}
|
|
200
|
+
proof.mutation = `${o.mutation} — turned the run red (exit ${red}); changed ${changed.out.split("\n").join(", ")}`;
|
|
201
|
+
}
|
|
202
|
+
return proof;
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
// Write, then read back through the one reader. A cache it refuses is undone:
|
|
206
|
+
// the prior bytes restored, or the file removed when none stood before.
|
|
207
|
+
function writeCache(cache, recipe, repo) {
|
|
208
|
+
mkdirSync(dirname(cache), { recursive: true });
|
|
209
|
+
const prior = existsSync(cache) ? readFileSync(cache) : null;
|
|
210
|
+
const tmp = `${cache}.${process.pid}.tmp`;
|
|
211
|
+
try {
|
|
212
|
+
writeFileSync(tmp, `${JSON.stringify(recipe)}\n`);
|
|
213
|
+
renameSync(tmp, cache);
|
|
214
|
+
} catch (e) {
|
|
215
|
+
rmSync(tmp, { force: true });
|
|
216
|
+
throw e;
|
|
217
|
+
}
|
|
218
|
+
for (const field of ["install", "test"]) {
|
|
219
|
+
const r = spawnSync("sh", [READER, repo, field], { env: ENV, encoding: "utf8" });
|
|
220
|
+
if (r.status !== 0 || r.stdout.replace(/\n$/, "") !== recipe[field]) {
|
|
221
|
+
if (prior === null) unlinkSync(cache);
|
|
222
|
+
else writeFileSync(cache, prior);
|
|
223
|
+
throw notProven(`the Recipe cache reader refuses what was proven: ${(r.stderr || r.stdout).trim()}`);
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
}
|
|
227
|
+
|
|
228
|
+
// Best-effort, and never allowed to replace the proof's own outcome — a proof
|
|
229
|
+
// that held, or a Refusal that names why it did not, is what the caller reads.
|
|
230
|
+
// An Install step that leaves read-only directories behind (a Go module cache
|
|
231
|
+
// is the usual one; the repo ignores them) makes both git's removal and a
|
|
232
|
+
// plain recursive delete fail, so the fallback makes the tree writable first.
|
|
233
|
+
// The prune always runs: it is what drops the registration of a worktree
|
|
234
|
+
// whose directory is gone.
|
|
235
|
+
function removeWorktree(wt, repo) {
|
|
236
|
+
if (!git(["worktree", "remove", "--force", wt], repo).ok) {
|
|
237
|
+
spawnSync("chmod", ["-R", "u+w", wt], { stdio: "ignore" });
|
|
238
|
+
try {
|
|
239
|
+
rmSync(wt, { recursive: true, force: true });
|
|
240
|
+
} catch (e) {
|
|
241
|
+
process.stderr.write(`${NAME}: could not remove the throwaway worktree ${wt}: ${e.message}\n`);
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
git(["worktree", "prune"], repo);
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
function main(argv) {
|
|
248
|
+
const o = parseArgs(argv);
|
|
249
|
+
if (!git(["rev-parse", "--git-dir"], o.repo).ok) throw cannot(`${o.repo} is not a git repository`);
|
|
250
|
+
const common = git(["rev-parse", "--path-format=absolute", "--git-common-dir"], o.repo);
|
|
251
|
+
const workspace = common.ok ? workspaceDirFromGitCommonDir(common.out, o.repo) : null;
|
|
252
|
+
if (!workspace) throw cannot(`cannot resolve the common git dir of ${o.repo}: ${common.err}`);
|
|
253
|
+
const sha = git(["rev-parse", "--verify", "origin/main^{commit}"], o.repo);
|
|
254
|
+
if (!sha.ok) throw cannot("origin/main does not resolve to a commit — fetch it; the proof runs against it");
|
|
255
|
+
|
|
256
|
+
const logs = mkdtempSync(join(tmpdir(), `${NAME}-`));
|
|
257
|
+
const wt = join(logs, "wt");
|
|
258
|
+
const add = git(["worktree", "add", "--detach", wt, sha.out], o.repo);
|
|
259
|
+
if (!add.ok) throw cannot(`could not create the throwaway worktree: ${add.err}`);
|
|
260
|
+
let proof;
|
|
261
|
+
try {
|
|
262
|
+
proof = prove(o, wt, logs);
|
|
263
|
+
} finally {
|
|
264
|
+
removeWorktree(wt, o.repo);
|
|
265
|
+
}
|
|
266
|
+
|
|
267
|
+
const recipe = { install: o.install, test: o.test, derivedAt: sha.out, installClean: true, ...proof };
|
|
268
|
+
const cache = join(workspace, ".fleet", "recipe.json");
|
|
269
|
+
writeCache(cache, recipe, o.repo);
|
|
270
|
+
process.stdout.write(`${NAME}: PROVEN — Recipe cache written to ${cache}\n${JSON.stringify(recipe)}\n`);
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
try {
|
|
274
|
+
main(process.argv.slice(2));
|
|
275
|
+
} catch (e) {
|
|
276
|
+
// A fault outside the proof — the cache or its temp file could not be
|
|
277
|
+
// written — is "no verdict" (2), never the NOT PROVEN (1) a stack trace's
|
|
278
|
+
// exit status would read as.
|
|
279
|
+
if (!(e instanceof Refusal)) {
|
|
280
|
+
process.stderr.write(`${NAME}: ${e.message}\n`);
|
|
281
|
+
process.exit(2);
|
|
282
|
+
}
|
|
283
|
+
// The closing sentence on a line of its own: most reasons end in a log path,
|
|
284
|
+
// and a `.` riding on one would read as part of it.
|
|
285
|
+
const msg = e.code === 1 ? `NOT PROVEN — ${e.message}\nNo Recipe cache written.` : e.message;
|
|
286
|
+
process.stderr.write(`${NAME}: ${msg}\n`);
|
|
287
|
+
process.exit(e.code);
|
|
288
|
+
}
|
|
@@ -0,0 +1,537 @@
|
|
|
1
|
+
import { test } from "node:test";
|
|
2
|
+
import assert from "node:assert/strict";
|
|
3
|
+
import { execFileSync, spawnSync } from "node:child_process";
|
|
4
|
+
import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
|
|
5
|
+
import { join } from "node:path";
|
|
6
|
+
import { tempDir } from "./temp-dir.mjs";
|
|
7
|
+
import { writeExecStub } from "./exec-stub.mjs";
|
|
8
|
+
|
|
9
|
+
// recipe-prove.mjs is the Recipe derivation step's proof and the ONE writer of
|
|
10
|
+
// the Recipe cache: the deriving agent chooses the Install step and the Test
|
|
11
|
+
// entrypoint by reading the repository, and this script is what turns that
|
|
12
|
+
// guess into a cache — or refuses to. These fixtures stand in for two
|
|
13
|
+
// ecosystems the plugin keeps no knowledge of: a Maven-shaped repo and a
|
|
14
|
+
// Go-shaped one, each run through a stub of its own toolchain on PATH, so the
|
|
15
|
+
// suite needs neither `mvn` nor `go` and still exercises a command the
|
|
16
|
+
// plugin never names.
|
|
17
|
+
const SCRIPT = join(import.meta.dirname, "recipe-prove.mjs");
|
|
18
|
+
const READER = join(import.meta.dirname, "derive-testcmd.sh");
|
|
19
|
+
|
|
20
|
+
// Fixture construction must not inherit an ambient GIT_DIR: under one, `git
|
|
21
|
+
// init` re-inits whatever it names and the fixture is built nowhere.
|
|
22
|
+
const FIXTURE_ENV = { ...process.env, GIT_DIR: undefined, GIT_WORK_TREE: undefined };
|
|
23
|
+
|
|
24
|
+
// A stub `mvn`: `test` counts the @Test methods under src/test/java, prints
|
|
25
|
+
// Surefire's own summary line, and fails when a test asserts 1 == 2. Any other
|
|
26
|
+
// goal stands in for a dependency fetch and writes into `target/`, which the
|
|
27
|
+
// fixture ignores — the build output a real install leaves behind.
|
|
28
|
+
const MVN = `#!/bin/sh
|
|
29
|
+
if [ "$1" = -q ]; then shift; fi
|
|
30
|
+
case $1 in
|
|
31
|
+
test)
|
|
32
|
+
n=0; fail=0
|
|
33
|
+
for f in $(find src/test/java -name '*Test.java' 2>/dev/null); do
|
|
34
|
+
c=$(grep -c '@Test' "$f"); n=$((n + c))
|
|
35
|
+
if grep -q 'assertEquals(1, 2)' "$f"; then fail=1; fi
|
|
36
|
+
done
|
|
37
|
+
if [ "$n" -eq 0 ]; then echo "No tests to run."; exit 0; fi
|
|
38
|
+
echo "Tests run: $n, Failures: $fail, Errors: 0, Skipped: 0"
|
|
39
|
+
if [ "$fail" -ne 0 ]; then echo "BUILD FAILURE"; exit 1; fi
|
|
40
|
+
echo "BUILD SUCCESS" ;;
|
|
41
|
+
*) mkdir -p target && : > target/deps ;;
|
|
42
|
+
esac
|
|
43
|
+
`;
|
|
44
|
+
|
|
45
|
+
// A stub `go`: `go test ./...` reports "[no test files]" and passes when no
|
|
46
|
+
// _test.go is tracked — the vacuous green the real one gives — and otherwise
|
|
47
|
+
// passes only while Add still adds. It prints no test count, like the real
|
|
48
|
+
// one without -v: the mutation proof is the only proof this fixture offers.
|
|
49
|
+
const GO = `#!/bin/sh
|
|
50
|
+
[ "$1" = test ] || exit 0
|
|
51
|
+
if [ -z "$(find . -name '*_test.go' -not -path './.git/*')" ]; then
|
|
52
|
+
printf '? \\texample.com/calc\\t[no test files]\\n'; exit 0
|
|
53
|
+
fi
|
|
54
|
+
if grep -q 'return a + b' calc.go; then printf 'ok \\texample.com/calc\\t0.001s\\n'; exit 0; fi
|
|
55
|
+
printf -- '--- FAIL: TestAdd\\nFAIL\\texample.com/calc\\n'; exit 1
|
|
56
|
+
`;
|
|
57
|
+
|
|
58
|
+
function toolchain() {
|
|
59
|
+
const bin = tempDir("recipe-prove-bin-");
|
|
60
|
+
writeExecStub(join(bin, "mvn"), MVN);
|
|
61
|
+
writeExecStub(join(bin, "go"), GO);
|
|
62
|
+
return bin;
|
|
63
|
+
}
|
|
64
|
+
const BIN = toolchain();
|
|
65
|
+
|
|
66
|
+
const MAVEN_FILES = {
|
|
67
|
+
"pom.xml": "<project><modelVersion>4.0.0</modelVersion><groupId>x</groupId><artifactId>foo</artifactId><version>1</version></project>\n",
|
|
68
|
+
".gitignore": "target/\n",
|
|
69
|
+
"src/main/java/x/Foo.java": "package x; public class Foo { public int one() { return 1; } }\n",
|
|
70
|
+
"src/test/java/x/FooTest.java":
|
|
71
|
+
"package x;\nimport org.junit.jupiter.api.Test;\nimport static org.junit.jupiter.api.Assertions.assertEquals;\n" +
|
|
72
|
+
"class FooTest {\n @Test void one() { assertEquals(1, new Foo().one()); }\n}\n",
|
|
73
|
+
};
|
|
74
|
+
|
|
75
|
+
const GO_FILES = {
|
|
76
|
+
"go.mod": "module example.com/calc\n\ngo 1.22\n",
|
|
77
|
+
"calc.go": "package calc\n\nfunc Add(a, b int) int { return a + b }\n",
|
|
78
|
+
"calc_test.go":
|
|
79
|
+
'package calc\n\nimport "testing"\n\nfunc TestAdd(t *testing.T) {\n\tif Add(1, 2) != 3 {\n\t\tt.Fatal("Add")\n\t}\n}\n',
|
|
80
|
+
};
|
|
81
|
+
|
|
82
|
+
function git(dir, ...a) {
|
|
83
|
+
return execFileSync("git", a, { cwd: dir, stdio: "pipe", env: FIXTURE_ENV, encoding: "utf8" }).trim();
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
function write(dir, files) {
|
|
87
|
+
for (const [name, body] of Object.entries(files)) {
|
|
88
|
+
mkdirSync(join(dir, name, ".."), { recursive: true });
|
|
89
|
+
writeFileSync(join(dir, name), body);
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
// A repository whose `origin/main` is the commit holding `files` — the ref the
|
|
94
|
+
// proof runs against. Set with update-ref, so no remote is needed.
|
|
95
|
+
function repo(files) {
|
|
96
|
+
const dir = tempDir("recipe-prove-");
|
|
97
|
+
git(dir, "init", "-q");
|
|
98
|
+
git(dir, "config", "user.email", "t@t");
|
|
99
|
+
git(dir, "config", "user.name", "t");
|
|
100
|
+
write(dir, files);
|
|
101
|
+
git(dir, "add", "-A");
|
|
102
|
+
git(dir, "commit", "-q", "-m", "files");
|
|
103
|
+
git(dir, "update-ref", "refs/remotes/origin/main", "HEAD");
|
|
104
|
+
return { dir, head: git(dir, "rev-parse", "HEAD") };
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
function withoutTests(files) {
|
|
108
|
+
return Object.fromEntries(Object.entries(files).filter(([n]) => !/(^src\/test\/|_test\.go$)/.test(n)));
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
const cachePath = (dir) => join(dir, ".fleet", "recipe.json");
|
|
112
|
+
|
|
113
|
+
function prove(dir, args, { env = {}, cwd = dir } = {}) {
|
|
114
|
+
const tmp = tempDir("recipe-prove-tmp-");
|
|
115
|
+
const r = spawnSync(process.execPath, [SCRIPT, dir, ...args], {
|
|
116
|
+
encoding: "utf8",
|
|
117
|
+
cwd,
|
|
118
|
+
env: { ...FIXTURE_ENV, PATH: `${BIN}:${process.env.PATH}`, TMPDIR: tmp, ...env },
|
|
119
|
+
timeout: 60_000,
|
|
120
|
+
});
|
|
121
|
+
return { status: r.status, out: r.stdout, err: r.stderr, tmp };
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
function read(dir, field) {
|
|
125
|
+
const r = spawnSync("sh", [READER, dir, field], {
|
|
126
|
+
encoding: "utf8",
|
|
127
|
+
env: { ...FIXTURE_ENV, PATH: `${BIN}:${process.env.PATH}` },
|
|
128
|
+
});
|
|
129
|
+
return { status: r.status, out: r.stdout.replace(/\n$/, ""), err: r.stderr };
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
const MAVEN_PROOF = ["--install", "mvn -q dependency:go-offline", "--test", "mvn -q test", "--count-line", "Tests run: 1,", "--test-count", "1"];
|
|
133
|
+
const GO_PROOF = [
|
|
134
|
+
"--install", "go mod download",
|
|
135
|
+
"--test", "go test ./...",
|
|
136
|
+
"--mutate", "printf 'package calc\\n\\nfunc Add(a, b int) int { return a - b }\\n' > calc.go",
|
|
137
|
+
"--mutation", "Add subtracts instead of adding",
|
|
138
|
+
];
|
|
139
|
+
|
|
140
|
+
test("a Maven-shaped repo gets a proven cache: the count proof, derivedAt, installClean — and the reader accepts it", () => {
|
|
141
|
+
const { dir, head } = repo(MAVEN_FILES);
|
|
142
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
143
|
+
assert.equal(r.status, 0, r.err);
|
|
144
|
+
const cache = JSON.parse(readFileSync(cachePath(dir), "utf8"));
|
|
145
|
+
assert.deepEqual(cache, {
|
|
146
|
+
install: "mvn -q dependency:go-offline",
|
|
147
|
+
test: "mvn -q test",
|
|
148
|
+
derivedAt: head,
|
|
149
|
+
installClean: true,
|
|
150
|
+
testCount: 1,
|
|
151
|
+
});
|
|
152
|
+
assert.deepEqual([read(dir, "install").out, read(dir, "test").out], ["mvn -q dependency:go-offline", "mvn -q test"]);
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
test("a Go-shaped repo gets a proven cache through the mutation proof, which records what turned the run red", () => {
|
|
156
|
+
const { dir, head } = repo(GO_FILES);
|
|
157
|
+
const r = prove(dir, GO_PROOF);
|
|
158
|
+
assert.equal(r.status, 0, r.err);
|
|
159
|
+
const cache = JSON.parse(readFileSync(cachePath(dir), "utf8"));
|
|
160
|
+
assert.equal(cache.install, "go mod download");
|
|
161
|
+
assert.equal(cache.test, "go test ./...");
|
|
162
|
+
assert.equal(cache.derivedAt, head);
|
|
163
|
+
assert.equal(cache.installClean, true);
|
|
164
|
+
assert.equal(cache.testCount, undefined);
|
|
165
|
+
assert.match(cache.mutation, /^Add subtracts instead of adding — turned the run red \(exit 1\); changed calc\.go$/);
|
|
166
|
+
assert.equal(read(dir, "test").out, "go test ./...");
|
|
167
|
+
});
|
|
168
|
+
|
|
169
|
+
test("the Maven fixture with its tests deleted gets NO cache, and the refusal names vacuity", () => {
|
|
170
|
+
const { dir } = repo(withoutTests(MAVEN_FILES));
|
|
171
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
172
|
+
assert.equal(r.status, 1, r.err);
|
|
173
|
+
assert.match(r.err, /NOT PROVEN — vacuous: the Test entrypoint's output does not contain the count line 'Tests run: 1,'/);
|
|
174
|
+
assert.match(r.err, /No Recipe cache written/);
|
|
175
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
176
|
+
});
|
|
177
|
+
|
|
178
|
+
test("the Go fixture with its tests deleted gets NO cache: the mutation leaves the run green, which is vacuity", () => {
|
|
179
|
+
const { dir } = repo(withoutTests(GO_FILES));
|
|
180
|
+
const r = prove(dir, GO_PROOF);
|
|
181
|
+
assert.equal(r.status, 1, r.err);
|
|
182
|
+
assert.match(r.err, /NOT PROVEN — vacuous: the mutation \(Add subtracts instead of adding\) did not turn the run red/);
|
|
183
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
184
|
+
});
|
|
185
|
+
|
|
186
|
+
test("a count proof of zero tests is a failed proof, even when the runner prints exactly that", () => {
|
|
187
|
+
const { dir } = repo(withoutTests(MAVEN_FILES));
|
|
188
|
+
const r = prove(dir, ["--install", "true", "--test", "echo 'tests 0'", "--count-line", "tests 0", "--test-count", "0"]);
|
|
189
|
+
assert.equal(r.status, 1);
|
|
190
|
+
assert.match(r.err, /NOT PROVEN — vacuous: a test count of 0/);
|
|
191
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
192
|
+
});
|
|
193
|
+
|
|
194
|
+
test("the claimed count must be the number on the count line, not any number", () => {
|
|
195
|
+
const { dir } = repo(MAVEN_FILES);
|
|
196
|
+
const r = prove(dir, ["--install", "true", "--test", "mvn -q test", "--count-line", "Tests run: 1,", "--test-count", "7"]);
|
|
197
|
+
assert.equal(r.status, 1);
|
|
198
|
+
assert.match(r.err, /the count line 'Tests run: 1,' does not carry the test count 7/);
|
|
199
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
200
|
+
});
|
|
201
|
+
|
|
202
|
+
test("an Install step that changes a tracked file is not proven", () => {
|
|
203
|
+
const { dir } = repo(MAVEN_FILES);
|
|
204
|
+
const r = prove(dir, ["--install", "echo '<!-- -->' >> pom.xml", ...MAVEN_PROOF.slice(2)]);
|
|
205
|
+
assert.equal(r.status, 1);
|
|
206
|
+
assert.match(r.err, /NOT PROVEN — the Install step changed the tree \(first: M pom\.xml\)/);
|
|
207
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
208
|
+
});
|
|
209
|
+
|
|
210
|
+
test("an Install step that creates a file the tree neither tracks nor ignores is not proven", () => {
|
|
211
|
+
const { dir } = repo(MAVEN_FILES);
|
|
212
|
+
const r = prove(dir, ["--install", ": > deps.lock", ...MAVEN_PROOF.slice(2)]);
|
|
213
|
+
assert.equal(r.status, 1);
|
|
214
|
+
assert.match(r.err, /the Install step changed the tree \(first: \?\? deps\.lock\)/);
|
|
215
|
+
});
|
|
216
|
+
|
|
217
|
+
test("an Install step whose output the repo ignores IS proven — build output is not a dirty tree", () => {
|
|
218
|
+
// The stub mvn's dependency goal writes target/deps; the fixture ignores
|
|
219
|
+
// target/. Refusing here would refuse every repo whose install builds.
|
|
220
|
+
const { dir } = repo(MAVEN_FILES);
|
|
221
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
222
|
+
assert.equal(r.status, 0, r.err);
|
|
223
|
+
});
|
|
224
|
+
|
|
225
|
+
test("an Install step or Test entrypoint that does not RUN is named as such", () => {
|
|
226
|
+
const { dir } = repo(MAVEN_FILES);
|
|
227
|
+
const i = prove(dir, ["--install", "no-such-installer-xyz", ...MAVEN_PROOF.slice(2)]);
|
|
228
|
+
assert.equal(i.status, 1);
|
|
229
|
+
assert.match(i.err, /NOT PROVEN — the Install step 'no-such-installer-xyz' did not run \(exit 127: not executable or not found\)/);
|
|
230
|
+
const t = prove(dir, ["--install", "true", "--test", "no-such-runner-xyz", "--count-line", "x 1", "--test-count", "1"]);
|
|
231
|
+
assert.equal(t.status, 1);
|
|
232
|
+
assert.match(t.err, /NOT PROVEN — the Test entrypoint 'no-such-runner-xyz' did not run \(exit 127/);
|
|
233
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
234
|
+
});
|
|
235
|
+
|
|
236
|
+
test("an Install step that fails is not proven", () => {
|
|
237
|
+
const { dir } = repo(MAVEN_FILES);
|
|
238
|
+
const r = prove(dir, ["--install", "exit 3", ...MAVEN_PROOF.slice(2)]);
|
|
239
|
+
assert.equal(r.status, 1);
|
|
240
|
+
assert.match(r.err, /NOT PROVEN — the Install step 'exit 3' failed \(exit 3\)/);
|
|
241
|
+
});
|
|
242
|
+
|
|
243
|
+
test("the mutation proof refuses a red baseline: a mutation turning a red run red proves nothing", () => {
|
|
244
|
+
const { dir } = repo(GO_FILES);
|
|
245
|
+
const r = prove(dir, ["--install", "true", "--test", "go test ./... && exit 1", ...GO_PROOF.slice(4)]);
|
|
246
|
+
assert.equal(r.status, 1);
|
|
247
|
+
assert.match(r.err, /the unmutated run is already red \(exit 1\), so a mutation turning it red proves nothing — use the count proof/);
|
|
248
|
+
});
|
|
249
|
+
|
|
250
|
+
test("a mutation that changes no tracked file is refused, not counted as red", () => {
|
|
251
|
+
const { dir } = repo(GO_FILES);
|
|
252
|
+
const r = prove(dir, ["--install", "true", "--test", "go test ./...", "--mutate", ": > new_test.go", "--mutation", "adds a file"]);
|
|
253
|
+
assert.equal(r.status, 1);
|
|
254
|
+
assert.match(r.err, /the mutation \(adds a file\) changed no tracked file/);
|
|
255
|
+
});
|
|
256
|
+
|
|
257
|
+
test("a mutation that stages its edit still counts as changing a tracked file", () => {
|
|
258
|
+
const { dir } = repo(GO_FILES);
|
|
259
|
+
const r = prove(dir, [...GO_PROOF.slice(0, 4), "--mutate", `${GO_PROOF[5]} && git add calc.go`, "--mutation", "Add subtracts, staged"]);
|
|
260
|
+
assert.equal(r.status, 0, r.err);
|
|
261
|
+
assert.match(JSON.parse(readFileSync(cachePath(dir), "utf8")).mutation, /changed calc\.go$/);
|
|
262
|
+
});
|
|
263
|
+
|
|
264
|
+
test("no proof given is a refusal that still shows where the run's output is", () => {
|
|
265
|
+
const { dir } = repo(MAVEN_FILES);
|
|
266
|
+
const r = prove(dir, ["--install", "true", "--test", "mvn -q test"]);
|
|
267
|
+
assert.equal(r.status, 1);
|
|
268
|
+
assert.match(r.err, /NOT PROVEN — no proof given/);
|
|
269
|
+
const log = r.err.match(/test output: (\S+)/);
|
|
270
|
+
assert.ok(log, r.err);
|
|
271
|
+
assert.match(readFileSync(log[1], "utf8"), /Tests run: 1, Failures: 0/);
|
|
272
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
273
|
+
});
|
|
274
|
+
|
|
275
|
+
test("the proof runs against origin/main, not the checkout's own HEAD, and derivedAt says so", () => {
|
|
276
|
+
// The local branch has deleted the tests; origin/main still has them. A
|
|
277
|
+
// proof run against the checkout would be vacuous — against origin/main it
|
|
278
|
+
// is not.
|
|
279
|
+
const { dir, head } = repo(MAVEN_FILES);
|
|
280
|
+
rmSync(join(dir, "src", "test"), { recursive: true });
|
|
281
|
+
git(dir, "commit", "-q", "-am", "drop tests locally");
|
|
282
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
283
|
+
assert.equal(r.status, 0, r.err);
|
|
284
|
+
assert.equal(JSON.parse(readFileSync(cachePath(dir), "utf8")).derivedAt, head);
|
|
285
|
+
});
|
|
286
|
+
|
|
287
|
+
test("a failed proof leaves an existing cache byte-identical — it never writes on a guess", () => {
|
|
288
|
+
const { dir } = repo(withoutTests(MAVEN_FILES));
|
|
289
|
+
mkdirSync(join(dir, ".fleet"));
|
|
290
|
+
writeFileSync(cachePath(dir), "prior bytes");
|
|
291
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
292
|
+
assert.equal(r.status, 1);
|
|
293
|
+
assert.equal(readFileSync(cachePath(dir), "utf8"), "prior bytes");
|
|
294
|
+
});
|
|
295
|
+
|
|
296
|
+
test("a cache the reader would refuse is not left behind: it is rolled back to what stood before", () => {
|
|
297
|
+
// `mvn -q test;` runs fine under sh -c, but a trailing `;` would let a
|
|
298
|
+
// runner-appended argument run as its own command, so the reader refuses it.
|
|
299
|
+
const { dir } = repo(MAVEN_FILES);
|
|
300
|
+
mkdirSync(join(dir, ".fleet"));
|
|
301
|
+
writeFileSync(cachePath(dir), "prior bytes");
|
|
302
|
+
const r = prove(dir, ["--install", "true", "--test", "mvn -q test;", "--count-line", "Tests run: 1,", "--test-count", "1"]);
|
|
303
|
+
assert.equal(r.status, 1, r.err);
|
|
304
|
+
assert.match(r.err, /NOT PROVEN — the Recipe cache reader refuses what was proven: .*ends in ';'/);
|
|
305
|
+
assert.equal(readFileSync(cachePath(dir), "utf8"), "prior bytes");
|
|
306
|
+
|
|
307
|
+
const { dir: fresh } = repo(MAVEN_FILES);
|
|
308
|
+
const f = prove(fresh, ["--install", "true", "--test", "mvn -q test;", "--count-line", "Tests run: 1,", "--test-count", "1"]);
|
|
309
|
+
assert.equal(f.status, 1);
|
|
310
|
+
assert.equal(existsSync(cachePath(fresh)), false, "no prior cache: the refused one is removed, not left");
|
|
311
|
+
});
|
|
312
|
+
|
|
313
|
+
test("from a linked worktree the cache lands beside the common git dir — the main checkout the reader reads", () => {
|
|
314
|
+
const { dir, head } = repo(MAVEN_FILES);
|
|
315
|
+
const wt = join(tempDir("recipe-prove-wt-"), "wt");
|
|
316
|
+
git(dir, "worktree", "add", "-q", "--detach", wt);
|
|
317
|
+
const r = prove(wt, MAVEN_PROOF);
|
|
318
|
+
assert.equal(r.status, 0, r.err);
|
|
319
|
+
assert.equal(JSON.parse(readFileSync(cachePath(dir), "utf8")).derivedAt, head);
|
|
320
|
+
assert.equal(existsSync(cachePath(wt)), false);
|
|
321
|
+
});
|
|
322
|
+
|
|
323
|
+
test("the throwaway worktree is removed afterwards, proven or not", () => {
|
|
324
|
+
const { dir } = repo(MAVEN_FILES);
|
|
325
|
+
prove(dir, MAVEN_PROOF);
|
|
326
|
+
prove(dir, ["--install", "exit 3", ...MAVEN_PROOF.slice(2)]);
|
|
327
|
+
assert.equal(git(dir, "worktree", "list", "--porcelain").split("\n").filter((l) => l.startsWith("worktree ")).length, 1);
|
|
328
|
+
});
|
|
329
|
+
|
|
330
|
+
test("an ambient GIT_DIR naming another repository does not change the answer", () => {
|
|
331
|
+
const { dir, head } = repo(MAVEN_FILES);
|
|
332
|
+
const { dir: other } = repo(withoutTests(MAVEN_FILES));
|
|
333
|
+
const r = prove(dir, MAVEN_PROOF, { env: { GIT_DIR: join(other, ".git"), GIT_WORK_TREE: other } });
|
|
334
|
+
assert.equal(r.status, 0, r.err);
|
|
335
|
+
assert.equal(JSON.parse(readFileSync(cachePath(dir), "utf8")).derivedAt, head);
|
|
336
|
+
assert.equal(existsSync(cachePath(other)), false);
|
|
337
|
+
});
|
|
338
|
+
|
|
339
|
+
test("a proof that cannot be attempted exits 2, distinct from a Recipe that is not proven", () => {
|
|
340
|
+
const usage = spawnSync(process.execPath, [SCRIPT], { encoding: "utf8", env: FIXTURE_ENV });
|
|
341
|
+
assert.equal(usage.status, 2);
|
|
342
|
+
assert.match(usage.stderr, /usage: recipe-prove\.mjs <repo> --install <cmd> --test <cmd>/);
|
|
343
|
+
|
|
344
|
+
const notRepo = prove(tempDir("recipe-prove-norepo-"), MAVEN_PROOF);
|
|
345
|
+
assert.equal(notRepo.status, 2);
|
|
346
|
+
assert.match(notRepo.err, /is not a git repository/);
|
|
347
|
+
|
|
348
|
+
const { dir } = repo(MAVEN_FILES);
|
|
349
|
+
git(dir, "update-ref", "-d", "refs/remotes/origin/main");
|
|
350
|
+
const noMain = prove(dir, MAVEN_PROOF);
|
|
351
|
+
assert.equal(noMain.status, 2);
|
|
352
|
+
assert.match(noMain.err, /origin\/main does not resolve to a commit/);
|
|
353
|
+
|
|
354
|
+
const { dir: half } = repo(MAVEN_FILES);
|
|
355
|
+
const r = prove(half, ["--install", "true", "--test", "mvn -q test", "--count-line", "Tests run: 1,"]);
|
|
356
|
+
assert.equal(r.status, 2);
|
|
357
|
+
assert.match(r.err, /--count-line and --test-count go together/);
|
|
358
|
+
|
|
359
|
+
const nan = prove(half, ["--install", "true", "--test", "mvn -q test", "--count-line", "Tests run: 1,", "--test-count", "1e0"]);
|
|
360
|
+
assert.equal(nan.status, 2);
|
|
361
|
+
assert.match(nan.err, /--test-count must be a whole number, got '1e0'/);
|
|
362
|
+
|
|
363
|
+
// A flag where a value belongs is a missing value, never the command. The
|
|
364
|
+
// fixture must be one whose swallowed value would otherwise be ACCEPTED: with
|
|
365
|
+
// `--install --test "mvn -q test"` the later iteration refuses on its own
|
|
366
|
+
// and the guard under test is never what fires.
|
|
367
|
+
const missing = prove(half, ["--install", "--test", "--test", "mvn -q test"]);
|
|
368
|
+
assert.equal(missing.status, 2);
|
|
369
|
+
assert.match(missing.err, /^recipe-prove: usage:/);
|
|
370
|
+
assert.equal(git(half, "worktree", "list", "--porcelain").split("\n").filter((l) => l.startsWith("worktree ")).length, 1,
|
|
371
|
+
"a refusal at the argument boundary creates no worktree");
|
|
372
|
+
});
|
|
373
|
+
|
|
374
|
+
test("the proof leaves no worktree directory behind in its temp dir", () => {
|
|
375
|
+
const { dir } = repo(MAVEN_FILES);
|
|
376
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
377
|
+
assert.equal(r.status, 0, r.err);
|
|
378
|
+
for (const d of readdirSync(r.tmp)) {
|
|
379
|
+
assert.deepEqual(readdirSync(join(r.tmp, d)).filter((n) => n === "wt"), [], `${d} still holds the throwaway worktree`);
|
|
380
|
+
}
|
|
381
|
+
});
|
|
382
|
+
|
|
383
|
+
// Whether the process ignores file modes: root writes into a 0555 directory.
|
|
384
|
+
const IGNORES_MODES = process.getuid?.() === 0 && "root ignores file modes";
|
|
385
|
+
|
|
386
|
+
// An Install step whose output the repo ignores and leaves read-only — a Go
|
|
387
|
+
// module cache is the usual one. git's own removal of the worktree fails on it.
|
|
388
|
+
const READ_ONLY_INSTALL = "mkdir -p build/x && : > build/x/f && chmod 555 build/x";
|
|
389
|
+
const IGNORING_BUILD = { ...MAVEN_FILES, ".gitignore": "target/\nbuild/\n" };
|
|
390
|
+
|
|
391
|
+
test("read-only install output cannot turn a proof that held into a crash", { skip: IGNORES_MODES }, () => {
|
|
392
|
+
const { dir, head } = repo(IGNORING_BUILD);
|
|
393
|
+
const r = prove(dir, ["--install", READ_ONLY_INSTALL, ...MAVEN_PROOF.slice(2)]);
|
|
394
|
+
assert.equal(r.status, 0, r.err);
|
|
395
|
+
assert.equal(JSON.parse(readFileSync(cachePath(dir), "utf8")).derivedAt, head);
|
|
396
|
+
for (const d of readdirSync(r.tmp)) {
|
|
397
|
+
assert.deepEqual(readdirSync(join(r.tmp, d)).filter((n) => n === "wt"), [], `${d} still holds the throwaway worktree`);
|
|
398
|
+
}
|
|
399
|
+
assert.equal(git(dir, "worktree", "list", "--porcelain").split("\n").filter((l) => l.startsWith("worktree ")).length, 1);
|
|
400
|
+
});
|
|
401
|
+
|
|
402
|
+
test("read-only install output does not replace a refusal's reason with a stack trace", { skip: IGNORES_MODES }, () => {
|
|
403
|
+
const { dir } = repo(IGNORING_BUILD);
|
|
404
|
+
const r = prove(dir, ["--install", READ_ONLY_INSTALL, "--test", "echo 'tests 0'", "--count-line", "tests 0", "--test-count", "0"]);
|
|
405
|
+
assert.equal(r.status, 1);
|
|
406
|
+
assert.match(r.err, /NOT PROVEN — vacuous: a test count of 0/);
|
|
407
|
+
assert.doesNotMatch(r.err, /ENOTEMPTY/);
|
|
408
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
409
|
+
});
|
|
410
|
+
|
|
411
|
+
test("a cache that cannot be written is exit 2 with its reason, never a stack trace or a NOT PROVEN", { skip: IGNORES_MODES }, () => {
|
|
412
|
+
const { dir } = repo(MAVEN_FILES);
|
|
413
|
+
mkdirSync(join(dir, ".fleet"));
|
|
414
|
+
chmodSync(join(dir, ".fleet"), 0o555);
|
|
415
|
+
try {
|
|
416
|
+
const r = prove(dir, MAVEN_PROOF);
|
|
417
|
+
assert.equal(r.status, 2, r.err);
|
|
418
|
+
assert.match(r.err, /^recipe-prove: EACCES/);
|
|
419
|
+
assert.doesNotMatch(r.err, /\n\s+at /);
|
|
420
|
+
assert.doesNotMatch(r.err, /NOT PROVEN/);
|
|
421
|
+
assert.equal(r.out, "");
|
|
422
|
+
} finally {
|
|
423
|
+
chmodSync(join(dir, ".fleet"), 0o755);
|
|
424
|
+
}
|
|
425
|
+
});
|
|
426
|
+
|
|
427
|
+
test("a command killed by a signal is refused, never counted as a red or a green run", () => {
|
|
428
|
+
// The count proof: the output carries the count line, then the run dies.
|
|
429
|
+
const { dir } = repo(MAVEN_FILES);
|
|
430
|
+
const t = prove(dir, ["--install", "true", "--test", "echo 'tests 3'; kill -9 $$", "--count-line", "tests 3", "--test-count", "3"]);
|
|
431
|
+
assert.equal(t.status, 1, t.err);
|
|
432
|
+
assert.match(t.err, /NOT PROVEN — 'echo 'tests 3'; kill -9 \$\$' was killed by SIGKILL/);
|
|
433
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
434
|
+
|
|
435
|
+
// The mutation proof: the unmutated run is green, the mutated one dies — a
|
|
436
|
+
// crash is not a test failing.
|
|
437
|
+
const { dir: go } = repo(GO_FILES);
|
|
438
|
+
const dies = "if grep -q 'a - b' calc.go; then kill -9 $$; else go test ./...; fi";
|
|
439
|
+
const m = prove(go, ["--install", "true", "--test", dies, ...GO_PROOF.slice(4)]);
|
|
440
|
+
assert.equal(m.status, 1, m.err);
|
|
441
|
+
assert.match(m.err, /NOT PROVEN — '.*kill -9 \$\$.*' was killed by SIGKILL/);
|
|
442
|
+
assert.equal(existsSync(cachePath(go)), false);
|
|
443
|
+
});
|
|
444
|
+
|
|
445
|
+
test("an untracked file an Install step creates is seen even under status.showUntrackedFiles=no", () => {
|
|
446
|
+
const { dir } = repo(MAVEN_FILES);
|
|
447
|
+
git(dir, "config", "status.showUntrackedFiles", "no");
|
|
448
|
+
const r = prove(dir, ["--install", ": > deps.lock", ...MAVEN_PROOF.slice(2)]);
|
|
449
|
+
assert.equal(r.status, 1, r.err);
|
|
450
|
+
assert.match(r.err, /the Install step changed the tree \(first: \?\? deps\.lock\)/);
|
|
451
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
452
|
+
});
|
|
453
|
+
|
|
454
|
+
test("the claimed count is a whole number on the count line, leading zeros aside", () => {
|
|
455
|
+
const claim = (line, n) => {
|
|
456
|
+
const { dir } = repo(MAVEN_FILES);
|
|
457
|
+
return { dir, ...prove(dir, ["--install", "true", "--test", `echo '${line}'`, "--count-line", line, "--test-count", n]) };
|
|
458
|
+
};
|
|
459
|
+
// 1 is a substring of 11 and of 21, and is not the number either line carries.
|
|
460
|
+
for (const line of ["Tests run: 11", "21 passed"]) {
|
|
461
|
+
const r = claim(line, "1");
|
|
462
|
+
assert.equal(r.status, 1, `${line}: ${r.err}`);
|
|
463
|
+
assert.match(r.err, /does not carry the test count 1/);
|
|
464
|
+
assert.equal(existsSync(cachePath(r.dir)), false);
|
|
465
|
+
}
|
|
466
|
+
// A runner that zero-pads its count still carries it.
|
|
467
|
+
const padded = claim("tests 01", "1");
|
|
468
|
+
assert.equal(padded.status, 0, padded.err);
|
|
469
|
+
assert.equal(JSON.parse(readFileSync(cachePath(padded.dir), "utf8")).testCount, 1);
|
|
470
|
+
});
|
|
471
|
+
|
|
472
|
+
test("the Install step and the Test entrypoint run against their own worktree whatever GIT_DIR the caller exports", () => {
|
|
473
|
+
// Each command asks git where it is: with an ambient GIT_DIR/GIT_WORK_TREE
|
|
474
|
+
// reaching the `sh -c` child, the answer is the other repository.
|
|
475
|
+
const here = 'test "$(git rev-parse --show-toplevel)" = "$(pwd -P)"';
|
|
476
|
+
const args = ["--install", here, "--test", `${here} && mvn -q test`, "--count-line", "Tests run: 1,", "--test-count", "1"];
|
|
477
|
+
const { dir } = repo(MAVEN_FILES);
|
|
478
|
+
const control = prove(dir, args);
|
|
479
|
+
assert.equal(control.status, 0, `the commands pass without an ambient GIT_DIR: ${control.err}`);
|
|
480
|
+
|
|
481
|
+
const { dir: other } = repo(withoutTests(MAVEN_FILES));
|
|
482
|
+
const { dir: dir2 } = repo(MAVEN_FILES);
|
|
483
|
+
const r = prove(dir2, args, { env: { GIT_DIR: join(other, ".git"), GIT_WORK_TREE: other } });
|
|
484
|
+
assert.equal(r.status, 0, r.err);
|
|
485
|
+
});
|
|
486
|
+
|
|
487
|
+
test("exit 126 and 127 both mean a command did not RUN, in the Install step and in the Test entrypoint", () => {
|
|
488
|
+
const { dir } = repo({ ...MAVEN_FILES, "t.sh": "echo 'Tests run: 1,'\n" });
|
|
489
|
+
// t.sh is committed 0644: `sh -c ./t.sh` exits 126, "found but not executable".
|
|
490
|
+
const i = prove(dir, ["--install", "./t.sh", ...MAVEN_PROOF.slice(2)]);
|
|
491
|
+
assert.equal(i.status, 1);
|
|
492
|
+
assert.match(i.err, /the Install step '\.\/t\.sh' did not run \(exit 126/);
|
|
493
|
+
const t = prove(dir, ["--install", "true", "--test", "./t.sh", "--count-line", "Tests run: 1,", "--test-count", "1"]);
|
|
494
|
+
assert.equal(t.status, 1);
|
|
495
|
+
assert.match(t.err, /the Test entrypoint '\.\/t\.sh' did not run \(exit 126/);
|
|
496
|
+
});
|
|
497
|
+
|
|
498
|
+
test("a mutation command that fails, or that leaves nothing runnable, or that breaks git itself, is refused for that reason", () => {
|
|
499
|
+
const { dir } = repo(GO_FILES);
|
|
500
|
+
const failing = prove(dir, ["--install", "true", "--test", "go test ./...", "--mutate", "exit 3", "--mutation", "x"]);
|
|
501
|
+
assert.equal(failing.status, 1);
|
|
502
|
+
assert.match(failing.err, /the mutation command 'exit 3' failed \(exit 3\)/);
|
|
503
|
+
|
|
504
|
+
// The Test entrypoint is an executable the mutation deletes: the mutated run
|
|
505
|
+
// exits 127, which is not a red suite.
|
|
506
|
+
const { dir: tool } = repo({ tool: "#!/bin/sh\necho 'Tests run: 1,'\n" });
|
|
507
|
+
git(tool, "update-index", "--chmod=+x", "tool");
|
|
508
|
+
git(tool, "commit", "-q", "--amend", "--no-edit");
|
|
509
|
+
git(tool, "update-ref", "refs/remotes/origin/main", "HEAD");
|
|
510
|
+
const gone = prove(tool, ["--install", "true", "--test", "./tool", "--mutate", "rm tool", "--mutation", "delete the runner"]);
|
|
511
|
+
assert.equal(gone.status, 1);
|
|
512
|
+
assert.match(gone.err, /the Test entrypoint did not run on the mutated tree \(exit 127\)/);
|
|
513
|
+
|
|
514
|
+
// A mutation that removes the worktree's link to its repository leaves
|
|
515
|
+
// git unable to say what changed.
|
|
516
|
+
const { dir: blind } = repo(GO_FILES);
|
|
517
|
+
const lost = prove(blind, ["--install", "true", "--test", "go test ./...", "--mutate", "rm .git", "--mutation", "orphan the tree"]);
|
|
518
|
+
assert.equal(lost.status, 1);
|
|
519
|
+
assert.match(lost.err, /could not read what the mutation changed/);
|
|
520
|
+
});
|
|
521
|
+
|
|
522
|
+
test("the mutation is credited only with what it changed, never with what the unmutated Test run rewrote", () => {
|
|
523
|
+
// t.sh is red while state.txt holds anything, and otherwise writes into it:
|
|
524
|
+
// the unmutated run rewrites a tracked file.
|
|
525
|
+
const files = { "state.txt": "", "t.sh": "if [ -s state.txt ]; then exit 1; fi\necho run > state.txt\n" };
|
|
526
|
+
const { dir } = repo(files);
|
|
527
|
+
const noop = prove(dir, ["--install", "true", "--test", "sh t.sh", "--mutate", "true", "--mutation", "no-op"]);
|
|
528
|
+
assert.equal(noop.status, 1, noop.err);
|
|
529
|
+
assert.match(noop.err, /the mutation \(no-op\) changed no tracked file/);
|
|
530
|
+
assert.equal(existsSync(cachePath(dir)), false);
|
|
531
|
+
|
|
532
|
+
// A genuine mutation of a file the run also rewrites is still a mutation.
|
|
533
|
+
const { dir: real } = repo(files);
|
|
534
|
+
const r = prove(real, ["--install", "true", "--test", "sh t.sh", "--mutate", "echo x > state.txt", "--mutation", "poison the state"]);
|
|
535
|
+
assert.equal(r.status, 0, r.err);
|
|
536
|
+
assert.match(JSON.parse(readFileSync(cachePath(real), "utf8")).mutation, /changed state\.txt$/);
|
|
537
|
+
});
|
|
@@ -76,7 +76,7 @@ gh label create in-progress --color FBCA04 --force # first time only
|
|
|
76
76
|
gh issue edit <N> --add-label in-progress
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
Infer branch/worktree convention from `git worktree list` / `git branch -r` — commonly `feat|fix|refactor/<N>-slug` and `.worktrees/<N>-slug`. Branch off fresh `origin/main`,
|
|
79
|
+
Infer branch/worktree convention from `git worktree list` / `git branch -r` — commonly `feat|fix|refactor/<N>-slug` and `.worktrees/<N>-slug`. Branch off fresh `origin/main`, run the Install step, run the Test entrypoint as the baseline — both the repo's **Recipe**, read from its Recipe cache: `~/.fleet/bin/fleet-run derive-testcmd.sh . install` and `… . test` print them. **No usable cache (either refuses) → run the Recipe derivation step first:** dispatch a `task` member, agent `fleet-recipe-deriver`, its prompt naming the main checkout's absolute path as `<repo>` and a scratch directory of its own; it reads the repo, proves both commands in a throwaway worktree and writes the cache only on proof. `RECIPE NOT PROVEN` → stop and give the maintainer its reason; never pick a command yourself.
|
|
80
80
|
|
|
81
81
|
Abandoned before PR opens → remove `in-progress`.
|
|
82
82
|
|
|
@@ -90,7 +90,7 @@ Come back here. The sizing path hands off; this step always follows.
|
|
|
90
90
|
|
|
91
91
|
```bash
|
|
92
92
|
git fetch origin && git rebase origin/main # rebase, never merge main in
|
|
93
|
-
<test command> # re-run after rebasing
|
|
93
|
+
<test command> # the Recipe's Test entrypoint; re-run after rebasing
|
|
94
94
|
git push --force-with-lease -u origin HEAD
|
|
95
95
|
gh pr create --base main --body "…
|
|
96
96
|
|
package/skills/run-team/SKILL.md
CHANGED
|
@@ -361,6 +361,52 @@ phase, or in any later one, asks the maintainer which tickets to take.
|
|
|
361
361
|
before this run's own first tick or beat, either of which writes the mark
|
|
362
362
|
the guard reads. Older archives, hand-named ones included, are left alone.
|
|
363
363
|
|
|
364
|
+
**Derive the Recipe — the Recipe derivation step — before the fold-in below
|
|
365
|
+
and before the first claim.** Every claim runs this repository's Install
|
|
366
|
+
step and bakes its Test entrypoint into the member's runner, and every
|
|
367
|
+
review reads the Test entrypoint, all out of the **Recipe cache**
|
|
368
|
+
(`.fleet/recipe.json` beside the common git dir); no script infers either
|
|
369
|
+
command. Check it first: `~/.fleet/bin/fleet-run derive-testcmd.sh .
|
|
370
|
+
install` and `~/.fleet/bin/fleet-run derive-testcmd.sh . test`. **Both exit
|
|
371
|
+
0 → the cache is present and valid: use it as it stands and derive
|
|
372
|
+
nothing.** Either refuses → derive it. This step is yours: one derivation
|
|
373
|
+
per run start at most, never one per member, and a member never derives.
|
|
374
|
+
Dispatch a `task` member named `derive-recipe`, agent
|
|
375
|
+
`fleet-recipe-deriver`, and wait for its report — nothing in this run can
|
|
376
|
+
claim before it lands:
|
|
377
|
+
|
|
378
|
+
> Derive, prove and cache the Recipe of the repository at `<repo>`. Your
|
|
379
|
+
> scratch directory is `<scratch>/derive-recipe/`; write nowhere else.
|
|
380
|
+
> Follow your definition's procedure exactly, and end with its report line.
|
|
381
|
+
|
|
382
|
+
`<repo>` is this main checkout's absolute path and `<scratch>` is resolved
|
|
383
|
+
to an absolute path before you send it. The member reads the repository,
|
|
384
|
+
proposes both commands and proves them with `recipe-prove.mjs` in a
|
|
385
|
+
throwaway worktree of `origin/main`: the Install step must leave the tree
|
|
386
|
+
clean, and the Test entrypoint must be shown to run real tests — the
|
|
387
|
+
runner's own non-zero count, or a deliberate mutation of one test that
|
|
388
|
+
turns the run red. Only a proof writes the cache, with `derivedAt` (the
|
|
389
|
+
`origin/main` commit it ran against) and the proof itself. **`RECIPE
|
|
390
|
+
PROVEN`** → re-run both `derive-testcmd.sh` reads; their output is the
|
|
391
|
+
Recipe this run uses. **`RECIPE NOT PROVEN`** → stall, below.
|
|
392
|
+
|
|
393
|
+
**Mid-run, a cache that stops running is re-derived once, then the run
|
|
394
|
+
stalls.** A claim or a review that refuses because the Recipe cache is
|
|
395
|
+
invalid — a command that no longer runs, or an Install step that changed
|
|
396
|
+
the tree — is the one mid-run trigger: dispatch `derive-recipe` again
|
|
397
|
+
exactly as above, then retry what refused. The Install step's refusals
|
|
398
|
+
fire after the claim has labelled the ticket and created its worktree, so
|
|
399
|
+
release that claim with `release-ticket.sh` first (**Release the claims
|
|
400
|
+
that never became PRs**). A red suite never triggers it: a failing test is
|
|
401
|
+
a finding, not a stale Recipe. **Stall:** `RECIPE NOT PROVEN` from any
|
|
402
|
+
derivation, or a second invalid-cache refusal after this run's one
|
|
403
|
+
re-derivation, halts the run — no Pull, no claim, no review dispatch — and
|
|
404
|
+
you report `STALL — Recipe not proven: <cause>` to the maintainer with the
|
|
405
|
+
deriver's reason verbatim. A repository the fleet cannot run needs human
|
|
406
|
+
hands, the `ready-for-human` cause: a toolchain this machine lacks, or a
|
|
407
|
+
suite that runs no tests the proof can see. It is never a loop: no third
|
|
408
|
+
derivation, and never a cache written by hand.
|
|
409
|
+
|
|
364
410
|
**Fold in every PR a prior run left open, before shortlisting.** A chore PR
|
|
365
411
|
carrying that run's own metrics, or ticket work whose review was deferred —
|
|
366
412
|
both are reviewable work no member otherwise picks up, because the Shortlist
|
|
@@ -730,7 +776,7 @@ oldest-first slot; its row is rewritten to `impl-<N> …` when it is pulled,
|
|
|
730
776
|
because `row` replaces the whole line.
|
|
731
777
|
|
|
732
778
|
**Claiming.** `~/.fleet/bin/fleet-run claim-ticket.sh <N> impl-<N> implementer --apply` does
|
|
733
|
-
the label, worktree, branch,
|
|
779
|
+
the label, worktree, branch, the Recipe's Install step, the clean-tree assertion, and the
|
|
734
780
|
isolation runner in one serial pass. The two arguments are always that literal
|
|
735
781
|
pair — `impl-<N>` as `<slug>`, `implementer` as `<type>` — so the claim is
|
|
736
782
|
branch `implementer/<N>-impl-<N>`, worktree `.worktrees/<N>-impl-<N>`. Never
|
|
@@ -738,9 +784,12 @@ read them off `git branch -r` or `git worktree list`: human branches on origin
|
|
|
738
784
|
(`fix/…`, `docs/…`) have another shape, and the script takes both arguments as
|
|
739
785
|
free text, so a pair copied from one claims under that shape at exit 0.
|
|
740
786
|
|
|
741
|
-
**
|
|
742
|
-
|
|
743
|
-
|
|
787
|
+
**The Install step comes from the Recipe cache — never pass or infer one.** The
|
|
788
|
+
claim reads it, and the Test entrypoint, through `derive-testcmd.sh`, runs the
|
|
789
|
+
Install step in the fresh worktree and refuses if it left any file changed: an
|
|
790
|
+
install that rewrites the tree in a throwaway worktree corrupts it for everyone.
|
|
791
|
+
No cache is phase 0's Recipe derivation step not having run; an invalid one is
|
|
792
|
+
its re-derive-once rule.
|
|
744
793
|
|
|
745
794
|
**Materialize the isolation envelope as a file, not a briefing.** The runner is
|
|
746
795
|
tracked at the repo root since #55, so every checkout has one; it derives its
|
|
@@ -2128,14 +2177,15 @@ The fix-applier prompt and every refuter brief carry the rule and the safe
|
|
|
2128
2177
|
invocation form; the merge bot's brief is the wrong seat, because a bot
|
|
2129
2178
|
reads gates and never injects a fault.
|
|
2130
2179
|
|
|
2131
|
-
**Where `testCmd` comes from:** the
|
|
2132
|
-
|
|
2133
|
-
|
|
2134
|
-
|
|
2135
|
-
|
|
2136
|
-
the review
|
|
2137
|
-
|
|
2138
|
-
|
|
2180
|
+
**Where `testCmd` comes from:** the repository's Test entrypoint, out of the
|
|
2181
|
+
Recipe cache phase 0's Recipe derivation step proved —
|
|
2182
|
+
`~/.fleet/bin/fleet-run derive-testcmd.sh . test` prints it, in this repo
|
|
2183
|
+
`node --test plugin/scripts/*.test.mjs` — and the one you hand specialists per
|
|
2184
|
+
**Give specialists a stack-free test command** above. Pass the same string to
|
|
2185
|
+
the review, to the fix-applier, and to the finisher — whose duty-2 mutation
|
|
2186
|
+
gate runs it too — so every gate runs one command. Omit it from the review args
|
|
2187
|
+
and the review (`review-core.mjs`, in a runner) reads the same cache itself,
|
|
2188
|
+
refusing outright when there is none; the fix-applier has no such fallback, so
|
|
2139
2189
|
substituting `<testCmd>` with nothing leaves it no gate at all.
|
|
2140
2190
|
|
|
2141
2191
|
**Where `<branch>` comes from:** the PR's head branch —
|