@feigi/fleet-ctl 3.70.0 → 3.71.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -42,8 +42,11 @@ from the repository root by `sh -c`:
42
42
  and rewrites nothing: the frozen or locked mode where the tool has one.
43
43
  `true` when nothing needs installing. It must leave every file exactly as
44
44
  checked out; build output the repository ignores is fine.
45
- - **Test entrypoint** — the whole suite, the way CI runs it. Members run it
46
- with their own arguments appended (`sh -c '<test> "$@"'`), so prefer a form
45
+ - **Test entrypoint** — every test step in the merge-gating CI jobs that can
46
+ run in a fresh worktree on this host. Which steps qualify is yours to
47
+ decide, by reading CI and the repository. Members run it with their own
48
+ arguments appended (`sh -c '<test> "$@"'`), so it is one invocation: an
49
+ `&&` chain hands those arguments to its last command alone. Prefer a form
47
50
  where an appended test file or test name lands as an argument to the test
48
51
  runner. The cache reader refuses a command whose first word does not
49
52
  resolve from the repository root (a builtin, a `PATH` entry, or an
@@ -53,6 +56,13 @@ from the repository root by `sh -c`:
53
56
  resolves under its own directory option, e.g.
54
57
  `git -C web ls-files -- '*.test.ts'`: the reader accepts a quoted pattern
55
58
  unchecked.
59
+ - **Uncovered steps** — what the Test entrypoint leaves out, passed as
60
+ `--uncovered`: one JSON array of `{"step": "<the CI step>", "reason": "<why
61
+ it is left out>"}`, `'[]'` when it leaves out nothing. When no single
62
+ invocation covers every qualifying step, take the widest one and list every
63
+ qualifying step it leaves out; that is never a reason to report
64
+ `RECIPE NOT PROVEN`. A test step that cannot run here — it needs a service
65
+ this host lacks, secrets, or another OS — is listed too, with that reason.
56
66
 
57
67
  ## 2. Prove it — this procedure exactly
58
68
 
@@ -61,7 +71,7 @@ script creates and removes; `<repo>`'s own tree is never touched.
61
71
 
62
72
  **First run, to see the output** — no proof flags:
63
73
 
64
- ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>'
74
+ ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' --uncovered '<uncovered>'
65
75
 
66
76
  It exits 1 with `NOT PROVEN — no proof given` once both commands ran, and
67
77
  names the Test entrypoint's output file (`test output: <path>`). Any other
@@ -74,7 +84,7 @@ and revise.
74
84
  text off that summary line, as printed, into `--count-line`, and the number
75
85
  of tests that ran into `--test-count`:
76
86
 
77
- ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' \
87
+ ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' --uncovered '<uncovered>' \
78
88
  --count-line '<literal text of the count line>' --test-count <n>
79
89
 
80
90
  The number is the tests that ran — never the skipped, the failed, or a
@@ -92,14 +102,14 @@ and revise.
92
102
  must change a tracked file; the script runs it in the throwaway worktree,
93
103
  requires the suite to go red, and discards the worktree after:
94
104
 
95
- ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' \
105
+ ~/.fleet/bin/fleet-run recipe-prove.mjs <repo> --install '<install>' --test '<test>' --uncovered '<uncovered>' \
96
106
  --mutate '<shell command that breaks one test>' --mutation '<one line: what it breaks>'
97
107
 
98
108
  Exit 0 is the proof holding: the script wrote the Recipe cache and printed
99
109
  `PROVEN` and the cache's contents. Exit 1 is `NOT PROVEN`, with the reason;
100
110
  no cache was written. Exit 2 means no verdict on the proof: it could not be
101
- attempted (no `origin/main`, not a repository) or the cache could not be
102
- written — report it, do not work around it.
111
+ attempted (no `origin/main`, not a repository, a malformed `--uncovered`) or
112
+ the cache could not be written — report it, do not work around it.
103
113
 
104
114
  The script bounds each command it runs itself — the Install step, both Test
105
115
  runs and the mutation — and refuses one still running at that bound as
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@feigi/fleet-ctl",
3
- "version": "3.70.0",
3
+ "version": "3.71.0",
4
4
  "description": "Agent fleet: run-team controller, merge bot, PR reviewer, and the ticket pipeline they share",
5
5
  "license": "Apache-2.0",
6
6
  "repository": {
@@ -1,6 +1,7 @@
1
1
  #!/bin/sh
2
- # Read one command of a repository's proven Recipe out of its Recipe cache:
3
- # `install` (the Install step) or `test` (the Test entrypoint).
2
+ # Read one field of a repository's proven Recipe out of its Recipe cache:
3
+ # `install` (the Install step), `test` (the Test entrypoint) or `uncovered`
4
+ # (the CI test steps the Test entrypoint leaves out).
4
5
  #
5
6
  # The fleet keeps no table of technologies. A Recipe is DERIVED by an agent
6
7
  # reasoning over the repository and PROVEN before it is written — the Install
@@ -27,11 +28,16 @@
27
28
  # installClean true the Install step left `git status --porcelain` empty
28
29
  # testCount integer >0, the test count the proving run reported, and/or
29
30
  # mutation string the deliberate failing mutation that turned it red
31
+ # uncovered array every test step of the merge-gating CI jobs the Test
32
+ # entrypoint does not run, `[]` when it runs them all:
33
+ # each entry `{ "step": <string>, "reason": <string> }`,
34
+ # both non-empty
30
35
  #
31
36
  # At least one of testCount/mutation is the proof; a cache carrying neither,
32
- # or `installClean` anything but `true`, is an UNPROVEN Recipe and refused like
33
- # an absent one. Written only by recipe-prove.mjs, the derivation step's proof,
34
- # and only once that proof holds; this script never writes it.
37
+ # `installClean` anything but `true`, or no well-formed `uncovered`, is an
38
+ # UNPROVEN Recipe and refused like an absent one. Written only by
39
+ # recipe-prove.mjs, the derivation step's proof, and only once that proof
40
+ # holds; this script never writes it.
35
41
  #
36
42
  # INVALID on failure to RUN, never on failing tests: the command this script
37
43
  # is asked for must name something that resolves from <repo> (a builtin, a
@@ -44,8 +50,9 @@
44
50
  # tree that probe reads instead of <repo>'s own index, for a caller that runs
45
51
  # the command in a worktree cut from a ref.
46
52
  #
47
- # Exit status: 0 the command is printed; 1 a refusal about the cache, the
48
- # repository or the arguments; 3 a tool this script needs to read the cache
53
+ # Exit status: 0 the field is printed, `uncovered` as one line of compact
54
+ # JSON; 1 a refusal about the cache, the repository or the arguments; 3 a
55
+ # tool this script needs to read the cache
49
56
  # (git, node, mktemp, cat) could not be started, mktemp could not create its
50
57
  # temp file, git did not run to an answer, node died mid-read, or node exited
51
58
  # non-zero with no reason and a status other than 2, its only verdict on the
@@ -116,7 +123,7 @@ derive="run the Recipe derivation step (run-team phase 0, before the first claim
116
123
  # the checkout still lists the old layout until someone updates it. Without
117
124
  # `--at` the probe reads <repo>'s own index, as for a caller that runs the
118
125
  # command in <repo> itself. Three arguments is neither shape: a refusal.
119
- usage="usage: derive-testcmd.sh <repo> <install|test> [--at <rev>]"
126
+ usage="usage: derive-testcmd.sh <repo> <install|test|uncovered> [--at <rev>]"
120
127
  rev=
121
128
  case $# in
122
129
  2) ;;
@@ -139,8 +146,8 @@ gitout=$(git -C "$repo" rev-parse --git-dir 2>&1) || {
139
146
  }
140
147
 
141
148
  case $field in
142
- install|test) ;;
143
- *) die "unknown Recipe field '$field' — expected install or test" ;;
149
+ install|test|uncovered) ;;
150
+ *) die "unknown Recipe field '$field' — expected install, test or uncovered" ;;
144
151
  esac
145
152
 
146
153
  # A <rev> is only ever consulted by the probe, which only the Test entrypoint
@@ -223,7 +230,13 @@ if (r.installClean !== true) bad("`installClean` is not true — the Install ste
223
230
  const counted = Number.isInteger(r.testCount) && r.testCount > 0;
224
231
  const mutated = typeof r.mutation === "string" && r.mutation.trim() !== "";
225
232
  if (!counted && !mutated) bad("it carries no proof of real tests — neither a positive `testCount` nor a `mutation`");
226
- const value = r[field];
233
+ if (!Array.isArray(r.uncovered)) bad("`uncovered` is not a list of { step, reason } entries — the CI test steps the Test entrypoint leaves out, `[]` for none");
234
+ const filled = (s) => typeof s === "string" && s.trim() !== "";
235
+ r.uncovered.forEach((u, i) => {
236
+ if (u === null || typeof u !== "object" || Array.isArray(u) || !filled(u.step) || !filled(u.reason))
237
+ bad(`\`uncovered\` entry ${i + 1} is not an object with a non-empty \`step\` and \`reason\``);
238
+ });
239
+ const value = field === "uncovered" ? JSON.stringify(r.uncovered) : r[field];
227
240
  fs.writeFileSync(lenFile, String(Buffer.byteLength(open + value + close)));
228
241
  process.stdout.write(open + value + close);
229
242
  ' "$cache" "$field" "$open" "$close" "$lenf" 2>"$errf") || rc=$?
@@ -292,6 +305,14 @@ esac
292
305
  value=${framed#"$open"}
293
306
  value=${value%"$close"}
294
307
 
308
+ # `uncovered` is data, not a command: nothing runs it and nothing appends to
309
+ # it, so none of the command checks below apply. JSON.stringify above made it
310
+ # one line.
311
+ if [ "$field" = uncovered ]; then
312
+ printf '%s\n' "$value"
313
+ exit 0
314
+ fi
315
+
295
316
  # claim-ticket.sh appends the runner's own arguments after this string
296
317
  # textually (`exec sh -c '<cmd> "$@"' agent-test "$@"`), so a command ending in
297
318
  # `;`, `&`, or a newline lets a real shell read the caller's "$@" as an
@@ -5,10 +5,17 @@
5
5
  // reads: the deriving agent chooses both commands by reading the repository,
6
6
  // and this script is what turns that choice into a cache, or refuses to.
7
7
  //
8
- // recipe-prove.mjs <repo> --install <cmd> --test <cmd>
8
+ // recipe-prove.mjs <repo> --install <cmd> --test <cmd> --uncovered <json>
9
9
  // [--count-line <literal> --test-count <n>]
10
10
  // [--mutate <cmd> --mutation <description>]
11
11
  //
12
+ // `--uncovered` is one JSON array, `[]` when the Test entrypoint runs every
13
+ // test step of the merge-gating CI jobs, and otherwise one
14
+ // `{ "step": <string>, "reason": <string> }` per step it leaves out, both
15
+ // non-empty. It is not proven — nothing here can run CI — but required, so a
16
+ // cache never stays silent about a step the Test entrypoint skips; it is
17
+ // written as given, as the cache's `uncovered`.
18
+ //
12
19
  // Every check runs in a throwaway worktree of `origin/main`, detached and
13
20
  // removed afterwards, so neither the caller's checkout nor its uncommitted
14
21
  // state takes part — `derivedAt` is the commit the proof ran against:
@@ -48,7 +55,8 @@
48
55
  // On proof, the cache is written atomically to <workspace>/.fleet/recipe.json
49
56
  // — <workspace> being the directory holding the repository's common git dir,
50
57
  // the place derive-testcmd.sh reads it from — and then read back through
51
- // derive-testcmd.sh for both fields, the Test entrypoint against `derivedAt`'s
58
+ // derive-testcmd.sh for every field, `uncovered` byte for byte as the compact
59
+ // JSON of the list given, the Test entrypoint against `derivedAt`'s
52
60
  // tree (`--at`): the tree the proof ran in, and the one a claim probes, so a
53
61
  // main checkout still on an older layout cannot refuse a Recipe that holds
54
62
  // where it runs. A cache the reader refuses is rolled
@@ -84,7 +92,7 @@ import { isDigits } from "./arg.mjs";
84
92
 
85
93
  const NAME = "recipe-prove";
86
94
  const USAGE =
87
- "usage: recipe-prove.mjs <repo> --install <cmd> --test <cmd> [--count-line <literal> --test-count <n>] [--mutate <cmd> --mutation <description>]";
95
+ "usage: recipe-prove.mjs <repo> --install <cmd> --test <cmd> --uncovered <json> [--count-line <literal> --test-count <n>] [--mutate <cmd> --mutation <description>]";
88
96
  const READER = join(dirname(fileURLToPath(import.meta.url)), "derive-testcmd.sh");
89
97
 
90
98
  // Scrubbed once, for every child: an ambient GIT_DIR or GIT_WORK_TREE — set
@@ -132,7 +140,7 @@ const READER_COULD_NOT_RUN = 3;
132
140
  function parseArgs(argv) {
133
141
  const [repo, ...rest] = argv;
134
142
  if (!repo || repo.startsWith("--")) throw cannot(USAGE);
135
- const flags = new Set(["--install", "--test", "--count-line", "--test-count", "--mutate", "--mutation"]);
143
+ const flags = new Set(["--install", "--test", "--uncovered", "--count-line", "--test-count", "--mutate", "--mutation"]);
136
144
  const a = {};
137
145
  for (let i = 0; i < rest.length; i += 2) {
138
146
  const [flag, value] = [rest[i], rest[i + 1]];
@@ -145,6 +153,8 @@ function parseArgs(argv) {
145
153
  for (const f of ["--install", "--test"]) {
146
154
  if (!a[f] || !a[f].trim()) throw cannot(`${f} needs a non-empty command — ${USAGE}`);
147
155
  }
156
+ if (!("--uncovered" in a)) throw cannot(`--uncovered is required: the CI test steps the Test entrypoint leaves out, as one JSON array, \`[]\` for none — ${USAGE}`);
157
+ const uncovered = parseUncovered(a["--uncovered"]);
148
158
  if (("--count-line" in a) !== ("--test-count" in a)) throw cannot("--count-line and --test-count go together");
149
159
  if ("--test-count" in a && !isDigits(a["--test-count"])) {
150
160
  throw cannot(`--test-count must be a whole number, got '${a["--test-count"]}'`);
@@ -163,9 +173,30 @@ function parseArgs(argv) {
163
173
  testCount: a["--test-count"],
164
174
  mutate: a["--mutate"],
165
175
  mutation: a["--mutation"],
176
+ uncovered,
166
177
  };
167
178
  }
168
179
 
180
+ // The same rule derive-testcmd.sh holds the cache to, applied before any
181
+ // command runs, so a malformed list is a usage error rather than a proof that
182
+ // held and a cache the reader then refuses.
183
+ function parseUncovered(json) {
184
+ let list;
185
+ try {
186
+ list = JSON.parse(json);
187
+ } catch (e) {
188
+ throw cannot(`--uncovered is not JSON: ${e.message}`);
189
+ }
190
+ if (!Array.isArray(list)) throw cannot("--uncovered is not a JSON array of { step, reason } entries, `[]` for none");
191
+ const filled = (s) => typeof s === "string" && s.trim() !== "";
192
+ list.forEach((u, i) => {
193
+ if (u === null || typeof u !== "object" || Array.isArray(u) || !filled(u.step) || !filled(u.reason)) {
194
+ throw cannot(`--uncovered entry ${i + 1} is not an object with a non-empty \`step\` and \`reason\``);
195
+ }
196
+ });
197
+ return list;
198
+ }
199
+
169
200
  // A git that cannot be started is no verdict, as for `sh` below: read as a
170
201
  // failed git command, it would surface as whatever that command's failure
171
202
  // means to its caller — a missing repository, an unfetched origin/main, a
@@ -343,9 +374,11 @@ function writeCache(cache, recipe, repo) {
343
374
  rmSync(tmp, { force: true });
344
375
  throw e;
345
376
  }
346
- for (const field of ["install", "test"]) {
377
+ for (const field of ["install", "test", "uncovered"]) {
347
378
  const r = spawnSync("sh", [READER, repo, field, ...(field === "test" ? ["--at", recipe.derivedAt] : [])], { env: ENV, encoding: "utf8" });
348
- if (r.status === 0 && r.stdout.replace(/\n$/, "") === recipe[field]) continue;
379
+ const expected = field === "uncovered" ? JSON.stringify(recipe.uncovered) : recipe[field];
380
+ const got = r.status === 0 ? r.stdout.replace(/\n$/, "") : null;
381
+ if (got === expected) continue;
349
382
  if (prior === null) unlinkSync(cache);
350
383
  else writeFileSync(cache, prior);
351
384
  if (r.error && !r.pid) throw cannot(`could not start sh to read the Recipe cache back: ${r.error.message}`);
@@ -366,6 +399,9 @@ function writeCache(cache, recipe, repo) {
366
399
  const stderrNote = probe.err ? `: ${probe.err}` : " and wrote nothing to stderr";
367
400
  throw noVerdict(`git started but does not run: \`git --version\` ${exit}${stderrNote}`);
368
401
  }
402
+ if (got !== null) {
403
+ throw notProven(`the Recipe cache reader reads \`${field}\` back as ${JSON.stringify(got)}, not the ${JSON.stringify(expected)} written`);
404
+ }
369
405
  const said = r.stderr.trim() || r.stdout.trim();
370
406
  throw notProven(`the Recipe cache reader refuses what was proven: ${said || `it wrote nothing and exited with status ${r.status}`}`);
371
407
  }
@@ -437,7 +473,7 @@ function main(argv) {
437
473
  removeWorktree(wt, o.repo);
438
474
  }
439
475
 
440
- recipe = { install: o.install, test: o.test, derivedAt: sha.out, installClean: true, ...proof };
476
+ recipe = { install: o.install, test: o.test, derivedAt: sha.out, installClean: true, ...proof, uncovered: o.uncovered };
441
477
  writeCache(cache, recipe, o.repo);
442
478
  } catch (e) {
443
479
  if (!(e instanceof Refusal && e.message.includes(logs))) removeLogs(logs);
@@ -122,6 +122,12 @@ export const SNAPSHOT_SCHEMA = {
122
122
  prHead: { type: "string" },
123
123
  testCmd: { type: "string" },
124
124
  testCmdError: { type: "string" },
125
+ // The Recipe's `uncovered` as derive-testcmd.sh printed it: the CI test
126
+ // steps the Test entrypoint leaves out, one line of JSON, `[]` for none.
127
+ // `uncoveredError` carries the reader's refusal instead when it could not
128
+ // be read, so an unread list never reads as an empty one.
129
+ uncovered: { type: "string" },
130
+ uncoveredError: { type: "string" },
125
131
  // The Recipe's Install step as the snapshot agent read it, and its exit
126
132
  // status in the snapshot. Optional, so an absent `installExit` is read
127
133
  // as an install that did not succeed, never as one that did:
@@ -499,9 +505,36 @@ export function environmentNote(snap) {
499
505
  note += `\nAfter the Install step the snapshot's status could not be read (${snap.statusError.trim()}), so what it left untracked is not known.`;
500
506
  if (shown.length)
501
507
  note += `\nAfter the Install step the snapshot reads untracked under the repository's own ignore rules: ${shown.join(", ")}${more > 0 ? `, and ${more} more (the whole status is in ${snap.runRoot ?? "the run root"}/status.txt)` : ""}.`;
508
+ note += uncoveredNote(snap);
502
509
  return note;
503
510
  }
504
511
 
512
+ // The CI test steps the Recipe's Test entrypoint leaves out, as the snapshot
513
+ // agent read them through the cache's one reader, ride on the end as one line
514
+ // naming each step and why: a green run here says nothing about them, and no
515
+ // specialist runs them. An empty list adds nothing. A list the agent could
516
+ // not read, or reported in another shape, is said so — never left to read as
517
+ // an empty one.
518
+ const oneLine = (s) => s.replace(/\s+/g, " ").trim();
519
+ function uncoveredNote(snap) {
520
+ let list = null;
521
+ if (snap && typeof snap.uncovered === "string") {
522
+ try {
523
+ list = JSON.parse(snap.uncovered);
524
+ } catch {
525
+ list = null;
526
+ }
527
+ }
528
+ if (Array.isArray(list) && list.every((u) => u && typeof u.step === "string" && typeof u.reason === "string")) {
529
+ if (!list.length) return "";
530
+ return `\nThe Recipe's Test entrypoint does not run these merge-gating CI test steps, and nothing in this review ran them: ${list.map((u) => `${oneLine(u.step)} (${oneLine(u.reason)})`).join("; ")}.`;
531
+ }
532
+ const why =
533
+ (snap && typeof snap.uncoveredError === "string" && snap.uncoveredError.trim() && oneLine(snap.uncoveredError)) ||
534
+ (snap && typeof snap.uncovered === "string" ? `it reported ${JSON.stringify(oneLine(snap.uncovered))}, not a list of steps` : "the snapshot agent reported nothing for it");
535
+ return `\nWhich merge-gating CI test steps the Recipe's Test entrypoint leaves out could not be read (${why}), so a passing run here is not known to cover them.`;
536
+ }
537
+
505
538
  function repoNote(snap) {
506
539
  if (snap && snap.repoVerified === true)
507
540
  return `Test environment: the snapshot is a git repository whose single commit holds
@@ -1085,6 +1118,15 @@ guess a command yourself: a refusal about the cache names the step that
1085
1118
  derives it; an exit-3 refusal names an environment fault, and the caller fixes
1086
1119
  that instead of deriving.
1087
1120
 
1121
+ Then read, out of the same cache, the merge-gating CI test steps that Test
1122
+ entrypoint leaves out:
1123
+
1124
+ ~/.fleet/bin/fleet-run derive-testcmd.sh ${worktree} uncovered
1125
+
1126
+ Report \`uncovered\` = its stdout, one line of JSON copied verbatim, ONLY if it
1127
+ exited 0. If it exited non-zero, report \`uncoveredError\` = its stderr and omit
1128
+ \`uncovered\`. Run none of the steps it lists.
1129
+
1088
1130
  Then size the diff:
1089
1131
 
1090
1132
  ~/.fleet/bin/fleet-run diff-stats.mjs --pr ${pr}
@@ -1100,7 +1142,8 @@ specialists into another run's tree. Report the HEAD sha, and — in
1100
1142
  STDOUT, copied verbatim. Only runRoot, path, head, pathVerified and repoVerified
1101
1143
  are ever required — diffStats, diffPath, diffLines, refHead and prHead are each
1102
1144
  omitted independently when their command failed, repoError only accompanies
1103
- a false repoVerified, and installError only replaces installCmd and installExit.`,
1145
+ a false repoVerified, installError only replaces installCmd and installExit,
1146
+ and uncoveredError only replaces uncovered.`,
1104
1147
  { label: `snapshot${forPr}`, phase: "Snapshot", agentType: SNAPSHOT_AGENT_TYPE, schema: SNAPSHOT_SCHEMA },
1105
1148
  );
1106
1149