@feigi/fleet-ctl 3.47.5 → 3.47.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -28,8 +28,15 @@ and confirm the named test stays green. Vary the syntactic form of the
|
|
|
28
28
|
mutation — a guard written to catch `// whole-line` comments may let `code; //
|
|
29
29
|
trailing` through, and a test that only exercises one form is not proven to
|
|
30
30
|
discriminate the class. `<testCmd>` is the `command` your dispatch prompt's
|
|
31
|
-
Tests paragraph names, run only in your own copy of the snapshot.
|
|
32
|
-
|
|
31
|
+
Tests paragraph names, run only in your own copy of the snapshot. Make that
|
|
32
|
+
copy from the snapshot's tracked files only —
|
|
33
|
+
`git -C <snapshot> archive HEAD | tar -x -C <copy>`, or a `git clone` of the
|
|
34
|
+
snapshot repository — then `git init`, `git add -A -f` and commit in it, then
|
|
35
|
+
run the Install step your dispatch prompt names from the copy's root before
|
|
36
|
+
you measure anything there. Never `cp -R` the snapshot, and never symlink or
|
|
37
|
+
clone its ignored paths: a copy sharing the snapshot's installed output can
|
|
38
|
+
run the snapshot's source, and your mutation never runs. A kill is scored by
|
|
39
|
+
these rules and nothing else:
|
|
33
40
|
|
|
34
41
|
**Valid red:** a run that completed — the command returned on its own, with
|
|
35
42
|
no signal and no timeout, deadline or kill having fired — in which the test
|
|
@@ -82,8 +82,8 @@ It cannot ask the user anything and cannot wait on CI — rebasing, watching che
|
|
|
82
82
|
- **Say read-only.** No `fleet-review-*` definition's frontmatter restricts tool access — a `.agent.md` prompt saying "report only" is instruction, not enforcement — so an agent hand-dispatched here, named or general-purpose, has write tools it must not use: one committed and pushed during a report-only dispatch. No commits, no pushes, no worktree edits.
|
|
83
83
|
- **Cut TWO trees, not one — a shared snapshot is not enough.** Mutation probing is a *write*, so a mutating specialist and read-only specialists cannot share a copy: the readers then observe the mutant exactly as if it were real code. Give them `snap-ro`, pristine and **never written**, and one private copy **per mutating specialist**. Keep every one out of the worktree. Observed with a single shared snapshot: three specialists read two *different* in-flight mutations, one reporting the PR's own bug as still present; elsewhere three watched their file go clean → `M` mid-analysis; twice a reviewer came one step from filing a false finding. Reverting a probe does not help — it leaves a window where every concurrent reader sees a lie, and serializing does not close it because readers are concurrent with the *mutator*.
|
|
84
84
|
- **Do not trust the obvious contamination check.** `diff -rq` and `md5` against the mutating tree both returned *clean* — because the probe had been reverted between the two reads. A clean diff against a live tree is not evidence in either direction; only `snap-ro` and the object store settle it.
|
|
85
|
-
- **`git archive` carries tracked files only.** No installed dependencies, no gitignored runner or config. Provision in the same step — run the Install step in the copy (read it off the worktree or main checkout, `derive-testcmd.sh <worktree or main checkout> install`; pointed at the copy it refuses, since a copy holds no Recipe cache)
|
|
86
|
-
- **Initialize the copy before anyone measures in it — a bare `git archive` tree cannot validly run every suite.** It is not a git repository, so anything that asks git about the checkout breaks: `derive_workspace_id` shells out to `git rev-parse`, falls back to the directory basename and the hook tests fail on the *name* — naming the dir after the repo makes those pass **by coincidence, not correctness** — while every sweep that asks git what ships declines instead of running, which is the silent half (measured on this repo at `ffa9026`: 2401 tests, 2382 pass, 19 skipped in an extraction against 2401/2401/0 in a worktree, the totals identical). `git init && git add -A -f && git commit` in the extracted copy is what makes a count mean what it looks like; the review's snapshot agent does exactly that
|
|
85
|
+
- **`git archive` carries tracked files only.** No installed dependencies, no gitignored runner or config. Provision in the same step — run the Install step in the copy (read it off the worktree or main checkout, `derive-testcmd.sh <worktree or main checkout> install`; pointed at the copy it refuses, since a copy holds no Recipe cache), from the copy's root, before anyone measures in it — or specialists silently have no runnable suite and reason from source instead of measuring. That failure is invisible: you get confident prose where you asked for a measurement. Never `cp -R` an installed tree, and never symlink or clone its ignored paths into a copy: an install can record absolute paths back into the tree it ran in, so the copy runs that tree's source and a mutation made in the copy never runs.
|
|
86
|
+
- **Initialize the copy before anyone measures in it — a bare `git archive` tree cannot validly run every suite.** It is not a git repository, so anything that asks git about the checkout breaks: `derive_workspace_id` shells out to `git rev-parse`, falls back to the directory basename and the hook tests fail on the *name* — naming the dir after the repo makes those pass **by coincidence, not correctness** — while every sweep that asks git what ships declines instead of running, which is the silent half (measured on this repo at `ffa9026`: 2401 tests, 2382 pass, 19 skipped in an extraction against 2401/2401/0 in a worktree, the totals identical). `git init && git add -A -f && git commit` in the extracted copy is what makes a count mean what it looks like; the review's snapshot agent does exactly that, verifies it by comparing the new commit's tree hash against the reviewed commit's, and then runs the Install step in it, and a hand-cut tree needs the same before you read a count off it. **Every agent you send to work in its own copy of the snapshot makes that copy the same way:** copy the tracked files only (`git -C <snapshot> archive HEAD | tar -x -C <copy>`, or a `git clone` of the snapshot repository), `git init`, `git add -A -f` and commit, then run the Install step in the copy before measuring anything in it — never `cp -R` the snapshot. Suites that need the real worktree for some *other* reason — the stack below — still do.
|
|
87
87
|
- **Give specialists a stack-free test command, not the worktree's runner, and their own scratch dir** — `<scratch>/pr<N>/fleet-review-<key>/`, resolved to an absolute path that you write into that specialist's prompt; a specialist never derives its own. Specialists inherit no environment, and one falling back to the default config runs a `globalSetup` that brings the shared compose stack up and tears it down, recreating the DB mid-run for every sibling. **The snapshot does not cover this** — the compose project name comes from the environment, not the working directory, so three agents on three copies still collide. But `./agent-test` does not fix it either: it exports **one** `TEST_COMPOSE_PROJECT`/port triple per *worktree* (`ab-<issue>`), so N concurrent specialists sharing it tear down each other's postgres mid-run. Observed: a probe returned `No test files found / No such container` and would have read as a test failure. So tell specialists the repo's test command directly — the Test entrypoint, read off the Recipe cache the way step 3 reads it (`derive-testcmd.sh <worktree or main checkout> test` prints it; the snapshot carries no `.fleet/` cache, so pointing it at the snapshot refuses) — when it brings up no shared stack, and the repo's own stack-free config when it does; never name a config the repo does not have, which sends every specialist into a missing-file error that reads as a broken tree rather than a bad instruction. Keep `./agent-test` for your own verification. **Hand that command with its reading rule: `tests 0` is a failed run, not a pass.** A command is not self-checking: a path glob or filter that matches nothing can exit 0 having run nothing. Measured with `node --test` over a glob matching no file: bash passes the pattern through literally, node globs it itself and finds no files, and you get `ℹ tests 0` / exit 0 — and `# tests 0` to a pipe on older node, so a rule that greps only for `ℹ` finds nothing in a CI log and cannot tell a zero-test run from a full one. Do not count on the shell to catch it: zsh errors (`no matches found`, exit 1) and bash does not, so the guard has to be the reading rule. **And zero is not the only no-work count.** **0 passes with no failures** is everything skipped, and a count **materially below the full suite** is a partial copy of the tree — which is the *documented* behavior, since the prompt tells specialists to run their probe work inside their own copy of the snapshot and nothing anchors an expected count for them. `review-core.mjs` classifies the first itself, in `unrunReason`; the second it structurally cannot, because that function is kept pure and so never learns how many tests the tree has. Handing the rule over with the command is the only guard the partial-copy case has. Carry the rule, not the command: wherever a repo has a stack, N concurrent specialists sharing one runner collide, and the escape is whatever that repo's stack-free config is.
|
|
88
88
|
- **Completion is not delivery — but you can reach a specialist, so never wait idle.** A specialist you dispatched reports when it settles; if a report is missing, `hub send` it by name and ask for the report. Judge delivery by whether a report came out. **Do not settle a dimension you hold no report for**: ask the specialist, then the controller by name; only when neither answers, rule and name that dimension **unrun** — a killed specialist never reports, so waiting on one is unbounded. Never ship "dispatched, never delivered" as settled; a ruling citing a report you do not hold gets verified from source, never applied on trust.
|
|
89
89
|
- **Apply only once the fan-out is settled — every report either in hand or accounted for.** Collection is not guaranteed; a report that never arrived is an open dimension, not a smaller set. And editing while they read is the same defect as probing — one specialist reviewed uncommitted code that was never in the PR diff.
|
package/package.json
CHANGED
package/scripts/review-core.mjs
CHANGED
|
@@ -3,8 +3,10 @@
|
|
|
3
3
|
// This file holds every host-independent declaration PR review needs: the
|
|
4
4
|
// JSON schemas, `usableDiff`, `readRules`, `resolveTestCmd`,
|
|
5
5
|
// `decodeArgs`, `selectDimensions`, `resolveDimensions`, `snapshotMissing`,
|
|
6
|
-
// `unrunReason`/`unrunEntries`/`unrunCrashed`,
|
|
7
|
-
// `
|
|
6
|
+
// `installReason`/`installFinding`, `unrunReason`/`unrunEntries`/`unrunCrashed`,
|
|
7
|
+
// `sharedRunNote`, `copyRule`,
|
|
8
|
+
// `verdictFor`, `resumeFor`, and `runReview(host, args)` — the orchestration
|
|
9
|
+
// function itself.
|
|
8
10
|
//
|
|
9
11
|
// review-eval.mjs imports this file directly, a RELATIVE specifier: both
|
|
10
12
|
// ship together under the same Install root, so the Resolver is only needed
|
|
@@ -120,6 +122,23 @@ export const SNAPSHOT_SCHEMA = {
|
|
|
120
122
|
prHead: { type: "string" },
|
|
121
123
|
testCmd: { type: "string" },
|
|
122
124
|
testCmdError: { type: "string" },
|
|
125
|
+
// The Recipe's Install step as the snapshot agent read it, and its exit
|
|
126
|
+
// status in the snapshot. Optional, so an absent `installExit` is read
|
|
127
|
+
// as an install that did not succeed, never as one that did:
|
|
128
|
+
// `installError` carries derive-testcmd.sh's refusal when the step could
|
|
129
|
+
// not be read at all.
|
|
130
|
+
installCmd: { type: "string" },
|
|
131
|
+
installExit: { type: "integer" },
|
|
132
|
+
installError: { type: "string" },
|
|
133
|
+
// What `git status --porcelain --untracked-files=all` printed in the
|
|
134
|
+
// snapshot after the Install step, one entry per line, the first twenty
|
|
135
|
+
// at most; `untrackedCount` is how many it printed in all. The
|
|
136
|
+
// repository's own ignore rules decide it, so it is reported, never
|
|
137
|
+
// excluded. `statusError` says the read itself failed, which is not an
|
|
138
|
+
// empty answer.
|
|
139
|
+
untracked: { type: "array", items: { type: "string" } },
|
|
140
|
+
untrackedCount: { type: "integer" },
|
|
141
|
+
statusError: { type: "string" },
|
|
123
142
|
},
|
|
124
143
|
};
|
|
125
144
|
|
|
@@ -138,7 +157,7 @@ export const TEST_RUN_SCHEMA = {
|
|
|
138
157
|
tests: { type: "integer" },
|
|
139
158
|
pass: { type: "integer" },
|
|
140
159
|
fail: { type: "integer" },
|
|
141
|
-
expectedFail: { type: "integer", description: "Tests the runner reports as failing by design (vitest
|
|
160
|
+
expectedFail: { type: "integer", description: "Tests the runner reports as failing by design (vitest `expected fail`, pytest `xfailed`), never folded into `fail`." },
|
|
142
161
|
cancelled: { type: "integer" },
|
|
143
162
|
skipped: { type: "integer" },
|
|
144
163
|
error: { type: "string", description: "Only when no summary was printed: what happened instead." },
|
|
@@ -458,7 +477,32 @@ export function snapshotMissing(snap, runRootPrefix) {
|
|
|
458
477
|
// commit, no ancestry, no remotes, so `git log`/`git diff` now return
|
|
459
478
|
// plausible output about the snapshot in place of the `fatal:` that used to
|
|
460
479
|
// stop that question being asked.
|
|
480
|
+
//
|
|
481
|
+
// What the snapshot reads untracked after the Install step rides on the end
|
|
482
|
+
// as one line: the repository's own ignore rules left it, so it is a fact
|
|
483
|
+
// about the repository, and the review names it rather than hiding it. The
|
|
484
|
+
// line is interpolated into every specialist and refuter prompt and the
|
|
485
|
+
// payload, so it carries the first `LISTED_UNTRACKED` entries and a count of
|
|
486
|
+
// the rest — a PR that drops its dependency directory from `.gitignore` would
|
|
487
|
+
// otherwise put one entry per installed file in each of them; the whole
|
|
488
|
+
// status is in the run root's `status.txt`. A status the block could not read
|
|
489
|
+
// is said so, never left to read as an empty one.
|
|
490
|
+
const LISTED_UNTRACKED = 20;
|
|
491
|
+
|
|
461
492
|
export function environmentNote(snap) {
|
|
493
|
+
const untracked = snap && Array.isArray(snap.untracked) ? snap.untracked.filter((l) => typeof l === "string" && l.trim()) : [];
|
|
494
|
+
const total = Math.max(untracked.length, snap && Number.isInteger(snap.untrackedCount) ? snap.untrackedCount : 0);
|
|
495
|
+
const shown = untracked.slice(0, LISTED_UNTRACKED);
|
|
496
|
+
const more = total - shown.length;
|
|
497
|
+
let note = repoNote(snap);
|
|
498
|
+
if (snap && typeof snap.statusError === "string" && snap.statusError.trim())
|
|
499
|
+
note += `\nAfter the Install step the snapshot's status could not be read (${snap.statusError.trim()}), so what it left untracked is not known.`;
|
|
500
|
+
if (shown.length)
|
|
501
|
+
note += `\nAfter the Install step the snapshot reads untracked under the repository's own ignore rules: ${shown.join(", ")}${more > 0 ? `, and ${more} more (the whole status is in ${snap.runRoot ?? "the run root"}/status.txt)` : ""}.`;
|
|
502
|
+
return note;
|
|
503
|
+
}
|
|
504
|
+
|
|
505
|
+
function repoNote(snap) {
|
|
462
506
|
if (snap && snap.repoVerified === true)
|
|
463
507
|
return `Test environment: the snapshot is a git repository whose single commit holds
|
|
464
508
|
the reviewed tree — its tree hash was compared against the commit under review
|
|
@@ -482,10 +526,52 @@ validation of the tree: its counts are snapshot-measured, and a failure in it
|
|
|
482
526
|
cannot be told apart from a regression.`;
|
|
483
527
|
}
|
|
484
528
|
|
|
529
|
+
// The Install step's verdict on the snapshot: null when it ran there and
|
|
530
|
+
// exited 0, otherwise why no test of this review can run in it. An absent
|
|
531
|
+
// `installExit` is never a success — the step was unreadable (`installError`
|
|
532
|
+
// says why) or the snapshot agent dropped the line — because a snapshot
|
|
533
|
+
// nothing was installed into measures its own missing dependencies, not the
|
|
534
|
+
// tree.
|
|
535
|
+
export function installReason(snap) {
|
|
536
|
+
if (snap && snap.installExit === 0) return null;
|
|
537
|
+
if (snap && Number.isInteger(snap.installExit))
|
|
538
|
+
return `the Install step${snap.installCmd ? ` \`${snap.installCmd}\`` : ""} exited ${snap.installExit} in the snapshot (log ${snap.runRoot}/install.log) — nothing it installs is there, so the shared test run was skipped`;
|
|
539
|
+
const why = (snap && typeof snap.installError === "string" && snap.installError.trim()) || "the snapshot agent reported no exit status for it";
|
|
540
|
+
return `the Install step did not run in the snapshot — ${why} — so the shared test run was skipped`;
|
|
541
|
+
}
|
|
542
|
+
|
|
543
|
+
// The one finding an Install step that exits non-zero in the snapshot files
|
|
544
|
+
// on the PR, when the snapshot is the reviewed tree. It is the review's own
|
|
545
|
+
// measurement, an exit status, not a specialist's claim, so it carries no
|
|
546
|
+
// refuter vote — and it claims only what was measured: which tree the step
|
|
547
|
+
// ran on and how it exited. Why it exited is not known here, and the same
|
|
548
|
+
// status can come from the PR (a script it deleted exits 127) or from the host
|
|
549
|
+
// (a kill, a timeout), so the claim says the cause is not established rather
|
|
550
|
+
// than naming the PR's defect. The Recipe cache, proven on the base, stays as
|
|
551
|
+
// it is. A snapshot that is NOT the reviewed tree (`repoVerified` not true)
|
|
552
|
+
// files nothing: an exit there is a fact about a different tree, which
|
|
553
|
+
// `environmentNote` already says. An Install step that could not be READ files
|
|
554
|
+
// nothing either: that is the Recipe's state, not the PR's.
|
|
555
|
+
export function installFinding(snap) {
|
|
556
|
+
if (!snap || snap.repoVerified !== true || !Number.isInteger(snap.installExit) || snap.installExit === 0) return null;
|
|
557
|
+
const cmd = snap.installCmd ? `\`${snap.installCmd}\`` : "the Recipe's Install step";
|
|
558
|
+
return {
|
|
559
|
+
severity: "critical",
|
|
560
|
+
claim: `The Install step exited ${snap.installExit} on this PR's tree: ${cmd}, run from the root of a snapshot of ${snap.head}, so no test of this review ran. The review did not establish why it exited — the PR's tree or this host.`,
|
|
561
|
+
evidence: `The review ran ${cmd} once from the root of ${snap.path} and it exited ${snap.installExit}; its output is in ${snap.runRoot}/install.log.`,
|
|
562
|
+
dimension: "install",
|
|
563
|
+
verdict: "survived",
|
|
564
|
+
votes: [],
|
|
565
|
+
refutersDispatched: 0,
|
|
566
|
+
refutersInconclusive: 0,
|
|
567
|
+
};
|
|
568
|
+
}
|
|
569
|
+
|
|
485
570
|
// Every run-quality verdict below reads the review's ONE shared test
|
|
486
571
|
// run — `{command, logPath, exitCode?, tests?, pass?, fail?, expectedFail?,
|
|
487
572
|
// cancelled?, skipped?, error?}`, `command`/`logPath` the caller's own and the
|
|
488
|
-
// rest the test-run agent's report
|
|
573
|
+
// rest the test-run agent's report, or `{command, logPath, unprovisioned}`
|
|
574
|
+
// when `installReason` skipped the run — never a specialist's own run: no
|
|
489
575
|
// specialist runs the full suite any more, so a verdict on one would judge a
|
|
490
576
|
// copy of this.
|
|
491
577
|
// `run` null is a caller holding no run at all — runReview always hands one
|
|
@@ -498,6 +584,7 @@ cannot be told apart from a regression.`;
|
|
|
498
584
|
// TEST_RUN_EXIT line would otherwise read as green.
|
|
499
585
|
function testRunReason(run) {
|
|
500
586
|
if (!run) return "the review's shared test run returned nothing — no counts exist, so no dimension's tests ran";
|
|
587
|
+
if (run.unprovisioned) return run.unprovisioned;
|
|
501
588
|
// The wrapper's wait cap expired: it launched nothing, so there are no
|
|
502
589
|
// counts to judge and the exit status is the wrapper's, not the suite's. A
|
|
503
590
|
// run that printed counts and happened to exit 75 is the suite's own answer
|
|
@@ -567,13 +654,42 @@ function countsOf(run) {
|
|
|
567
654
|
.join(", ");
|
|
568
655
|
}
|
|
569
656
|
|
|
657
|
+
// How a specialist or refuter makes "your own copy of the snapshot", in
|
|
658
|
+
// every prompt that sends one to work there. Tracked files only, then the
|
|
659
|
+
// Install step in the copy: what an install writes can record absolute paths
|
|
660
|
+
// back into the tree it ran in, so a copy that shares the snapshot's
|
|
661
|
+
// installed output imports the snapshot's source, and a mutation made in the
|
|
662
|
+
// copy never runs.
|
|
663
|
+
export function copyRule(snap) {
|
|
664
|
+
const step = snap && snap.installCmd
|
|
665
|
+
? `the Install step, \`${snap.installCmd}\`,`
|
|
666
|
+
: "the Install step (this review read none, so a copy has nothing installed and a suite run in it does not measure the tree)";
|
|
667
|
+
return `Make your own copy of the snapshot from its tracked files only —
|
|
668
|
+
\`git -C ${snap && snap.path} archive HEAD | tar -x -C <copy>\`, or a \`git clone\`
|
|
669
|
+
of the snapshot repository — then make it a repository with \`git init\`,
|
|
670
|
+
\`git add -A -f\` and a commit (a clone already is one), then run
|
|
671
|
+
${step} from the copy's root before you measure anything in it.
|
|
672
|
+
Never \`cp -R\` the snapshot, and never symlink or clone its ignored paths: an
|
|
673
|
+
install can record absolute paths back into the tree it ran in, so a copy
|
|
674
|
+
sharing the snapshot's installed output runs the snapshot's source, and a
|
|
675
|
+
mutation in the copy never runs.`;
|
|
676
|
+
}
|
|
677
|
+
|
|
570
678
|
// The Tests paragraph of every specialist prompt: the shared run's
|
|
571
679
|
// command, counts, exit status and log path, in place of the instruction to
|
|
572
680
|
// run the full suite that each specialist used to follow — one review was up
|
|
573
681
|
// to six full sweeps of one immutable snapshot. `owner` is the one dimension
|
|
574
682
|
// told to file the finding about failing tests, so the others are told not
|
|
575
|
-
// to duplicate it rather than left to race each other to it.
|
|
683
|
+
// to duplicate it rather than left to race each other to it. A run the
|
|
684
|
+
// Install step skipped has no counts and no log to describe, so it gets a
|
|
685
|
+
// paragraph of its own.
|
|
576
686
|
export function sharedRunNote(run, owner, key) {
|
|
687
|
+
if (run.unprovisioned)
|
|
688
|
+
return `Tests: this review did NOT run its test command: ${run.unprovisioned}.
|
|
689
|
+
Every dimension of this review is reported unrun for it — do not run the full
|
|
690
|
+
command yourself to replace it, and file nothing about the install: the review
|
|
691
|
+
reports it itself.
|
|
692
|
+
command: ${run.command}`;
|
|
577
693
|
const lines = [
|
|
578
694
|
`Tests: this review ran its test command ONCE, from the snapshot's root, before
|
|
579
695
|
dispatching you. Do NOT run that full command yourself — every dimension reads
|
|
@@ -873,15 +989,37 @@ export async function runReview(host, args) {
|
|
|
873
989
|
( cd "$SNAP" && git init -q && git add -A -f && git -c user.name=fleet -c user.email=fleet@invalid commit -q --no-verify -m "review snapshot of $SHA" ) || echo SNAPSHOT_INIT_FAILED
|
|
874
990
|
SNAPTREE=$(git -C "$SNAP" rev-parse 'HEAD^{tree}' 2>/dev/null); SRCTREE=$(git -C ${worktree} rev-parse 'HEAD^{tree}')
|
|
875
991
|
{ [ -d "$SNAP/.git" ] && [ -n "$SNAPTREE" ] && [ "$SNAPTREE" = "$SRCTREE" ]; } && echo SNAPSHOT_TREE_MATCH || echo SNAPSHOT_TREE_MISMATCH="snapshot $SNAPTREE vs commit $SRCTREE"
|
|
876
|
-
|
|
877
|
-
|
|
992
|
+
INSTALL=$(~/.fleet/bin/fleet-run derive-testcmd.sh ${worktree} install) && echo SNAPSHOT_INSTALL_STEP="$INSTALL" || { INSTALL=; echo SNAPSHOT_INSTALL_UNREAD; }
|
|
993
|
+
[ -z "$INSTALL" ] || { ( cd "$SNAP" && sh -c "$INSTALL" ) >"$RUN/install.log" 2>&1; echo SNAPSHOT_INSTALL_EXIT=$?; }
|
|
994
|
+
if [ -d "$SNAP/.git" ] && git -C "$SNAP" status --porcelain --untracked-files=all >"$RUN/status.txt"; then head -n 20 "$RUN/status.txt" | sed 's/^/SNAPSHOT_UNTRACKED=/'; echo SNAPSHOT_UNTRACKED_COUNT=$(wc -l <"$RUN/status.txt" | tr -d ' '); else echo SNAPSHOT_STATUS_FAILED; fi
|
|
995
|
+
|
|
996
|
+
The Install step can take minutes: run the block as ONE blocking foreground
|
|
997
|
+
command with a command deadline of 3600 seconds, never backgrounded and never
|
|
998
|
+
polled. The Install step takes no suite slot.
|
|
999
|
+
|
|
1000
|
+
Report \`installCmd\` = the SNAPSHOT_INSTALL_STEP value and \`installExit\` = the
|
|
1001
|
+
SNAPSHOT_INSTALL_EXIT value, both copied as printed. If the block printed
|
|
1002
|
+
SNAPSHOT_INSTALL_UNREAD instead, omit both and report \`installError\` = what
|
|
1003
|
+
derive-testcmd.sh printed on stderr. The Install step runs once, in the
|
|
1004
|
+
snapshot, from its root: never rerun it, never repair a failure, and never run
|
|
1005
|
+
it anywhere else. A non-zero exit is reported as printed, and the caller files
|
|
1006
|
+
it as a finding on the PR.
|
|
1007
|
+
|
|
1008
|
+
Report \`untracked\` = every SNAPSHOT_UNTRACKED= value, copied as printed, in
|
|
1009
|
+
order, and omit it when the block printed none; the block prints twenty at most,
|
|
1010
|
+
so report \`untrackedCount\` = the SNAPSHOT_UNTRACKED_COUNT value, copied as
|
|
1011
|
+
printed. If the block printed SNAPSHOT_STATUS_FAILED, report \`statusError\` =
|
|
1012
|
+
SNAPSHOT_STATUS_FAILED: the status could not be read, which is not an empty
|
|
1013
|
+
status. The repository's own ignore rules decide what reads untracked after the
|
|
1014
|
+
Install step: list it, and add nothing to the snapshot's \`.git/info/exclude\` or
|
|
1015
|
+
anywhere else to hide it.
|
|
878
1016
|
|
|
879
1017
|
Report \`pathVerified\` = true ONLY if the 'ls -A' line printed SNAPSHOT_NONEMPTY,
|
|
880
|
-
run in that order — BEFORE the 'git init'
|
|
881
|
-
|
|
882
|
-
|
|
1018
|
+
run in that order — BEFORE the 'git init', never after it. The init creates an
|
|
1019
|
+
entry of its own, so a probe below it prints SNAPSHOT_NONEMPTY on a
|
|
1020
|
+
'git archive' that extracted nothing at all.
|
|
883
1021
|
|
|
884
|
-
Report \`repoVerified\` = true ONLY if the
|
|
1022
|
+
Report \`repoVerified\` = true ONLY if the tree-compare line above printed
|
|
885
1023
|
SNAPSHOT_TREE_MATCH; otherwise report it false and put the line it printed
|
|
886
1024
|
instead — SNAPSHOT_INIT_FAILED, or the whole SNAPSHOT_TREE_MISMATCH= value with
|
|
887
1025
|
both hashes — in \`repoError\`. The init is what makes the snapshot MEASURABLE:
|
|
@@ -891,8 +1029,9 @@ extraction than in a checkout at the same commit, with nothing in the payload
|
|
|
891
1029
|
saying the measurement happened somewhere else. The tree-hash compare is
|
|
892
1030
|
the verification: two commits whose trees hash the same hold byte-identical
|
|
893
1031
|
content, so a match settles that the snapshot IS the reviewed tree — a stronger
|
|
894
|
-
check than reading a few files, and the reason the commit
|
|
895
|
-
|
|
1032
|
+
check than reading a few files, and the reason the commit and the compare come
|
|
1033
|
+
BEFORE the Install step: 'git add -A -f' stages ignored files too, so whatever
|
|
1034
|
+
the Install step writes would enter the index and change the hash.
|
|
896
1035
|
|
|
897
1036
|
Report \`head\` = the full sha 'git -C ${worktree} rev-parse HEAD' prints — the
|
|
898
1037
|
destination path carries only the short form. The byte-identity spot check this
|
|
@@ -960,8 +1099,8 @@ specialists into another run's tree. Report the HEAD sha, and — in
|
|
|
960
1099
|
\`diffStats\` — the SINGLE-LINE JSON object diff-stats.mjs prints to
|
|
961
1100
|
STDOUT, copied verbatim. Only runRoot, path, head, pathVerified and repoVerified
|
|
962
1101
|
are ever required — diffStats, diffPath, diffLines, refHead and prHead are each
|
|
963
|
-
omitted independently when their command failed,
|
|
964
|
-
a false repoVerified.`,
|
|
1102
|
+
omitted independently when their command failed, repoError only accompanies
|
|
1103
|
+
a false repoVerified, and installError only replaces installCmd and installExit.`,
|
|
965
1104
|
{ label: `snapshot${forPr}`, phase: "Snapshot", agentType: SNAPSHOT_AGENT_TYPE, schema: SNAPSHOT_SCHEMA },
|
|
966
1105
|
);
|
|
967
1106
|
|
|
@@ -979,6 +1118,11 @@ a false repoVerified.`,
|
|
|
979
1118
|
|
|
980
1119
|
const testCmd = resolveTestCmd(A.testCmd, snap);
|
|
981
1120
|
log(`testCmd ${A.testCmd ? "(caller override)" : "(derived)"} ${testCmd}`);
|
|
1121
|
+
// An Install step that failed or never ran leaves a snapshot that cannot
|
|
1122
|
+
// measure the tree: the shared run is skipped, every dimension is reported
|
|
1123
|
+
// unrun through `testRunReason`, and a non-zero exit is filed as a finding.
|
|
1124
|
+
const notInstalled = installReason(snap);
|
|
1125
|
+
log(notInstalled ? `install: ${notInstalled}` : `install: ${snap.installCmd ?? "the Install step"} exited 0 in the snapshot`);
|
|
982
1126
|
|
|
983
1127
|
const usable = usableDiff(snap);
|
|
984
1128
|
log(
|
|
@@ -1012,7 +1156,7 @@ a false repoVerified.`,
|
|
|
1012
1156
|
// every dimension, never a pass.
|
|
1013
1157
|
const logPath = `${snap.runRoot}/test-run.log`;
|
|
1014
1158
|
phase("Test run");
|
|
1015
|
-
const ran = await agent(
|
|
1159
|
+
const ran = notInstalled ? null : await agent(
|
|
1016
1160
|
`Run this repository's test command ONCE, for the review of PR #${pr}, and
|
|
1017
1161
|
report what it printed. Every review specialist reads your counts instead of
|
|
1018
1162
|
running the suite itself, so this is the review's only full run.
|
|
@@ -1057,7 +1201,7 @@ Then read the summary the runner printed at the end of the log —
|
|
|
1057
1201
|
\`exitCode\` = the TEST_RUN_EXIT value, and \`tests\`, \`pass\`, \`fail\`,
|
|
1058
1202
|
\`expectedFail\`, \`cancelled\`, \`skipped\` = the counts that summary states, each
|
|
1059
1203
|
copied as printed; omit any the runner does not print. A test the runner reports
|
|
1060
|
-
as failing by design (vitest
|
|
1204
|
+
as failing by design (vitest \`expected fail\`, pytest \`xfailed\`) goes in \`expectedFail\`, never in
|
|
1061
1205
|
\`fail\`. Copy the counts, never judge them: 'tests 0' is a FAILED run, not a
|
|
1062
1206
|
pass, and 0 passes with no failures is everything skipped — report both
|
|
1063
1207
|
exactly as printed, and the caller reports every dimension unrun for them.
|
|
@@ -1068,8 +1212,10 @@ write 0 for a count the log does not state: an absent count is how the caller
|
|
|
1068
1212
|
learns the run produced none.`,
|
|
1069
1213
|
{ label: `test-run${forPr}`, phase: "Test run", agentType: TEST_RUN_AGENT_TYPE, schema: TEST_RUN_SCHEMA },
|
|
1070
1214
|
).catch((e) => ({ error: `the test-run dispatch threw: ${e?.message ?? e}` }));
|
|
1071
|
-
const sharedRun =
|
|
1072
|
-
|
|
1215
|
+
const sharedRun = notInstalled
|
|
1216
|
+
? { command: testCmd, logPath, unprovisioned: notInstalled }
|
|
1217
|
+
: { ...(ran || { error: "the test-run agent returned nothing" }), command: testCmd, logPath };
|
|
1218
|
+
log(notInstalled ? `shared test run: skipped — ${notInstalled}` : `shared test run: ${countsOf(sharedRun) || "no counts"} — exit ${sharedRun.exitCode ?? "(absent)"} — log ${logPath}`);
|
|
1073
1219
|
// Failing tests are the review's to report, not each dimension's: the first
|
|
1074
1220
|
// selected dimension is told to file them, and every other one not to.
|
|
1075
1221
|
const failureOwner = dimensions[0].key;
|
|
@@ -1110,6 +1256,7 @@ READ ONLY FROM THE SNAPSHOT: ${snap.path} (HEAD ${snap.head}) — plus the diff
|
|
|
1110
1256
|
file named below, if one is given.
|
|
1111
1257
|
Never read or write ${worktree} — other agents are using it.
|
|
1112
1258
|
Run any mutation or probe work inside your own copy of the snapshot.
|
|
1259
|
+
${copyRule(snap)}
|
|
1113
1260
|
|
|
1114
1261
|
Your shell starts in NEITHER of those directories, and what it does start in is
|
|
1115
1262
|
a tree you must not write to: this dispatch carries no working directory of its
|
|
@@ -1206,6 +1353,8 @@ decide it and nothing else:
|
|
|
1206
1353
|
|
|
1207
1354
|
${KILL_RULES}
|
|
1208
1355
|
|
|
1356
|
+
${copyRule(snap)}
|
|
1357
|
+
|
|
1209
1358
|
Abstain when your own check could produce neither a valid red nor a valid
|
|
1210
1359
|
green: set \`inconclusive\` to true and name the runs in \`reason\`, each with
|
|
1211
1360
|
its evidence line. \`refuted\` is still required, and the tally ignores it on
|
|
@@ -1291,7 +1440,8 @@ how three reviews from one cell left four files modified in that checkout.`,
|
|
|
1291
1440
|
},
|
|
1292
1441
|
);
|
|
1293
1442
|
|
|
1294
|
-
const
|
|
1443
|
+
const installed = installFinding(snap);
|
|
1444
|
+
const all = [...(installed ? [installed] : []), ...reviewed.flat().filter(Boolean)];
|
|
1295
1445
|
const survived = all.filter((f) => f.verdict === "survived");
|
|
1296
1446
|
const refuted = all.filter((f) => f.verdict === "refuted");
|
|
1297
1447
|
const unverified = all.filter((f) => f.verdict === "unverified");
|
package/skills/run-team/SKILL.md
CHANGED
|
@@ -2173,7 +2173,7 @@ PRs**, because the two causes live at different levels — the budget is one
|
|
|
2173
2173
|
account, the payload is one PR:
|
|
2174
2174
|
|
|
2175
2175
|
```sh
|
|
2176
|
-
budget_out= list_out= cap_out= blind=
|
|
2176
|
+
budget_out= hidden_out= list_out= cap_out= blind= # seed the latches: set -u reads each below before tick 1 ever assigns one
|
|
2177
2177
|
pr_cap=200 # the open-PR read's --limit; gh stops there with exit 0 and no warning
|
|
2178
2178
|
while :; do # one tick
|
|
2179
2179
|
rl=$(gh api rate_limit --jq '.resources.core.remaining' 2>/dev/null || echo ERR)
|
|
@@ -2210,34 +2210,52 @@ while :; do # one tick
|
|
|
2210
2210
|
# ceiling of one ci-state.mjs call on the counter `gh api rate_limit`
|
|
2211
2211
|
# reports, for single-page reads: 2 on the common path, 4 for a PR whose
|
|
2212
2212
|
# head dropped required jobs.
|
|
2213
|
-
#
|
|
2214
|
-
# does not report and this gate does not guard. A shortfall pauses this
|
|
2215
|
-
# tick's pass.
|
|
2213
|
+
# A shortfall here, or at the Actions-bucket gate below, pauses this tick's pass.
|
|
2216
2214
|
if [ "$rl" -lt $((200 + 4 * count)) ]; then
|
|
2217
2215
|
if [ -z "$budget_out" ]; then
|
|
2218
2216
|
echo "WATCHER DEGRADED: core REST budget=$rl < $((200 + 4 * count)) needed for $count PRs — CI polling paused, silence is NOT green"
|
|
2219
2217
|
budget_out=1
|
|
2220
2218
|
fi
|
|
2221
2219
|
else
|
|
2222
|
-
|
|
2223
|
-
|
|
2224
|
-
|
|
2225
|
-
|
|
2226
|
-
|
|
2227
|
-
|
|
2228
|
-
|
|
2229
|
-
|
|
2230
|
-
|
|
2231
|
-
|
|
2232
|
-
|
|
2233
|
-
|
|
2220
|
+
# The workflow and run lookups bill a second core bucket that
|
|
2221
|
+
# `rate_limit` does not report. Its count is only in the response
|
|
2222
|
+
# headers of a lookup that bills it: one 1-unit read per tick, 4 units
|
|
2223
|
+
# per listed PR on the same 200 floor. Recovery for both buckets waits
|
|
2224
|
+
# until this gate clears too, so no line says polling resumed while
|
|
2225
|
+
# this one still pauses it.
|
|
2226
|
+
hrl=$(gh api -i 'repos/{owner}/{repo}/actions/workflows?per_page=1' 2>/dev/null | awk 'tolower($1)=="x-ratelimit-remaining:" {sub(/\r$/, "", $2); print $2; exit}')
|
|
2227
|
+
case "$hrl" in ''|*[!0-9]*) hrl=ERR;; esac # a missing header is an outage, not a budget
|
|
2228
|
+
if [ "$hrl" = ERR ] || [ "$hrl" -lt $((200 + 4 * count)) ]; then
|
|
2229
|
+
if [ -z "$hidden_out" ]; then
|
|
2230
|
+
need=; [ "$hrl" = ERR ] || need=" < $((200 + 4 * count)) needed for $count PRs" # an unreadable header has no amount to compare
|
|
2231
|
+
echo "WATCHER DEGRADED: unreported Actions core REST bucket=$hrl$need — CI polling paused, silence is NOT green"
|
|
2232
|
+
hidden_out=1
|
|
2233
|
+
fi
|
|
2234
|
+
else
|
|
2235
|
+
if [ -n "$budget_out" ]; then
|
|
2236
|
+
echo "WATCHER RECOVERED: core budget=$rl — CI polling resumed"
|
|
2237
|
+
budget_out=
|
|
2234
2238
|
fi
|
|
2235
|
-
|
|
2236
|
-
echo "WATCHER RECOVERED:
|
|
2237
|
-
|
|
2238
|
-
|
|
2239
|
-
|
|
2240
|
-
|
|
2239
|
+
if [ -n "$hidden_out" ]; then
|
|
2240
|
+
echo "WATCHER RECOVERED: unreported Actions core REST bucket=$hrl — CI polling resumed"
|
|
2241
|
+
hidden_out=
|
|
2242
|
+
fi
|
|
2243
|
+
for pr in $(printf '%s\n' "$prs"); do # inline $(...): `for pr in $prs` is ONE iteration under zsh
|
|
2244
|
+
st=$(~/.fleet/bin/fleet-run ci-state.mjs --pr "$pr" 2>/dev/null) || : # not-green exits non-zero; the payload is the verdict
|
|
2245
|
+
if ! printf '%s' "$st" | jq -e '.verdict and .verdict != "rate-limited"' >/dev/null 2>&1; then
|
|
2246
|
+
case " $blind " in *" $pr "*) ;; *) # latch keyed BY PR: this cause is per-PR
|
|
2247
|
+
echo "WATCHER DEGRADED: no usable ci-state reading for #$pr — silence is NOT green"
|
|
2248
|
+
blind="$blind $pr" ;;
|
|
2249
|
+
esac
|
|
2250
|
+
continue # inner continue — still reaches the tick sleep
|
|
2251
|
+
fi
|
|
2252
|
+
case " $blind " in *" $pr "*)
|
|
2253
|
+
echo "WATCHER RECOVERED: ci-state reading #$pr again — CI polling resumed"
|
|
2254
|
+
keep=; for b in $(printf '%s\n' "$blind"); do [ "$b" = "$pr" ] || keep="$keep $b"; done; blind=$keep ;;
|
|
2255
|
+
esac
|
|
2256
|
+
: # your normal handling of $st for this PR
|
|
2257
|
+
done
|
|
2258
|
+
fi
|
|
2241
2259
|
fi
|
|
2242
2260
|
fi
|
|
2243
2261
|
fi
|
|
@@ -2263,8 +2281,8 @@ as a CI reading. Require the field, then compare it.
|
|
|
2263
2281
|
never one per tick. An unlatched line at a 120s poll gets the Monitor
|
|
2264
2282
|
auto-stopped for volume during even a two-minute outage, reintroducing the same
|
|
2265
2283
|
blindness by another route. **Latch each cause at the level its cause lives
|
|
2266
|
-
at**: the budget, the open-PR list and that
|
|
2267
|
-
each — all
|
|
2284
|
+
at**: the budget, the unreported Actions bucket, the open-PR list and that
|
|
2285
|
+
list's cap are one global flag each — all account-level — but the payload latch is keyed by PR, and a
|
|
2268
2286
|
shared scalar for those is an unlatched line wearing a latch's clothes — one
|
|
2269
2287
|
persistently blind PR alongside one healthy PR re-clears the flag on every tick,
|
|
2270
2288
|
emitting a DEGRADED and a false RECOVERED pair forever. The recovery line is
|
|
@@ -2280,7 +2298,7 @@ probe pairs during exactly the outage it is reporting. Every degraded
|
|
|
2280
2298
|
branch above falls through to the same single sleep instead.
|
|
2281
2299
|
|
|
2282
2300
|
**Guard every probe, not just the ones with a verdict field.** The block makes
|
|
2283
|
-
|
|
2301
|
+
several API calls, and `gh pr list` is the one that looks like
|
|
2284
2302
|
plumbing rather than a probe. A non-quota failure there (secondary/abuse limit,
|
|
2285
2303
|
5xx, network blip, expired token) does not lower `.resources.core.remaining`, so
|
|
2286
2304
|
the budget gate clears, and an unguarded `for pr in $(gh pr list …)` then
|
|
@@ -2306,8 +2324,9 @@ enforce. It latches globally like the list failure but does not pause polling:
|
|
|
2306
2324
|
the PRs it did return are real, so the pass still watches them, and the
|
|
2307
2325
|
DEGRADED line names the rest as unwatched until a pass comes back under the cap.
|
|
2308
2326
|
|
|
2309
|
-
**The budget is gated
|
|
2310
|
-
tick-top floor (`200`) covers the
|
|
2327
|
+
**The budget is gated at the floor and again at the pass, and only the pass
|
|
2328
|
+
gates announce recovery.** The tick-top floor (`200`) covers the `rate_limit`
|
|
2329
|
+
and `gh pr list` reads. Once the open-PR list is
|
|
2311
2330
|
read, the pass gate reserves `200 + 4 × count` core REST units — 4 per listed PR
|
|
2312
2331
|
on top of the floor, so the reserve grows with the list instead of staying a
|
|
2313
2332
|
flat 200 that a long list drains mid-pass. The 4 is the measured ceiling of one
|
|
@@ -2316,21 +2335,36 @@ flat 200 that a long list drains mid-pass. The 4 is the measured ceiling of one
|
|
|
2316
2335
|
to 100 jobs and 100 rules — more pages cost more. That counter is not the only
|
|
2317
2336
|
core bucket one call bills. The lookups `actions/workflows`,
|
|
2318
2337
|
`actions/workflows/<id>`, `actions/workflows/<id>/runs` and `actions/runs/<id>`
|
|
2319
|
-
bill a second one that `rate_limit` does not report
|
|
2320
|
-
|
|
2321
|
-
|
|
2322
|
-
|
|
2323
|
-
|
|
2324
|
-
|
|
2325
|
-
|
|
2338
|
+
bill a second one that `rate_limit` does not report; the boundary is per route,
|
|
2339
|
+
so `actions/runs/<id>/jobs` is on the reported counter. Every response says
|
|
2340
|
+
`X-Ratelimit-Resource: core`, so only the reset epoch and the used sequence tell
|
|
2341
|
+
the two apart, and the unreported bucket's remaining count is readable only from
|
|
2342
|
+
the `X-Ratelimit-Remaining` header of a call that bills it. **A second pass gate
|
|
2343
|
+
therefore probes it too**: one `gh api -i` read of
|
|
2344
|
+
`actions/workflows?per_page=1` per tick, 1 unit of that bucket (measured
|
|
2345
|
+
2026-10-13: its remaining count fell by 1 per call while `rate_limit` stood
|
|
2346
|
+
still, and its reset epoch differed from `rate_limit`'s). A `ci-state` call
|
|
2347
|
+
spends 4 units of it on the common path — one `gh run list` and one `gh run view`
|
|
2348
|
+
cost 2 each (measured the same day) — so it is held to the same `200 + 4 × count`
|
|
2349
|
+
reserve. It has its own global latch (`hidden_out`) and its DEGRADED and
|
|
2350
|
+
RECOVERED lines name it, so a pause on this bucket is not misread as the
|
|
2351
|
+
reported budget running low. An unreadable header is an outage here exactly as a
|
|
2352
|
+
non-numeric `rate_limit` read is. The probe runs only when the reported-bucket
|
|
2353
|
+
gate has cleared, so a tick costs at most one probe call. The gate pauses before
|
|
2354
|
+
the per-PR pass, so it never touches the per-PR blind latch: a PR is either not
|
|
2355
|
+
polled — no per-PR line, and one already latched blind stays latched — or polled
|
|
2356
|
+
and reported through that latch alone, never by both. A shortfall at any of
|
|
2357
|
+
these gates pauses the **whole** pass and never polls a prefix of the list: a prefix runs
|
|
2326
2358
|
in list order, so the same tail PRs would go unread every tick with no line at
|
|
2327
2359
|
all (never polled, so nothing marks them blind), while the PRs that were polled
|
|
2328
|
-
would run dry mid-pass and each emit a blind line. The DEGRADED line from
|
|
2329
|
-
pass gate names the amount it needed and the count it needed it for
|
|
2330
|
-
|
|
2331
|
-
|
|
2332
|
-
|
|
2333
|
-
|
|
2360
|
+
would run dry mid-pass and each emit a blind line. The DEGRADED line from each
|
|
2361
|
+
pass gate names the amount it needed and the count it needed it for — except an
|
|
2362
|
+
unreadable header, which has no amount to compare and prints `bucket=ERR` alone
|
|
2363
|
+
— so a healthy-looking `budget=999` is not read as an outage with no cause. Only the
|
|
2364
|
+
pass gates emit RECOVERED lines, and both only once the pass will actually run:
|
|
2365
|
+
announcing recovery at the floor, or for one bucket while the other still pauses
|
|
2366
|
+
the pass, would report resumption on a tick whose other gate then pauses it
|
|
2367
|
+
again — a DEGRADED/RECOVERED pair every tick.
|
|
2334
2368
|
|
|
2335
2369
|
**Judge `ci-state` on its payload, never its exit code** — the same rule as
|
|
2336
2370
|
`merge-gate.mjs` applies on the merge path, applied to the watcher. `not-green` is an
|