@feigi/fleet-ctl 3.69.2 → 3.71.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agents/fleet-recipe-deriver.agent.md +17 -7
- package/commands/review-and-fix.md +1 -1
- package/commands/run-merge-bot.md +1 -1
- package/package.json +1 -1
- package/scripts/derive-testcmd.sh +32 -11
- package/scripts/recipe-prove.mjs +43 -7
- package/scripts/review-core.mjs +44 -1
- package/skills/run-team/SKILL.md +68 -625
- package/skills/run-team/references/ci-and-staleness.md +27 -1
- package/skills/run-team/references/ci-watcher.md +88 -0
- package/skills/run-team/references/correction-tickets.md +27 -1
- package/skills/run-team/references/failure-handling.md +35 -0
- package/skills/run-team/references/finisher.md +93 -0
- package/skills/run-team/references/instruments.md +19 -0
- package/skills/run-team/references/isolation.md +140 -1
- package/skills/run-team/references/ledger.md +57 -0
- package/skills/run-team/references/member-lifecycle.md +43 -1
- package/skills/run-team/references/merge-bot.md +14 -0
- package/skills/run-team/references/pull.md +60 -0
- package/skills/run-team/references/reviewers.md +98 -0
- package/skills/run-team/references/shell-traps.md +55 -0
|
@@ -60,6 +60,23 @@ currency check. So never cache conclusion; key watchers on
|
|
|
60
60
|
re-marked, not re-run. Nothing in `ci-state.mjs` cached for same reason: only
|
|
61
61
|
safe design is re-query at moment of decision.
|
|
62
62
|
|
|
63
|
+
**A conclusion is not stable, even for a fixed run id on an unchanged head.** A
|
|
64
|
+
rerun rewrites the run in place, so never cache a conclusion; key watchers on
|
|
65
|
+
`<run-id>:<attempt>:<conclusion>`. Leaving `attempt` out of the key is the bug
|
|
66
|
+
that looks like it works: a rerun that lands on the *same* conclusion regenerates
|
|
67
|
+
an already-seen key and fires nothing, so the second failure is silent and reads
|
|
68
|
+
exactly like a run still in progress. **Building that key with jq's `//` is the
|
|
69
|
+
same bug in the opposite direction**: `ci-state` reports an in-progress run as
|
|
70
|
+
`conclusion: ""`, and `//` catches `null` and `false` but never the empty
|
|
71
|
+
string, so `.conclusion // "-"` passes `""` straight through, keys a RUNNING job
|
|
72
|
+
as terminal and fires a spurious not-green — measured in four separate runs,
|
|
73
|
+
including one where the controller then had to re-arm blind. Test any fallback
|
|
74
|
+
you write against `""`, `null` AND a real conclusion, and check it still lets
|
|
75
|
+
the real one through: `map(if . == null or . == "" then "-" else tostring end)`.
|
|
76
|
+
Also: the newest
|
|
77
|
+
run on a branch is frequently *not* CI, so `--limit 1` can hide the CI result
|
|
78
|
+
entirely.
|
|
79
|
+
|
|
63
80
|
## `--limit 1` hides CI
|
|
64
81
|
|
|
65
82
|
Newest run on branch frequently *not* CI — label or policy workflow often lands
|
|
@@ -73,4 +90,13 @@ green* until something re-triggers, then flip to `rebase-check: FAILURE` with
|
|
|
73
90
|
stale-**green**. Behind-count only honest signal. And one rebase per PR at merge
|
|
74
91
|
time enough: reviewers label on own green without requiring currency, and
|
|
75
92
|
requiring it earlier costs full CI cycle per sibling merge — six wasted cycles
|
|
76
|
-
in one run. Any behind-count you hand bot expired on arrival.
|
|
93
|
+
in one run. Any behind-count you hand bot expired on arrival.
|
|
94
|
+
|
|
95
|
+
## Beat when there is nothing to do, and never end your turn on a drained queue
|
|
96
|
+
|
|
97
|
+
**Beat when there is nothing to do, and never end your turn on a drained
|
|
98
|
+
queue.** You cannot be woken. A backgrounded Monitor does not wake an idle
|
|
99
|
+
agent — measured twice, see **A member runs its test suite in the foreground** in references/member-lifecycle.md, where `fix-pr-<M>` backgrounded a suite plus a Monitor and sat idle for
|
|
100
|
+
roughly two hours — and `run-merge-bot.md`'s CI gate states the general rule:
|
|
101
|
+
"whatever wakes you is external and may never come". So hold the wait yourself,
|
|
102
|
+
exactly the way that gate does:
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# The CI watcher: why its block is shaped the way it is
|
|
2
|
+
|
|
3
|
+
Evidence for the rules SKILL.md states beside Phase 3's watcher block, keyed by the rule each justifies.
|
|
4
|
+
|
|
5
|
+
## The pass is budgeted at 4 core REST units per listed PR
|
|
6
|
+
|
|
7
|
+
The 4 is the measured ceiling of one ci-state.mjs call on the counter `gh api rate_limit` reports, for single-page reads: 2 on the common path, 4 for a PR whose head dropped required jobs.
|
|
8
|
+
|
|
9
|
+
## One gate, not a gate plus a caveat
|
|
10
|
+
|
|
11
|
+
**One gate, not a gate plus a caveat.** `jq -e '.verdict and .verdict !=
|
|
12
|
+
"rate-limited"'` exits non-zero on unparseable input *and* on a false result, so
|
|
13
|
+
that single test covers empty, garbage AND a self-named quota refusal. Empty
|
|
14
|
+
stdin produces no result at all, which `-e` reports as exit 4 (measured), so a
|
|
15
|
+
`[ -z "$st" ] ||` pre-check in front of it is a *second* gate testing what this
|
|
16
|
+
one already tests — the caveat this heading is about. Leaving
|
|
17
|
+
the named verdict to prose instead is how a watcher clears its latch on an
|
|
18
|
+
outage payload and hands it downstream as CI state — the very blindness this
|
|
19
|
+
section exists to remove, one level in. **The `.verdict and` half is load-bearing
|
|
20
|
+
and not belt-and-braces**: inequality alone never tests that a verdict *exists*,
|
|
21
|
+
so `{}`, `null` and `{"verdict":null}` all evaluate `null != "rate-limited"` →
|
|
22
|
+
true and clear the gate (measured), handing an empty or error object downstream
|
|
23
|
+
as a CI reading. Require the field, then compare it.
|
|
24
|
+
|
|
25
|
+
## All pacing in exactly one place — the tick loop's own tail `sleep`
|
|
26
|
+
|
|
27
|
+
**All pacing in exactly one place — the tick loop's own tail `sleep`.** A sleep
|
|
28
|
+
inside the per-PR pass multiplies by the number of open PRs (8 open PRs → 960s
|
|
29
|
+
of pause per tick, against the 24s outage reset measured below), and a degraded
|
|
30
|
+
branch that `continue`s the *outer* loop skips the tail sleep and busy-spins
|
|
31
|
+
probe pairs during exactly the outage it is reporting. Every degraded branch of SKILL.md's watcher block falls through to the same single sleep instead.
|
|
32
|
+
|
|
33
|
+
## The budget is gated at the floor and again at the pass
|
|
34
|
+
|
|
35
|
+
**The budget is gated at the floor and again at the pass, and only the pass
|
|
36
|
+
gates announce recovery.** The tick-top floor (`200`) covers the `rate_limit`
|
|
37
|
+
and `gh pr list` reads. Once the open-PR list is
|
|
38
|
+
read, the pass gate reserves `200 + 4 × count` core REST units — 4 per listed PR
|
|
39
|
+
on top of the floor, so the reserve grows with the list instead of staying a
|
|
40
|
+
flat 200 that a long list drains mid-pass. The 4 is the measured ceiling of one
|
|
41
|
+
`ci-state.mjs` call's cost on the counter `gh api rate_limit` reports (measured
|
|
42
|
+
2026-10-13): 2 on the common path, 4 when the head dropped required jobs, for up
|
|
43
|
+
to 100 jobs and 100 rules — more pages cost more. That counter is not the only
|
|
44
|
+
core bucket one call bills. The lookups `actions/workflows`,
|
|
45
|
+
`actions/workflows/<id>`, `actions/workflows/<id>/runs` and `actions/runs/<id>`
|
|
46
|
+
bill a second one that `rate_limit` does not report; the boundary is per route,
|
|
47
|
+
so `actions/runs/<id>/jobs` is on the reported counter. Every response says
|
|
48
|
+
`X-Ratelimit-Resource: core`, so only the reset epoch and the used sequence tell
|
|
49
|
+
the two apart, and the unreported bucket's remaining count is readable only from
|
|
50
|
+
the `X-Ratelimit-Remaining` header of a call that bills it. **A second pass gate
|
|
51
|
+
therefore probes it too**: one `gh api -i` read of
|
|
52
|
+
`actions/workflows?per_page=1` per tick, 1 unit of that bucket (measured
|
|
53
|
+
2026-10-13: its remaining count fell by 1 per call while `rate_limit` stood
|
|
54
|
+
still, and its reset epoch differed from `rate_limit`'s). A `ci-state` call
|
|
55
|
+
spends 4 units of it on the common path — one `gh run list` and one `gh run view`
|
|
56
|
+
cost 2 each (measured the same day) — so it is held to the same `200 + 4 × count`
|
|
57
|
+
reserve. It has its own global latch (`hidden_out`) and its DEGRADED and
|
|
58
|
+
RECOVERED lines name it, so a pause on this bucket is not misread as the
|
|
59
|
+
reported budget running low. An unreadable header is an outage here exactly as a
|
|
60
|
+
non-numeric `rate_limit` read is. The probe runs only when the reported-bucket
|
|
61
|
+
gate has cleared, so a tick costs at most one probe call. The gate pauses before
|
|
62
|
+
the per-PR pass, so it never touches the per-PR blind latch: a PR is either not
|
|
63
|
+
polled — no per-PR line, and one already latched blind stays latched — or polled
|
|
64
|
+
and reported through that latch alone, never by both. A shortfall at any of
|
|
65
|
+
these gates pauses the **whole** pass and never polls a prefix of the list: a prefix runs
|
|
66
|
+
in list order, so the same tail PRs would go unread every tick with no line at
|
|
67
|
+
all (never polled, so nothing marks them blind), while the PRs that were polled
|
|
68
|
+
would run dry mid-pass and each emit a blind line. The DEGRADED line from each
|
|
69
|
+
pass gate names the amount it needed and the count it needed it for — except an
|
|
70
|
+
unreadable header, which has no amount to compare and prints `bucket=ERR` alone
|
|
71
|
+
— so a healthy-looking `budget=999` is not read as an outage with no cause. Only the
|
|
72
|
+
pass gates emit RECOVERED lines — the budget's and the unreported bucket's, and the
|
|
73
|
+
open-PR list's and its cap's with them — and all of them only once the pass will
|
|
74
|
+
actually run: announcing recovery at the floor, or for one bucket while the other
|
|
75
|
+
still pauses the pass, would report resumption on a tick whose other gate then
|
|
76
|
+
pauses it again — a DEGRADED/RECOVERED pair every tick — and announcing the list or
|
|
77
|
+
its cap on the tick that reads it back would say polling resumed on a tick that
|
|
78
|
+
polls nothing. A list or cap latch held across a paused tick stays latched and says
|
|
79
|
+
nothing more.
|
|
80
|
+
|
|
81
|
+
## Do not back off and wait
|
|
82
|
+
|
|
83
|
+
**Do not back off and wait.** These outages are short — a rolling window; one
|
|
84
|
+
measured reset came 24 seconds after `remaining: 0` — and pausing members for a
|
|
85
|
+
fixed interval stalls the fleet longer than the outage itself would have. A
|
|
86
|
+
retry-and-wait loop is what turns a 60s outage into a stalled member. Report it,
|
|
87
|
+
keep the local work moving (git, tests and mutation runs are all unaffected),
|
|
88
|
+
re-probe.
|
|
@@ -34,4 +34,30 @@ A bare `#N` copied from another repo's run reads as **this** repo's N — to a h
|
|
|
34
34
|
- **Any other host** (GHE, GitLab, internal) → prose naming host and repo, plus explicit *cannot be settled from this repo*. Qualified form is inert there: `gh api repos/<owner>/<repo>` returns 404 for a repo that lives only on a GHE host, so `owner/repo#N` would point at nothing.
|
|
35
35
|
- **This repo's own issues and PRs stay bare `#N`.** Convention covers foreign evidence only; qualifying local reference is noise.
|
|
36
36
|
|
|
37
|
-
Disclaimer must reach every foreign token it covers — numbers, ordinals, file names, test vocabulary — and must say unverifiable, because the clause-by-clause duty demands settling command and none exists to run. Where lesson itself is host-neutral, name behaviour generically instead of scoping it foreign: foreign-scoped lesson invites reader to discard one that does apply here.
|
|
37
|
+
Disclaimer must reach every foreign token it covers — numbers, ordinals, file names, test vocabulary — and must say unverifiable, because the clause-by-clause duty demands settling command and none exists to run. Where lesson itself is host-neutral, name behaviour generically instead of scoping it foreign: foreign-scoped lesson invites reader to discard one that does apply here.
|
|
38
|
+
|
|
39
|
+
## Every implementer body carries the claim discipline
|
|
40
|
+
|
|
41
|
+
**Every implementer body carries the claim discipline, because tickets that
|
|
42
|
+
restate facts ship new wrong claims** — inherited from the ticket, and minted
|
|
43
|
+
in prose nobody asked for. It was measured on tickets that would have been
|
|
44
|
+
judged routine too: an unasked-for comment asserting a silent-failure mode
|
|
45
|
+
that does not exist, a false comparative plus positional references replacing
|
|
46
|
+
a name-based one, and two present-tense counts already false when they were
|
|
47
|
+
written.
|
|
48
|
+
Nothing before dispatch can see the diff, so the discipline goes to every
|
|
49
|
+
implementer. A pushed commit body cannot be edited, so its claims are settled
|
|
50
|
+
at write time.
|
|
51
|
+
|
|
52
|
+
## A settling command over literal text uses `grep -F`
|
|
53
|
+
|
|
54
|
+
**A settling command over literal text uses `grep -F`, and its zero counts only
|
|
55
|
+
after the same command finds a line known to match.** Prose fragments and
|
|
56
|
+
code snippets routinely carry `\`, `.`, `*`, `[` or `$`, and without `-F` those
|
|
57
|
+
are read as a pattern rather than as the text itself, so the command answers
|
|
58
|
+
for a different string from the one the claim names — and a wrong zero is the
|
|
59
|
+
benign-looking answer. Measured once: `grep -c 'settled by:\s+x'` answers
|
|
60
|
+
0 against a file holding that exact line, and `grep -cF` answers 1. A zero or
|
|
61
|
+
an absence counts only after the same command, with the same flags, finds a
|
|
62
|
+
line known to match: a command that cannot find a match settles nothing, and
|
|
63
|
+
under `Verified:` its zero reads as a checked absence.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Failure handling: the evidence behind its rules
|
|
2
|
+
|
|
3
|
+
Evidence for the rules SKILL.md states in **Failure handling**, keyed by the rule each justifies.
|
|
4
|
+
|
|
5
|
+
## A frozen transcript does not establish that a member is dead
|
|
6
|
+
|
|
7
|
+
**A frozen transcript does not establish that a member is dead — the
|
|
8
|
+
discriminator is its scratch dir.** A member wedged on a blocked tool call
|
|
9
|
+
emits the whole signature of a dead one: record counts unchanged from one poll
|
|
10
|
+
of its transcript to the next. So read the mtimes of the files anywhere under
|
|
11
|
+
the scratch directory its own dispatch prompt assigned it, subdirectories
|
|
12
|
+
included — `<scratch>/impl-<N>/` for an implementer, that member's own
|
|
13
|
+
run-scoped subdirectory under `<scratch>/pr<N>/` on the review side, never
|
|
14
|
+
the bare `pr<N>/` tree those siblings share — since a member waiting on a
|
|
15
|
+
child it dispatched writes nothing itself while that child writes below it.
|
|
16
|
+
Read them at two observations a poll apart — `touch` a marker file outside
|
|
17
|
+
that directory at the first, and
|
|
18
|
+
`find <dir> -type f -newer <marker>` at the second lists every file that
|
|
19
|
+
moved: **mtimes that moved between the two mean BLOCKED, a live member
|
|
20
|
+
still holding its claim and its worktree, so the killed row of SKILL.md's Failure handling table does not
|
|
21
|
+
apply to it; mtimes unchanged at both are the dead signature.** Only the first
|
|
22
|
+
of those two verdicts is conclusive — a member wedged on a call that writes
|
|
23
|
+
nothing freezes its scratch dir too — so settle the dead one on the Member unaccounted row of that table, never on the two listings alone. Measured once: a refuter
|
|
24
|
+
wedged forever on `until grep -q` for a marker `node --test` never writes, its
|
|
25
|
+
transcript's record counts unchanged across two polls while its own scratch
|
|
26
|
+
files carried fresh mtimes throughout — alive the entire time it read as dead.
|
|
27
|
+
The filenames are not a contract: they are whatever that member happens to be
|
|
28
|
+
writing, and they differ by member and by role. That second listing is cheap
|
|
29
|
+
at the price, because the cost of the wrong call is asymmetric — a live member
|
|
30
|
+
written off gets killed and re-dispatched, which discards the work in flight
|
|
31
|
+
and, where it holds a claim or a worktree, collides the fleet with itself,
|
|
32
|
+
while one poll spent on a member that had in fact stopped costs only the poll.
|
|
33
|
+
The refuter's own observation rule in the fix-applier body stops a
|
|
34
|
+
member from creating that state; this one stops you from misreading whoever
|
|
35
|
+
reaches it anyway, and neither replaces the other.
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# The finisher: why its gate and duties are shaped the way they are
|
|
2
|
+
|
|
3
|
+
Evidence for the finisher rules SKILL.md states in **Reviewers**, keyed by the rule each justifies.
|
|
4
|
+
|
|
5
|
+
## An unanswered question from the member is an outbox item
|
|
6
|
+
|
|
7
|
+
**An unanswered question from the member is an outbox item, and it blocks
|
|
8
|
+
dispatch with the same weight as a ruling you have already made.** It does not
|
|
9
|
+
feel like one — you have decided nothing yet, so the outbox reads empty — but a
|
|
10
|
+
member waiting on you is a member that will push once you answer, and the SHA
|
|
11
|
+
you are about to pin is exactly what that push moves. Measured on one run:
|
|
12
|
+
a `fix-pr-<M>` member asked whether to add two pins, and the finisher was held until the
|
|
13
|
+
member acknowledged the answer. That worked because the controller chose to
|
|
14
|
+
hold, not because a rule required it. Answer it, or tell the member you are not
|
|
15
|
+
ruling and it should proceed on its own default — either empties the outbox.
|
|
16
|
+
What does not empty it is noticing the question and dispatching anyway.
|
|
17
|
+
|
|
18
|
+
## Duty 2: every deferral, and every claimed APPLY, has a home that outlives the merge
|
|
19
|
+
|
|
20
|
+
**Confirm every deferral — and every claimed APPLY — has a home that outlives
|
|
21
|
+
the merge.** The test is a durable home, not a tracker number: a tracker
|
|
22
|
+
issue and a committed in-tree comment both qualify, and for a finding whose
|
|
23
|
+
remedy was measured worse than the defect the in-tree note is the better one.
|
|
24
|
+
What fails is a promise living only in a PR body, or on the PR's *own* source
|
|
25
|
+
issue, which `Closes #N` buries on merge. Caught once at seven findings. A
|
|
26
|
+
comment on an existing *follow-up* issue is filed: that is
|
|
27
|
+
`review-and-fix.md` step 5, not a violation. Not filed → file it or halt,
|
|
28
|
+
never label over it — and file it the way step 5 does, through
|
|
29
|
+
`ledger.mjs check "<subject>"`, never a bare search, labelled `needs-triage`
|
|
30
|
+
as every issue the plugin creates is, its body carrying the refuter's
|
|
31
|
+
verdict or the review's votes and the candidate remedy. A duplicate is a
|
|
32
|
+
comment citing that verdict on the existing issue, never a relabel: an open
|
|
33
|
+
`ready-for-agent` issue triage already promoted holds the deferral as it
|
|
34
|
+
stands. Only triage sets `ready-for-agent`, and step 5's guard on this rule —
|
|
35
|
+
open `needs-triage` issues citing `Deferred from PR` above 40 on 3
|
|
36
|
+
consecutive run dates send it back to the maintainer — counts these
|
|
37
|
+
filings too. The finisher files last,
|
|
38
|
+
off its own read of what the reviewer left behind, so a deferral it reads as
|
|
39
|
+
unfiled may already be on the tracker under someone else's wording. Then
|
|
40
|
+
record it the way step 5 does: `ledger.mjs filed <N> "<subject>"`
|
|
41
|
+
immediately after `gh issue create`, with the same subject string you passed
|
|
42
|
+
to `check`. If `filed` fails, retry once; if it fails again, leave the issue
|
|
43
|
+
alone — it exists and is correct — and report it `unrecorded: #N <subject>`.
|
|
44
|
+
|
|
45
|
+
## `origin/main...HEAD` is right for the positive check and wrong for a negative one
|
|
46
|
+
|
|
47
|
+
**That range is right for the positive check and wrong for a negative one.**
|
|
48
|
+
It spans the implementer's commits too, so a constraint of the form *file X
|
|
49
|
+
was not touched* has to be scoped to the apply commit. Measured on one PR:
|
|
50
|
+
a controller wrote "neither document was touched" against
|
|
51
|
+
`origin/main...HEAD` on a two-commit branch whose documents the IMPLEMENTER
|
|
52
|
+
had correctly edited — the PR's whole point — and a finisher reading that
|
|
53
|
+
literally halts a correct PR. It caught the error instead, and said so; do
|
|
54
|
+
not rely on that.
|
|
55
|
+
|
|
56
|
+
## Whatever you verify by running, run in a tree nobody else owns
|
|
57
|
+
|
|
58
|
+
**Whatever you verify by RUNNING, run in a tree nobody else owns.**
|
|
59
|
+
Independently re-running the ticket's acceptance mutation — loosen the
|
|
60
|
+
regex, delete the guard clause, flip the per-entry reset, run `<testCmd>`,
|
|
61
|
+
confirm a kill by the test-run verdict rule in SKILL.md, discard — is the half of this duty reading a diff cannot
|
|
62
|
+
do, and it stays. But it **writes**, and the PR's worktree belongs to
|
|
63
|
+
another member. Measured once: a finisher was inside that worktree at the
|
|
64
|
+
instant the fix-applier announced one more clause to land, and neither
|
|
65
|
+
ordering is detectable afterwards from the diff — a restore lands on the
|
|
66
|
+
uncommitted clause and destroys it silently, or the mutant is still in the
|
|
67
|
+
tree when the fix-applier commits and a deliberately broken guard ships as
|
|
68
|
+
the fix, in a commit nothing re-reviews. So take your own:
|
|
69
|
+
`git worktree add --detach <scratch>/pr<N>/finish-<your member name> <your dispatch pin>`,
|
|
70
|
+
mutate and run `<testCmd>` in there, then
|
|
71
|
+
`git worktree remove --force <scratch>/pr<N>/finish-<your member name>`.
|
|
72
|
+
**At your dispatch pin, never at the fetched branch tip** — duty 1 just
|
|
73
|
+
proved the owned worktree sits at that pin, and `PR headRefOid` may carry
|
|
74
|
+
commits no reviewer read, the substitution the halt block's head-equality
|
|
75
|
+
check rejects for the same reason. `<scratch>` is the absolute root your dispatch prompt names. The member name is in the path because
|
|
76
|
+
the scratch root is shared with every sibling, so `<scratch>/pr<N>` alone
|
|
77
|
+
collides whenever a PR gets a second finisher, and `worktree add` on an
|
|
78
|
+
occupied path fails closed into a halt with a purely mechanical cause.
|
|
79
|
+
|
|
80
|
+
## Remove the verification worktree on every path
|
|
81
|
+
|
|
82
|
+
**Remove it on every path, the failing ones included** — a mutant that comes
|
|
83
|
+
back green, a suite that will not start, your own halt at any duty. `--force`
|
|
84
|
+
is required and is right only here: the tree is deliberately dirty when you
|
|
85
|
+
are done, so a plain `git worktree remove` refuses it (`contains modified or
|
|
86
|
+
untracked files`, rc 128, directory left on disk — measured, git 2.50.1),
|
|
87
|
+
while everywhere else in this runbook that refusal is a finding precisely
|
|
88
|
+
because the dirt may be someone's only copy. Here nothing in it is anyone's.
|
|
89
|
+
Nothing sweeps a leak for you either: `reap.sh`'s branchless sweep is bounded
|
|
90
|
+
to the fleet's worktree home, so a tree under `<scratch>` is a `kept` entry
|
|
91
|
+
it reports and never collects. No claim built the tree you added, so it has
|
|
92
|
+
`./agent-test` only if the repository tracks one — run that or `<testCmd>`
|
|
93
|
+
in it, per `references/isolation.md`, **A reused worktree may lack the runner**.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Instruments: why they are re-checked
|
|
2
|
+
|
|
3
|
+
Evidence for the rule SKILL.md states in **Rules that fail silently**, keyed by the rule it justifies.
|
|
4
|
+
|
|
5
|
+
## Re-check your instruments before every gate decision
|
|
6
|
+
|
|
7
|
+
**You read your instruments out of a tree every member can write to — re-check
|
|
8
|
+
them before every gate decision.** Per-member worktrees and `./agent-test`'s port
|
|
9
|
+
derivation protect members from each other; neither protects the main checkout,
|
|
10
|
+
which you run every gate probe out of and which any member can edit — except
|
|
11
|
+
through a `write`/`edit`/`ast_edit` the `member-write-guard` extension refuses;
|
|
12
|
+
a `bash` or `eval` write still lands. Measured: a
|
|
13
|
+
member edited two files there instead of in its worktree while the CI monitor was
|
|
14
|
+
polling one of them every 120s and a live finisher was using it to decide a
|
|
15
|
+
label. It caught itself; nothing in the fleet would have. That is the silent
|
|
16
|
+
direction — a modified instrument raises no error, shows in no PR and leaves
|
|
17
|
+
nothing in the ledger, and it still returns a verdict, just not necessarily the
|
|
18
|
+
right one, so you have no reason to re-check a script you did not know had
|
|
19
|
+
changed.
|
|
@@ -123,4 +123,143 @@ The axis is two-sided: the member partitions, and the parent assigns each child'
|
|
|
123
123
|
|
|
124
124
|
## IDE/harness diagnostics attribute by bare filename, with no path
|
|
125
125
|
|
|
126
|
-
Probe copies carry same filenames as real tree, so specialist's throwaway mutation surfaces as errors that read exactly like live worktree's — and line numbers can plausibly line up with real in-flight edits. Never relay diagnostic without reproducing it in that member's specific worktree (re-run the check that raised it — the typechecker, the linter — from there). Ran twice in one session: clean first time (sibling's probe), genuinely broken second. Telling implementer to chase phantom in file it is mid-rewrite on is expensive failure.
|
|
126
|
+
Probe copies carry same filenames as real tree, so specialist's throwaway mutation surfaces as errors that read exactly like live worktree's — and line numbers can plausibly line up with real in-flight edits. Never relay diagnostic without reproducing it in that member's specific worktree (re-run the check that raised it — the typechecker, the linter — from there). Ran twice in one session: clean first time (sibling's probe), genuinely broken second. Telling implementer to chase phantom in file it is mid-rewrite on is expensive failure.
|
|
127
|
+
|
|
128
|
+
## Where the omp probes ran
|
|
129
|
+
|
|
130
|
+
Measured 2026-09-09 in a throwaway clone (`/tmp/fleet-probe`)
|
|
131
|
+
that never touched the real checkout.
|
|
132
|
+
|
|
133
|
+
## A fleet member is never dispatched with omp's `isolated: true`
|
|
134
|
+
|
|
135
|
+
**A fleet member is never dispatched with omp's `isolated: true`.** Measured:
|
|
136
|
+
an isolated spawn builds its workspace under `~/.omp/wt/<hash>/m` — an APFS
|
|
137
|
+
clone (on this box), never a registered `git worktree` of the repo — and,
|
|
138
|
+
because `apply` defaults to `true`, its changes are patch-applied into the
|
|
139
|
+
**calling session's own cwd** the instant the spawn completes: a file an
|
|
140
|
+
isolated probe created there appeared as `?? probe-isolated.txt` in the probe
|
|
141
|
+
session's own cwd — which for an actual fleet run is the controller's main
|
|
142
|
+
checkout — not in the worktree the prompt named. That is the exact
|
|
143
|
+
silent-spill hazard this section exists to guard against, on by default, so
|
|
144
|
+
`isolated` stays unused for every member and `task.isolation.enabled` stays
|
|
145
|
+
off.
|
|
146
|
+
|
|
147
|
+
## A member's tree is the claimed worktree, addressed by absolute path
|
|
148
|
+
|
|
149
|
+
**A member's tree is the claimed worktree, addressed by absolute path.** An
|
|
150
|
+
`edit` header without the worktree prefix resolves into the main checkout,
|
|
151
|
+
where `member-write-guard` refuses it (**You read your instruments out of a
|
|
152
|
+
tree every member can write to** in SKILL.md). How a
|
|
153
|
+
member gets its cwd: the `task` tool's item schema — `name`/`agent`/`task`/
|
|
154
|
+
`outputSchema`/`schemaMode` always, `effort`/`isolated` only when their own
|
|
155
|
+
settings enable them — carries no working-directory field; measured
|
|
156
|
+
directly: a member dispatched through it with none of those fields, told
|
|
157
|
+
only to report `pwd`, returned the calling session's own cwd, not the
|
|
158
|
+
claimed worktree, the same output an un-`cwd`-set `bash` call gives from
|
|
159
|
+
that session, so the recipe is the dispatch-prompt absolute path, plus
|
|
160
|
+
`bash`'s own `cwd` parameter set to it on every call and absolute paths for
|
|
161
|
+
`write`/`edit`.
|
|
162
|
+
|
|
163
|
+
## `task` does not accept a per-dispatch working directory
|
|
164
|
+
|
|
165
|
+
**The one open gap that probe measured: `task` does not accept a per-dispatch
|
|
166
|
+
working directory.** `omp://tools/task.md` documents non-isolated spawns as
|
|
167
|
+
running `runSubprocess(...)` "directly with parent cwd" — no `cwd` field
|
|
168
|
+
appears anywhere in the item schema — and the probe above confirms it in
|
|
169
|
+
practice: the recipe is the absolute-path discipline above, not a placeholder
|
|
170
|
+
for a `cwd` field that does not exist. The gap is now enforced rather than only
|
|
171
|
+
written down: the plugin's `member-write-guard` omp extension
|
|
172
|
+
refuses a fleet member's `write`/`edit`/`ast_edit` into the main checkout's
|
|
173
|
+
non-ignored tree and an implementer's `bash` from a main-checkout cwd, and the
|
|
174
|
+
missing field itself is requested upstream in `can1357/oh-my-pi`.
|
|
175
|
+
|
|
176
|
+
## `task`-dispatched members share one `eval` kernel
|
|
177
|
+
|
|
178
|
+
**The same gap reaches past the filesystem: `task`-dispatched members share one
|
|
179
|
+
`eval` kernel.** Measured 2026-09-21, in a live run rather than a
|
|
180
|
+
probe clone: two sibling implementers dispatched in the same batch, and the
|
|
181
|
+
controller that dispatched both, reported the same Python kernel pid and the
|
|
182
|
+
same runner file under `$TMPDIR/omp-python-runner/`, and each read the others'
|
|
183
|
+
top-level bindings out of its own `eval` cell — a bare `WT` bound by one member
|
|
184
|
+
was read back by a sibling and by the controller, the collision a ticket was filed
|
|
185
|
+
on as an unproven self-report. `omp://tools/eval.md` states the mechanism:
|
|
186
|
+
retained kernels are keyed by `python:${sessionId}`, normalized cwd and
|
|
187
|
+
interpreter — `js:${sessionId}` for the JS VM — and "Parent and ordinary task
|
|
188
|
+
subagents may share an inherited eval executor id; children created by eval's
|
|
189
|
+
own `agent()` explicitly do not." Neither key component separates two members
|
|
190
|
+
of one run: the session id is inherited from the controller, and the cwd is the
|
|
191
|
+
main checkout for all of them, because `eval`, unlike `bash`, takes no `cwd`
|
|
192
|
+
parameter to set. So the absolute-path discipline above is load-bearing inside
|
|
193
|
+
`eval` too, and it needs one rule `write`/`edit` do not — member-prefixed
|
|
194
|
+
binding names, since a shared global has no path to be absolute about. The
|
|
195
|
+
corollary is `reset`: it "is likewise destructive to concurrent work sharing
|
|
196
|
+
that backend session", so a member resetting its own kernel resets every
|
|
197
|
+
concurrent sibling's. No per-dispatch knob turns any of this off: `task`'s item
|
|
198
|
+
shape carries no kernel, executor or cwd field, and `python.kernelMode` set to
|
|
199
|
+
`"per-call"` — the one documented lever that would hand every call a fresh
|
|
200
|
+
kernel — is a session setting a child inherits, not something one dispatch can
|
|
201
|
+
set for one member. Isolating a member's kernel is an omp-side change, so
|
|
202
|
+
discipline is the whole of the remedy here — unlike the missing `cwd` field
|
|
203
|
+
above, which `member-write-guard` enforces for `write`/`edit`/`ast_edit` and
|
|
204
|
+
`eval` bypasses. **Phase 2** carries all three rules to every member.
|
|
205
|
+
|
|
206
|
+
## `release-ticket.sh` is not a no-op on omp, and a running member hides nothing from `inflight.sh`
|
|
207
|
+
|
|
208
|
+
**`release-ticket.sh` is not a no-op on omp, and `inflight.sh`'s probes are
|
|
209
|
+
unaffected by a running member.** Measured against a hand-built claim (a
|
|
210
|
+
branch plus a `git worktree add`) with a member sleeping inside it: a `bash`
|
|
211
|
+
call with `cwd` set to the worktree, and a non-isolated `task` spawn told to
|
|
212
|
+
work there, both operate inside the existing worktree and leave `git worktree
|
|
213
|
+
list` unchanged — they coexist with the claim rather than duplicating it. An
|
|
214
|
+
isolated spawn is architecturally blind to the worktree — its relative writes
|
|
215
|
+
land in its own `~/.omp/wt/…` workspace regardless of the prompt — but it
|
|
216
|
+
neither removes nor hides the claim, and `~/.omp/wt/` was empty after every
|
|
217
|
+
isolated run: omp's own teardown touches only omp's own workspace, never the
|
|
218
|
+
claim. `release-ticket.sh`'s three-artifact teardown (label, worktree, branch)
|
|
219
|
+
is therefore still necessary. With the member live inside
|
|
220
|
+
the worktree, `git worktree list` and `git branch --list` from the main
|
|
221
|
+
checkout reported the claim unchanged before, during and after — `inflight.sh`
|
|
222
|
+
hides nothing from omp.
|
|
223
|
+
|
|
224
|
+
## A pattern kill reaches siblings
|
|
225
|
+
|
|
226
|
+
**A pattern kill reaches siblings — the implementer, fix-applier and finisher briefs forbid it.** `pkill -f
|
|
227
|
+
<pattern>` matches on the whole command line, machine-wide. Members hold
|
|
228
|
+
separate worktrees, but the process table is not partitioned by them and a
|
|
229
|
+
test file's name is identical in every one, so a pattern naming a test file
|
|
230
|
+
matches whichever member happens to be running it. Measured: an implementer
|
|
231
|
+
chasing its own stalled run issued a `pkill -f` on its test file's name and
|
|
232
|
+
killed a `node --test` over two test files that was not
|
|
233
|
+
its own.
|
|
234
|
+
|
|
235
|
+
## What a killed run's output looks like depends on the signal
|
|
236
|
+
|
|
237
|
+
**What the victim's output looks like depends on the signal — and one of the
|
|
238
|
+
cases is a clean green.** Each measured on node v26.8.1/darwin against a
|
|
239
|
+
purpose-spawned fixture killed by its own PID. `pkill -f` **defaults to
|
|
240
|
+
SIGTERM**, and a SIGTERMed runner prints `Interrupted while running:` followed
|
|
241
|
+
by a *complete* summary block with `cancelled` non-zero, exiting `rc=1` — so
|
|
242
|
+
`cancelled` is exactly the thing to search for in the default case. `pkill -9
|
|
243
|
+
-f` leaves no summary block at all and the shell reports `rc=137`: SIGKILL
|
|
244
|
+
means the harness never lived long enough to report. A kill landing on one of
|
|
245
|
+
node's per-file **child** test processes leaves a complete block with `fail`
|
|
246
|
+
non-zero and `cancelled 0` at `rc=1`, indistinguishable from an ordinary test
|
|
247
|
+
failure. And a kill landing on a process a **test itself spawns** leaves a
|
|
248
|
+
complete `pass N / fail 0 / cancelled 0` at the expected N over a corrupted
|
|
249
|
+
run: `spawnSync` reports a SIGKILLed child as `status null`, and a test
|
|
250
|
+
asserting only that the process it spawned exited non-zero —
|
|
251
|
+
`assert.notEqual(r.status, 0)`, a form used across this suite, in files that
|
|
252
|
+
spawn the shell scripts they cover — passes vacuously on `null`.
|
|
253
|
+
|
|
254
|
+
## The command line is no partition either
|
|
255
|
+
|
|
256
|
+
**The command line is no partition either — not reliably.** Ports are derived
|
|
257
|
+
per issue precisely so collisions are impossible; argv isolates a run only
|
|
258
|
+
sometimes. Measured with `ps -Ao pid,ppid,args -ww` (without `-ww` macOS `ps`
|
|
259
|
+
truncates, and the truncation reads as an absence): an invocation carrying an
|
|
260
|
+
**absolute** path puts a member-unique string in argv, while one carrying
|
|
261
|
+
repo-relative filenames is byte-identical across members. Which of the two you
|
|
262
|
+
get is not yours to choose — members are briefed on `./agent-test` with the
|
|
263
|
+
arguments the Test entrypoint takes, and the runner hands them on exactly as
|
|
264
|
+
its caller's shell expanded them. Clearing your own leftover process needs its
|
|
265
|
+
PID, not a pattern.
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# The run ledger: the evidence behind its rules
|
|
2
|
+
|
|
3
|
+
Evidence for the rules SKILL.md states in **Run ledger**, keyed by the rule each justifies.
|
|
4
|
+
|
|
5
|
+
## Writer policy: every write is serialized by `<file>.lock`
|
|
6
|
+
|
|
7
|
+
**Writer policy: the controller owns run state (`row`/`token`/`settle`/`dispatch`/`review`/`drain`/`ruled`);
|
|
8
|
+
filers append filings with `filed`; all writes are serialized by `<file>.lock`.**
|
|
9
|
+
Every write rewrites the whole file from what it read, so without the lock two
|
|
10
|
+
concurrent writers are last-writer-wins — measured, 8 concurrent `filed` kept
|
|
11
|
+
2–5 of 8 rows. A writer that cannot take the lock within 10 s exits 2
|
|
12
|
+
naming the holder's pid, having written nothing; a lock whose holder is dead is
|
|
13
|
+
taken over, and a leftover `<file>.lock.reap` is never reclaimed. One other
|
|
14
|
+
shape wedges the lock permanently the same way, needing the same fix: an empty
|
|
15
|
+
`<file>.lock` — its writer was killed between creating it and writing its pid
|
|
16
|
+
into it — is unparseable and therefore, by design, always read as live rather
|
|
17
|
+
than handed to the death check; its timeout names `<file>.lock` itself, since
|
|
18
|
+
there is no pid to name. A zombie holder — a pid whose process exited
|
|
19
|
+
but whose parent never `wait()`-ed on it — also answers a liveness check as
|
|
20
|
+
alive, because the OS keeps a reserved pid until it is reaped, but that is
|
|
21
|
+
transient, not a wedge: once the pid is reaped (the parent's own exit is
|
|
22
|
+
normally enough to trigger this), the next writer's death check reads it as
|
|
23
|
+
dead and takes the lock over the same as any other dead holder — no hand
|
|
24
|
+
removal needed. The other two shapes do not clear on their own — remove
|
|
25
|
+
`<file>.lock` or `<file>.lock.reap` by hand once nothing is writing. Reads
|
|
26
|
+
(`check`, `read`) take no lock.
|
|
27
|
+
|
|
28
|
+
## The gap the lock does not close
|
|
29
|
+
|
|
30
|
+
**The gap the lock does not close:** `check` → `gh issue create` → `filed` is
|
|
31
|
+
three commands, and the lock covers each one alone, so two filers can both read
|
|
32
|
+
`clean` in the seconds before either records. Accepted — the measured
|
|
33
|
+
same-run accidental duplicate was filed minutes apart — and no
|
|
34
|
+
claim/reserve step exists to close it.
|
|
35
|
+
|
|
36
|
+
## `check`'s exits and its `verdict`
|
|
37
|
+
|
|
38
|
+
`check` exits **0** clean, **1** already in this run's filed list, **2** usage
|
|
39
|
+
error (no JSON on stdout), **3** the ledger is clean but open or closed tracker issues match — read
|
|
40
|
+
those and decide. Exit **3** is scored: some row gh returned has to score above
|
|
41
|
+
zero against the subject, because the ANDed query also matches issues the
|
|
42
|
+
overlap rates 0.00, and every measured stop over those was survived only by a
|
|
43
|
+
reader who overrode it and searched the tracker by hand — obeying it would have
|
|
44
|
+
dropped a real deferral. It also prints the closest filed rows with an overlap score.
|
|
45
|
+
Those never move the exit code — the same finding gets worded differently by
|
|
46
|
+
whoever finds it second — but once the tracker read succeeds, a filed row at or
|
|
47
|
+
above `NEAR_SOFT_HIT` in `ledger.mjs` does move the verdict to `soft-hit`, as
|
|
48
|
+
does a tracker set with no scoring row in it; a read that failed stays
|
|
49
|
+
`unverified`, which outranks both. **Exit 0 is not automatically "safe to file":** when `gh`
|
|
50
|
+
cannot be reached the answer is ledger-only, and it says `TRACKER NOT CHECKED` —
|
|
51
|
+
an issue filed by an earlier run is invisible to it; and `soft-hit` means rows
|
|
52
|
+
worth reading were found while none of them established a duplicate, so read
|
|
53
|
+
them before filing. The stdout JSON names all of that in one field:
|
|
54
|
+
`verdict` is `already-filed`, `tracker-hit`, `clean`, `soft-hit` or
|
|
55
|
+
`unverified`. Only `already-filed` and `tracker-hit` carry a non-zero code, so
|
|
56
|
+
`verdict` is the only thing that tells a searched-and-clean tracker from one
|
|
57
|
+
that was never read and from one whose rows you have to read yourself.
|
|
@@ -56,4 +56,46 @@ Reviewers go idle waiting on CI, won't resume alone — rebased, pushed, stopped
|
|
|
56
56
|
|
|
57
57
|
## React to artifacts, not agents
|
|
58
58
|
|
|
59
|
-
Liveness says nothing: `idle` means "not currently executing", not "done"; finished member's finding may never arrive. In one run every member that looked dead had its work in git or on PR — pushed commits, self-removed label, applied ruling. Read `gh pr view`, `gh run view`, `git -C <worktree> status` before messaging: one command settles what round-trip usually doesn't, and member that ignored one ping tends to ignore second. When you message, send specific next action, never "what is your status".
|
|
59
|
+
Liveness says nothing: `idle` means "not currently executing", not "done"; finished member's finding may never arrive. In one run every member that looked dead had its work in git or on PR — pushed commits, self-removed label, applied ruling. Read `gh pr view`, `gh run view`, `git -C <worktree> status` before messaging: one command settles what round-trip usually doesn't, and member that ignored one ping tends to ignore second. When you message, send specific next action, never "what is your status".
|
|
60
|
+
|
|
61
|
+
## A missing report is not evidence the member failed to send one
|
|
62
|
+
|
|
63
|
+
**But a missing report is NOT evidence the member failed to send one — the
|
|
64
|
+
inbound direction drops AND delays messages.** Both were measured in one run,
|
|
65
|
+
and they are indistinguishable at the moment you look: one finisher's report was
|
|
66
|
+
accepted with a msg_id and never arrived at all, while another's arrived late —
|
|
67
|
+
its original and its re-send landing together, after the controller had already
|
|
68
|
+
concluded it was missing. Measured: a finisher completed every
|
|
69
|
+
duty, labelled the PR, and sent its report; the send was accepted with a msg_id
|
|
70
|
+
and never arrived. The controller read the silence as the member ending its turn
|
|
71
|
+
without reporting, said so, and was wrong. The re-send it asked for by name is
|
|
72
|
+
the only reason the report exists. So when a report is missing, **ask by name —
|
|
73
|
+
that is what the "unless the controller asks by name" clause is for — and do not
|
|
74
|
+
attribute the gap to the member until it answers.** A member re-sending on a
|
|
75
|
+
named request should state the original's msg_id and that it is a re-send, which
|
|
76
|
+
is what distinguishes a lost message from a member that never sent one; a
|
|
77
|
+
controller that skips the ask cannot tell those apart and will guess wrong in
|
|
78
|
+
whichever direction its expectations point. The tick prints `SETTLE <member>`
|
|
79
|
+
for a report file on disk that passes its schema and `REPORT-INVALID` for one
|
|
80
|
+
that fails it, so a missing message is read off those lines before anyone is
|
|
81
|
+
pinged.
|
|
82
|
+
|
|
83
|
+
Note this cuts against the paragraph above only in appearance: the default stays
|
|
84
|
+
*send once*, because unprompted re-sends were measured 2-to-1 false. The ask is
|
|
85
|
+
the controller's move, not the member's.
|
|
86
|
+
|
|
87
|
+
## A member runs its test suite in the foreground and blocks on it
|
|
88
|
+
|
|
89
|
+
**A member runs its test suite in the foreground and blocks on it.**
|
|
90
|
+
Backgrounding a suite and polling for it makes the member's own progress
|
|
91
|
+
depend on a wake-up nothing in the fleet guarantees, and an idle member is
|
|
92
|
+
indistinguishable from a working one — you learn only by pinging. Measured
|
|
93
|
+
twice in one run: `fix-pr-<M>` backgrounded its suite plus a Monitor and sat
|
|
94
|
+
idle for roughly two hours; neither ever woke it, and the only reason no work
|
|
95
|
+
was lost is that you pinged it by name with a concrete next action.
|
|
96
|
+
`finisher-pr-<M>` reached for the same pattern minutes later — a proactive
|
|
97
|
+
warning is the only reason it did not repeat the stall. Its
|
|
98
|
+
`pgrep -f 'finish-finisher-pr-<M>'` could never have matched any process, so
|
|
99
|
+
the poll it was waiting on was structurally incapable of firing. This file is
|
|
100
|
+
yours, not theirs: the implementer, fix-applier and finisher briefs carry the
|
|
101
|
+
rule, as the line `A member runs its test suite in the foreground and blocks on it.`
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# Landing a change on main
|
|
2
|
+
|
|
3
|
+
Evidence for the rule SKILL.md states in **Fix the tooling mid-run**, keyed by the rule it justifies.
|
|
4
|
+
|
|
5
|
+
## A mid-run tooling fix lands through a PR
|
|
6
|
+
|
|
7
|
+
**Landing it.** The file being right is not the end of it. `main` carries a
|
|
8
|
+
ruleset with no bypass actors, so a direct `git push origin main` is **refused** —
|
|
9
|
+
`! [remote rejected] main -> main (push declined due to repository rule
|
|
10
|
+
violations)`, measured 2026-08-10 by a controller that tried exactly that at the
|
|
11
|
+
end of a 7-PR run. A mid-run tooling fix goes through a PR like every other
|
|
12
|
+
change, and that PR faces every required check any other PR faces — the set
|
|
13
|
+
`.github/rulesets/main.json` declares — so it carries a release label too.
|
|
14
|
+
Plan it as a PR the run's own merge queue lands, never as an edit-and-continue.
|