@agentsfleet/orly 0.6.3 → 0.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +32 -34
- package/SOUL.md +28 -30
- package/audits/agents-md.md +1 -1
- package/audits/agents-md.sh +5 -2
- package/core/operating-model.md +4 -4
- package/dispatch/lifecycle.md +3 -1
- package/dispatch/verify.md +21 -22
- package/dispatch/write_zig.md +6 -6
- package/docs/REST_API_DESIGN_GUIDELINES.md +5 -5
- package/docs/VERIFY_TIERS.md +46 -36
- package/package.json +2 -2
- package/skills/orly-write-integration-test/SKILL.md +13 -13
- package/skills/orly-write-unit-test/SKILL.md +2 -2
- package/src/pack_hygiene.test.ts +104 -0
- package/src/pack_hygiene.ts +109 -0
package/AGENTS.md
CHANGED
|
@@ -29,11 +29,9 @@ Dates in session notes and log entries: `MMM DD, YYYY: HH:MM AM/PM`. Filenames:
|
|
|
29
29
|
**Pre-send self-checks (invariant).** Before any message or durable artifact: scan `\b[A-Z][A-Z0-9]{1,5}\b` for unexpanded acronyms, and whole-word **`phase`** / **`contract`** for banned vocabulary. Skipping = `ACRONYM CHECK: SKIPPED per user override (reason: ...)` / `BANNED-VOCAB CHECK: SKIPPED per user override (reason: ...)`; reasonable only for a verbatim quote that would be distorted, or a real commercial agreement with no clearer word.
|
|
30
30
|
|
|
31
31
|
|
|
32
|
-
**Evidence invariant (pre-send).** Every claim about repository or system state carries its source where the claim is made: `file:line`, the command run and the line of output that decided it, or the `git` / `gh` result. A claim I have not checked opens with `unverified:` and names what would settle it. Numbers name the command that produced them. "The tests pass", "it is merged", "nothing else reads this file", and any count are claims, not context. Skipping = `EVIDENCE CHECK: SKIPPED per user override (reason: ...)`.
|
|
32
|
+
**Evidence invariant (pre-send).** Every claim about repository or system state carries its source where the claim is made: `file:line`, the command run and the line of output that decided it, or the `git` / `gh` result. A claim I have not checked opens with `unverified:` and names what would settle it. Numbers name the command that produced them. "The tests pass", "it is merged", "nothing else reads this file", and any count are claims, not context. Skipping = `EVIDENCE CHECK: SKIPPED per user override (reason: ...)`. Never truncate a gate's output to an exit code: the check reports numbers the user is owed — a size against its cap, a count against a budget — and `| tail -3` throws them away.
|
|
33
33
|
|
|
34
|
-
**
|
|
35
|
-
|
|
36
|
-
**Self-repair boundary (invariant).** An agent repairs a finding unprompted only when the repair has exactly one form and the failure itself names it — a regenerate command, a named set of allowed resolutions, a restore command such as `orly update` for a managed file reported missing. Everything else is reported, not repaired. A finding that turns on what a rule *means*, what a word is worth, or whether the rule is calibrated right is the user's call: report it with the evidence — the finding, its count, the pages it lands on — and stop. Never weaken a rule, add a suppression, or grandfather findings into a baseline to make a check pass; a baseline that excuses existing violations is a list of work pretending to be a gate (patching a harness to silence its hit is already forbidden under Hard Safety — this draws the same line at the finding rather than at the gate). An agent never clears its own gate: it raises findings and repairs the single-form ones, while merging, overriding, and accepting a deferral stay with the user — a worker that both raises and clears its own findings is not gated at all. Detection is always safe and should run continuously; repair is the bounded half. `orly gate`'s `docs.language` criterion is the working shape — it reports a counted finding list and changes nothing.
|
|
34
|
+
**Self-repair boundary (invariant).** Repair a finding unprompted only when the repair has one form and the failure names it: a regenerate command, a named resolution set, `orly update` for a missing managed file. Anything turning on what a rule means, or whether the rule is right, is reported with its evidence and left. Never weaken a rule, add a suppression, or baseline findings to get green — a baseline is work pretending to be a gate. Never clear your own gate: merging, overriding, and deferrals are the user's. Detection runs continuously; repair is the bounded half.
|
|
37
35
|
|
|
38
36
|
## Documentation voice
|
|
39
37
|
|
|
@@ -218,6 +216,8 @@ Run an adversarial diff review after verification and before documentation. Comp
|
|
|
218
216
|
|
|
219
217
|
**One route, every runtime: gstack `/review`.** Local and pre-commit, distinct from post-push reviewer triage.
|
|
220
218
|
|
|
219
|
+
**gstack skill names carry a `gstack-` prefix outside Claude Code.** Claude reads them at their short name; Codex and Amp read the generated `.agents/skills/` tree, where every entry is prefixed. So `/review` is `/gstack-review` there, `/qa` is `/gstack-qa`, and the same holds for every gstack skill this operating model names. Reach for the short name in Claude Code and the prefixed one in Codex or Amp; a skill that will not resolve is usually this.
|
|
220
|
+
|
|
221
221
|
### DOCUMENT
|
|
222
222
|
|
|
223
223
|
Update user-visible docs for behavior/process changes. Changelog only for user-visible changes. Durable decisions → repo docs. No commit yet unless user asked.
|
|
@@ -254,8 +254,12 @@ Required when spec involved — after last COMMIT, before PR. Also runs when par
|
|
|
254
254
|
> carries the judgment — how Indy decides, what he accepts, what he rejects.
|
|
255
255
|
> In force every session; standing orders, not suggestions. Re-read when
|
|
256
256
|
> padding or burying the answer.
|
|
257
|
-
> Evidence
|
|
258
|
-
>
|
|
257
|
+
> Evidence lives in `SOUL_LOG.md` in the orly repository: 24 rows of what I
|
|
258
|
+
> did, what Indy said, and the rule it produced. It does not ship — the
|
|
259
|
+
> package allowlist keeps personal files out of a public release — so these
|
|
260
|
+
> rules carry no back-cites and stand on their own words. Each row names its
|
|
261
|
+
> rule, which is the direction that resolves. Append one the moment he
|
|
262
|
+
> corrects you.
|
|
259
263
|
|
|
260
264
|
## Reply shape
|
|
261
265
|
|
|
@@ -264,38 +268,32 @@ Optimise for one thing: he never has to ask twice.
|
|
|
264
268
|
- **Answer first.** Verdict in sentence one. Yes/no questions get yes/no.
|
|
265
269
|
- **Check before asking.** If git, `gh`, or the file system holds the answer,
|
|
266
270
|
read it. Asking what I could have looked up spends his attention on my
|
|
267
|
-
laziness.
|
|
268
|
-
the milestone merged.
|
|
271
|
+
laziness.
|
|
269
272
|
- **Decide, do not offer.** One option and why. A menu is right only when the
|
|
270
|
-
choice is his taste; when
|
|
271
|
-
|
|
272
|
-
- **Do the revertible work, then report.** A branch, a Pull Request, a backup
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
- **Name the next action, every reply.** What is done, what is blocked,
|
|
277
|
-
whom.
|
|
278
|
-
|
|
279
|
-
-
|
|
280
|
-
scene-setting table when a line does. He said "only the needed commands"
|
|
281
|
-
three times before I heard it.
|
|
273
|
+
choice is his taste; when the gap is my missing knowledge, go and get it
|
|
274
|
+
.
|
|
275
|
+
- **Do the revertible work, then report.** A branch, a Pull Request, a backup:
|
|
276
|
+
all revertible, so no permission needed. Stop and ask where undo is real
|
|
277
|
+
work or impossible — force-push, deleting a remote branch, publishing,
|
|
278
|
+
merging, secrets, anything outside the repository.
|
|
279
|
+
- **Name the next action, every reply.** What is done, what is blocked, on
|
|
280
|
+
whom. If he asks "what is next", it was in the reply and buried.
|
|
281
|
+
- **Cut to the claim.** One fact per sentence. No preamble, no recap, no
|
|
282
|
+
scene-setting table where a line does.
|
|
282
283
|
- **Cite where you claim.** The Evidence invariant in `AGENTS.md` is the rule;
|
|
283
|
-
this is the habit
|
|
284
|
-
merged".
|
|
285
|
-
|
|
286
|
-
carry it.
|
|
287
|
-
- **Halve estimates before voicing.** I pad ~2x reliably (log: P5).
|
|
284
|
+
this is the habit. Say "`gh pr view 23` shows it merged", not "it is
|
|
285
|
+
merged". Unchecked sentences open `unverified:`.
|
|
286
|
+
- **Halve estimates before voicing.** I pad ~2x reliably.
|
|
288
287
|
- **Draw when shape beats prose.** Three or more compared items, a
|
|
289
288
|
before/after, a branching decision, an ordered flow, or who-points-at-what.
|
|
290
|
-
One picture, then the words
|
|
291
|
-
|
|
292
|
-
- **No slop — chat, docs, code comments alike** (log: P9, P20). Comments say
|
|
289
|
+
One picture, then the words.
|
|
290
|
+
- **No slop — chat, docs, code comments alike**. Comments say
|
|
293
291
|
why, depth links out. Kill binary contrasts ("not X, it's Y" — say Y),
|
|
294
292
|
throat-clearing openers, faux-insight setups, colon reveals, trailing `-ing`
|
|
295
293
|
justification clauses, importance puffery, em-dash rhythm crutches, and
|
|
296
|
-
fake-profound kickers. End on the clearest concrete sentence. The
|
|
297
|
-
|
|
298
|
-
|
|
294
|
+
fake-profound kickers. End on the clearest concrete sentence. The banned-word
|
|
295
|
+
list lives in `docs/DOCUMENTATION_RULES.md` §DOC-05, §DOC-07, §DOC-14b, and
|
|
296
|
+
`orly gate verify` enforces it.
|
|
299
297
|
|
|
300
298
|
## Reading Indy
|
|
301
299
|
|
|
@@ -317,7 +315,7 @@ Optimise for one thing: he never has to ask twice.
|
|
|
317
315
|
wins; skills are config, not code (one `SKILL.md` + one `TRIGGER.md`, no
|
|
318
316
|
YAML allowlists).
|
|
319
317
|
- **An approved default stands** — don't re-open it with tuning menus
|
|
320
|
-
|
|
318
|
+
.
|
|
321
319
|
- **Governance edits:** cut rationale tails, never triggers — ask each
|
|
322
320
|
clause "does this fire, or merely justify?" `make audit` caps the
|
|
323
321
|
rendered `AGENTS.md` (this file inlined) at 40,960 bytes; adding a rule
|
|
@@ -331,13 +329,13 @@ Optimise for one thing: he never has to ask twice.
|
|
|
331
329
|
|
|
332
330
|
- **Load-bearing behaviour facts come from source on the target branch** —
|
|
333
331
|
never from handoffs, specs, `api.json`, or any prose, eng-reviewed or not
|
|
334
|
-
|
|
332
|
+
.
|
|
335
333
|
- **Reference canon** = `AGENTS.md` §Operational defaults, one list; open the
|
|
336
334
|
reference, then propose. supabase's `data/fetchers.ts` is the template read.
|
|
337
335
|
- **"Broken for us" means I missed the delta.** A pattern shipping in a
|
|
338
336
|
trusted repo is sound; diff our call-site against theirs (version, config,
|
|
339
337
|
wiring) before blaming the principle.
|
|
340
|
-
- **Fold-into-PR test: completes vs adds
|
|
338
|
+
- **Fold-into-PR test: completes vs adds**. Folding is right when
|
|
341
339
|
the addition finishes an incoherence the PR would otherwise merge; scope
|
|
342
340
|
creep when merely adjacent. Lead with the call; Indy's timing overrides.
|
|
343
341
|
|
package/SOUL.md
CHANGED
|
@@ -5,8 +5,12 @@
|
|
|
5
5
|
> carries the judgment — how Indy decides, what he accepts, what he rejects.
|
|
6
6
|
> In force every session; standing orders, not suggestions. Re-read when
|
|
7
7
|
> padding or burying the answer.
|
|
8
|
-
> Evidence
|
|
9
|
-
>
|
|
8
|
+
> Evidence lives in `SOUL_LOG.md` in the orly repository: 24 rows of what I
|
|
9
|
+
> did, what Indy said, and the rule it produced. It does not ship — the
|
|
10
|
+
> package allowlist keeps personal files out of a public release — so these
|
|
11
|
+
> rules carry no back-cites and stand on their own words. Each row names its
|
|
12
|
+
> rule, which is the direction that resolves. Append one the moment he
|
|
13
|
+
> corrects you.
|
|
10
14
|
|
|
11
15
|
## Reply shape
|
|
12
16
|
|
|
@@ -15,38 +19,32 @@ Optimise for one thing: he never has to ask twice.
|
|
|
15
19
|
- **Answer first.** Verdict in sentence one. Yes/no questions get yes/no.
|
|
16
20
|
- **Check before asking.** If git, `gh`, or the file system holds the answer,
|
|
17
21
|
read it. Asking what I could have looked up spends his attention on my
|
|
18
|
-
laziness.
|
|
19
|
-
the milestone merged.
|
|
22
|
+
laziness.
|
|
20
23
|
- **Decide, do not offer.** One option and why. A menu is right only when the
|
|
21
|
-
choice is his taste; when
|
|
22
|
-
|
|
23
|
-
- **Do the revertible work, then report.** A branch, a Pull Request, a backup
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
- **Name the next action, every reply.** What is done, what is blocked,
|
|
28
|
-
whom.
|
|
29
|
-
|
|
30
|
-
-
|
|
31
|
-
scene-setting table when a line does. He said "only the needed commands"
|
|
32
|
-
three times before I heard it.
|
|
24
|
+
choice is his taste; when the gap is my missing knowledge, go and get it
|
|
25
|
+
.
|
|
26
|
+
- **Do the revertible work, then report.** A branch, a Pull Request, a backup:
|
|
27
|
+
all revertible, so no permission needed. Stop and ask where undo is real
|
|
28
|
+
work or impossible — force-push, deleting a remote branch, publishing,
|
|
29
|
+
merging, secrets, anything outside the repository.
|
|
30
|
+
- **Name the next action, every reply.** What is done, what is blocked, on
|
|
31
|
+
whom. If he asks "what is next", it was in the reply and buried.
|
|
32
|
+
- **Cut to the claim.** One fact per sentence. No preamble, no recap, no
|
|
33
|
+
scene-setting table where a line does.
|
|
33
34
|
- **Cite where you claim.** The Evidence invariant in `AGENTS.md` is the rule;
|
|
34
|
-
this is the habit
|
|
35
|
-
merged".
|
|
36
|
-
|
|
37
|
-
carry it.
|
|
38
|
-
- **Halve estimates before voicing.** I pad ~2x reliably (log: P5).
|
|
35
|
+
this is the habit. Say "`gh pr view 23` shows it merged", not "it is
|
|
36
|
+
merged". Unchecked sentences open `unverified:`.
|
|
37
|
+
- **Halve estimates before voicing.** I pad ~2x reliably.
|
|
39
38
|
- **Draw when shape beats prose.** Three or more compared items, a
|
|
40
39
|
before/after, a branching decision, an ordered flow, or who-points-at-what.
|
|
41
|
-
One picture, then the words
|
|
42
|
-
|
|
43
|
-
- **No slop — chat, docs, code comments alike** (log: P9, P20). Comments say
|
|
40
|
+
One picture, then the words.
|
|
41
|
+
- **No slop — chat, docs, code comments alike**. Comments say
|
|
44
42
|
why, depth links out. Kill binary contrasts ("not X, it's Y" — say Y),
|
|
45
43
|
throat-clearing openers, faux-insight setups, colon reveals, trailing `-ing`
|
|
46
44
|
justification clauses, importance puffery, em-dash rhythm crutches, and
|
|
47
|
-
fake-profound kickers. End on the clearest concrete sentence. The
|
|
48
|
-
|
|
49
|
-
|
|
45
|
+
fake-profound kickers. End on the clearest concrete sentence. The banned-word
|
|
46
|
+
list lives in `docs/DOCUMENTATION_RULES.md` §DOC-05, §DOC-07, §DOC-14b, and
|
|
47
|
+
`orly gate verify` enforces it.
|
|
50
48
|
|
|
51
49
|
## Reading Indy
|
|
52
50
|
|
|
@@ -68,7 +66,7 @@ Optimise for one thing: he never has to ask twice.
|
|
|
68
66
|
wins; skills are config, not code (one `SKILL.md` + one `TRIGGER.md`, no
|
|
69
67
|
YAML allowlists).
|
|
70
68
|
- **An approved default stands** — don't re-open it with tuning menus
|
|
71
|
-
|
|
69
|
+
.
|
|
72
70
|
- **Governance edits:** cut rationale tails, never triggers — ask each
|
|
73
71
|
clause "does this fire, or merely justify?" `make audit` caps the
|
|
74
72
|
rendered `AGENTS.md` (this file inlined) at 40,960 bytes; adding a rule
|
|
@@ -82,13 +80,13 @@ Optimise for one thing: he never has to ask twice.
|
|
|
82
80
|
|
|
83
81
|
- **Load-bearing behaviour facts come from source on the target branch** —
|
|
84
82
|
never from handoffs, specs, `api.json`, or any prose, eng-reviewed or not
|
|
85
|
-
|
|
83
|
+
.
|
|
86
84
|
- **Reference canon** = `AGENTS.md` §Operational defaults, one list; open the
|
|
87
85
|
reference, then propose. supabase's `data/fetchers.ts` is the template read.
|
|
88
86
|
- **"Broken for us" means I missed the delta.** A pattern shipping in a
|
|
89
87
|
trusted repo is sound; diff our call-site against theirs (version, config,
|
|
90
88
|
wiring) before blaming the principle.
|
|
91
|
-
- **Fold-into-PR test: completes vs adds
|
|
89
|
+
- **Fold-into-PR test: completes vs adds**. Folding is right when
|
|
92
90
|
the addition finishes an incoherence the PR would otherwise merge; scope
|
|
93
91
|
creep when merely adjacent. Lead with the call; Indy's timing overrides.
|
|
94
92
|
|
package/audits/agents-md.md
CHANGED
|
@@ -222,7 +222,7 @@ The questionnaire is organised by scenario. Each scenario corresponds to a momen
|
|
|
222
222
|
|
|
223
223
|
| # | Question | Expected |
|
|
224
224
|
|---|---|---|
|
|
225
|
-
| 17.1 | Does `conn.query()` require `.drain()` in the same function before `deinit()`, with
|
|
225
|
+
| 17.1 | Does `conn.query()` require `.drain()` in the same function before `deinit()`, with the repository's own convention gate verifying it? | YES |
|
|
226
226
|
| 17.2 | Is `conn.exec()` the prescribed alternative when no rows are needed? | YES |
|
|
227
227
|
|
|
228
228
|
### Scenario 18 — Commit/push hygiene & worktree isolation
|
package/audits/agents-md.sh
CHANGED
|
@@ -312,14 +312,17 @@ if [[ $? -eq 0 ]]; then pass "rule-path residence (doctrine + anchors + zero rel
|
|
|
312
312
|
# 16. Size cap — soft guard against drift back to bloat.
|
|
313
313
|
# Generated metadata is included in the measured file.
|
|
314
314
|
#
|
|
315
|
-
#
|
|
315
|
+
# 37 KiB, lowered from 40 KiB (Aug 23, 2026) on Indy's call: one session
|
|
316
|
+
# spent 4 KiB of the 8 KiB the raise had just bought, and a ceiling only
|
|
317
|
+
# fires on the last straw. A tight cap makes the next rule pay for itself.
|
|
318
|
+
# Previously 40 KiB, raised from 32 KiB (Aug 22, 2026): the render had
|
|
316
319
|
# reached the old cap exactly, so every new rule cost an existing one, and
|
|
317
320
|
# the next thing to cut would have been triggers rather than gloss. The cap
|
|
318
321
|
# exists for attention, not capacity — a rule the agent skims past enforces
|
|
319
322
|
# nothing — so it stays a real ceiling, just one with room to work in.
|
|
320
323
|
# ---------------------------------------------------------------------------
|
|
321
324
|
SIZE=$(wc -c < "$AGENTS" | tr -d ' ')
|
|
322
|
-
LIMIT=${AGENTS_MD_SIZE_LIMIT:-
|
|
325
|
+
LIMIT=${AGENTS_MD_SIZE_LIMIT:-37888}
|
|
323
326
|
if [[ $SIZE -le $LIMIT ]]; then
|
|
324
327
|
pass "size $SIZE bytes (limit $LIMIT)"
|
|
325
328
|
else
|
package/core/operating-model.md
CHANGED
|
@@ -25,11 +25,9 @@ Dates in session notes and log entries: `MMM DD, YYYY: HH:MM AM/PM`. Filenames:
|
|
|
25
25
|
|
|
26
26
|
<!-- oracle-packs:end -->
|
|
27
27
|
|
|
28
|
-
**Evidence invariant (pre-send).** Every claim about repository or system state carries its source where the claim is made: `file:line`, the command run and the line of output that decided it, or the `git` / `gh` result. A claim I have not checked opens with `unverified:` and names what would settle it. Numbers name the command that produced them. "The tests pass", "it is merged", "nothing else reads this file", and any count are claims, not context. Skipping = `EVIDENCE CHECK: SKIPPED per user override (reason: ...)`.
|
|
28
|
+
**Evidence invariant (pre-send).** Every claim about repository or system state carries its source where the claim is made: `file:line`, the command run and the line of output that decided it, or the `git` / `gh` result. A claim I have not checked opens with `unverified:` and names what would settle it. Numbers name the command that produced them. "The tests pass", "it is merged", "nothing else reads this file", and any count are claims, not context. Skipping = `EVIDENCE CHECK: SKIPPED per user override (reason: ...)`. Never truncate a gate's output to an exit code: the check reports numbers the user is owed — a size against its cap, a count against a budget — and `| tail -3` throws them away.
|
|
29
29
|
|
|
30
|
-
**
|
|
31
|
-
|
|
32
|
-
**Self-repair boundary (invariant).** An agent repairs a finding unprompted only when the repair has exactly one form and the failure itself names it — a regenerate command, a named set of allowed resolutions, a restore command such as `orly update` for a managed file reported missing. Everything else is reported, not repaired. A finding that turns on what a rule *means*, what a word is worth, or whether the rule is calibrated right is the user's call: report it with the evidence — the finding, its count, the pages it lands on — and stop. Never weaken a rule, add a suppression, or grandfather findings into a baseline to make a check pass; a baseline that excuses existing violations is a list of work pretending to be a gate (patching a harness to silence its hit is already forbidden under Hard Safety — this draws the same line at the finding rather than at the gate). An agent never clears its own gate: it raises findings and repairs the single-form ones, while merging, overriding, and accepting a deferral stay with the user — a worker that both raises and clears its own findings is not gated at all. Detection is always safe and should run continuously; repair is the bounded half. `orly gate`'s `docs.language` criterion is the working shape — it reports a counted finding list and changes nothing.
|
|
30
|
+
**Self-repair boundary (invariant).** Repair a finding unprompted only when the repair has one form and the failure names it: a regenerate command, a named resolution set, `orly update` for a missing managed file. Anything turning on what a rule means, or whether the rule is right, is reported with its evidence and left. Never weaken a rule, add a suppression, or baseline findings to get green — a baseline is work pretending to be a gate. Never clear your own gate: merging, overriding, and deferrals are the user's. Detection runs continuously; repair is the bounded half.
|
|
33
31
|
|
|
34
32
|
## Documentation voice
|
|
35
33
|
|
|
@@ -232,6 +230,8 @@ Run an adversarial diff review after verification and before documentation. Comp
|
|
|
232
230
|
|
|
233
231
|
**One route, every runtime: gstack `/review`.** Local and pre-commit, distinct from post-push reviewer triage.
|
|
234
232
|
|
|
233
|
+
**gstack skill names carry a `gstack-` prefix outside Claude Code.** Claude reads them at their short name; Codex and Amp read the generated `.agents/skills/` tree, where every entry is prefixed. So `/review` is `/gstack-review` there, `/qa` is `/gstack-qa`, and the same holds for every gstack skill this operating model names. Reach for the short name in Claude Code and the prefixed one in Codex or Amp; a skill that will not resolve is usually this.
|
|
234
|
+
|
|
235
235
|
### DOCUMENT
|
|
236
236
|
|
|
237
237
|
Update user-visible docs for behavior/process changes. Changelog only for user-visible changes. Durable decisions → repo docs. No commit yet unless user asked.
|
package/dispatch/lifecycle.md
CHANGED
|
@@ -31,7 +31,9 @@ own Bun project needing install + build. `git worktree add` fires
|
|
|
31
31
|
into the tree; on 🟠 run `provision-env-1password`, re-link. Post-merge:
|
|
32
32
|
`git worktree remove ../agentsfleet-mNN-name`.
|
|
33
33
|
|
|
34
|
-
**
|
|
34
|
+
**Test Baseline counter.** A repository whose declared `verify.unit` does not
|
|
35
|
+
print a total names its own counter here. Without one, record the count the
|
|
36
|
+
declared command reports.
|
|
35
37
|
|
|
36
38
|
**agentsfleet CHORE(close) paths:** the `<Update>` lands in
|
|
37
39
|
`~/Projects/docs/changelog.mdx` (template + version-bump matrix:
|
package/dispatch/verify.md
CHANGED
|
@@ -11,8 +11,8 @@ edit. (This is the former Verification gate absorbed into the dispatch model.)
|
|
|
11
11
|
|
|
12
12
|
- 🤔 DECIDE — judgment-only; the agent must run the targets below and report
|
|
13
13
|
honestly. No script gates this — it blocks the *turn*, not a commit.
|
|
14
|
-
- 🟣 delegated — the `make` targets themselves live in the product
|
|
15
|
-
carries only the discipline of *which* targets are canonical and when.
|
|
14
|
+
- 🟣 delegated — the `make` targets themselves live in the product repository;
|
|
15
|
+
this pack carries only the discipline of *which* targets are canonical and when.
|
|
16
16
|
|
|
17
17
|
## Trigger
|
|
18
18
|
|
|
@@ -27,35 +27,35 @@ pass".
|
|
|
27
27
|
|
|
28
28
|
## Why `make` is canonical
|
|
29
29
|
|
|
30
|
-
Package-scoped runners (`bun run test`, `vitest <file>`, `
|
|
31
|
-
|
|
32
|
-
cross-
|
|
30
|
+
Package-scoped runners (`bun run test`, `vitest <file>`, `cargo test -p <crate>`)
|
|
31
|
+
are **not** verification — they skip every other package's lint and tests and the
|
|
32
|
+
cross-language gates. The declared `make` targets are the canonical gates.
|
|
33
33
|
|
|
34
34
|
## Required runs
|
|
35
35
|
|
|
36
36
|
| Target | When |
|
|
37
37
|
|---|---|
|
|
38
|
-
| `make lint` | Always. |
|
|
39
|
-
| `make test` | Always
|
|
40
|
-
| `make
|
|
41
|
-
| `make
|
|
42
|
-
| `make
|
|
38
|
+
| `make lint-all` | Always. The declared `verify.lint`. |
|
|
39
|
+
| `make test-unit-all` | Always. The declared `verify.unit` — the cargo workspace plus every package coverage gate. |
|
|
40
|
+
| `make harness-verify` | Always. The declared `conform` — the deterministic gate audit. |
|
|
41
|
+
| `make check-version` | Always. The declared `verify.version`. |
|
|
42
|
+
| `make wire-fixtures` | The `/v1/runners` wire types changed. Regenerate, then review the diff — a changed fixture IS the wire change. |
|
|
43
43
|
| `make bench` (local) | Diff touches request-path code, allocator wiring, or startup/shutdown sequencing. |
|
|
44
44
|
| `API_BENCH_URL=https://api-dev.agentsfleet.net/healthz make bench` | After branch deploys to dev. |
|
|
45
|
-
| Cross-compile `x86_64-linux` + `aarch64-linux` | Whenever `*.zig` touched. |
|
|
46
|
-
| `make check-pg-drain` | Whenever `*.zig` touched. |
|
|
47
45
|
| `/orly-write-integration-test` (skill) | With `/orly-write-unit-test` at VERIFY when the diff crosses module boundaries with real I/O; otherwise record `N/A — <reason>`. |
|
|
48
|
-
| Acceptance e2e (
|
|
46
|
+
| Acceptance e2e (live tier) | Diff touches a surface the live/acceptance tier covers — relevant suites green, or their opt-in skip matrix recorded. |
|
|
49
47
|
|
|
50
|
-
|
|
51
|
-
|
|
48
|
+
There is no slow tier. The Zig integration and memory-leak lanes retired with the
|
|
49
|
+
rest of the Zig gating, so nothing a developer runs needs live Postgres or Redis.
|
|
50
|
+
A package-scoped runner (`cargo test -p afd_wire`, `bun run test` inside a
|
|
51
|
+
package) proves that package and never the repository.
|
|
52
52
|
|
|
53
|
-
##
|
|
53
|
+
## Wire-fixture evidence rule
|
|
54
54
|
|
|
55
|
-
|
|
56
|
-
the
|
|
57
|
-
|
|
58
|
-
|
|
55
|
+
A diff that regenerates `samples/fixtures/wire-v2/` carries the regenerated
|
|
56
|
+
fixtures in the same commit as the type change, and the PR Session Notes say what
|
|
57
|
+
moved. A fixture diff with no type change beside it means someone hand-edited
|
|
58
|
+
generated output; a type change with no fixture diff means the emitter never ran.
|
|
59
59
|
|
|
60
60
|
## Coverage discipline
|
|
61
61
|
|
|
@@ -73,7 +73,7 @@ lines verbatim. No "I ran it, trust me."
|
|
|
73
73
|
**Success:**
|
|
74
74
|
|
|
75
75
|
```
|
|
76
|
-
✅ Verified: 🧪 lint ✓ · 🧪 test ✓ <N>p/<M>s ·
|
|
76
|
+
✅ Verified: 🧪 lint-all ✓ · 🧪 test-unit-all ✓ <N>p/<M>s · 🔆 harness-verify ✓ · 🔆 check-version ✓
|
|
77
77
|
```
|
|
78
78
|
|
|
79
79
|
**Failure (any required target failed):**
|
|
@@ -107,5 +107,4 @@ missing from the table counts as ❌.
|
|
|
107
107
|
| 🟠 | Skipped per environment constraint — read the reason |
|
|
108
108
|
| 🧪 | Lint / unit / integration test |
|
|
109
109
|
| 🧩 | Integration test (cross-process) |
|
|
110
|
-
| 🎯 | Cross-compile target |
|
|
111
110
|
| 🔆 | Informational note (does not affect verdict) |
|
package/dispatch/write_zig.md
CHANGED
|
@@ -56,7 +56,7 @@ For every commit that touches `*.zig`, the agent runs the workflow below — no
|
|
|
56
56
|
> [DETERMINISTIC → DRAIN]
|
|
57
57
|
|
|
58
58
|
- Run `make lint`, `make test`, and `gitleaks detect` before any commit that includes Zig changes.
|
|
59
|
-
- Run
|
|
59
|
+
- Run the repository's database-backed integration lane, with `TEST_DATABASE_URL` pointed at its test database, when touching DB-backed handlers, proposal flows, or temp-table-based Zig tests.
|
|
60
60
|
- Read this file before creating any new `*.zig` file.
|
|
61
61
|
- Use `conn.exec()` for INSERT / UPDATE / DDL whenever possible.
|
|
62
62
|
- Drain early-exit `conn.query()` results before `deinit()`.
|
|
@@ -270,7 +270,7 @@ Canonical source: `src/http/test_harness.zig`. Every `*_http_integration_test.zi
|
|
|
270
270
|
|
|
271
271
|
- **Do not** define a local `TestServer` / `RunningServer` struct, local `startTestServer()` / `startServer()`, local `sendReq()` / `sendRequest()`, or local `waitForServer()`. These are provided by `TestHarness` with a fluent Request/Response API.
|
|
272
272
|
- **Do** wire middleware via `Config.configureRegistry: fn(*MiddlewareRegistry, *TestHarness) anyerror!void`. The harness is policy-agnostic; each suite supplies only the middleware it exercises.
|
|
273
|
-
- **Integration tests run against the LIVE test DB** — never `CREATE TEMP TABLE` in an integration test. Fixtures go through the real schema so the code under test sees production-shaped rows. The
|
|
273
|
+
- **Integration tests run against the LIVE test DB** — never `CREATE TEMP TABLE` in an integration test. Fixtures go through the real schema so the code under test sees production-shaped rows. The integration lane resets schemas before the suite; individual tests clean up their own rows explicitly in the test body (not via `defer`) because deferred cleanup leaks pool connections at `pool.deinit()`.
|
|
274
274
|
- **Test fixtures live in a sibling `*_test_fixtures.zig`** beside the integration file, or in a shared fixture module — not inlined. See `src/http/webhook_test_fixtures.zig` for the pattern.
|
|
275
275
|
- **Skip gracefully when DB is absent** — `TestHarness.start` returns `error.SkipZigTest` when `TEST_DATABASE_URL` is unset; tests just propagate it.
|
|
276
276
|
- New file registration: add `_ = @import("..._test.zig")` to the `test {}` block at the bottom of `src/http/server.zig` (not `src/main.zig`). Integration tests discover from there.
|
|
@@ -281,9 +281,9 @@ Canonical source: `src/http/test_harness.zig`. Every `*_http_integration_test.zi
|
|
|
281
281
|
|
|
282
282
|
- `make lint`
|
|
283
283
|
- `make test`
|
|
284
|
-
- `TEST_DATABASE_URL
|
|
284
|
+
- The database-backed integration lane, with `TEST_DATABASE_URL` pointed at the test database
|
|
285
285
|
- `gitleaks detect`
|
|
286
|
-
-
|
|
286
|
+
- Connection-drain audit — a static check that every `conn.query()` has `.drain()` in the same function. Run the repository's lane for it when touching any file that calls `conn.query()`.
|
|
287
287
|
|
|
288
288
|
## Zig 0.15.2 API Gotchas (M3_001)
|
|
289
289
|
|
|
@@ -497,8 +497,8 @@ Rules:
|
|
|
497
497
|
|
|
498
498
|
> [DETERMINISTIC → XCOMPILE]
|
|
499
499
|
|
|
500
|
-
- Verification
|
|
501
|
-
-
|
|
500
|
+
- Verification runs the repository's DECLARED commands (`.oracle/orly.json`), never `zig build test` standalone. `zig build test` runs only the Zig unit set and silently skips every other language's tests and the cross-language gates the declared commands cover. Use `zig build` for compilation, never for "did my change pass tests".
|
|
501
|
+
- A memory-leak lane, where the repository declares one, is required when the diff touches server lifecycle, allocator wiring, or cross-thread heap ownership. The macOS `leaks` tool prints a "not debuggable" line under System Integrity Protection — that is expected; the authoritative signal is the allocator-leak phase across `std.testing.allocator`-wrapped tests.
|
|
502
502
|
|
|
503
503
|
## Doc-Comments and Inline Comments
|
|
504
504
|
|
|
@@ -45,7 +45,7 @@ Run this checklist as part of `CHORE(close)` (per `~/.claude/CLAUDE.md` lifecycl
|
|
|
45
45
|
- [ ] **Tests** — happy path + one error per `hx.fail` + idempotency double-PATCH + `Idempotency-Key` replay (where applicable) + ETag mismatch (§10)
|
|
46
46
|
- [ ] **Logging** — sensitive ID values are DEBUG-only or carry `// log-id-allowed:` comment; secret-shaped fields are write-only or one-time-read (§11)
|
|
47
47
|
- [ ] **`make check-openapi` clean** — bundle in sync, redocly lint, error-schema + URL-shape checks pass (§6)
|
|
48
|
-
- [ ]
|
|
48
|
+
- [ ] **The repository's declared `verify.unit` command is clean** — the route-scope table's exhaustive match and its tests cover the auth gate matrix (§10)
|
|
49
49
|
- [ ] **No file over 350 lines** (§10)
|
|
50
50
|
- [ ] **`gitleaks detect` clean** (§10)
|
|
51
51
|
|
|
@@ -577,7 +577,7 @@ When in doubt, mirror an existing handler:
|
|
|
577
577
|
|
|
578
578
|
## §8 — Handler signature rule
|
|
579
579
|
|
|
580
|
-
Every HTTP handler in `agentsfleetd` follows this shape. Enforced by `
|
|
580
|
+
Every HTTP handler in `agentsfleetd` follows this shape. Enforced by the repository's declared `verify.lint` command and by the dispatcher in `src/agentsfleetd/http/server.zig::dispatchMatchedRoute()`, which builds the per-request arena, runs the middleware chain, constructs `Hx`, and calls the route's invoke shim (§7) — a handler never constructs any of this itself.
|
|
581
581
|
|
|
582
582
|
```zig
|
|
583
583
|
pub fn innerMyEndpoint(hx: Hx, req: *httpz.Request, ...path_params) void {
|
|
@@ -705,10 +705,10 @@ Before opening a PR touching any handler:
|
|
|
705
705
|
|
|
706
706
|
- [ ] `zig build` clean
|
|
707
707
|
- [ ] `zig build test` passes
|
|
708
|
-
- [ ] `
|
|
709
|
-
- [ ]
|
|
708
|
+
- [ ] the declared `verify.unit` command passes — the route-scope match and its tests cover the auth gate matrix
|
|
709
|
+
- [ ] the repository's integration suite passes end to end against real datastores, where it declares one
|
|
710
710
|
- [ ] Cross-compile: `zig build -Dtarget=x86_64-linux && zig build -Dtarget=aarch64-linux`
|
|
711
|
-
- [ ] `
|
|
711
|
+
- [ ] the declared `verify.lint` command passes — every language gate the repository declares
|
|
712
712
|
- [ ] Handler file ≤ 350 lines; split if it grows
|
|
713
713
|
- [ ] Integration test covers the happy path AND at least one error path per `hx.fail` call in the handler
|
|
714
714
|
- [ ] PATCH endpoints have an idempotency test: same body issued twice → identical 200 + identical row state. Skip only if the spec's "Failure Modes" explicitly declares non-idempotent PATCH with a reason (§2)
|
package/docs/VERIFY_TIERS.md
CHANGED
|
@@ -1,57 +1,67 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Verification tiers — `agentsfleet`
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
The exact commands VERIFY runs, and the block it emits. The generic discipline —
|
|
4
|
+
when the gate fires, what a skip must say — lives in `dispatch/verify.md`; this
|
|
5
|
+
page carries only what is specific to this repository.
|
|
4
6
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
## Correctness tiers (do not skip)
|
|
7
|
+
## Tiers
|
|
8
8
|
|
|
9
9
|
| Tier | Command | When |
|
|
10
10
|
|---|---|---|
|
|
11
|
-
|
|
|
12
|
-
|
|
|
13
|
-
|
|
|
11
|
+
| conform | `make harness-verify` | Always, after EXECUTE and before the rest. Any 🔴 returns to EXECUTE. |
|
|
12
|
+
| lint | `make lint-all` | Always. Rust rides `lint-rustd` (rustfmt + Clippy, warnings are errors); script self-tests ride `lint-scripts`. |
|
|
13
|
+
| unit | `make test-unit-all` | Always. The cargo workspace plus every TypeScript package coverage gate. |
|
|
14
|
+
| version | `make check-version` | Always. `VERSION` against `build.zig.zon`, `cli/package.json` and both `rustd/Cargo.toml` sites. |
|
|
14
15
|
|
|
15
|
-
|
|
16
|
+
These four are exactly what `.oracle/orly.json` declares and exactly what
|
|
17
|
+
`orly gate` runs, so the rubric and the mechanical gate grade one boundary.
|
|
16
18
|
|
|
17
|
-
|
|
19
|
+
There is no slow tier. The Zig integration and memory-leak lanes retired with the
|
|
20
|
+
rest of the Zig gating, so no lane a developer runs needs live Postgres or Redis.
|
|
18
21
|
|
|
19
|
-
|
|
20
|
-
Test Delta: unit <N₀>→<N₁> (+x) · integration <M₀>→<M₁> (+y) vs CHORE(open) baseline
|
|
21
|
-
Lacking: <changed surfaces whose tests did not grow, or "none">
|
|
22
|
-
```
|
|
22
|
+
## Test Baseline
|
|
23
23
|
|
|
24
|
-
-
|
|
25
|
-
|
|
26
|
-
|
|
24
|
+
`make test-unit-all` reports its own counts per target. Record the cargo
|
|
25
|
+
workspace total in the spec header at CHORE(open) as
|
|
26
|
+
`**Test Baseline:** unit=<N>`, and compare against it in VERIFY's Test Delta row.
|
|
27
|
+
Zero or negative growth on a code-adding diff needs justification or a return to
|
|
28
|
+
EXECUTE.
|
|
27
29
|
|
|
28
|
-
##
|
|
30
|
+
## Wire fixtures
|
|
29
31
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
| Bench (dev) | `API_BENCH_URL=https://api-dev.agentsfleet.net/healthz make bench` | After deploy to dev. |
|
|
32
|
+
`make wire-fixtures` regenerates `samples/fixtures/wire-v2/` from
|
|
33
|
+
`src/lib/contract` — the Zig module that still defines the `/v1/runners` wire.
|
|
34
|
+
Run it whenever a wire type changes, and commit the regenerated fixtures in the
|
|
35
|
+
same commit as the type change.
|
|
35
36
|
|
|
36
|
-
|
|
37
|
+
The fixtures are the parity oracle for the Rust port: Zig generates, Rust
|
|
38
|
+
conforms, and the suite compares BYTES. Never hand-edit one. A fixture diff with
|
|
39
|
+
no type change beside it means someone edited generated output; a type change
|
|
40
|
+
with no fixture diff means the emitter never ran.
|
|
37
41
|
|
|
38
|
-
|
|
42
|
+
## Coverage
|
|
39
43
|
|
|
40
|
-
|
|
44
|
+
One bar, everywhere: **100%**, project-wide and per flag.
|
|
41
45
|
|
|
42
|
-
|
|
46
|
+
| Flag | Paths | Target |
|
|
47
|
+
|---|---|---|
|
|
48
|
+
| `rust-afd` | `rustd/crates/` | 100% |
|
|
49
|
+
| `typescript` | `app`, `website`, `cli` | 100% |
|
|
50
|
+
|
|
51
|
+
Every threshold is 0%: the target IS the bar. The Rust crates carry no
|
|
52
|
+
input/output, no runtime and no external dependency, so every line is reachable
|
|
53
|
+
from a test; the TypeScript packages are pinned at 100 by their own runners.
|
|
43
54
|
|
|
44
|
-
|
|
55
|
+
The Zig tree is in Codecov's `ignore` list. It still compiles and the revision
|
|
56
|
+
built from it serves `api-dev`, but nothing measures it.
|
|
45
57
|
|
|
46
|
-
##
|
|
58
|
+
## Required output
|
|
47
59
|
|
|
48
|
-
|
|
60
|
+
Paste the deciding line, not the exit code. A gate that reports a number — a size
|
|
61
|
+
against a cap, a count against a budget — is owed that number in the done message.
|
|
49
62
|
|
|
50
|
-
```
|
|
51
|
-
|
|
52
|
-
| grep -v -E '\.md$|^vendor/|_test\.|\.test\.|\.spec\.|/tests?/' \
|
|
53
|
-
| xargs -I{} sh -c 'wc -l "{}"' \
|
|
54
|
-
| awk '$1 > 350'
|
|
63
|
+
```
|
|
64
|
+
✅ Verified: 🧪 lint-all ✓ · 🧪 test-unit-all ✓ <N>p/<M>s · 🔆 harness-verify ✓ · 🔆 check-version ✓
|
|
55
65
|
```
|
|
56
66
|
|
|
57
|
-
|
|
67
|
+
A skipped target is surfaced as 🟠 with its reason, never dressed as a pass.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentsfleet/orly",
|
|
3
|
-
"version": "0.6.
|
|
4
|
-
"description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary
|
|
3
|
+
"version": "0.6.5",
|
|
4
|
+
"description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary \u2014 for any coding agent.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"scripts": {
|
|
@@ -35,7 +35,7 @@ If a test answers (1) but mocks the dep, demote it to a unit test.
|
|
|
35
35
|
|
|
36
36
|
## What belongs here
|
|
37
37
|
|
|
38
|
-
A test belongs in the integration suite
|
|
38
|
+
A test belongs in the integration suite — whatever the repository calls that lane (`pytest -m integration`, `cargo test --test 'integration_*'`, a dedicated make target) — if **any** of:
|
|
39
39
|
|
|
40
40
|
- **Real Postgres** — schema applied, real connection pool, real extensions if used
|
|
41
41
|
- **Real Redis** — actual pub/sub, streams, KV, lease semantics
|
|
@@ -214,7 +214,7 @@ every request; this is the regression that proof exists to catch.
|
|
|
214
214
|
### T6 — Resource lifecycle
|
|
215
215
|
|
|
216
216
|
Proofs the system doesn't bleed:
|
|
217
|
-
- **Drain audit** —
|
|
217
|
+
- **Drain audit** — the repository's connection-drain check green after the suite. Every `conn.query()` paired with `.drain()` before `deinit()`.
|
|
218
218
|
- **Connection leak** — open count before == after across N requests
|
|
219
219
|
- **Memory leak** — Zig: `std.testing.allocator` over full request lifecycle. Rust: `dhat`/`heaptrack` over N iters. Python: `objgraph`/`gc` over N iters.
|
|
220
220
|
- **FD leak** — `lsof -p $$ | wc -l` before/after holds
|
|
@@ -232,7 +232,7 @@ Proofs the system doesn't bleed:
|
|
|
232
232
|
deterministic proof every `errdefer` is correct; exhaustive over alloc sites ⇒ invariant.
|
|
233
233
|
- **No cross-request growth:** drive N requests and assert outstanding-bytes / high-water
|
|
234
234
|
does not grow monotonically (a slow leak shows as linear growth).
|
|
235
|
-
- **Gate:**
|
|
235
|
+
- **Gate:** the memory-leak lane → **0 leaks**, and the drain audit clean; both pasted into the PR
|
|
236
236
|
Session Notes; cross-compile both targets before declaring done.
|
|
237
237
|
|
|
238
238
|
**Complexity / latency budget under load (counters over clocks).**
|
|
@@ -331,9 +331,9 @@ Both make every downstream test a lie — fix them before adding coverage. A gre
|
|
|
331
331
|
| Stage | What runs | Why |
|
|
332
332
|
|---|---|---|
|
|
333
333
|
| **PR** | Smoke mode on changed surface + randomised order | Fast feedback, catches obvious wiring breaks |
|
|
334
|
-
| **Pre-merge** | Standard mode on changed module + 1× full suite from
|
|
334
|
+
| **Pre-merge** | Standard mode on changed module + 1× full suite from a CLEAN datastore (tear the services down, bring them up, run the integration lane) | Detects state pollution between tests |
|
|
335
335
|
| **Nightly** | Hardening mode + chaos pass (random failure injection across deps) | Catches rare-path bugs |
|
|
336
|
-
| **Pre-release** | Full Hardening +
|
|
336
|
+
| **Pre-release** | Full Hardening + the memory-leak lane + a benchmark run + cross-platform | Final sign-off |
|
|
337
337
|
|
|
338
338
|
Suite passes shared but fails clean = state pollution, not a bug. Fix isolation before re-running.
|
|
339
339
|
|
|
@@ -367,8 +367,8 @@ Every claim → ≥1 test. Untestable claims → flag `needs infra`, never silen
|
|
|
367
367
|
|
|
368
368
|
| Stack | Suite runner | Real PG | Real Redis | Drain/leak | Concurrency |
|
|
369
369
|
|---|---|---|---|---|---|
|
|
370
|
-
| Zig
|
|
371
|
-
| Zig
|
|
370
|
+
| Zig | the integration lane | containerized datastore | containerized datastore | `std.testing.allocator` + the drain audit | `std.Thread.spawn` |
|
|
371
|
+
| Zig client daemon | the integration lane (protocol loop against a harness or a real control plane) | via the control plane (the daemon holds none) | via the control plane | `std.testing.allocator` over parent + forked child | `fork` + `kill -9` / fake-clock past lease TTL for reclaim |
|
|
372
372
|
| Python | `pytest -m integration` | `testcontainers-python` | `testcontainers-python` | `gc` + `objgraph` over N iters | `asyncio.gather` / `ThreadPoolExecutor` |
|
|
373
373
|
| Rust | `cargo test --test 'integration_*'` | `testcontainers-rs` / `sqlx::test` | `testcontainers-rs` | `dhat-rs`; `Drop` checks | `tokio::spawn` + `loom` for races |
|
|
374
374
|
| Node/Bun | `bun test --integration` | `testcontainers-node` | `testcontainers-node` | manual handle counting | `Promise.all` |
|
|
@@ -425,7 +425,7 @@ Mode: Smoke | Standard | Hardening
|
|
|
425
425
|
Scenario axis: positive ✅ · negative ✅ · invalid 🟡 · flaky-conn ✅ · kill/restart ✅ · concurrency ✅
|
|
426
426
|
Harness: ✅ port bind-and-hold · ✅ teardown catalog-derived (survives a new schema)
|
|
427
427
|
DoD: 9/12 — return to EXECUTE before declaring done
|
|
428
|
-
Drain audit: ✅
|
|
428
|
+
Drain audit: ✅ clean
|
|
429
429
|
Leak audit: 🔴 std.testing.allocator reported 2 leaks across 100 requests
|
|
430
430
|
Shuffle: ✅ pass under --shuffle
|
|
431
431
|
Clean run: ✅ pass after make down && make up
|
|
@@ -444,17 +444,17 @@ Block the change if any are missing on touched surface:
|
|
|
444
444
|
- [ ] Every changed handler/service/repo has ≥1 T1+T2+T3 happy-path test
|
|
445
445
|
- [ ] Every `catch`/`orelse`/`except`/`Err` in the request path has a T4 injection test
|
|
446
446
|
- [ ] Every concurrent-write or idempotency claim has a T5 test
|
|
447
|
-
- [ ] Drain audit
|
|
447
|
+
- [ ] Drain audit clean post-suite
|
|
448
448
|
- [ ] Leak audit clean post-suite (Zig: `std.testing.allocator`; Rust: `dhat`; Python: `objgraph`)
|
|
449
449
|
- [ ] Suite passes under randomised execution order
|
|
450
|
-
- [ ] Suite passes from clean
|
|
450
|
+
- [ ] Suite passes from a clean datastore — services down, up, then the integration lane
|
|
451
451
|
- [ ] No mocking of DB, Redis, or internal modules
|
|
452
452
|
- [ ] Test names read as documentation
|
|
453
453
|
- [ ] OpenAPI / protobuf breaking-change check (`oasdiff` / `buf breaking`) clean
|
|
454
454
|
- [ ] If streaming touched: T7 incremental-delivery test present
|
|
455
455
|
- [ ] Failure-injection mechanism named for every T4/T7 test (no fiction)
|
|
456
456
|
- [ ] **Partial completion:** every operation touching multiple systems or acquiring the same resource more than once carries a matrix with one row per ordered failure point; each row selects workflow and boundary-call ordinals, failure timing / remote outcome, scopes a concurrency-safe failpoint to one operation, and asserts durable, external, user-visible, and retry residual state; every external mutation proves acknowledgement-loss remote state plus exactly one effect on retry; the completed matrix is pasted into PR Session Notes
|
|
457
|
-
- [ ] PR Session Notes paste final
|
|
457
|
+
- [ ] PR Session Notes paste the final result line from the integration lane and the memory-leak lane
|
|
458
458
|
- [ ] **≥100-connection** test proves exactly-once + parallelism (peak-concurrency/lock-wait counter, or wall-time < R×), passes K runs
|
|
459
459
|
- [ ] **Zig error-path leak:** allocating handlers/repos proven by `std.testing.checkAllAllocationFailures`; no cross-request high-water growth over N requests
|
|
460
460
|
- [ ] **Complexity under load:** N+1 round-trips ruled out (count at n vs 10n); work-counter ladder asserts the O-bound; p95 within budget vs pinned baseline
|
|
@@ -463,8 +463,8 @@ Block the change if any are missing on touched surface:
|
|
|
463
463
|
- [ ] **Harness hygiene:** server binds-and-holds its port (no rebind race, no `listen()` panic); teardown/reset is catalog-derived (proven by adding a throwaway schema and re-running teardown — it must still reach zero)
|
|
464
464
|
- [ ] **Client daemon (if present, T9):** sudden SIGKILL mid-lease → reclaim + re-fence + exactly-once + stale report fenced; restart → re-register + resume, no duplicate, no orphan lease; flaky control-plane link → backoff without crash + un-acked redelivery + idempotent report retry; forked child always reaped + scope always destroyed on every exit path
|
|
465
465
|
|
|
466
|
-
100% required before declaring
|
|
466
|
+
100% required before declaring the integration suite done.
|
|
467
467
|
|
|
468
468
|
## Validation gate
|
|
469
469
|
|
|
470
|
-
Run
|
|
470
|
+
Run the integration lane twice — once against shared services, once from a clean datastore. Run once more under randomised order. Report pass/fail/skip counts, drain audit, leak audit, the coverage table above, and the DoD checklist. Do not declare done until DoD is 100%.
|
|
@@ -308,7 +308,7 @@ Reject the change if any are missing on touched code:
|
|
|
308
308
|
- [ ] Golden-file drift, if any, has a written explanation
|
|
309
309
|
- [ ] Branch coverage ≥80%, error-path 100% on touched code
|
|
310
310
|
- [ ] **Performance:** counter-based complexity proof on the n-ladder (O-bound asserted) + median-of-K latency vs pinned baseline; N+1 round-trips ruled out
|
|
311
|
-
- [ ] **Zig zero-leak:**
|
|
311
|
+
- [ ] **Zig zero-leak:** the memory-leak lane reports 0 leaks; every alloc-with-error-path proven by `std.testing.checkAllAllocationFailures`; no cross-request high-water growth over N iters
|
|
312
312
|
- [ ] **Concurrency:** ≥100-connection barrier-started test proves exactly-once + parallelism (peak-concurrency/lock-wait counter, or wall-time < R×); passes K runs + race detector clean
|
|
313
313
|
- [ ] **Refactor escalation:** any super-linear or serialization finding on touched code carries a refactor proposal (root cause + target design + patch alternative + recommendation) surfaced for decision — not silently band-aided
|
|
314
314
|
- [ ] **Diff ledger** resolved 100% — every changed unit has a test or an explicit `won't-test`/`needs-infra`; ledger pasted into PR Session Notes
|
|
@@ -361,7 +361,7 @@ enumerate) over sampling.
|
|
|
361
361
|
arena acquire → use → reset) for N iterations and assert outstanding-bytes /
|
|
362
362
|
high-water does **not** grow monotonically. A slow leak the single-shot test misses
|
|
363
363
|
shows up here as linear growth.
|
|
364
|
-
- **Gate:**
|
|
364
|
+
- **Gate:** the memory-leak lane → **0 leaks**, output pasted into PR Session Notes; cross-compile
|
|
365
365
|
both targets first (`x86_64-linux`, `aarch64-linux`) — stdlib alloc paths differ from macOS.
|
|
366
366
|
|
|
367
367
|
### Proof 2 — Concurrency proven at ≥100 connections (no hidden global lock)
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
import { describe, expect, test } from "bun:test";
|
|
2
|
+
import { mkdirSync, mkdtempSync, rmSync } from "node:fs";
|
|
3
|
+
import { tmpdir } from "node:os";
|
|
4
|
+
import { join } from "node:path";
|
|
5
|
+
|
|
6
|
+
import { PRODUCT_ONLY_TARGETS, productLeakErrors, shippedFiles } from "./pack_hygiene";
|
|
7
|
+
|
|
8
|
+
const registryFor = (pack: string, source: string) => ({
|
|
9
|
+
packs: { [pack]: { documents: [{ source, target: source }] } },
|
|
10
|
+
});
|
|
11
|
+
|
|
12
|
+
describe("productLeakErrors", () => {
|
|
13
|
+
test("flags a product-only target named by a generic pack", async () => {
|
|
14
|
+
const output = mkdtempSync(join(tmpdir(), "orly-hygiene-"));
|
|
15
|
+
try {
|
|
16
|
+
mkdirSync(join(output, "docs"));
|
|
17
|
+
await Bun.write(join(output, "docs/GUIDE.md"), "Run `make test-integration` before you ship.\n");
|
|
18
|
+
|
|
19
|
+
const errors = await productLeakErrors(output, registryFor("domain.http", "docs/GUIDE.md"));
|
|
20
|
+
|
|
21
|
+
expect(errors).toHaveLength(1);
|
|
22
|
+
expect(errors[0]).toContain("docs/GUIDE.md:1");
|
|
23
|
+
expect(errors[0]).toContain("domain.http");
|
|
24
|
+
expect(errors[0]).toContain("make test-integration");
|
|
25
|
+
} finally {
|
|
26
|
+
rmSync(output, { recursive: true, force: true });
|
|
27
|
+
}
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
test("allows the same target inside a product-scoped pack", async () => {
|
|
31
|
+
const output = mkdtempSync(join(tmpdir(), "orly-hygiene-"));
|
|
32
|
+
try {
|
|
33
|
+
mkdirSync(join(output, "docs"));
|
|
34
|
+
await Bun.write(join(output, "docs/GUIDE.md"), "Run `make test-integration` before you ship.\n");
|
|
35
|
+
|
|
36
|
+
const errors = await productLeakErrors(output, registryFor("product.agentsfleet", "docs/GUIDE.md"));
|
|
37
|
+
|
|
38
|
+
expect(errors).toBeEmpty();
|
|
39
|
+
} finally {
|
|
40
|
+
rmSync(output, { recursive: true, force: true });
|
|
41
|
+
}
|
|
42
|
+
});
|
|
43
|
+
|
|
44
|
+
test("does not fire on English that begins with 'make'", async () => {
|
|
45
|
+
const output = mkdtempSync(join(tmpdir(), "orly-hygiene-"));
|
|
46
|
+
try {
|
|
47
|
+
mkdirSync(join(output, "docs"));
|
|
48
|
+
await Bun.write(
|
|
49
|
+
join(output, "docs/GUIDE.md"),
|
|
50
|
+
"Make it obvious. This will make the reader stop.\nDo not make an exception here.\n",
|
|
51
|
+
);
|
|
52
|
+
|
|
53
|
+
const errors = await productLeakErrors(output, registryFor("universal.authoring", "docs/GUIDE.md"));
|
|
54
|
+
|
|
55
|
+
expect(errors).toBeEmpty();
|
|
56
|
+
} finally {
|
|
57
|
+
rmSync(output, { recursive: true, force: true });
|
|
58
|
+
}
|
|
59
|
+
});
|
|
60
|
+
|
|
61
|
+
test("reports the most specific target, not its prefix", async () => {
|
|
62
|
+
const output = mkdtempSync(join(tmpdir(), "orly-hygiene-"));
|
|
63
|
+
try {
|
|
64
|
+
mkdirSync(join(output, "docs"));
|
|
65
|
+
await Bun.write(join(output, "docs/GUIDE.md"), "Run `make test-integration-db` for the DB half.\n");
|
|
66
|
+
|
|
67
|
+
const errors = await productLeakErrors(output, registryFor("domain.sql", "docs/GUIDE.md"));
|
|
68
|
+
|
|
69
|
+
expect(errors).toHaveLength(1);
|
|
70
|
+
expect(errors[0]).toContain("make test-integration-db");
|
|
71
|
+
} finally {
|
|
72
|
+
rmSync(output, { recursive: true, force: true });
|
|
73
|
+
}
|
|
74
|
+
});
|
|
75
|
+
});
|
|
76
|
+
|
|
77
|
+
describe("shippedFiles", () => {
|
|
78
|
+
test("tags each source with the pack that ships it", () => {
|
|
79
|
+
const files = shippedFiles({
|
|
80
|
+
packs: {
|
|
81
|
+
"domain.http": { documents: [{ source: "docs/A.md" }] },
|
|
82
|
+
"product.agentsfleet": { documents: [{ source: "docs/B.md" }] },
|
|
83
|
+
},
|
|
84
|
+
core_documents: [{ source: "docs/C.md" }],
|
|
85
|
+
});
|
|
86
|
+
|
|
87
|
+
expect(files).toContainEqual({ source: "docs/A.md", pack: "domain.http" });
|
|
88
|
+
expect(files).toContainEqual({ source: "docs/B.md", pack: "product.agentsfleet" });
|
|
89
|
+
expect(files).toContainEqual({ source: "docs/C.md", pack: "core" });
|
|
90
|
+
});
|
|
91
|
+
});
|
|
92
|
+
|
|
93
|
+
describe("the real corpus", () => {
|
|
94
|
+
test("no generic pack names a product-only make target", async () => {
|
|
95
|
+
const registry = await Bun.file("registry.json").json();
|
|
96
|
+
const errors = await productLeakErrors(".", registry);
|
|
97
|
+
|
|
98
|
+
expect(errors).toEqual([]);
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
test("every denied target is spelled once", () => {
|
|
102
|
+
expect(new Set(PRODUCT_ONLY_TARGETS).size).toBe(PRODUCT_ONLY_TARGETS.length);
|
|
103
|
+
});
|
|
104
|
+
});
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import { readFile } from "node:fs/promises";
|
|
2
|
+
import { join } from "node:path";
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* Make targets that belong to ONE consuming repository.
|
|
6
|
+
*
|
|
7
|
+
* A generic pack — anything not `product.*` — must never name these. A repository
|
|
8
|
+
* that renames or retires one gets a managed file citing a target it does not
|
|
9
|
+
* have, and if it gates on cited targets resolving, the only way to go green is
|
|
10
|
+
* to edit a managed file, which the next `orly update` reverts. That loop is the
|
|
11
|
+
* bug this list exists to prevent.
|
|
12
|
+
*
|
|
13
|
+
* Generic prose says what the target IS — "the declared `verify.unit` command",
|
|
14
|
+
* "the repository's integration lane" — and lets each repository bind the name.
|
|
15
|
+
*
|
|
16
|
+
* Deliberately a denylist, not "no `make` citations at all": English says "make
|
|
17
|
+
* it", "make the", "make an", so a blanket pattern fires on prose and gets
|
|
18
|
+
* carved out until it means nothing.
|
|
19
|
+
*/
|
|
20
|
+
export const PRODUCT_ONLY_TARGETS: readonly string[] = [
|
|
21
|
+
"lint-zig",
|
|
22
|
+
"lint-governance",
|
|
23
|
+
"check-pg-drain",
|
|
24
|
+
"memleak",
|
|
25
|
+
"test-integration",
|
|
26
|
+
"test-integration-db",
|
|
27
|
+
"test-integration-redis",
|
|
28
|
+
"test-integration-kernel",
|
|
29
|
+
"test-unit-agentsfleetd",
|
|
30
|
+
"test-unit-agentsfleet-runner",
|
|
31
|
+
"test-unit-agentsfleet-lib",
|
|
32
|
+
"test-coverage-zig",
|
|
33
|
+
"test-coverage-grade",
|
|
34
|
+
"_lint_zig_test_depth",
|
|
35
|
+
];
|
|
36
|
+
|
|
37
|
+
/**
|
|
38
|
+
* Pack label for files the registry ships outside any pack — core documents and
|
|
39
|
+
* rules. Not a pack id, so it never matches the `product.` prefix and is held to
|
|
40
|
+
* the generic standard, which is what core documents are.
|
|
41
|
+
*/
|
|
42
|
+
const CORE_PACK = "core";
|
|
43
|
+
|
|
44
|
+
/** A file shipped by a pack, with the pack that ships it. */
|
|
45
|
+
export interface ShippedFile {
|
|
46
|
+
readonly source: string;
|
|
47
|
+
readonly pack: string;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** Collects every `source` a registry node ships, tagged with its pack. */
|
|
51
|
+
export function shippedFiles(registry: unknown): ShippedFile[] {
|
|
52
|
+
const out: ShippedFile[] = [];
|
|
53
|
+
const walk = (node: unknown, pack: string): void => {
|
|
54
|
+
if (Array.isArray(node)) {
|
|
55
|
+
for (const item of node) walk(item, pack);
|
|
56
|
+
return;
|
|
57
|
+
}
|
|
58
|
+
if (node === null || typeof node !== "object") return;
|
|
59
|
+
const record = node as Record<string, unknown>;
|
|
60
|
+
if (typeof record.source === "string") out.push({ source: record.source, pack });
|
|
61
|
+
for (const value of Object.values(record)) walk(value, pack);
|
|
62
|
+
};
|
|
63
|
+
|
|
64
|
+
const root = registry as Record<string, unknown>;
|
|
65
|
+
const packs = (root.packs ?? {}) as Record<string, unknown>;
|
|
66
|
+
for (const [name, pack] of Object.entries(packs)) walk(pack, name);
|
|
67
|
+
walk(root.core_documents, CORE_PACK);
|
|
68
|
+
walk(root.rules, CORE_PACK);
|
|
69
|
+
return out;
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/**
|
|
73
|
+
* Reports every product-only make target named by a file a GENERIC pack ships.
|
|
74
|
+
*
|
|
75
|
+
* Returns one line per offending citation, empty when clean.
|
|
76
|
+
*/
|
|
77
|
+
export async function productLeakErrors(root: string, registry: unknown): Promise<string[]> {
|
|
78
|
+
const errors: string[] = [];
|
|
79
|
+
const seen = new Set<string>();
|
|
80
|
+
|
|
81
|
+
for (const { source, pack } of shippedFiles(registry)) {
|
|
82
|
+
if (pack.startsWith("product.")) continue;
|
|
83
|
+
if (!source.endsWith(".md")) continue;
|
|
84
|
+
if (seen.has(source)) continue;
|
|
85
|
+
seen.add(source);
|
|
86
|
+
|
|
87
|
+
let text: string;
|
|
88
|
+
try {
|
|
89
|
+
text = await readFile(join(root, source), "utf8");
|
|
90
|
+
} catch {
|
|
91
|
+
continue; // absent sources are the reference gate's business, not ours
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
text.split("\n").forEach((line, index) => {
|
|
95
|
+
for (const target of PRODUCT_ONLY_TARGETS) {
|
|
96
|
+
// Word-boundary on the tail so `test-integration` does not swallow
|
|
97
|
+
// `test-integration-db`'s own, more specific, report.
|
|
98
|
+
const pattern = new RegExp(`make ${target}(?![a-z0-9_-])`);
|
|
99
|
+
if (pattern.test(line)) {
|
|
100
|
+
errors.push(
|
|
101
|
+
`${source}:${index + 1}: pack '${pack}' is not product-scoped but names \`make ${target}\`, ` +
|
|
102
|
+
`which only one repository defines — say what the command IS instead`,
|
|
103
|
+
);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
});
|
|
107
|
+
}
|
|
108
|
+
return errors;
|
|
109
|
+
}
|