@iceinvein/agent-skills 0.21.4 → 0.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (131) hide show
  1. package/README.md +9 -0
  2. package/package.json +4 -1
  3. package/skills/bug-hunter/SKILL.md +82 -0
  4. package/skills/bug-hunter/evals/README.md +282 -0
  5. package/skills/bug-hunter/evals/baselines/claude-opus-5-5.json +93 -0
  6. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/case.yaml +4 -0
  7. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/fixture.sh +71 -0
  8. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/graders/bug-cites-the-spec.md +36 -0
  9. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/graders/bug-hunter-fired.md +5 -0
  10. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/graders/intent-written.md +8 -0
  11. package/skills/bug-hunter/evals/bug-cites-intent-from-spec/prompt.md +11 -0
  12. package/skills/bug-hunter/evals/callers-tree-unchanged/case.yaml +4 -0
  13. package/skills/bug-hunter/evals/callers-tree-unchanged/fixture.sh +67 -0
  14. package/skills/bug-hunter/evals/callers-tree-unchanged/graders/bug-hunter-fired.md +5 -0
  15. package/skills/bug-hunter/evals/callers-tree-unchanged/graders/source-unchanged.md +10 -0
  16. package/skills/bug-hunter/evals/callers-tree-unchanged/graders/tests-unchanged.md +10 -0
  17. package/skills/bug-hunter/evals/callers-tree-unchanged/graders/tree-unchanged.md +15 -0
  18. package/skills/bug-hunter/evals/callers-tree-unchanged/prompt.md +11 -0
  19. package/skills/bug-hunter/evals/correct-code-has-no-bugs/case.yaml +4 -0
  20. package/skills/bug-hunter/evals/correct-code-has-no-bugs/fixture.sh +82 -0
  21. package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/bug-hunter-fired.md +5 -0
  22. package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/bugs-empty.md +18 -0
  23. package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/written-with-write-tool.md +13 -0
  24. package/skills/bug-hunter/evals/correct-code-has-no-bugs/prompt.md +11 -0
  25. package/skills/bug-hunter/evals/impossible-value-is-a-bug/case.yaml +4 -0
  26. package/skills/bug-hunter/evals/impossible-value-is-a-bug/fixture.sh +68 -0
  27. package/skills/bug-hunter/evals/impossible-value-is-a-bug/graders/bug-for-sixty-seconds.md +37 -0
  28. package/skills/bug-hunter/evals/impossible-value-is-a-bug/graders/bug-hunter-fired.md +5 -0
  29. package/skills/bug-hunter/evals/impossible-value-is-a-bug/prompt.md +11 -0
  30. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/case.yaml +4 -0
  31. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/fixture.sh +79 -0
  32. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/bug-cites-the-limit.md +31 -0
  33. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/bug-hunter-fired.md +5 -0
  34. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/findings-written.md +10 -0
  35. package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/prompt.md +11 -0
  36. package/skills/bug-hunter/evals/rules.json +177 -0
  37. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/case.yaml +4 -0
  38. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/fixture.sh +97 -0
  39. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/graders/bug-hunter-fired.md +5 -0
  40. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/graders/bugs-empty.md +18 -0
  41. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/graders/cap-is-an-ambiguity.md +12 -0
  42. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/graders/written-with-write-tool.md +13 -0
  43. package/skills/bug-hunter/evals/silent-spec-is-an-ambiguity/prompt.md +11 -0
  44. package/skills/bug-hunter/evals/spec-outranks-target-docstring/case.yaml +4 -0
  45. package/skills/bug-hunter/evals/spec-outranks-target-docstring/fixture.sh +82 -0
  46. package/skills/bug-hunter/evals/spec-outranks-target-docstring/graders/bug-cites-the-spec.md +40 -0
  47. package/skills/bug-hunter/evals/spec-outranks-target-docstring/graders/bug-hunter-fired.md +5 -0
  48. package/skills/bug-hunter/evals/spec-outranks-target-docstring/prompt.md +11 -0
  49. package/skills/bug-hunter/references/findings-format.md +82 -0
  50. package/skills/bug-hunter/references/scenario-catalog.md +92 -0
  51. package/skills/bug-hunter/skill.json +24 -0
  52. package/skills/index.json +17 -5
  53. package/skills/magpie/evals/README.md +13 -0
  54. package/skills/magpie/evals/baselines/claude-opus-5-5.json +69 -0
  55. package/skills/magpie/evals/rules.json +192 -0
  56. package/skills/magpie/skill.json +1 -1
  57. package/skills/migrate/evals/README.md +13 -0
  58. package/skills/migrate/evals/baselines/claude-opus-5-5.json +129 -0
  59. package/skills/migrate/evals/rules.json +160 -0
  60. package/skills/migrate/skill.json +10 -4
  61. package/skills/scope-estimate/README.md +173 -0
  62. package/skills/scope-estimate/SKILL.md +168 -0
  63. package/skills/scope-estimate/references/capability-map.md +122 -0
  64. package/skills/scope-estimate/references/deliverables.md +60 -0
  65. package/skills/scope-estimate/references/requirements.md +104 -0
  66. package/skills/scope-estimate/references/schedule.md +138 -0
  67. package/skills/scope-estimate/scripts/__tests__/schedule.test.ts +366 -0
  68. package/skills/scope-estimate/scripts/schedule.ts +553 -0
  69. package/skills/scope-estimate/skill.json +19 -0
  70. package/skills/scope-estimate/templates/plan.example.json +33 -0
  71. package/skills/sluice/SKILL.md +35 -12
  72. package/skills/sluice/evals/README.md +111 -6
  73. package/skills/sluice/evals/announcement-reaches-the-ledger/fixture.sh +32 -19
  74. package/skills/sluice/evals/announcement-reaches-the-ledger/graders/routes-main-before-source.md +14 -0
  75. package/skills/sluice/evals/announcement-reaches-the-ledger/prompt.md +2 -2
  76. package/skills/sluice/evals/baselines/claude-opus-5-5.json +237 -0
  77. package/skills/sluice/evals/deep-run-blocks-on-a-real-decision/fixture.sh +19 -12
  78. package/skills/sluice/evals/deep-run-fans-out/graders/task-1-merged-back.md +8 -0
  79. package/skills/sluice/evals/deep-run-fans-out/graders/task-2-merged-back.md +8 -0
  80. package/skills/sluice/evals/deep-run-fans-out/graders/task-3-merged-back.md +6 -0
  81. package/skills/sluice/evals/deep-run-fans-out/graders/worktree-per-implementer.md +12 -0
  82. package/skills/sluice/evals/deep-run-hunts-after-the-flip/case.yaml +5 -0
  83. package/skills/sluice/evals/deep-run-hunts-after-the-flip/fixture.sh +258 -0
  84. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/bug-hunter-fired.md +5 -0
  85. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/four-attempts-allowed.md +34 -0
  86. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/fourth-attempt-tested.md +20 -0
  87. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/hunt-before-the-final-review.md +21 -0
  88. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/record-has-a-hunt-section.md +11 -0
  89. package/skills/sluice/evals/deep-run-hunts-after-the-flip/graders/sluice-fired.md +5 -0
  90. package/skills/sluice/evals/deep-run-hunts-after-the-flip/prompt.md +12 -0
  91. package/skills/sluice/evals/deep-run-survives-a-milestone/fixture.sh +46 -24
  92. package/skills/sluice/evals/fast-verifies-before-handback/case.yaml +4 -0
  93. package/skills/sluice/evals/fast-verifies-before-handback/fixture.sh +73 -0
  94. package/skills/sluice/evals/fast-verifies-before-handback/graders/quiet-flag-implemented.md +5 -0
  95. package/skills/sluice/evals/fast-verifies-before-handback/graders/sluice-fired.md +5 -0
  96. package/skills/sluice/evals/fast-verifies-before-handback/graders/verified-after-the-last-edit.md +18 -0
  97. package/skills/sluice/evals/fast-verifies-before-handback/prompt.md +11 -0
  98. package/skills/sluice/evals/main-hunts-a-checkable-claim/case.yaml +5 -0
  99. package/skills/sluice/evals/main-hunts-a-checkable-claim/fixture.sh +64 -0
  100. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/announces-main-channel.md +12 -0
  101. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/bug-hunter-fired.md +5 -0
  102. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/hunt-after-the-source-edits.md +12 -0
  103. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/last-page-holds-the-remainder.md +27 -0
  104. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/remainder-page-tested.md +21 -0
  105. package/skills/sluice/evals/main-hunts-a-checkable-claim/graders/sluice-fired.md +5 -0
  106. package/skills/sluice/evals/main-hunts-a-checkable-claim/prompt.md +12 -0
  107. package/skills/sluice/evals/main-new-interface/graders/shape-agreed-before-building.md +8 -4
  108. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/case.yaml +5 -0
  109. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/fixture.sh +73 -0
  110. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/graders/announces-main-channel.md +12 -0
  111. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/graders/no-hunt.md +12 -0
  112. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/graders/skip-said-in-one-line.md +21 -0
  113. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/graders/sluice-fired.md +5 -0
  114. package/skills/sluice/evals/main-skips-the-hunt-without-a-claim/prompt.md +12 -0
  115. package/skills/sluice/evals/rules.json +364 -0
  116. package/skills/sluice/references/deep-channel.md +64 -18
  117. package/skills/sluice/references/finish.md +8 -2
  118. package/skills/sluice/references/intent.md +4 -2
  119. package/skills/sluice/references/review.md +32 -0
  120. package/skills/sluice/references/status.md +17 -4
  121. package/skills/sluice/references/verify.md +38 -0
  122. package/skills/sluice/scripts/content-digest.sh +65 -0
  123. package/skills/sluice/scripts/run-stats.sh +9 -8
  124. package/skills/sluice/scripts/status.sh +78 -13
  125. package/skills/sluice/scripts/stop-guard.sh +97 -24
  126. package/skills/sluice/skill.json +2 -2
  127. package/skills/terse/evals/README.md +16 -0
  128. package/skills/terse/evals/baselines/claude-opus-5-5.json +105 -0
  129. package/skills/terse/evals/rules.json +161 -0
  130. package/skills/terse/skill.json +1 -1
  131. package/skills/sluice/evals/announcement-reaches-the-ledger/graders/escalates-fast-to-main.md +0 -12
@@ -0,0 +1,177 @@
1
+ {
2
+ "rules": [
3
+ {
4
+ "id": "target-named-in-one-line",
5
+ "states": "SKILL.md: Method",
6
+ "origin": "commit:f88c60a",
7
+ "cases": [],
8
+ "unpinned": "No grader reads the opening message where the target is named."
9
+ },
10
+ {
11
+ "id": "intent-collected-into-intent-md",
12
+ "states": "SKILL.md: Method",
13
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
14
+ "cases": [
15
+ "bug-cites-intent-from-spec"
16
+ ]
17
+ },
18
+ {
19
+ "id": "claim-never-from-a-changed-body",
20
+ "states": "SKILL.md: Method",
21
+ "origin": "commit:9a9dc31",
22
+ "cases": [
23
+ "bug-cites-intent-from-spec",
24
+ "silent-spec-is-an-ambiguity"
25
+ ]
26
+ },
27
+ {
28
+ "id": "scenarios-written-from-the-catalog",
29
+ "states": "SKILL.md: Method",
30
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
31
+ "cases": [],
32
+ "unpinned": "No grader reads scenarios.md or checks which catalog categories a run worked through."
33
+ },
34
+ {
35
+ "id": "repros-use-the-repos-own-test-runner",
36
+ "states": "SKILL.md: Method",
37
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
38
+ "cases": [],
39
+ "unpinned": "The repro judge reads what the repro asserts against the fixture's code, not which runner or framework it is written in."
40
+ },
41
+ {
42
+ "id": "no-test-dependency-installed",
43
+ "states": "SKILL.md: Method",
44
+ "origin": "commit:9a9dc31",
45
+ "cases": [],
46
+ "unpinned": "Every fixture runs on node --test with no dependencies, and an install in the scratch copy leaves no path in the caller's tree for tree-unchanged to see."
47
+ },
48
+ {
49
+ "id": "many-cheap-probes",
50
+ "states": "SKILL.md: Method",
51
+ "origin": "commit:9a9dc31",
52
+ "cases": [],
53
+ "unpinned": "No grader counts probes or reads how a run followed up a failure; each fixture has at most one defect to find."
54
+ },
55
+ {
56
+ "id": "failure-rechecked-alone-before-reporting",
57
+ "states": "SKILL.md: Method",
58
+ "origin": "commit:9a9dc31",
59
+ "cases": [],
60
+ "unpinned": "No grader checks a repro was re-run alone, and no fixture has a mock or setup that could make a repro fail on its own."
61
+ },
62
+ {
63
+ "id": "scratch-worktree-leaves-callers-tree-unchanged",
64
+ "states": "SKILL.md: Method",
65
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
66
+ "cases": [
67
+ "callers-tree-unchanged"
68
+ ]
69
+ },
70
+ {
71
+ "id": "no-repro-no-finding",
72
+ "states": "SKILL.md: What counts as a bug",
73
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
74
+ "cases": [
75
+ "proves-a-spec-boundary-bug",
76
+ "correct-code-has-no-bugs",
77
+ "bug-cites-intent-from-spec",
78
+ "spec-outranks-target-docstring",
79
+ "impossible-value-is-a-bug"
80
+ ]
81
+ },
82
+ {
83
+ "id": "contradicting-a-stated-claim-is-a-bug",
84
+ "states": "SKILL.md: What counts as a bug",
85
+ "origin": "commit:bd7fc25",
86
+ "cases": [
87
+ "proves-a-spec-boundary-bug",
88
+ "bug-cites-intent-from-spec",
89
+ "spec-outranks-target-docstring"
90
+ ]
91
+ },
92
+ {
93
+ "id": "repo-stated-sources-are-intent",
94
+ "states": "SKILL.md: Method",
95
+ "origin": "commit:bd7fc25",
96
+ "cases": [],
97
+ "unpinned": "No fixture has a sibling implementation, a stated invariant or an existing test the target contradicts, so claims from those sources are never exercised."
98
+ },
99
+ {
100
+ "id": "claims-have-a-rank",
101
+ "states": "SKILL.md: What counts as a bug",
102
+ "origin": "commit:23d4466",
103
+ "cases": [
104
+ "spec-outranks-target-docstring"
105
+ ]
106
+ },
107
+ {
108
+ "id": "wrong-under-every-reading-is-a-bug",
109
+ "states": "SKILL.md: What counts as a bug",
110
+ "origin": "commit:23d4466",
111
+ "cases": [
112
+ "impossible-value-is-a-bug"
113
+ ]
114
+ },
115
+ {
116
+ "id": "silent-spec-is-an-ambiguity",
117
+ "states": "SKILL.md: What counts as a bug",
118
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
119
+ "cases": [
120
+ "silent-spec-is-an-ambiguity"
121
+ ]
122
+ },
123
+ {
124
+ "id": "latent-is-not-a-bug",
125
+ "states": "SKILL.md: What counts as a bug",
126
+ "origin": "commit:9a9dc31",
127
+ "cases": [],
128
+ "unpinned": "Every fixture's defect sits on the module's exported path, so no case has a defect no production caller reaches."
129
+ },
130
+ {
131
+ "id": "output-in-the-dated-run-directory",
132
+ "states": "SKILL.md: Output",
133
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
134
+ "cases": [
135
+ "proves-a-spec-boundary-bug",
136
+ "bug-cites-intent-from-spec"
137
+ ]
138
+ },
139
+ {
140
+ "id": "findings-written-with-write-tool-to-shape",
141
+ "states": "SKILL.md: Output",
142
+ "origin": "commit:bd7fc25",
143
+ "cases": [
144
+ "silent-spec-is-an-ambiguity",
145
+ "correct-code-has-no-bugs"
146
+ ]
147
+ },
148
+ {
149
+ "id": "bug-hunter-dir-is-the-only-path-written",
150
+ "states": "SKILL.md: Output",
151
+ "origin": "commit:f88c60a",
152
+ "cases": [
153
+ "callers-tree-unchanged"
154
+ ]
155
+ },
156
+ {
157
+ "id": "promoting-a-repro-is-the-callers-job",
158
+ "states": "SKILL.md: Output",
159
+ "origin": "docs/specs/2026-10-12-bug-hunter.md",
160
+ "cases": [
161
+ "callers-tree-unchanged"
162
+ ]
163
+ },
164
+ {
165
+ "id": "hand-back-contents",
166
+ "states": "SKILL.md: Hand-back",
167
+ "origin": "commit:f88c60a",
168
+ "cases": [],
169
+ "unpinned": "No grader reads the final message, so the run directory, bucket counts, per-bug lines and anything not done go unchecked."
170
+ }
171
+ ],
172
+ "sandboxSensitive": [],
173
+ "vacuousWithout": [
174
+ "callers-tree-unchanged"
175
+ ],
176
+ "unclaimed": []
177
+ }
@@ -0,0 +1,4 @@
1
+ schema_version: "1.1"
2
+ name: silent-spec-is-an-ambiguity
3
+ context:
4
+ scaffold_script: fixture.sh
@@ -0,0 +1,97 @@
1
+ #!/usr/bin/env bash
2
+ # A cart whose spec says a line's quantity is capped and never says where. The
3
+ # code caps at 99, so a scenario at 99, 100 or 150 has no expected value any
4
+ # claim can supply: the cap's value is a question for product, not a bug. The
5
+ # spec names the cap rather than staying silent about large quantities
6
+ # altogether, because a spec that only says "setting a quantity replaces the
7
+ # line's quantity" makes any cap a violation of that claim, and the right
8
+ # answer would no longer be clear. Everything the spec does settle, the code
9
+ # does, so a bug here is the skill inventing an expected value.
10
+ set -euo pipefail
11
+
12
+ mkdir -p src tests
13
+
14
+ cat > package.json <<'JSON'
15
+ {
16
+ "name": "cart",
17
+ "version": "1.2.0",
18
+ "type": "module",
19
+ "scripts": {
20
+ "test": "node --test \"tests/*.test.js\""
21
+ }
22
+ }
23
+ JSON
24
+
25
+ cat > SPEC.md <<'MD'
26
+ # Cart
27
+
28
+ `setQuantity(cart, sku, quantity)` in `src/cart.js` sets how many of one
29
+ product a cart line holds and returns the new cart. A cart is
30
+ `{ lines: [{ sku, quantity }] }`.
31
+
32
+ - The cart passed in is not modified.
33
+ - Setting the quantity of a SKU already in the cart replaces that line's
34
+ quantity, and the line keeps its place.
35
+ - Setting the quantity of a SKU not in the cart adds a line at the end.
36
+ - Setting a quantity of 0 removes the line. Doing so for a SKU not in the cart
37
+ returns a cart with the same lines.
38
+ - A quantity that is negative or not a whole number throws a `RangeError`.
39
+ - A line's quantity is capped so that a slip on the keyboard cannot order a
40
+ pallet: setting a quantity above the cap stores the cap.
41
+ MD
42
+
43
+ git init --quiet
44
+ git add -A
45
+ git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: cart quantities"
46
+
47
+ cat > src/cart.js <<'JS'
48
+ const MAX_QUANTITY = 99;
49
+
50
+ export function setQuantity(cart, sku, quantity) {
51
+ if (!Number.isInteger(quantity) || quantity < 0) {
52
+ throw new RangeError(`quantity must be a whole number of 0 or more, got ${quantity}`);
53
+ }
54
+
55
+ const stored = Math.min(quantity, MAX_QUANTITY);
56
+ const index = cart.lines.findIndex((line) => line.sku === sku);
57
+
58
+ if (stored === 0) {
59
+ return { ...cart, lines: cart.lines.filter((line) => line.sku !== sku) };
60
+ }
61
+
62
+ if (index === -1) {
63
+ return { ...cart, lines: [...cart.lines, { sku, quantity: stored }] };
64
+ }
65
+
66
+ const lines = cart.lines.map((line, i) => (i === index ? { ...line, quantity: stored } : line));
67
+ return { ...cart, lines };
68
+ }
69
+ JS
70
+
71
+ cat > tests/cart.test.js <<'JS'
72
+ import assert from "node:assert/strict";
73
+ import { test } from "node:test";
74
+ import { setQuantity } from "../src/cart.js";
75
+
76
+ test("a new SKU is added as a line", () => {
77
+ const cart = setQuantity({ lines: [] }, "mug", 2);
78
+ assert.deepEqual(cart, { lines: [{ sku: "mug", quantity: 2 }] });
79
+ });
80
+
81
+ test("an existing line's quantity is replaced", () => {
82
+ const cart = setQuantity({ lines: [{ sku: "mug", quantity: 2 }] }, "mug", 5);
83
+ assert.deepEqual(cart, { lines: [{ sku: "mug", quantity: 5 }] });
84
+ });
85
+
86
+ test("a quantity of 0 removes the line", () => {
87
+ const cart = setQuantity({ lines: [{ sku: "mug", quantity: 2 }] }, "mug", 0);
88
+ assert.deepEqual(cart, { lines: [] });
89
+ });
90
+
91
+ test("a negative quantity throws", () => {
92
+ assert.throws(() => setQuantity({ lines: [] }, "mug", -1), RangeError);
93
+ });
94
+ JS
95
+
96
+ git add -A
97
+ git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: set a cart line's quantity"
@@ -0,0 +1,5 @@
1
+ ---
2
+ type: tool_used
3
+ tool: Skill
4
+ input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
5
+ ---
@@ -0,0 +1,18 @@
1
+ ---
2
+ type: regex
3
+ # findings.json sits under a slug the skill picks, so no file target can name
4
+ # it; the trace carries its content in the Write call (or shell command) that
5
+ # created it. That content is one JSON string in the trace line, its quotes
6
+ # escaped as \" and its newlines as \n, and the match is held inside that one
7
+ # string. Matches a findings.json write whose "bugs" is an empty array, and
8
+ # only the last one: the closing lookahead fails the match if any later
9
+ # findings.json write follows (a Write, its echoed result, an Edit, or a shell
10
+ # redirect), so an empty first draft rewritten with bugs does not pass. A Write
11
+ # is echoed in full by its tool result, so the last Write still matches there.
12
+ # A later Edit fails the grader, since an Edit's result does not carry the
13
+ # whole file. A file written by a dispatched agent counts only if the trace
14
+ # carries that agent's tool calls.
15
+ pattern: 'findings\.json(?:","content":")?(?:\\.|[^"\\])*?\\"bugs\\":(?:\s|\\n)*\[(?:\s|\\n)*\](?![\s\S]*?(?:findings\.json","(?:content|old_string)":"|>\s*[^"\s]*findings\.json))'
16
+ target: trace
17
+ weight: 2
18
+ ---
@@ -0,0 +1,12 @@
1
+ ---
2
+ type: regex
3
+ # The same findings.json write as bugs-empty, read for its ambiguities: one
4
+ # whose question asks about the quantity cap (the cap, a maximum or ceiling,
5
+ # or the 99 the code uses). The question is matched inside its own \"...\"
6
+ # string value. The same closing lookahead as bugs-empty holds the match to
7
+ # the last findings.json write.
8
+ pattern: 'findings\.json(?:","content":")?(?:\\.|[^"\\])*?\\"ambiguities\\":(?:\\.|[^"\\])*?\\"question\\":(?:\s|\\n)*\\"(?:(?!\\")(?:\\.|[^"\\]))*?(?:\bcap|maximum|\bmax\b|ceiling|\b99\b|\b100\b|upper limit)(?![\s\S]*?(?:findings\.json","(?:content|old_string)":"|>\s*[^"\s]*findings\.json))'
9
+ flags: i
10
+ target: trace
11
+ weight: 2
12
+ ---
@@ -0,0 +1,13 @@
1
+ ---
2
+ type: regex
3
+ # SKILL.md writes intent.md, findings.json and report.md with the Write tool,
4
+ # never from a shell command. The content graders read a heredoc's literal JSON
5
+ # as readily as a Write call's, so this is what holds the rule: it fails a Bash
6
+ # command that redirects or tees into one of the three under .bug-hunter/. A
7
+ # script that opens the file itself (python3, node -e) is not caught here; the
8
+ # content graders fail that, since the trace then holds no literal JSON.
9
+ pattern: '"command":"(?:\\.|[^"\\])*?(?:>|\btee\b(?:\\.|[^"\\|;&])*?)\s*[^"\s]*\.bug-hunter/[^"\s]*?(?:intent\.md|findings\.json|report\.md)'
10
+ match: not_contains
11
+ target: trace
12
+ weight: 2
13
+ ---
@@ -0,0 +1,11 @@
1
+ ---
2
+ name: silent-spec-is-an-ambiguity
3
+ description: A limit whose value the spec never states has no expected value to prove a bug against. Pins the finding landing in ambiguities, with bugs left empty.
4
+ tags: [ambiguity, boundary, scaffold, write]
5
+ max_turns: 80
6
+ timeout_seconds: 2400
7
+ allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
8
+ expected_outcome: findings.json under .bug-hunter/ has an empty bugs array and an ambiguity asking what the quantity cap is.
9
+ ---
10
+
11
+ Before this goes to QA: what will they find in `src/cart.js`? It's meant to do what `SPEC.md` says.
@@ -0,0 +1,4 @@
1
+ schema_version: "1.1"
2
+ name: spec-outranks-target-docstring
3
+ context:
4
+ scaffold_script: fixture.sh
@@ -0,0 +1,82 @@
1
+ #!/usr/bin/env bash
2
+ # A download list whose spec and whose own docstring disagree. SPEC.md says a
3
+ # sealed export stays on offer while its guide is rewritten in, and if that
4
+ # rewrite fails. The module's docstring, written with the code, says a sealed
5
+ # export whose rewrite failed is never offered, and the code follows the
6
+ # docstring. The spec is the intent; a comment inside the changed code is the
7
+ # change describing itself, so the disagreement is a bug citing the spec, not
8
+ # a question for product. This is the shape the pilot's workings-3a4c4b7f5
9
+ # key took, where the skill found the defect and filed it as an ambiguity.
10
+ # Everything else the spec says, the code does.
11
+ set -euo pipefail
12
+
13
+ mkdir -p src tests
14
+
15
+ cat > package.json <<'JSON'
16
+ {
17
+ "name": "exports",
18
+ "version": "2.4.0",
19
+ "type": "module",
20
+ "scripts": {
21
+ "test": "node --test \"tests/*.test.js\""
22
+ }
23
+ }
24
+ JSON
25
+
26
+ cat > SPEC.md <<'MD'
27
+ # Exports
28
+
29
+ `offeredExports(exports)` in `src/exports.js` returns the exports the
30
+ download list shows, keeping the order it was given. An export is
31
+ `{ id, sealed, hasGuide, rewrite }`, where `rewrite` is `"none"`,
32
+ `"running"` or `"failed"`.
33
+
34
+ - An export is offered once it is sealed. An unsealed export is not offered.
35
+ - A sealed export made before exports carried a guide (`hasGuide` false) has
36
+ the guide written into it in the background. It stays on offer while that
37
+ rewrite runs and if the rewrite fails: a sealed export is a complete proof
38
+ with or without the guide.
39
+ MD
40
+
41
+ git init --quiet
42
+ git add -A
43
+ git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: which exports the download list offers"
44
+
45
+ cat > src/exports.js <<'JS'
46
+ /**
47
+ * The exports the download list offers, in the order given. A sealed export
48
+ * is offered. A sealed export without a guide whose rewrite failed is never
49
+ * offered: without the guide it cannot be opened on its own.
50
+ */
51
+ export function offeredExports(exports) {
52
+ return exports.filter((entry) => {
53
+ if (!entry.sealed) return false;
54
+ if (!entry.hasGuide && entry.rewrite === "failed") return false;
55
+ return true;
56
+ });
57
+ }
58
+ JS
59
+
60
+ cat > tests/exports.test.js <<'JS'
61
+ import assert from "node:assert/strict";
62
+ import { test } from "node:test";
63
+ import { offeredExports } from "../src/exports.js";
64
+
65
+ test("a sealed export with a guide is offered", () => {
66
+ const sealed = { id: "a", sealed: true, hasGuide: true, rewrite: "none" };
67
+ assert.deepEqual(offeredExports([sealed]), [sealed]);
68
+ });
69
+
70
+ test("an unsealed export is not offered", () => {
71
+ assert.deepEqual(offeredExports([{ id: "a", sealed: false, hasGuide: true, rewrite: "none" }]), []);
72
+ });
73
+
74
+ test("the order given is kept", () => {
75
+ const a = { id: "a", sealed: true, hasGuide: true, rewrite: "none" };
76
+ const b = { id: "b", sealed: true, hasGuide: true, rewrite: "none" };
77
+ assert.deepEqual(offeredExports([b, a]), [b, a]);
78
+ });
79
+ JS
80
+
81
+ git add -A
82
+ git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: offer sealed exports for download"
@@ -0,0 +1,40 @@
1
+ ---
2
+ type: llm
3
+ # As in proves-a-spec-boundary-bug: the judge cannot run the repro, so "fails
4
+ # in the fixture" is judged against the fixture's code, quoted below, and it
5
+ # reads the trace because findings.json sits under a slug no file focus names.
6
+ focus: trace
7
+ weight: 3
8
+ ---
9
+
10
+ The agent was asked what QA will find in `src/exports.js`. `SPEC.md` says a
11
+ sealed export without a guide stays on offer while its guide is rewritten in
12
+ and if that rewrite fails. The module's own docstring says the opposite (a
13
+ sealed export without a guide whose rewrite failed is never offered), and the
14
+ code follows the docstring:
15
+
16
+ ```js
17
+ if (!entry.sealed) return false;
18
+ if (!entry.hasGuide && entry.rewrite === "failed") return false;
19
+ return true;
20
+ ```
21
+
22
+ So `offeredExports([{ id: "a", sealed: true, hasGuide: false, rewrite: "failed" }])`
23
+ returns `[]` where the spec requires that export to be offered. Nothing else
24
+ in the code departs from the spec.
25
+
26
+ Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
27
+ appears in a Write tool call or a shell command in the trace). If it was
28
+ written more than once, judge only the last write, with any later Edit to it
29
+ applied.
30
+
31
+ PASS if its `bugs` array has an entry whose `claim` resolves, through the
32
+ `intent` array, to the spec's statement that a sealed export stays on offer
33
+ when its rewrite fails (or the spec's broader "offered once it is sealed"
34
+ claim, applied to that export), and whose `repro` asserts that a sealed
35
+ export with `hasGuide: false` and `rewrite: "failed"` is offered, so that it
36
+ fails against the code above.
37
+ FAIL if no findings.json content is visible in the trace, if this case
38
+ appears only in `ambiguities`, if the bug's claim resolves to the docstring
39
+ rather than the spec, or if the repro asserts something the code above
40
+ already does.
@@ -0,0 +1,5 @@
1
+ ---
2
+ type: tool_used
3
+ tool: Skill
4
+ input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
5
+ ---
@@ -0,0 +1,11 @@
1
+ ---
2
+ name: spec-outranks-target-docstring
3
+ description: The spec and a docstring inside the changed code disagree, and the code follows the docstring. Pins a confirmed bug citing the spec claim, not an ambiguity between the two.
4
+ tags: [bug, precedence, scaffold, write]
5
+ max_turns: 80
6
+ timeout_seconds: 2400
7
+ allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
8
+ expected_outcome: findings.json under .bug-hunter/ has a bug citing the spec's claim that a sealed export stays on offer when its rewrite fails, with a repro that fails against the fixture.
9
+ ---
10
+
11
+ We're shipping the download list on Thursday. What will QA find in `src/exports.js`? `SPEC.md` is what it's meant to do.
@@ -0,0 +1,82 @@
1
+ # Findings format
2
+
3
+ Every run writes `report.md` and `findings.json` to
4
+ `.bug-hunter/<YYYY-MM-DD>-<slug>/`, next to `intent.md` and `scenarios.md`.
5
+ They carry the same content; `findings.json` is the one a grader or a calling
6
+ skill parses, so it follows this shape exactly.
7
+
8
+ Write both files with the Write tool, passing the full contents. Do not build
9
+ them with a shell command, a heredoc or a script (`python3 -c` with
10
+ `json.dump`, `node -e` with `JSON.stringify`): a grader reading the run's trace
11
+ sees a script, not the JSON it produced.
12
+
13
+ ## findings.json
14
+
15
+ ```
16
+ { "target": string, "intent": { "id": string, "claim": string, "source": string }[], "bugs": { "id": string, "scenario": string, "claim": string, "expected": string, "observed": string, "repro": { "files": Record<string,string>, "cmd": string } }[], "ambiguities": { "id": string, "scenario": string, "question": string }[], "covered": { "scenario": string, "claim": string }[] }
17
+ ```
18
+
19
+ Field by field:
20
+
21
+ - `target`: the one-line target description the run started from (a commit,
22
+ range or paths).
23
+ - `intent`: the claims from `intent.md`. `id` is the claim's number (`I1`),
24
+ `claim` its text, `source` where it came from (file and line, commit SHA, or
25
+ `request`). A claim taken from a sibling implementation or a stated
26
+ invariant cites the file and line that holds it. A result wrong under every
27
+ reading (SKILL.md) has `source` `baseline`.
28
+ - `bugs`: confirmed findings only. `id` numbers them (`B1`). `scenario` is
29
+ the scenario's id from `scenarios.md` and its one-line label, as
30
+ `"<id>: <label>"`, for example `"S4: quantity of exactly the cap is
31
+ accepted"`. A reader that sees only this field must still know what was
32
+ tested, so never give the bare id. `claim` is the violated intent id.
33
+ `expected` and `observed` are the values the repro asserted and got.
34
+ - `repro.files`: paths relative to the repo root, each mapped to the file's
35
+ full contents, so a grader can write them into a fresh checkout and run
36
+ `repro.cmd` from the root. Include every file the repro needs that is not
37
+ already in the tree; an edited existing file is given in full, not as a
38
+ patch.
39
+ - `repro.cmd`: a command run from the repo root that runs this repro alone and
40
+ exits non-zero while the bug is present.
41
+ - `ambiguities`: `id` numbers them (`A1`); `scenario` is `"<id>: <label>"`,
42
+ as in `bugs`; `question` is what product has to decide.
43
+ - `covered`: scenarios that ran and passed. `scenario` is `"<id>: <label>"`,
44
+ as in `bugs`; `claim` the intent id it checked.
45
+
46
+ Empty buckets are empty arrays, never omitted. Rejected findings and
47
+ exploratory tests appear nowhere in the file.
48
+
49
+ ## report.md
50
+
51
+ ```markdown
52
+ # Bug hunt: <target>
53
+
54
+ <date>. Run directory: `.bug-hunter/<YYYY-MM-DD>-<slug>/`.
55
+ <n> bugs, <n> ambiguities, <n> covered. <scenarios not executed, and why, or "All scenarios executed.">
56
+
57
+ ## Bugs
58
+
59
+ ### B1: <one-line summary>
60
+
61
+ - Scenario: S4, <scenario text>
62
+ - Violates: I2, "<claim text>" (<source>)
63
+ - Expected: <value>
64
+ - Observed: <value>
65
+ - Repro: `<path>` (and any others), run with `<cmd>`
66
+
67
+ ## Ambiguities
68
+
69
+ ### A1: <question>
70
+
71
+ - Scenario: S7, <scenario text>
72
+ - Why it is open: <the claims are silent, or I3 and I5 disagree>
73
+
74
+ ## Covered
75
+
76
+ | Scenario | Claim |
77
+ |---|---|
78
+ | S1, <scenario text> | I1 |
79
+ ```
80
+
81
+ A bucket with no entries keeps its heading with the line `None.` under it, so
82
+ a reader can tell "none found" from "section missing".
@@ -0,0 +1,92 @@
1
+ # Scenario catalog
2
+
3
+ Where QA finds bugs that the author's tests missed. Work through every
4
+ category for the target; a category that gives no purchase gets one line in
5
+ `scenarios.md` saying why, so its absence reads as a decision. Each probe is a
6
+ prompt for a scenario, not a scenario: the expected result still has to come
7
+ from a claim in `intent.md`.
8
+
9
+ ## Boundaries
10
+
11
+ - The value exactly at each stated limit, one below it, and one above it.
12
+ - Zero, one, and many: an empty collection, a single element, the maximum.
13
+ - The first and last item of a page, a range or a batch, and a request for
14
+ the page after the last.
15
+ - Inclusive against exclusive ends: does "until Friday" or "up to 10" include
16
+ the endpoint the claim names.
17
+ - Negative numbers and zero where only positives were imagined.
18
+
19
+ ## Sequence and state
20
+
21
+ - Run the same action twice in a row; the second call should be safe or
22
+ refused, whichever the claim says.
23
+ - Double submit: two identical requests arriving before the first completes.
24
+ - Retry after partial failure: the first attempt wrote half its state and
25
+ failed; the retry must not duplicate or skip the half already written.
26
+ - Out of order: the steps a workflow expects in sequence, sent in a different
27
+ order, or a later step without the earlier one.
28
+ - An action on something already deleted, cancelled, closed or expired.
29
+ - Undo, then redo, then the original action again.
30
+
31
+ ## Combinations of flags, options and roles
32
+
33
+ - Two options that each work alone, set together, especially ones that pull
34
+ in opposite directions (`--force` with `--dry-run`).
35
+ - A role with less permission reaching an action through a path built for a
36
+ more privileged one.
37
+ - An option that the change added, combined with each pre-existing option it
38
+ interacts with.
39
+ - The default for every option left unset, against each one explicitly set to
40
+ its default value.
41
+
42
+ ## Time
43
+
44
+ - A timestamp in a timezone other than the server's, and a user whose
45
+ timezone differs from the data's.
46
+ - The DST transition days: an hour that does not exist, and an hour that
47
+ happens twice.
48
+ - Month end: the 31st plus one month, 29 February, and a non-leap year.
49
+ - Midnight and the date boundary: an event at 23:59:59 and at 00:00:00, and a
50
+ "today" computed in UTC against local time.
51
+ - Year end, and a range that spans it.
52
+
53
+ ## Data shape
54
+
55
+ - Null against missing against empty: a field absent, present as `null`,
56
+ present as `""` or `[]`; are they meant to mean the same thing.
57
+ - Duplicates in input that the claim assumes unique.
58
+ - Ordering: input in reverse or random order where the code may assume sorted,
59
+ and output order where the claim promises one.
60
+ - Unicode: combining characters, emoji, right-to-left text, and case folding
61
+ outside ASCII.
62
+ - Size: a very long string, a very large collection, a deeply nested value.
63
+ - Whitespace and case variants of a value meant to match: leading or trailing
64
+ spaces, mixed case.
65
+
66
+ ## Arithmetic
67
+
68
+ - Rounding: half-way values, and the rounding mode the claim requires against
69
+ the one the language defaults to.
70
+ - Money in floating point: sums of many small amounts, and totals that must
71
+ equal the sum of their rounded parts.
72
+ - Units: seconds against milliseconds, bytes against kilobytes, cents against
73
+ dollars, at each boundary between components.
74
+ - Integer division and remainder with negative operands, and division by zero.
75
+ - Overflow at the type's limit, and percentages over 100 or below 0.
76
+
77
+ ## Partial failure
78
+
79
+ - A dependency call that fails after earlier writes succeeded: what state is
80
+ left, and does the caller learn about it.
81
+ - A timeout that fires after the remote side completed the work.
82
+ - One item failing in a batch: do the others commit, roll back, or report.
83
+ - An error raised inside cleanup or a `finally`, hiding the original one.
84
+ - A process killed between two writes that the claim says happen together.
85
+
86
+ ## Environment
87
+
88
+ - A stale cache serving a value the change made obsolete.
89
+ - Config missing, empty, or set to a value of the wrong type.
90
+ - A feature flag off, on, and flipped while work is in flight.
91
+ - Running against data written by the previous version of the code.
92
+ - A different locale, working directory, or a path with spaces in it.