@iceinvein/agent-skills 0.23.3 → 0.23.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/bug-hunter/SKILL.md +5 -0
- package/skills/bug-hunter/evals/README.md +26 -2
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/case.yaml +4 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/fixture.sh +116 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-for-unlogged-cap-cut.md +50 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-hunter-fired.md +5 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/prompt.md +11 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/case.yaml +4 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/fixture.sh +112 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/graders/bug-for-close-after-rename.md +50 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/graders/bug-hunter-fired.md +5 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/prompt.md +11 -0
- package/skills/bug-hunter/evals/rules.json +14 -3
- package/skills/bug-hunter/skill.json +1 -1
- package/skills/index.json +3 -3
- package/skills/migrate/scripts/__tests__/e2e-express.test.ts +2 -1
- package/skills/migrate/scripts/__tests__/e2e-webforms.test.ts +2 -1
- package/skills/migrate/skill.json +1 -1
- package/skills/sluice/scripts/status.sh +23 -10
- package/skills/sluice/scripts/stop-guard.sh +18 -5
- package/skills/sluice/skill.json +1 -1
package/package.json
CHANGED
|
@@ -74,6 +74,11 @@ command), **ambiguities** (the question for product), **covered** (scenarios
|
|
|
74
74
|
that passed, with the claim each checked). Write `findings.json` and
|
|
75
75
|
`report.md` with the Write tool, never from a shell command or script.
|
|
76
76
|
|
|
77
|
+
Run as a dispatched agent, the Write tool may refuse `report.md` and
|
|
78
|
+
`findings.json` as report files. Then put each refused file's full content in
|
|
79
|
+
the hand-back under its path, for the caller to write unchanged with the Write
|
|
80
|
+
tool; never fall back to a shell write.
|
|
81
|
+
|
|
77
82
|
The `.bug-hunter/` directory is the one path in the caller's tree the skill
|
|
78
83
|
writes. Promoting a repro into the tree as a regression test is the caller's
|
|
79
84
|
job.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# bug-hunter evals
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Nine cases for `claude plugin eval`. Each scaffolds a small Node repo with a
|
|
4
4
|
`SPEC.md`, one module and a passing `node --test` suite, committed in two steps
|
|
5
5
|
(the spec, then the module) so the target reads as a change. The prompt asks
|
|
6
6
|
what QA will find in the module, the way a user would, and never names the
|
|
@@ -15,6 +15,8 @@ skill.
|
|
|
15
15
|
| `bug-cites-intent-from-spec` | A coupon that floors where the spec rounds half up | `intent.md` is written, and the bug cites a claim sourced from `SPEC.md`, not from the code's floor |
|
|
16
16
|
| `spec-outranks-target-docstring` | The spec keeps a sealed export on offer when its rewrite fails; the module's docstring and code drop it | A bug citing the spec claim, not an ambiguity between spec and docstring |
|
|
17
17
|
| `impossible-value-is-a-bug` | The spec is silent on rounding; the code shows `60s` and `1m 60s` | A bug for the seconds field of 60, which no rounding rule gives |
|
|
18
|
+
| `false-success-is-a-bug` | Closing a renamed project answers `{ ok: true }` and leaves its workspace | A bug, not an ambiguity about renames |
|
|
19
|
+
| `every-member-of-a-set-claim` | Every dropped cited source is logged, except one cut by the cap after attachments fill the slots | A bug for the unlogged cut, which no existing test reaches |
|
|
18
20
|
|
|
19
21
|
Every case also has `bug-hunter-fired`, a `tool_used: Skill` grader, so a
|
|
20
22
|
score can be told apart from the model's own habits.
|
|
@@ -30,7 +32,7 @@ claude plugin eval skills/bug-hunter --scaffold --allow-tools Bash Write Edit \
|
|
|
30
32
|
```
|
|
31
33
|
|
|
32
34
|
Add `--keep-temp` to keep each run's sandbox and trace, which is the only way
|
|
33
|
-
to read back why a grader failed. `-j
|
|
35
|
+
to read back why a grader failed. `-j 9` runs the nine cases at once.
|
|
34
36
|
|
|
35
37
|
To check the skill against its last accepted baseline, use `eval:fresh`:
|
|
36
38
|
|
|
@@ -113,6 +115,28 @@ a file needs its exact path and throws on a glob. So:
|
|
|
113
115
|
|
|
114
116
|
## Baseline
|
|
115
117
|
|
|
118
|
+
### Skill 0.4.2, Opus 5.5, 2026-10-20: two guards and a field-only rule
|
|
119
|
+
|
|
120
|
+
A read of 14 hunts in the snowvault transcripts (2026-10-17 to 10-20) found
|
|
121
|
+
three misses. Two became cases, each copying the shape of a real one:
|
|
122
|
+
`false-success-is-a-bug` follows SNO-114, which filed `wt:remove -- SNO-n`
|
|
123
|
+
after a rename as an ambiguity when it exited 0 having released nothing, and
|
|
124
|
+
`every-member-of-a-set-claim` follows SNO-54, which reported 0 bugs on "every
|
|
125
|
+
dropped citation is logged" while one drop site logged nothing. Both scored
|
|
126
|
+
1.00 on the unchanged 0.4.2, two runs each, every judge vote PASS (USD 1.70
|
|
127
|
+
and 1.76, `results/2026-10-20T06-22-11-560Z` and
|
|
128
|
+
`results/2026-10-20T06-22-10-824Z`). As with 0.3.0's cases, the fixture makes
|
|
129
|
+
the defect plain enough that the existing rules catch it, so neither
|
|
130
|
+
reproduces the field miss. They stay as guards, and the skill gained no rule
|
|
131
|
+
for either.
|
|
132
|
+
|
|
133
|
+
The third miss became the rule that a dispatched agent's refused `report.md`
|
|
134
|
+
and `findings.json` go in its hand-back. Every one of the 10 hunts run as a
|
|
135
|
+
dispatched agent had both Writes refused ("Subagents should return findings
|
|
136
|
+
as text, not write report files"), so `findings.json` existed for 4 of the 14
|
|
137
|
+
runs. No case pins it: an eval runs the skill at the top level, where the
|
|
138
|
+
Write succeeds.
|
|
139
|
+
|
|
116
140
|
### Skill 0.4.0, Opus 5.5, 2026-10-13
|
|
117
141
|
|
|
118
142
|
Same command, `-j 7`. Skill 0.4.0 drops the three dispatched roles for one
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A draft-source picker whose spec says every cited source it does not return
|
|
3
|
+
# is logged once with a reason. Sources are dropped at three sites: a cited id
|
|
4
|
+
# with no document, a document with no file name, and the five-slot cap. The
|
|
5
|
+
# cap is applied after the reader's own attachments are merged in first, but
|
|
6
|
+
# over_cap is only logged when the cited sources alone exceed five, so a cited
|
|
7
|
+
# source cut because attachments filled the slots leaves no log line. The
|
|
8
|
+
# existing tests cover each reason on the cited-only path. This is the shape
|
|
9
|
+
# of the snowvault SNO-54 hunt, which was given exactly this claim, checked
|
|
10
|
+
# behaviour against the base with 20,000 random cases, reported 0 bugs, and
|
|
11
|
+
# missed the attachment cut that the reviewer then found.
|
|
12
|
+
set -euo pipefail
|
|
13
|
+
|
|
14
|
+
mkdir -p src tests
|
|
15
|
+
|
|
16
|
+
cat > package.json <<'JSON'
|
|
17
|
+
{
|
|
18
|
+
"name": "draft-sources",
|
|
19
|
+
"version": "1.4.0",
|
|
20
|
+
"type": "module",
|
|
21
|
+
"scripts": {
|
|
22
|
+
"test": "node --test \"tests/*.test.js\""
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
JSON
|
|
26
|
+
|
|
27
|
+
cat > SPEC.md <<'MD'
|
|
28
|
+
# Draft sources
|
|
29
|
+
|
|
30
|
+
`pickSources({ attachments, citedIds, documents, log })` in `src/sources.js`
|
|
31
|
+
picks the documents that seed a draft.
|
|
32
|
+
|
|
33
|
+
- `attachments` are documents the reader attached; they always come first.
|
|
34
|
+
- `citedIds` are document ids the earlier answer cited, looked up in
|
|
35
|
+
`documents` (an object keyed by id). Cited sources fill what is left.
|
|
36
|
+
- At most 5 documents are returned in all.
|
|
37
|
+
- A cited document with no `fileName` is not used.
|
|
38
|
+
- Every cited source that is not returned is logged once, as
|
|
39
|
+
`log({ id, reason })`, where `reason` is one of `not_found`,
|
|
40
|
+
`no_file_name` or `over_cap`.
|
|
41
|
+
MD
|
|
42
|
+
|
|
43
|
+
git init --quiet
|
|
44
|
+
git add -A
|
|
45
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: draft sources"
|
|
46
|
+
|
|
47
|
+
cat > src/sources.js <<'JS'
|
|
48
|
+
const MAX_SOURCES = 5;
|
|
49
|
+
|
|
50
|
+
function citedSources(citedIds, documents, log) {
|
|
51
|
+
const picked = [];
|
|
52
|
+
for (const id of citedIds) {
|
|
53
|
+
const doc = documents[id];
|
|
54
|
+
if (!doc) {
|
|
55
|
+
log({ id, reason: "not_found" });
|
|
56
|
+
continue;
|
|
57
|
+
}
|
|
58
|
+
if (!doc.fileName) {
|
|
59
|
+
log({ id, reason: "no_file_name" });
|
|
60
|
+
continue;
|
|
61
|
+
}
|
|
62
|
+
picked.push(doc);
|
|
63
|
+
}
|
|
64
|
+
for (const doc of picked.slice(MAX_SOURCES)) {
|
|
65
|
+
log({ id: doc.id, reason: "over_cap" });
|
|
66
|
+
}
|
|
67
|
+
return picked;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
export function pickSources({ attachments, citedIds, documents, log }) {
|
|
71
|
+
const cited = citedSources(citedIds, documents, log);
|
|
72
|
+
return [...attachments, ...cited].slice(0, MAX_SOURCES);
|
|
73
|
+
}
|
|
74
|
+
JS
|
|
75
|
+
|
|
76
|
+
cat > tests/sources.test.js <<'JS'
|
|
77
|
+
import assert from "node:assert/strict";
|
|
78
|
+
import { test } from "node:test";
|
|
79
|
+
import { pickSources } from "../src/sources.js";
|
|
80
|
+
|
|
81
|
+
const doc = (id) => ({ id, fileName: `${id}.pdf` });
|
|
82
|
+
|
|
83
|
+
function run(input) {
|
|
84
|
+
const lines = [];
|
|
85
|
+
const picked = pickSources({ attachments: [], documents: {}, ...input, log: (l) => lines.push(l) });
|
|
86
|
+
return { ids: picked.map((d) => d.id), lines };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
test("attachments come before cited sources", () => {
|
|
90
|
+
const { ids } = run({ attachments: [doc("a1")], citedIds: ["c1"], documents: { c1: doc("c1") } });
|
|
91
|
+
assert.deepEqual(ids, ["a1", "c1"]);
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
test("a cited id with no document is logged as not_found", () => {
|
|
95
|
+
const { ids, lines } = run({ citedIds: ["c9"] });
|
|
96
|
+
assert.deepEqual(ids, []);
|
|
97
|
+
assert.deepEqual(lines, [{ id: "c9", reason: "not_found" }]);
|
|
98
|
+
});
|
|
99
|
+
|
|
100
|
+
test("a cited document with no file name is logged as no_file_name", () => {
|
|
101
|
+
const { ids, lines } = run({ citedIds: ["c1"], documents: { c1: { id: "c1" } } });
|
|
102
|
+
assert.deepEqual(ids, []);
|
|
103
|
+
assert.deepEqual(lines, [{ id: "c1", reason: "no_file_name" }]);
|
|
104
|
+
});
|
|
105
|
+
|
|
106
|
+
test("cited sources beyond five are logged as over_cap", () => {
|
|
107
|
+
const ids6 = ["c1", "c2", "c3", "c4", "c5", "c6"];
|
|
108
|
+
const documents = Object.fromEntries(ids6.map((id) => [id, doc(id)]));
|
|
109
|
+
const { ids, lines } = run({ citedIds: ids6, documents });
|
|
110
|
+
assert.deepEqual(ids, ["c1", "c2", "c3", "c4", "c5"]);
|
|
111
|
+
assert.deepEqual(lines, [{ id: "c6", reason: "over_cap" }]);
|
|
112
|
+
});
|
|
113
|
+
JS
|
|
114
|
+
|
|
115
|
+
git add -A
|
|
116
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: pick draft sources"
|
package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-for-unlogged-cap-cut.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
# As in impossible-value-is-a-bug: the judge cannot run the repro, so "fails
|
|
4
|
+
# in the fixture" is judged against the fixture's code, quoted below.
|
|
5
|
+
focus: trace
|
|
6
|
+
weight: 3
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
The agent was asked what QA will find in `src/sources.js`. `SPEC.md` says
|
|
10
|
+
attachments come first, cited sources fill what is left, at most 5 documents
|
|
11
|
+
are returned, and every cited source that is not returned is logged once with
|
|
12
|
+
reason `not_found`, `no_file_name` or `over_cap`. The fixture's code is:
|
|
13
|
+
|
|
14
|
+
```js
|
|
15
|
+
function citedSources(citedIds, documents, log) {
|
|
16
|
+
const picked = [];
|
|
17
|
+
for (const id of citedIds) {
|
|
18
|
+
const doc = documents[id];
|
|
19
|
+
if (!doc) { log({ id, reason: "not_found" }); continue; }
|
|
20
|
+
if (!doc.fileName) { log({ id, reason: "no_file_name" }); continue; }
|
|
21
|
+
picked.push(doc);
|
|
22
|
+
}
|
|
23
|
+
for (const doc of picked.slice(MAX_SOURCES)) {
|
|
24
|
+
log({ id: doc.id, reason: "over_cap" });
|
|
25
|
+
}
|
|
26
|
+
return picked;
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
export function pickSources({ attachments, citedIds, documents, log }) {
|
|
30
|
+
const cited = citedSources(citedIds, documents, log);
|
|
31
|
+
return [...attachments, ...cited].slice(0, MAX_SOURCES);
|
|
32
|
+
}
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
So with, for example, 4 attachments and 2 valid cited sources, the second
|
|
36
|
+
cited source is not returned and no log line names it, because `over_cap` is
|
|
37
|
+
only logged for cited sources beyond five on their own.
|
|
38
|
+
|
|
39
|
+
Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
|
|
40
|
+
appears in a Write tool call or a shell command in the trace, or, if the Write
|
|
41
|
+
was refused, in the agent's final message). If it was written more than once,
|
|
42
|
+
judge only the last write, with any later Edit to it applied.
|
|
43
|
+
|
|
44
|
+
PASS if its `bugs` array has an entry for a cited source cut because
|
|
45
|
+
attachments took slots, which is neither returned nor logged, and whose
|
|
46
|
+
`repro` asserts a log line (or a returned source) that the code above does not
|
|
47
|
+
give for that input, so that it fails against the code.
|
|
48
|
+
FAIL if no findings.json content is visible in the trace, if `bugs` has no
|
|
49
|
+
entry for the attachment case, if it appears only in `ambiguities`, or if the
|
|
50
|
+
repro asserts something the code above already does.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: every-member-of-a-set-claim
|
|
3
|
+
description: A claim that every dropped cited source is logged, held at two of three drop sites. The existing tests cover each reason on the cited-only path; the cap cut after attachments are merged in logs nothing. Pins the hunt checking the claim at every site that drops a source, not only the ones the tests reach.
|
|
4
|
+
tags: [bug, coverage, scaffold, write]
|
|
5
|
+
max_turns: 80
|
|
6
|
+
timeout_seconds: 2400
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
|
|
8
|
+
expected_outcome: findings.json under .bug-hunter/ has a bug for a cited source cut by the five-slot cap when attachments fill slots, returned nowhere and logged nowhere, with a repro that fails against the fixture.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Before this goes to QA: what will they find in `src/sources.js`? It's meant to do what `SPEC.md` says.
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A project workspace registry whose close step derives the workspace name from
|
|
3
|
+
# the project's current title. The workspace was named from the title at open,
|
|
4
|
+
# so after a rename close removes nothing and still answers { ok: true }. The
|
|
5
|
+
# spec never mentions renaming a project and then closing it, which makes "should close
|
|
6
|
+
# find a workspace opened under the old title?" look like a question for
|
|
7
|
+
# product. It is not: whichever answer product gives, a close that reports
|
|
8
|
+
# success and left the workspace in place is wrong. This is the shape of the
|
|
9
|
+
# snowvault SNO-114 hunt, which filed `wt:remove -- SNO-n` after a rename as an
|
|
10
|
+
# ambiguity; review later found it released nothing and exited 0.
|
|
11
|
+
set -euo pipefail
|
|
12
|
+
|
|
13
|
+
mkdir -p src tests
|
|
14
|
+
|
|
15
|
+
cat > package.json <<'JSON'
|
|
16
|
+
{
|
|
17
|
+
"name": "workspaces",
|
|
18
|
+
"version": "2.3.0",
|
|
19
|
+
"type": "module",
|
|
20
|
+
"scripts": {
|
|
21
|
+
"test": "node --test \"tests/*.test.js\""
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
JSON
|
|
25
|
+
|
|
26
|
+
cat > SPEC.md <<'MD'
|
|
27
|
+
# Project workspaces
|
|
28
|
+
|
|
29
|
+
`src/workspaces.js` keeps a workspace for each open project. A project has a
|
|
30
|
+
key (`P-12`) that never changes and a title that can be renamed at any time.
|
|
31
|
+
|
|
32
|
+
- `openProject(state, key, title)` registers the project and creates its
|
|
33
|
+
workspace.
|
|
34
|
+
- `renameProject(state, key, title)` changes the project's title.
|
|
35
|
+
- `closeProject(state, key)` removes the project's workspace and returns
|
|
36
|
+
`{ ok: true }`. For a key that names no project it returns
|
|
37
|
+
`{ ok: false, error: "unknown project" }`.
|
|
38
|
+
- `workspaceCount(state)` is the number of workspaces that exist.
|
|
39
|
+
MD
|
|
40
|
+
|
|
41
|
+
git init --quiet
|
|
42
|
+
git add -A
|
|
43
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: project workspaces"
|
|
44
|
+
|
|
45
|
+
cat > src/workspaces.js <<'JS'
|
|
46
|
+
export function createState() {
|
|
47
|
+
return { projects: new Map(), workspaces: new Set() };
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function workspaceName(key, title) {
|
|
51
|
+
const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "");
|
|
52
|
+
return `${key.toLowerCase()}-${slug}`;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
export function openProject(state, key, title) {
|
|
56
|
+
state.projects.set(key, { title });
|
|
57
|
+
state.workspaces.add(workspaceName(key, title));
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export function renameProject(state, key, title) {
|
|
61
|
+
const project = state.projects.get(key);
|
|
62
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
63
|
+
project.title = title;
|
|
64
|
+
return { ok: true };
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
export function closeProject(state, key) {
|
|
68
|
+
const project = state.projects.get(key);
|
|
69
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
70
|
+
state.workspaces.delete(workspaceName(key, project.title));
|
|
71
|
+
state.projects.delete(key);
|
|
72
|
+
return { ok: true };
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
export function workspaceCount(state) {
|
|
76
|
+
return state.workspaces.size;
|
|
77
|
+
}
|
|
78
|
+
JS
|
|
79
|
+
|
|
80
|
+
cat > tests/workspaces.test.js <<'JS'
|
|
81
|
+
import assert from "node:assert/strict";
|
|
82
|
+
import { test } from "node:test";
|
|
83
|
+
import { createState, openProject, renameProject, closeProject, workspaceCount } from "../src/workspaces.js";
|
|
84
|
+
|
|
85
|
+
test("opening a project creates its workspace", () => {
|
|
86
|
+
const state = createState();
|
|
87
|
+
openProject(state, "P-1", "Launch plan");
|
|
88
|
+
assert.equal(workspaceCount(state), 1);
|
|
89
|
+
});
|
|
90
|
+
|
|
91
|
+
test("closing a project removes its workspace", () => {
|
|
92
|
+
const state = createState();
|
|
93
|
+
openProject(state, "P-1", "Launch plan");
|
|
94
|
+
assert.deepEqual(closeProject(state, "P-1"), { ok: true });
|
|
95
|
+
assert.equal(workspaceCount(state), 0);
|
|
96
|
+
});
|
|
97
|
+
|
|
98
|
+
test("renaming a project keeps it open", () => {
|
|
99
|
+
const state = createState();
|
|
100
|
+
openProject(state, "P-1", "Launch plan");
|
|
101
|
+
assert.deepEqual(renameProject(state, "P-1", "Launch plan v2"), { ok: true });
|
|
102
|
+
assert.equal(workspaceCount(state), 1);
|
|
103
|
+
});
|
|
104
|
+
|
|
105
|
+
test("closing an unknown key is refused", () => {
|
|
106
|
+
const state = createState();
|
|
107
|
+
assert.deepEqual(closeProject(state, "P-9"), { ok: false, error: "unknown project" });
|
|
108
|
+
});
|
|
109
|
+
JS
|
|
110
|
+
|
|
111
|
+
git add -A
|
|
112
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: project workspaces"
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
# As in impossible-value-is-a-bug: the judge cannot run the repro, so "fails
|
|
4
|
+
# in the fixture" is judged against the fixture's code, quoted below.
|
|
5
|
+
focus: trace
|
|
6
|
+
weight: 3
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
The agent was asked what QA will find in `src/workspaces.js`. `SPEC.md` says a
|
|
10
|
+
project's title can be renamed at any time, and that `closeProject(state, key)`
|
|
11
|
+
removes the project's workspace and returns `{ ok: true }`. It never mentions
|
|
12
|
+
renaming a project and then closing it. The fixture's code is:
|
|
13
|
+
|
|
14
|
+
```js
|
|
15
|
+
function workspaceName(key, title) {
|
|
16
|
+
const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "");
|
|
17
|
+
return `${key.toLowerCase()}-${slug}`;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export function openProject(state, key, title) {
|
|
21
|
+
state.projects.set(key, { title });
|
|
22
|
+
state.workspaces.add(workspaceName(key, title));
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
export function closeProject(state, key) {
|
|
26
|
+
const project = state.projects.get(key);
|
|
27
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
28
|
+
state.workspaces.delete(workspaceName(key, project.title));
|
|
29
|
+
state.projects.delete(key);
|
|
30
|
+
return { ok: true };
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
So after `renameProject`, `closeProject` returns `{ ok: true }` while the
|
|
35
|
+
workspace opened under the old title still exists (`workspaceCount` stays 1).
|
|
36
|
+
|
|
37
|
+
Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
|
|
38
|
+
appears in a Write tool call or a shell command in the trace, or, if the Write
|
|
39
|
+
was refused, in the agent's final message). If it was written more than once,
|
|
40
|
+
judge only the last write, with any later Edit to it applied.
|
|
41
|
+
|
|
42
|
+
PASS if its `bugs` array has an entry for closing a renamed project that
|
|
43
|
+
reports success but leaves the workspace, and whose `repro` asserts a result
|
|
44
|
+
the code above does not give (for example `workspaceCount` of 0 after rename
|
|
45
|
+
then close, or a result other than `{ ok: true }` with the workspace left), so
|
|
46
|
+
that it fails against the code. An ambiguity about renames may also be present;
|
|
47
|
+
it does not affect the verdict.
|
|
48
|
+
FAIL if no findings.json content is visible in the trace, if close-after-rename
|
|
49
|
+
appears only in `ambiguities` or only in the report's latent notes, or if the
|
|
50
|
+
repro asserts something the code above already does.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: false-success-is-a-bug
|
|
3
|
+
description: A close that answers ok after a rename but leaves the workspace in place. The spec never mentions renaming and then closing, so the case reads like a product question, but a reported success that did not do the work is wrong under every answer. Pins a bug rather than an ambiguity.
|
|
4
|
+
tags: [bug, baseline, scaffold, write]
|
|
5
|
+
max_turns: 80
|
|
6
|
+
timeout_seconds: 2400
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
|
|
8
|
+
expected_outcome: "findings.json under .bug-hunter/ has a bug for closing a renamed project, which returns { ok: true } and leaves its workspace, with a repro that fails against the fixture."
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Before this goes to QA: what will they find in `src/workspaces.js`? It's meant to do what `SPEC.md` says.
|
|
@@ -76,7 +76,9 @@
|
|
|
76
76
|
"correct-code-has-no-bugs",
|
|
77
77
|
"bug-cites-intent-from-spec",
|
|
78
78
|
"spec-outranks-target-docstring",
|
|
79
|
-
"impossible-value-is-a-bug"
|
|
79
|
+
"impossible-value-is-a-bug",
|
|
80
|
+
"false-success-is-a-bug",
|
|
81
|
+
"every-member-of-a-set-claim"
|
|
80
82
|
]
|
|
81
83
|
},
|
|
82
84
|
{
|
|
@@ -86,7 +88,8 @@
|
|
|
86
88
|
"cases": [
|
|
87
89
|
"proves-a-spec-boundary-bug",
|
|
88
90
|
"bug-cites-intent-from-spec",
|
|
89
|
-
"spec-outranks-target-docstring"
|
|
91
|
+
"spec-outranks-target-docstring",
|
|
92
|
+
"every-member-of-a-set-claim"
|
|
90
93
|
]
|
|
91
94
|
},
|
|
92
95
|
{
|
|
@@ -109,7 +112,8 @@
|
|
|
109
112
|
"states": "SKILL.md: What counts as a bug",
|
|
110
113
|
"origin": "commit:23d4466",
|
|
111
114
|
"cases": [
|
|
112
|
-
"impossible-value-is-a-bug"
|
|
115
|
+
"impossible-value-is-a-bug",
|
|
116
|
+
"false-success-is-a-bug"
|
|
113
117
|
]
|
|
114
118
|
},
|
|
115
119
|
{
|
|
@@ -145,6 +149,13 @@
|
|
|
145
149
|
"correct-code-has-no-bugs"
|
|
146
150
|
]
|
|
147
151
|
},
|
|
152
|
+
{
|
|
153
|
+
"id": "refused-report-writes-go-in-the-hand-back",
|
|
154
|
+
"states": "SKILL.md: Output",
|
|
155
|
+
"origin": "skills/bug-hunter/evals/README.md",
|
|
156
|
+
"cases": [],
|
|
157
|
+
"unpinned": "The refusal comes from the harness only inside a dispatched agent, and an eval case runs the skill at the top level, where the Write succeeds."
|
|
158
|
+
},
|
|
148
159
|
{
|
|
149
160
|
"id": "bug-hunter-dir-is-the-only-path-written",
|
|
150
161
|
"states": "SKILL.md: Output",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bug-hunter",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.3",
|
|
4
4
|
"description": "Hunts logic and correctness bugs in a change or area and proves each one with a failing reproduction checked against stated intent. One agent collects intent from the request, specs, docs and unchanged code, runs QA-style scenarios from a catalog as repros in the repo's own test runner inside a scratch copy, and re-runs each failure clean before reporting it. Counts a contradicted claim, or a result wrong under every reading, as a bug; keeps ambiguities for questions with more than one acceptable answer; reports latent defects without filing them. No repro, no finding. Writes report.md and findings.json (bugs, ambiguities, covered) to .bug-hunter/<date>-<slug>/ and never modifies the caller's tree. Language-agnostic; ships no per-language scripts.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|
package/skills/index.json
CHANGED
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
"name": "bug-hunter",
|
|
14
14
|
"description": "Hunts logic and correctness bugs in a change or area and proves each one with a failing reproduction checked against stated intent. One agent collects intent from the request, specs, docs and unchanged code, runs QA-style scenarios from a catalog as repros in the repo's own test runner inside a scratch copy, and re-runs each failure clean before reporting it. Counts a contradicted claim, or a result wrong under every reading, as a bug; keeps ambiguities for questions with more than one acceptable answer; reports latent defects without filing them. No repro, no finding. Writes report.md and findings.json (bugs, ambiguities, covered) to .bug-hunter/<date>-<slug>/ and never modifies the caller's tree. Language-agnostic; ships no per-language scripts.",
|
|
15
15
|
"type": "prompt",
|
|
16
|
-
"version": "0.4.
|
|
16
|
+
"version": "0.4.3"
|
|
17
17
|
},
|
|
18
18
|
{
|
|
19
19
|
"name": "codebase-architecture",
|
|
@@ -233,7 +233,7 @@
|
|
|
233
233
|
"name": "migrate",
|
|
234
234
|
"description": "Source-agnostic legacy migration mapping. Walks a legacy codebase through probe, enumerate, seam, extract, parity, queue, adjudicate and handoff, building an auditable requirements ledger with mandatory citations and a `migrate check` gate in place of self-reported completeness. Use when the user asks to migrate, re-specify, replatform, or map a legacy system onto a new stack, to hand mapped requirements to a delivery team, or to resume, check, report, or forecast a mapping run already under way.",
|
|
235
235
|
"type": "prompt",
|
|
236
|
-
"version": "0.4.
|
|
236
|
+
"version": "0.4.4"
|
|
237
237
|
},
|
|
238
238
|
{
|
|
239
239
|
"name": "module-secret-auditor",
|
|
@@ -295,7 +295,7 @@
|
|
|
295
295
|
"name": "sluice",
|
|
296
296
|
"description": "Routes work by change shape into four channels (bypass, fast, main, deep) and applies only the rules each channel needs, so a one-line fix does not pay the cost of a multi-subsystem build. Carries nine rules as one-liners in the router and the full treatment in references read only on friction. Checks the finished plan with plan.sh validate rather than trusting it to memory, seeds the run state from it, keeps a deep run's task breakdown in .sluice/run.json so a statusline segment, one status command and a SessionStart hook can answer where the run is (the hook prints a live run at every session start, compaction included), and closes each run with a ledger read out of the session transcript: elapsed, tools, tokens, and what each dispatched agent cost where the transcript recorded it. Two Stop hooks hold a handback: one refuses a tree that moved since its last recorded check, the other refuses a changed existing test not proven to fail on the session's starting code (or recorded with a reason). Claude Code only; stands down where the superpowers pipeline governs the repo.",
|
|
297
297
|
"type": "prompt",
|
|
298
|
-
"version": "0.25.
|
|
298
|
+
"version": "0.25.5"
|
|
299
299
|
},
|
|
300
300
|
{
|
|
301
301
|
"name": "temporal-coupling-detector",
|
|
@@ -755,6 +755,7 @@ function readTomlString(text: string, key: string): string {
|
|
|
755
755
|
return match[1]
|
|
756
756
|
}
|
|
757
757
|
|
|
758
|
+
// Drives a whole run through several gates; it takes about 5s, which is bun's default limit.
|
|
758
759
|
test('contract-only run driven probe through handoff, ending green at a plain check', async () => {
|
|
759
760
|
const groundTruthPath = join(source, 'GROUND-TRUTH.md')
|
|
760
761
|
const rows = await parseGroundTruth(groundTruthPath)
|
|
@@ -1250,4 +1251,4 @@ test('contract-only run driven probe through handoff, ending green at a plain ch
|
|
|
1250
1251
|
expect(reopened.out).toContain(QUEUE_ID)
|
|
1251
1252
|
await writeFile(reopenPath, ruledText)
|
|
1252
1253
|
expect((await migrate(['check'])).code).toBe(0)
|
|
1253
|
-
})
|
|
1254
|
+
}, 60000)
|
|
@@ -935,6 +935,7 @@ function readTomlString(text: string, key: string): string {
|
|
|
935
935
|
return match[1]
|
|
936
936
|
}
|
|
937
937
|
|
|
938
|
+
// Drives a whole run through several gates; it takes about 4s, close to bun's 5s default limit.
|
|
938
939
|
test('aspnet recipe run driven probe through queue, ending green at check --phase queue', async () => {
|
|
939
940
|
const groundTruthPath = join(source, 'GROUND-TRUTH.md')
|
|
940
941
|
const rows = await parseGroundTruth(groundTruthPath)
|
|
@@ -1278,4 +1279,4 @@ test('aspnet recipe run driven probe through queue, ending green at check --phas
|
|
|
1278
1279
|
expect(afterRemoval.out).toContain(
|
|
1279
1280
|
`lens census for ${mutatedSurface} claims in_ledger 0 + added ${claimed} = ${claimed} element(s) in the ledger, but elements.jsonl has ${actual}`,
|
|
1280
1281
|
)
|
|
1281
|
-
})
|
|
1282
|
+
}, 60000)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "migrate",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.4",
|
|
4
4
|
"description": "Source-agnostic legacy migration mapping. Walks a legacy codebase through probe, enumerate, seam, extract, parity, queue, adjudicate and handoff, building an auditable requirements ledger with mandatory citations and a `migrate check` gate in place of self-reported completeness. Use when the user asks to migrate, re-specify, replatform, or map a legacy system onto a new stack, to hand mapped requirements to a delivery team, or to resume, check, report, or forecast a mapping run already under way.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|
|
@@ -596,12 +596,17 @@ $(cd "$MAIN_ROOT" && pwd -P)"
|
|
|
596
596
|
exit 0
|
|
597
597
|
fi
|
|
598
598
|
|
|
599
|
-
|
|
600
|
-
#
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
599
|
+
# In the repo's common git dir rather than the tree, so every worktree reads
|
|
600
|
+
# the same records: a test recorded where it was edited still counts in the
|
|
601
|
+
# tree it is merged into. Inside the git dir it is never content of the tree,
|
|
602
|
+
# so it cannot move the tree it records. One file per call, because sessions
|
|
603
|
+
# in several worktrees record at once and bash splits a long append into
|
|
604
|
+
# several writes that interleave. The tree's own .sluice/tests-changed is
|
|
605
|
+
# where records were written before, and is still read.
|
|
606
|
+
COMMON="$(git -C "$TREE" rev-parse --path-format=absolute --git-common-dir 2>/dev/null)"
|
|
607
|
+
[ -n "$COMMON" ] || { err "could not find the git dir of $TREE, so nothing was recorded"; exit 1; }
|
|
608
|
+
RECORDS="$COMMON/sluice/tests-changed"
|
|
609
|
+
mkdir -p "$RECORDS" || { err "could not make $RECORDS"; exit 1; }
|
|
605
610
|
|
|
606
611
|
# Hashed path-aware, so filters apply, the way the stop gate hashes it. A
|
|
607
612
|
# directory where the file was (a fixture split into many) is that file's
|
|
@@ -630,14 +635,16 @@ $(cd "$MAIN_ROOT" && pwd -P)"
|
|
|
630
635
|
while IFS= read -r p; do
|
|
631
636
|
sha="$(current_blob "$p")" || exit 1
|
|
632
637
|
found=""
|
|
633
|
-
|
|
638
|
+
for rec in "$RECORDS"/* "$TREE/.sluice/tests-changed"; do
|
|
639
|
+
[ -f "$rec" ] || continue
|
|
634
640
|
while IFS=$'\t' read -r rec_sha rec_path rest; do
|
|
635
641
|
if [ "$rec_sha" = "$sha" ] && [ "$rec_path" = "$p" ]; then
|
|
636
642
|
found=1
|
|
637
643
|
break
|
|
638
644
|
fi
|
|
639
|
-
done <"$
|
|
640
|
-
|
|
645
|
+
done <"$rec"
|
|
646
|
+
[ -n "$found" ] && break
|
|
647
|
+
done
|
|
641
648
|
[ -n "$found" ] || UNRECORDED="$UNRECORDED$p"$'\n'
|
|
642
649
|
done <<<"$FLAGGED"
|
|
643
650
|
FLAGGED="${UNRECORDED%$'\n'}"
|
|
@@ -754,7 +761,13 @@ $(cd "$MAIN_ROOT" && pwd -P)"
|
|
|
754
761
|
"
|
|
755
762
|
COUNT=$((COUNT + 1))
|
|
756
763
|
done <<<"$FLAGGED"
|
|
757
|
-
|
|
764
|
+
# Written under a dot name and renamed, so a reader never sees half of it.
|
|
765
|
+
NEW="$(mktemp "$RECORDS/.XXXXXX")" || { err "could not make a record file in $RECORDS"; exit 1; }
|
|
766
|
+
printf '%s' "$LINES" >"$NEW" && mv "$NEW" "$RECORDS/${NEW##*/.}" || {
|
|
767
|
+
rm -f "$NEW"
|
|
768
|
+
err "could not write a record to $RECORDS"
|
|
769
|
+
exit 1
|
|
770
|
+
}
|
|
758
771
|
echo "sluice: recorded $COUNT changed test(s) as $STATUS"
|
|
759
772
|
[ -z "$SKIPPED" ] || echo "sluice: not recorded, since no run can prove a deletion: $SKIPPED; record it with --none <reason>"
|
|
760
773
|
exit 0
|
|
@@ -240,7 +240,7 @@ fi
|
|
|
240
240
|
# A test edited until it agrees with the new code proves nothing about that
|
|
241
241
|
# code, and a passing check cannot tell that edit from an honest one. So a
|
|
242
242
|
# changed existing test (a removed line, a deletion, a skip or focus marker,
|
|
243
|
-
# as test-changes.sh decides) needs a line in
|
|
243
|
+
# as test-changes.sh decides) needs a line in the repo's tests-changed record naming its
|
|
244
244
|
# current content: either proved red on the session's base by
|
|
245
245
|
# `status.sh tests-changed -- <cmd>`, or recorded unproven with a reason. An
|
|
246
246
|
# edit made after its record names other content and is refused again. A new
|
|
@@ -261,6 +261,17 @@ test_gate() {
|
|
|
261
261
|
flagged="$(bash "$here/test-changes.sh" "$moved_tree" "$base" 2>/dev/null)" || return 0
|
|
262
262
|
[ -n "$flagged" ] || return 0
|
|
263
263
|
|
|
264
|
+
# Records live in the repo's common git dir, one file per status.sh call,
|
|
265
|
+
# shared by every worktree, so a test recorded in the worktree it was edited
|
|
266
|
+
# in still counts once merged here. A record names content, not a tree, so it
|
|
267
|
+
# holds wherever that content is. The tree's own .sluice/tests-changed is
|
|
268
|
+
# where records were written before, and those still stand.
|
|
269
|
+
local common records rec
|
|
270
|
+
common="$(git -C "$moved_tree" rev-parse --path-format=absolute --git-common-dir 2>/dev/null)"
|
|
271
|
+
records=()
|
|
272
|
+
[ -n "$common" ] && records+=("$common/sluice/tests-changed"/*)
|
|
273
|
+
records+=("$moved_tree/.sluice/tests-changed")
|
|
274
|
+
|
|
264
275
|
missing=""
|
|
265
276
|
while IFS= read -r p; do
|
|
266
277
|
if [ -f "$moved_tree/$p" ]; then
|
|
@@ -278,14 +289,16 @@ test_gate() {
|
|
|
278
289
|
sha="-"
|
|
279
290
|
fi
|
|
280
291
|
found=""
|
|
281
|
-
|
|
292
|
+
for rec in "${records[@]}"; do
|
|
293
|
+
[ -f "$rec" ] || continue
|
|
282
294
|
while IFS=$'\t' read -r rec_sha rec_path rest; do
|
|
283
295
|
if [ "$rec_sha" = "$sha" ] && [ "$rec_path" = "$p" ]; then
|
|
284
296
|
found=1
|
|
285
297
|
break
|
|
286
298
|
fi
|
|
287
|
-
done <"$
|
|
288
|
-
|
|
299
|
+
done <"$rec"
|
|
300
|
+
[ -n "$found" ] && break
|
|
301
|
+
done
|
|
289
302
|
[ -n "$found" ] || missing="$missing$p"$'\n'
|
|
290
303
|
done <<<"$flagged"
|
|
291
304
|
[ -n "$missing" ] || return 0
|
|
@@ -296,7 +309,7 @@ test_gate() {
|
|
|
296
309
|
decision: "block",
|
|
297
310
|
reason: ("sluice: this session changed existing tests in a way that asks less of the code (an edited or removed assertion, a deleted test, a skip or focus marker), and nothing records why: "
|
|
298
311
|
+ ($paths | split("\n") | map(select(. != "") | gsub("[\u0000-\u001f\u007f]"; "")) | join(", "))
|
|
299
|
-
+ ". A test bent to fit new code proves nothing about it, so each one needs a record against the session base \($base). Prove it fails on the old code: status.sh tests-changed --because \"<the behaviour the change replaces>\" -- <the suite, scoped to these files> (the run is in a scratch worktree of the base, so a node suite usually needs --with node_modules). If it cannot run on the old code, record why: status.sh tests-changed --because \"<the behaviour the change replaces>\" --none \"<why it cannot run on the old code>\". No run can prove a deleted test, so prove the edited ones first, then record each deletion with --none. --only <path> proves or records just those paths, so a fixture that cannot be proved can go to --none --only <fixture> on its own. Then name each changed test, and why it changed, in your reply.")
|
|
312
|
+
+ ". A test bent to fit new code proves nothing about it, so each one needs a record against the session base \($base). Prove it fails on the old code: status.sh tests-changed --because \"<the behaviour the change replaces>\" -- <the suite, scoped to these files> (the run is in a scratch worktree of the base, so a node suite usually needs --with node_modules). If it cannot run on the old code, record why: status.sh tests-changed --because \"<the behaviour the change replaces>\" --none \"<why it cannot run on the old code>\". No run can prove a deleted test, so prove the edited ones first, then record each deletion with --none. --only <path> proves or records just those paths, so a fixture that cannot be proved can go to --none --only <fixture> on its own. Then name each changed test, and why it changed, in your reply. The base is where this tree stood when the session opened, so a test that arrived through a merge of another session into this tree since then is listed too. Do not give a reason for a change that is not yours: record it as what it is, status.sh tests-changed --because \"arrived through the merge of <branch>, not from this session\" --none \"not this session to prove\" --only <path>, and name it in your reply as not from this session.")
|
|
300
313
|
}'
|
|
301
314
|
return 1
|
|
302
315
|
}
|
package/skills/sluice/skill.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sluice",
|
|
3
|
-
"version": "0.25.
|
|
3
|
+
"version": "0.25.5",
|
|
4
4
|
"description": "Routes work by change shape into four channels (bypass, fast, main, deep) and applies only the rules each channel needs, so a one-line fix does not pay the cost of a multi-subsystem build. Carries nine rules as one-liners in the router and the full treatment in references read only on friction. Checks the finished plan with plan.sh validate rather than trusting it to memory, seeds the run state from it, keeps a deep run's task breakdown in .sluice/run.json so a statusline segment, one status command and a SessionStart hook can answer where the run is (the hook prints a live run at every session start, compaction included), and closes each run with a ledger read out of the session transcript: elapsed, tools, tokens, and what each dispatched agent cost where the transcript recorded it. Two Stop hooks hold a handback: one refuses a tree that moved since its last recorded check, the other refuses a changed existing test not proven to fail on the session's starting code (or recorded with a reason). Claude Code only; stands down where the superpowers pipeline governs the repo.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|