@iceinvein/agent-skills 0.23.4 → 0.23.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/bug-hunter/SKILL.md +5 -0
- package/skills/bug-hunter/evals/README.md +26 -2
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/case.yaml +4 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/fixture.sh +116 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-for-unlogged-cap-cut.md +50 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-hunter-fired.md +5 -0
- package/skills/bug-hunter/evals/every-member-of-a-set-claim/prompt.md +11 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/case.yaml +4 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/fixture.sh +112 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/graders/bug-for-close-after-rename.md +50 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/graders/bug-hunter-fired.md +5 -0
- package/skills/bug-hunter/evals/false-success-is-a-bug/prompt.md +11 -0
- package/skills/bug-hunter/evals/rules.json +14 -3
- package/skills/bug-hunter/skill.json +1 -1
- package/skills/index.json +3 -3
- package/skills/migrate/scripts/__tests__/e2e-express.test.ts +2 -1
- package/skills/migrate/scripts/__tests__/e2e-webforms.test.ts +2 -1
- package/skills/migrate/skill.json +1 -1
- package/skills/sluice/skill.json +1 -1
package/package.json
CHANGED
|
@@ -74,6 +74,11 @@ command), **ambiguities** (the question for product), **covered** (scenarios
|
|
|
74
74
|
that passed, with the claim each checked). Write `findings.json` and
|
|
75
75
|
`report.md` with the Write tool, never from a shell command or script.
|
|
76
76
|
|
|
77
|
+
Run as a dispatched agent, the Write tool may refuse `report.md` and
|
|
78
|
+
`findings.json` as report files. Then put each refused file's full content in
|
|
79
|
+
the hand-back under its path, for the caller to write unchanged with the Write
|
|
80
|
+
tool; never fall back to a shell write.
|
|
81
|
+
|
|
77
82
|
The `.bug-hunter/` directory is the one path in the caller's tree the skill
|
|
78
83
|
writes. Promoting a repro into the tree as a regression test is the caller's
|
|
79
84
|
job.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# bug-hunter evals
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Nine cases for `claude plugin eval`. Each scaffolds a small Node repo with a
|
|
4
4
|
`SPEC.md`, one module and a passing `node --test` suite, committed in two steps
|
|
5
5
|
(the spec, then the module) so the target reads as a change. The prompt asks
|
|
6
6
|
what QA will find in the module, the way a user would, and never names the
|
|
@@ -15,6 +15,8 @@ skill.
|
|
|
15
15
|
| `bug-cites-intent-from-spec` | A coupon that floors where the spec rounds half up | `intent.md` is written, and the bug cites a claim sourced from `SPEC.md`, not from the code's floor |
|
|
16
16
|
| `spec-outranks-target-docstring` | The spec keeps a sealed export on offer when its rewrite fails; the module's docstring and code drop it | A bug citing the spec claim, not an ambiguity between spec and docstring |
|
|
17
17
|
| `impossible-value-is-a-bug` | The spec is silent on rounding; the code shows `60s` and `1m 60s` | A bug for the seconds field of 60, which no rounding rule gives |
|
|
18
|
+
| `false-success-is-a-bug` | Closing a renamed project answers `{ ok: true }` and leaves its workspace | A bug, not an ambiguity about renames |
|
|
19
|
+
| `every-member-of-a-set-claim` | Every dropped cited source is logged, except one cut by the cap after attachments fill the slots | A bug for the unlogged cut, which no existing test reaches |
|
|
18
20
|
|
|
19
21
|
Every case also has `bug-hunter-fired`, a `tool_used: Skill` grader, so a
|
|
20
22
|
score can be told apart from the model's own habits.
|
|
@@ -30,7 +32,7 @@ claude plugin eval skills/bug-hunter --scaffold --allow-tools Bash Write Edit \
|
|
|
30
32
|
```
|
|
31
33
|
|
|
32
34
|
Add `--keep-temp` to keep each run's sandbox and trace, which is the only way
|
|
33
|
-
to read back why a grader failed. `-j
|
|
35
|
+
to read back why a grader failed. `-j 9` runs the nine cases at once.
|
|
34
36
|
|
|
35
37
|
To check the skill against its last accepted baseline, use `eval:fresh`:
|
|
36
38
|
|
|
@@ -113,6 +115,28 @@ a file needs its exact path and throws on a glob. So:
|
|
|
113
115
|
|
|
114
116
|
## Baseline
|
|
115
117
|
|
|
118
|
+
### Skill 0.4.2, Opus 5.5, 2026-10-20: two guards and a field-only rule
|
|
119
|
+
|
|
120
|
+
A read of 14 hunts in the snowvault transcripts (2026-10-17 to 10-20) found
|
|
121
|
+
three misses. Two became cases, each copying the shape of a real one:
|
|
122
|
+
`false-success-is-a-bug` follows SNO-114, which filed `wt:remove -- SNO-n`
|
|
123
|
+
after a rename as an ambiguity when it exited 0 having released nothing, and
|
|
124
|
+
`every-member-of-a-set-claim` follows SNO-54, which reported 0 bugs on "every
|
|
125
|
+
dropped citation is logged" while one drop site logged nothing. Both scored
|
|
126
|
+
1.00 on the unchanged 0.4.2, two runs each, every judge vote PASS (USD 1.70
|
|
127
|
+
and 1.76, `results/2026-10-20T06-22-11-560Z` and
|
|
128
|
+
`results/2026-10-20T06-22-10-824Z`). As with 0.3.0's cases, the fixture makes
|
|
129
|
+
the defect plain enough that the existing rules catch it, so neither
|
|
130
|
+
reproduces the field miss. They stay as guards, and the skill gained no rule
|
|
131
|
+
for either.
|
|
132
|
+
|
|
133
|
+
The third miss became the rule that a dispatched agent's refused `report.md`
|
|
134
|
+
and `findings.json` go in its hand-back. Every one of the 10 hunts run as a
|
|
135
|
+
dispatched agent had both Writes refused ("Subagents should return findings
|
|
136
|
+
as text, not write report files"), so `findings.json` existed for 4 of the 14
|
|
137
|
+
runs. No case pins it: an eval runs the skill at the top level, where the
|
|
138
|
+
Write succeeds.
|
|
139
|
+
|
|
116
140
|
### Skill 0.4.0, Opus 5.5, 2026-10-13
|
|
117
141
|
|
|
118
142
|
Same command, `-j 7`. Skill 0.4.0 drops the three dispatched roles for one
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A draft-source picker whose spec says every cited source it does not return
|
|
3
|
+
# is logged once with a reason. Sources are dropped at three sites: a cited id
|
|
4
|
+
# with no document, a document with no file name, and the five-slot cap. The
|
|
5
|
+
# cap is applied after the reader's own attachments are merged in first, but
|
|
6
|
+
# over_cap is only logged when the cited sources alone exceed five, so a cited
|
|
7
|
+
# source cut because attachments filled the slots leaves no log line. The
|
|
8
|
+
# existing tests cover each reason on the cited-only path. This is the shape
|
|
9
|
+
# of the snowvault SNO-54 hunt, which was given exactly this claim, checked
|
|
10
|
+
# behaviour against the base with 20,000 random cases, reported 0 bugs, and
|
|
11
|
+
# missed the attachment cut that the reviewer then found.
|
|
12
|
+
set -euo pipefail
|
|
13
|
+
|
|
14
|
+
mkdir -p src tests
|
|
15
|
+
|
|
16
|
+
cat > package.json <<'JSON'
|
|
17
|
+
{
|
|
18
|
+
"name": "draft-sources",
|
|
19
|
+
"version": "1.4.0",
|
|
20
|
+
"type": "module",
|
|
21
|
+
"scripts": {
|
|
22
|
+
"test": "node --test \"tests/*.test.js\""
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
JSON
|
|
26
|
+
|
|
27
|
+
cat > SPEC.md <<'MD'
|
|
28
|
+
# Draft sources
|
|
29
|
+
|
|
30
|
+
`pickSources({ attachments, citedIds, documents, log })` in `src/sources.js`
|
|
31
|
+
picks the documents that seed a draft.
|
|
32
|
+
|
|
33
|
+
- `attachments` are documents the reader attached; they always come first.
|
|
34
|
+
- `citedIds` are document ids the earlier answer cited, looked up in
|
|
35
|
+
`documents` (an object keyed by id). Cited sources fill what is left.
|
|
36
|
+
- At most 5 documents are returned in all.
|
|
37
|
+
- A cited document with no `fileName` is not used.
|
|
38
|
+
- Every cited source that is not returned is logged once, as
|
|
39
|
+
`log({ id, reason })`, where `reason` is one of `not_found`,
|
|
40
|
+
`no_file_name` or `over_cap`.
|
|
41
|
+
MD
|
|
42
|
+
|
|
43
|
+
git init --quiet
|
|
44
|
+
git add -A
|
|
45
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: draft sources"
|
|
46
|
+
|
|
47
|
+
cat > src/sources.js <<'JS'
|
|
48
|
+
const MAX_SOURCES = 5;
|
|
49
|
+
|
|
50
|
+
function citedSources(citedIds, documents, log) {
|
|
51
|
+
const picked = [];
|
|
52
|
+
for (const id of citedIds) {
|
|
53
|
+
const doc = documents[id];
|
|
54
|
+
if (!doc) {
|
|
55
|
+
log({ id, reason: "not_found" });
|
|
56
|
+
continue;
|
|
57
|
+
}
|
|
58
|
+
if (!doc.fileName) {
|
|
59
|
+
log({ id, reason: "no_file_name" });
|
|
60
|
+
continue;
|
|
61
|
+
}
|
|
62
|
+
picked.push(doc);
|
|
63
|
+
}
|
|
64
|
+
for (const doc of picked.slice(MAX_SOURCES)) {
|
|
65
|
+
log({ id: doc.id, reason: "over_cap" });
|
|
66
|
+
}
|
|
67
|
+
return picked;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
export function pickSources({ attachments, citedIds, documents, log }) {
|
|
71
|
+
const cited = citedSources(citedIds, documents, log);
|
|
72
|
+
return [...attachments, ...cited].slice(0, MAX_SOURCES);
|
|
73
|
+
}
|
|
74
|
+
JS
|
|
75
|
+
|
|
76
|
+
cat > tests/sources.test.js <<'JS'
|
|
77
|
+
import assert from "node:assert/strict";
|
|
78
|
+
import { test } from "node:test";
|
|
79
|
+
import { pickSources } from "../src/sources.js";
|
|
80
|
+
|
|
81
|
+
const doc = (id) => ({ id, fileName: `${id}.pdf` });
|
|
82
|
+
|
|
83
|
+
function run(input) {
|
|
84
|
+
const lines = [];
|
|
85
|
+
const picked = pickSources({ attachments: [], documents: {}, ...input, log: (l) => lines.push(l) });
|
|
86
|
+
return { ids: picked.map((d) => d.id), lines };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
test("attachments come before cited sources", () => {
|
|
90
|
+
const { ids } = run({ attachments: [doc("a1")], citedIds: ["c1"], documents: { c1: doc("c1") } });
|
|
91
|
+
assert.deepEqual(ids, ["a1", "c1"]);
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
test("a cited id with no document is logged as not_found", () => {
|
|
95
|
+
const { ids, lines } = run({ citedIds: ["c9"] });
|
|
96
|
+
assert.deepEqual(ids, []);
|
|
97
|
+
assert.deepEqual(lines, [{ id: "c9", reason: "not_found" }]);
|
|
98
|
+
});
|
|
99
|
+
|
|
100
|
+
test("a cited document with no file name is logged as no_file_name", () => {
|
|
101
|
+
const { ids, lines } = run({ citedIds: ["c1"], documents: { c1: { id: "c1" } } });
|
|
102
|
+
assert.deepEqual(ids, []);
|
|
103
|
+
assert.deepEqual(lines, [{ id: "c1", reason: "no_file_name" }]);
|
|
104
|
+
});
|
|
105
|
+
|
|
106
|
+
test("cited sources beyond five are logged as over_cap", () => {
|
|
107
|
+
const ids6 = ["c1", "c2", "c3", "c4", "c5", "c6"];
|
|
108
|
+
const documents = Object.fromEntries(ids6.map((id) => [id, doc(id)]));
|
|
109
|
+
const { ids, lines } = run({ citedIds: ids6, documents });
|
|
110
|
+
assert.deepEqual(ids, ["c1", "c2", "c3", "c4", "c5"]);
|
|
111
|
+
assert.deepEqual(lines, [{ id: "c6", reason: "over_cap" }]);
|
|
112
|
+
});
|
|
113
|
+
JS
|
|
114
|
+
|
|
115
|
+
git add -A
|
|
116
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: pick draft sources"
|
package/skills/bug-hunter/evals/every-member-of-a-set-claim/graders/bug-for-unlogged-cap-cut.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
# As in impossible-value-is-a-bug: the judge cannot run the repro, so "fails
|
|
4
|
+
# in the fixture" is judged against the fixture's code, quoted below.
|
|
5
|
+
focus: trace
|
|
6
|
+
weight: 3
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
The agent was asked what QA will find in `src/sources.js`. `SPEC.md` says
|
|
10
|
+
attachments come first, cited sources fill what is left, at most 5 documents
|
|
11
|
+
are returned, and every cited source that is not returned is logged once with
|
|
12
|
+
reason `not_found`, `no_file_name` or `over_cap`. The fixture's code is:
|
|
13
|
+
|
|
14
|
+
```js
|
|
15
|
+
function citedSources(citedIds, documents, log) {
|
|
16
|
+
const picked = [];
|
|
17
|
+
for (const id of citedIds) {
|
|
18
|
+
const doc = documents[id];
|
|
19
|
+
if (!doc) { log({ id, reason: "not_found" }); continue; }
|
|
20
|
+
if (!doc.fileName) { log({ id, reason: "no_file_name" }); continue; }
|
|
21
|
+
picked.push(doc);
|
|
22
|
+
}
|
|
23
|
+
for (const doc of picked.slice(MAX_SOURCES)) {
|
|
24
|
+
log({ id: doc.id, reason: "over_cap" });
|
|
25
|
+
}
|
|
26
|
+
return picked;
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
export function pickSources({ attachments, citedIds, documents, log }) {
|
|
30
|
+
const cited = citedSources(citedIds, documents, log);
|
|
31
|
+
return [...attachments, ...cited].slice(0, MAX_SOURCES);
|
|
32
|
+
}
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
So with, for example, 4 attachments and 2 valid cited sources, the second
|
|
36
|
+
cited source is not returned and no log line names it, because `over_cap` is
|
|
37
|
+
only logged for cited sources beyond five on their own.
|
|
38
|
+
|
|
39
|
+
Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
|
|
40
|
+
appears in a Write tool call or a shell command in the trace, or, if the Write
|
|
41
|
+
was refused, in the agent's final message). If it was written more than once,
|
|
42
|
+
judge only the last write, with any later Edit to it applied.
|
|
43
|
+
|
|
44
|
+
PASS if its `bugs` array has an entry for a cited source cut because
|
|
45
|
+
attachments took slots, which is neither returned nor logged, and whose
|
|
46
|
+
`repro` asserts a log line (or a returned source) that the code above does not
|
|
47
|
+
give for that input, so that it fails against the code.
|
|
48
|
+
FAIL if no findings.json content is visible in the trace, if `bugs` has no
|
|
49
|
+
entry for the attachment case, if it appears only in `ambiguities`, or if the
|
|
50
|
+
repro asserts something the code above already does.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: every-member-of-a-set-claim
|
|
3
|
+
description: A claim that every dropped cited source is logged, held at two of three drop sites. The existing tests cover each reason on the cited-only path; the cap cut after attachments are merged in logs nothing. Pins the hunt checking the claim at every site that drops a source, not only the ones the tests reach.
|
|
4
|
+
tags: [bug, coverage, scaffold, write]
|
|
5
|
+
max_turns: 80
|
|
6
|
+
timeout_seconds: 2400
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
|
|
8
|
+
expected_outcome: findings.json under .bug-hunter/ has a bug for a cited source cut by the five-slot cap when attachments fill slots, returned nowhere and logged nowhere, with a repro that fails against the fixture.
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Before this goes to QA: what will they find in `src/sources.js`? It's meant to do what `SPEC.md` says.
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# A project workspace registry whose close step derives the workspace name from
|
|
3
|
+
# the project's current title. The workspace was named from the title at open,
|
|
4
|
+
# so after a rename close removes nothing and still answers { ok: true }. The
|
|
5
|
+
# spec never mentions renaming a project and then closing it, which makes "should close
|
|
6
|
+
# find a workspace opened under the old title?" look like a question for
|
|
7
|
+
# product. It is not: whichever answer product gives, a close that reports
|
|
8
|
+
# success and left the workspace in place is wrong. This is the shape of the
|
|
9
|
+
# snowvault SNO-114 hunt, which filed `wt:remove -- SNO-n` after a rename as an
|
|
10
|
+
# ambiguity; review later found it released nothing and exited 0.
|
|
11
|
+
set -euo pipefail
|
|
12
|
+
|
|
13
|
+
mkdir -p src tests
|
|
14
|
+
|
|
15
|
+
cat > package.json <<'JSON'
|
|
16
|
+
{
|
|
17
|
+
"name": "workspaces",
|
|
18
|
+
"version": "2.3.0",
|
|
19
|
+
"type": "module",
|
|
20
|
+
"scripts": {
|
|
21
|
+
"test": "node --test \"tests/*.test.js\""
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
JSON
|
|
25
|
+
|
|
26
|
+
cat > SPEC.md <<'MD'
|
|
27
|
+
# Project workspaces
|
|
28
|
+
|
|
29
|
+
`src/workspaces.js` keeps a workspace for each open project. A project has a
|
|
30
|
+
key (`P-12`) that never changes and a title that can be renamed at any time.
|
|
31
|
+
|
|
32
|
+
- `openProject(state, key, title)` registers the project and creates its
|
|
33
|
+
workspace.
|
|
34
|
+
- `renameProject(state, key, title)` changes the project's title.
|
|
35
|
+
- `closeProject(state, key)` removes the project's workspace and returns
|
|
36
|
+
`{ ok: true }`. For a key that names no project it returns
|
|
37
|
+
`{ ok: false, error: "unknown project" }`.
|
|
38
|
+
- `workspaceCount(state)` is the number of workspaces that exist.
|
|
39
|
+
MD
|
|
40
|
+
|
|
41
|
+
git init --quiet
|
|
42
|
+
git add -A
|
|
43
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: project workspaces"
|
|
44
|
+
|
|
45
|
+
cat > src/workspaces.js <<'JS'
|
|
46
|
+
export function createState() {
|
|
47
|
+
return { projects: new Map(), workspaces: new Set() };
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function workspaceName(key, title) {
|
|
51
|
+
const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "");
|
|
52
|
+
return `${key.toLowerCase()}-${slug}`;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
export function openProject(state, key, title) {
|
|
56
|
+
state.projects.set(key, { title });
|
|
57
|
+
state.workspaces.add(workspaceName(key, title));
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export function renameProject(state, key, title) {
|
|
61
|
+
const project = state.projects.get(key);
|
|
62
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
63
|
+
project.title = title;
|
|
64
|
+
return { ok: true };
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
export function closeProject(state, key) {
|
|
68
|
+
const project = state.projects.get(key);
|
|
69
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
70
|
+
state.workspaces.delete(workspaceName(key, project.title));
|
|
71
|
+
state.projects.delete(key);
|
|
72
|
+
return { ok: true };
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
export function workspaceCount(state) {
|
|
76
|
+
return state.workspaces.size;
|
|
77
|
+
}
|
|
78
|
+
JS
|
|
79
|
+
|
|
80
|
+
cat > tests/workspaces.test.js <<'JS'
|
|
81
|
+
import assert from "node:assert/strict";
|
|
82
|
+
import { test } from "node:test";
|
|
83
|
+
import { createState, openProject, renameProject, closeProject, workspaceCount } from "../src/workspaces.js";
|
|
84
|
+
|
|
85
|
+
test("opening a project creates its workspace", () => {
|
|
86
|
+
const state = createState();
|
|
87
|
+
openProject(state, "P-1", "Launch plan");
|
|
88
|
+
assert.equal(workspaceCount(state), 1);
|
|
89
|
+
});
|
|
90
|
+
|
|
91
|
+
test("closing a project removes its workspace", () => {
|
|
92
|
+
const state = createState();
|
|
93
|
+
openProject(state, "P-1", "Launch plan");
|
|
94
|
+
assert.deepEqual(closeProject(state, "P-1"), { ok: true });
|
|
95
|
+
assert.equal(workspaceCount(state), 0);
|
|
96
|
+
});
|
|
97
|
+
|
|
98
|
+
test("renaming a project keeps it open", () => {
|
|
99
|
+
const state = createState();
|
|
100
|
+
openProject(state, "P-1", "Launch plan");
|
|
101
|
+
assert.deepEqual(renameProject(state, "P-1", "Launch plan v2"), { ok: true });
|
|
102
|
+
assert.equal(workspaceCount(state), 1);
|
|
103
|
+
});
|
|
104
|
+
|
|
105
|
+
test("closing an unknown key is refused", () => {
|
|
106
|
+
const state = createState();
|
|
107
|
+
assert.deepEqual(closeProject(state, "P-9"), { ok: false, error: "unknown project" });
|
|
108
|
+
});
|
|
109
|
+
JS
|
|
110
|
+
|
|
111
|
+
git add -A
|
|
112
|
+
git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: project workspaces"
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: llm
|
|
3
|
+
# As in impossible-value-is-a-bug: the judge cannot run the repro, so "fails
|
|
4
|
+
# in the fixture" is judged against the fixture's code, quoted below.
|
|
5
|
+
focus: trace
|
|
6
|
+
weight: 3
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
The agent was asked what QA will find in `src/workspaces.js`. `SPEC.md` says a
|
|
10
|
+
project's title can be renamed at any time, and that `closeProject(state, key)`
|
|
11
|
+
removes the project's workspace and returns `{ ok: true }`. It never mentions
|
|
12
|
+
renaming a project and then closing it. The fixture's code is:
|
|
13
|
+
|
|
14
|
+
```js
|
|
15
|
+
function workspaceName(key, title) {
|
|
16
|
+
const slug = title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "");
|
|
17
|
+
return `${key.toLowerCase()}-${slug}`;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
export function openProject(state, key, title) {
|
|
21
|
+
state.projects.set(key, { title });
|
|
22
|
+
state.workspaces.add(workspaceName(key, title));
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
export function closeProject(state, key) {
|
|
26
|
+
const project = state.projects.get(key);
|
|
27
|
+
if (!project) return { ok: false, error: "unknown project" };
|
|
28
|
+
state.workspaces.delete(workspaceName(key, project.title));
|
|
29
|
+
state.projects.delete(key);
|
|
30
|
+
return { ok: true };
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
So after `renameProject`, `closeProject` returns `{ ok: true }` while the
|
|
35
|
+
workspace opened under the old title still exists (`workspaceCount` stays 1).
|
|
36
|
+
|
|
37
|
+
Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
|
|
38
|
+
appears in a Write tool call or a shell command in the trace, or, if the Write
|
|
39
|
+
was refused, in the agent's final message). If it was written more than once,
|
|
40
|
+
judge only the last write, with any later Edit to it applied.
|
|
41
|
+
|
|
42
|
+
PASS if its `bugs` array has an entry for closing a renamed project that
|
|
43
|
+
reports success but leaves the workspace, and whose `repro` asserts a result
|
|
44
|
+
the code above does not give (for example `workspaceCount` of 0 after rename
|
|
45
|
+
then close, or a result other than `{ ok: true }` with the workspace left), so
|
|
46
|
+
that it fails against the code. An ambiguity about renames may also be present;
|
|
47
|
+
it does not affect the verdict.
|
|
48
|
+
FAIL if no findings.json content is visible in the trace, if close-after-rename
|
|
49
|
+
appears only in `ambiguities` or only in the report's latent notes, or if the
|
|
50
|
+
repro asserts something the code above already does.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: false-success-is-a-bug
|
|
3
|
+
description: A close that answers ok after a rename but leaves the workspace in place. The spec never mentions renaming and then closing, so the case reads like a product question, but a reported success that did not do the work is wrong under every answer. Pins a bug rather than an ambiguity.
|
|
4
|
+
tags: [bug, baseline, scaffold, write]
|
|
5
|
+
max_turns: 80
|
|
6
|
+
timeout_seconds: 2400
|
|
7
|
+
allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
|
|
8
|
+
expected_outcome: "findings.json under .bug-hunter/ has a bug for closing a renamed project, which returns { ok: true } and leaves its workspace, with a repro that fails against the fixture."
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
Before this goes to QA: what will they find in `src/workspaces.js`? It's meant to do what `SPEC.md` says.
|
|
@@ -76,7 +76,9 @@
|
|
|
76
76
|
"correct-code-has-no-bugs",
|
|
77
77
|
"bug-cites-intent-from-spec",
|
|
78
78
|
"spec-outranks-target-docstring",
|
|
79
|
-
"impossible-value-is-a-bug"
|
|
79
|
+
"impossible-value-is-a-bug",
|
|
80
|
+
"false-success-is-a-bug",
|
|
81
|
+
"every-member-of-a-set-claim"
|
|
80
82
|
]
|
|
81
83
|
},
|
|
82
84
|
{
|
|
@@ -86,7 +88,8 @@
|
|
|
86
88
|
"cases": [
|
|
87
89
|
"proves-a-spec-boundary-bug",
|
|
88
90
|
"bug-cites-intent-from-spec",
|
|
89
|
-
"spec-outranks-target-docstring"
|
|
91
|
+
"spec-outranks-target-docstring",
|
|
92
|
+
"every-member-of-a-set-claim"
|
|
90
93
|
]
|
|
91
94
|
},
|
|
92
95
|
{
|
|
@@ -109,7 +112,8 @@
|
|
|
109
112
|
"states": "SKILL.md: What counts as a bug",
|
|
110
113
|
"origin": "commit:23d4466",
|
|
111
114
|
"cases": [
|
|
112
|
-
"impossible-value-is-a-bug"
|
|
115
|
+
"impossible-value-is-a-bug",
|
|
116
|
+
"false-success-is-a-bug"
|
|
113
117
|
]
|
|
114
118
|
},
|
|
115
119
|
{
|
|
@@ -145,6 +149,13 @@
|
|
|
145
149
|
"correct-code-has-no-bugs"
|
|
146
150
|
]
|
|
147
151
|
},
|
|
152
|
+
{
|
|
153
|
+
"id": "refused-report-writes-go-in-the-hand-back",
|
|
154
|
+
"states": "SKILL.md: Output",
|
|
155
|
+
"origin": "skills/bug-hunter/evals/README.md",
|
|
156
|
+
"cases": [],
|
|
157
|
+
"unpinned": "The refusal comes from the harness only inside a dispatched agent, and an eval case runs the skill at the top level, where the Write succeeds."
|
|
158
|
+
},
|
|
148
159
|
{
|
|
149
160
|
"id": "bug-hunter-dir-is-the-only-path-written",
|
|
150
161
|
"states": "SKILL.md: Output",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bug-hunter",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.3",
|
|
4
4
|
"description": "Hunts logic and correctness bugs in a change or area and proves each one with a failing reproduction checked against stated intent. One agent collects intent from the request, specs, docs and unchanged code, runs QA-style scenarios from a catalog as repros in the repo's own test runner inside a scratch copy, and re-runs each failure clean before reporting it. Counts a contradicted claim, or a result wrong under every reading, as a bug; keeps ambiguities for questions with more than one acceptable answer; reports latent defects without filing them. No repro, no finding. Writes report.md and findings.json (bugs, ambiguities, covered) to .bug-hunter/<date>-<slug>/ and never modifies the caller's tree. Language-agnostic; ships no per-language scripts.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|
package/skills/index.json
CHANGED
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
"name": "bug-hunter",
|
|
14
14
|
"description": "Hunts logic and correctness bugs in a change or area and proves each one with a failing reproduction checked against stated intent. One agent collects intent from the request, specs, docs and unchanged code, runs QA-style scenarios from a catalog as repros in the repo's own test runner inside a scratch copy, and re-runs each failure clean before reporting it. Counts a contradicted claim, or a result wrong under every reading, as a bug; keeps ambiguities for questions with more than one acceptable answer; reports latent defects without filing them. No repro, no finding. Writes report.md and findings.json (bugs, ambiguities, covered) to .bug-hunter/<date>-<slug>/ and never modifies the caller's tree. Language-agnostic; ships no per-language scripts.",
|
|
15
15
|
"type": "prompt",
|
|
16
|
-
"version": "0.4.
|
|
16
|
+
"version": "0.4.3"
|
|
17
17
|
},
|
|
18
18
|
{
|
|
19
19
|
"name": "codebase-architecture",
|
|
@@ -233,7 +233,7 @@
|
|
|
233
233
|
"name": "migrate",
|
|
234
234
|
"description": "Source-agnostic legacy migration mapping. Walks a legacy codebase through probe, enumerate, seam, extract, parity, queue, adjudicate and handoff, building an auditable requirements ledger with mandatory citations and a `migrate check` gate in place of self-reported completeness. Use when the user asks to migrate, re-specify, replatform, or map a legacy system onto a new stack, to hand mapped requirements to a delivery team, or to resume, check, report, or forecast a mapping run already under way.",
|
|
235
235
|
"type": "prompt",
|
|
236
|
-
"version": "0.4.
|
|
236
|
+
"version": "0.4.4"
|
|
237
237
|
},
|
|
238
238
|
{
|
|
239
239
|
"name": "module-secret-auditor",
|
|
@@ -295,7 +295,7 @@
|
|
|
295
295
|
"name": "sluice",
|
|
296
296
|
"description": "Routes work by change shape into four channels (bypass, fast, main, deep) and applies only the rules each channel needs, so a one-line fix does not pay the cost of a multi-subsystem build. Carries nine rules as one-liners in the router and the full treatment in references read only on friction. Checks the finished plan with plan.sh validate rather than trusting it to memory, seeds the run state from it, keeps a deep run's task breakdown in .sluice/run.json so a statusline segment, one status command and a SessionStart hook can answer where the run is (the hook prints a live run at every session start, compaction included), and closes each run with a ledger read out of the session transcript: elapsed, tools, tokens, and what each dispatched agent cost where the transcript recorded it. Two Stop hooks hold a handback: one refuses a tree that moved since its last recorded check, the other refuses a changed existing test not proven to fail on the session's starting code (or recorded with a reason). Claude Code only; stands down where the superpowers pipeline governs the repo.",
|
|
297
297
|
"type": "prompt",
|
|
298
|
-
"version": "0.25.
|
|
298
|
+
"version": "0.25.5"
|
|
299
299
|
},
|
|
300
300
|
{
|
|
301
301
|
"name": "temporal-coupling-detector",
|
|
@@ -755,6 +755,7 @@ function readTomlString(text: string, key: string): string {
|
|
|
755
755
|
return match[1]
|
|
756
756
|
}
|
|
757
757
|
|
|
758
|
+
// Drives a whole run through several gates; it takes about 5s, which is bun's default limit.
|
|
758
759
|
test('contract-only run driven probe through handoff, ending green at a plain check', async () => {
|
|
759
760
|
const groundTruthPath = join(source, 'GROUND-TRUTH.md')
|
|
760
761
|
const rows = await parseGroundTruth(groundTruthPath)
|
|
@@ -1250,4 +1251,4 @@ test('contract-only run driven probe through handoff, ending green at a plain ch
|
|
|
1250
1251
|
expect(reopened.out).toContain(QUEUE_ID)
|
|
1251
1252
|
await writeFile(reopenPath, ruledText)
|
|
1252
1253
|
expect((await migrate(['check'])).code).toBe(0)
|
|
1253
|
-
})
|
|
1254
|
+
}, 60000)
|
|
@@ -935,6 +935,7 @@ function readTomlString(text: string, key: string): string {
|
|
|
935
935
|
return match[1]
|
|
936
936
|
}
|
|
937
937
|
|
|
938
|
+
// Drives a whole run through several gates; it takes about 4s, close to bun's 5s default limit.
|
|
938
939
|
test('aspnet recipe run driven probe through queue, ending green at check --phase queue', async () => {
|
|
939
940
|
const groundTruthPath = join(source, 'GROUND-TRUTH.md')
|
|
940
941
|
const rows = await parseGroundTruth(groundTruthPath)
|
|
@@ -1278,4 +1279,4 @@ test('aspnet recipe run driven probe through queue, ending green at check --phas
|
|
|
1278
1279
|
expect(afterRemoval.out).toContain(
|
|
1279
1280
|
`lens census for ${mutatedSurface} claims in_ledger 0 + added ${claimed} = ${claimed} element(s) in the ledger, but elements.jsonl has ${actual}`,
|
|
1280
1281
|
)
|
|
1281
|
-
})
|
|
1282
|
+
}, 60000)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "migrate",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.4",
|
|
4
4
|
"description": "Source-agnostic legacy migration mapping. Walks a legacy codebase through probe, enumerate, seam, extract, parity, queue, adjudicate and handoff, building an auditable requirements ledger with mandatory citations and a `migrate check` gate in place of self-reported completeness. Use when the user asks to migrate, re-specify, replatform, or map a legacy system onto a new stack, to hand mapped requirements to a delivery team, or to resume, check, report, or forecast a mapping run already under way.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|
package/skills/sluice/skill.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "sluice",
|
|
3
|
-
"version": "0.25.
|
|
3
|
+
"version": "0.25.5",
|
|
4
4
|
"description": "Routes work by change shape into four channels (bypass, fast, main, deep) and applies only the rules each channel needs, so a one-line fix does not pay the cost of a multi-subsystem build. Carries nine rules as one-liners in the router and the full treatment in references read only on friction. Checks the finished plan with plan.sh validate rather than trusting it to memory, seeds the run state from it, keeps a deep run's task breakdown in .sluice/run.json so a statusline segment, one status command and a SessionStart hook can answer where the run is (the hook prints a live run at every session start, compaction included), and closes each run with a ledger read out of the session transcript: elapsed, tools, tokens, and what each dispatched agent cost where the transcript recorded it. Two Stop hooks hold a handback: one refuses a tree that moved since its last recorded check, the other refuses a changed existing test not proven to fail on the session's starting code (or recorded with a reason). Claude Code only; stands down where the superpowers pipeline governs the repo.",
|
|
5
5
|
"author": "iceinvein",
|
|
6
6
|
"type": "prompt",
|