@iceinvein/agent-skills 0.21.4 → 0.22.0

package/skills/bug-hunter/evals/callers-tree-unchanged/fixture.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# A retry backoff with a real bug (attempt 1 waits 200 ms where the spec says
+# 100), so the hunt has repros to write, run and confirm, which is when files
+# are most likely to leak into the caller's tree. The case is about where
+# those files land, not about the bug.
+set -euo pipefail
+
+mkdir -p src tests
+
+cat > package.json <<'JSON'
+{
+  "name": "uplink",
+  "version": "0.9.0",
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\""
+  }
+}
+JSON
+
+cat > SPEC.md <<'MD'
+# Uplink retries
+
+`backoffDelay(attempt)` in `src/backoff.js` returns how many milliseconds to
+wait before retry number `attempt`.
+
+- Attempt 1 waits 100 ms. Each later attempt waits twice as long as the one
+  before it.
+- No wait is longer than 5000 ms: an attempt whose doubled delay would pass
+  5000 waits exactly 5000.
+- `attempt` is a whole number of 1 or more; anything else throws a
+  `RangeError`.
+MD
+
+git init --quiet
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: uplink retry backoff"
+
+cat > src/backoff.js <<'JS'
+const BASE_MS = 100;
+const MAX_MS = 5000;
+
+export function backoffDelay(attempt) {
+  if (!Number.isInteger(attempt) || attempt < 1) {
+    throw new RangeError(`attempt must be a whole number of 1 or more, got ${attempt}`);
+  }
+
+  return Math.min(BASE_MS * 2 ** attempt, MAX_MS);
+}
+JS
+
+cat > tests/backoff.test.js <<'JS'
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { backoffDelay } from "../src/backoff.js";
+
+test("a late attempt waits the maximum", () => {
+  assert.equal(backoffDelay(10), 5000);
+});
+
+test("attempt 0 throws", () => {
+  assert.throws(() => backoffDelay(0), RangeError);
+});
+JS
+
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: back off between uplink retries"

package/skills/bug-hunter/evals/callers-tree-unchanged/graders/bug-hunter-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
+---

package/skills/bug-hunter/evals/callers-tree-unchanged/graders/source-unchanged.md

@@ -0,0 +1,10 @@
+---
+type: regex
+# tree-unchanged reads the files target, which lists only paths Claude
+# created, so an edit to a scaffold file never reaches it. A fix made in place
+# would rewrite the buggy line, so its survival is what shows src/backoff.js
+# was left alone.
+pattern: 'return Math\.min\(BASE_MS \* 2 \*\* attempt, MAX_MS\);'
+target: { source: file, path: src/backoff.js }
+weight: 3
+---

package/skills/bug-hunter/evals/callers-tree-unchanged/graders/tests-unchanged.md

@@ -0,0 +1,10 @@
+---
+type: regex
+# As source-unchanged: a repro appended to the scaffold's own test file is an
+# edit, which the files target never lists. The scaffold has two tests, so a
+# third means the hunt wrote into the caller's suite.
+pattern: '\b(?:test|it)\('
+match: 'count:2'
+target: { source: file, path: tests/backoff.test.js }
+weight: 3
+---

package/skills/bug-hunter/evals/callers-tree-unchanged/graders/tree-unchanged.md

@@ -0,0 +1,15 @@
+---
+type: regex
+# A grader cannot run `git status --porcelain`, so this reads the harness's own
+# list of files the run created in the working directory, one path per line,
+# relative to it. A file .gitignore hides still shows here. .git/ is excluded
+# because git keeps its worktree bookkeeping there and porcelain never reports
+# it. Fails on any line naming a path outside .bug-hunter/. The list leaves out
+# files the run only modified, so source-unchanged and tests-unchanged cover
+# edits to the scaffold's own files.
+pattern: '^(?!\.bug-hunter/|\.git/).+'
+flags: m
+match: not_contains
+target: files
+weight: 3
+---

package/skills/bug-hunter/evals/callers-tree-unchanged/prompt.md

@@ -0,0 +1,11 @@
+---
+name: callers-tree-unchanged
+description: The skill hunts in a scratch worktree and writes only under .bug-hunter/. Pins the caller's tree left as it was, with repros written and run along the way.
+tags: [tree, worktree, scaffold, write]
+max_turns: 80
+timeout_seconds: 2400
+allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
+expected_outcome: Every file the run created or changed in the caller's tree is under .bug-hunter/.
+---
+
+Hunt for bugs in `src/backoff.js` against `SPEC.md` before QA gets to it.

package/skills/bug-hunter/evals/correct-code-has-no-bugs/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: correct-code-has-no-bugs
+context:
+  scaffold_script: fixture.sh

package/skills/bug-hunter/evals/correct-code-has-no-bugs/fixture.sh

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+# A pagination helper that does everything SPEC.md says, limits included: the
+# spec states every range and every error, and the code meets each one at its
+# edge. The existing tests cover only the middle, so the boundaries are left
+# for the hunt to probe and find correct. A bug reported here is a false
+# positive.
+set -euo pipefail
+
+mkdir -p src tests
+
+cat > package.json <<'JSON'
+{
+  "name": "listing",
+  "version": "2.0.1",
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\""
+  }
+}
+JSON
+
+cat > SPEC.md <<'MD'
+# Listing
+
+`paginate(items, page, perPage)` in `src/paginate.js` returns one page of a
+list. Callers always pass an array as `items`; anything else is outside this
+function's contract.
+
+- Pages are numbered from 1. Page `n` holds the items at indexes
+  `(n - 1) * perPage` up to but not including `n * perPage`.
+- `perPage` is a whole number from 1 to 50, inclusive.
+- `page` is a whole number of 1 or more.
+- A `page` or `perPage` outside those ranges, or not a whole number, throws a
+  `RangeError`.
+- The last page may hold fewer than `perPage` items.
+- A page past the last one returns an empty array.
+- The result is a new array, and `items` is not modified.
+MD
+
+git init --quiet
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: listing pagination"
+
+cat > src/paginate.js <<'JS'
+const MAX_PER_PAGE = 50;
+
+export function paginate(items, page, perPage) {
+  if (!Number.isInteger(perPage) || perPage < 1 || perPage > MAX_PER_PAGE) {
+    throw new RangeError(`perPage must be a whole number from 1 to ${MAX_PER_PAGE}, got ${perPage}`);
+  }
+
+  if (!Number.isInteger(page) || page < 1) {
+    throw new RangeError(`page must be a whole number of 1 or more, got ${page}`);
+  }
+
+  const start = (page - 1) * perPage;
+  return items.slice(start, start + perPage);
+}
+JS
+
+cat > tests/paginate.test.js <<'JS'
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { paginate } from "../src/paginate.js";
+
+const letters = ["a", "b", "c", "d", "e", "f", "g"];
+
+test("the first page holds the first perPage items", () => {
+  assert.deepEqual(paginate(letters, 1, 3), ["a", "b", "c"]);
+});
+
+test("the second page continues where the first stopped", () => {
+  assert.deepEqual(paginate(letters, 2, 3), ["d", "e", "f"]);
+});
+
+test("a fractional page throws", () => {
+  assert.throws(() => paginate(letters, 1.5, 3), RangeError);
+});
+JS
+
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: paginate listings"

package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/bug-hunter-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
+---

package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/bugs-empty.md

@@ -0,0 +1,18 @@
+---
+type: regex
+# findings.json sits under a slug the skill picks, so no file target can name
+# it; the trace carries its content in the Write call (or shell command) that
+# created it. That content is one JSON string in the trace line, its quotes
+# escaped as \" and its newlines as \n, and the match is held inside that one
+# string. Matches a findings.json write whose "bugs" is an empty array, and
+# only the last one: the closing lookahead fails the match if any later
+# findings.json write follows (a Write, its echoed result, an Edit, or a shell
+# redirect), so an empty first draft rewritten with bugs does not pass. A Write
+# is echoed in full by its tool result, so the last Write still matches there.
+# A later Edit fails the grader, since an Edit's result does not carry the
+# whole file. A file written by a dispatched agent counts only if the trace
+# carries that agent's tool calls.
+pattern: 'findings\.json(?:","content":")?(?:\\.|[^"\\])*?\\"bugs\\":(?:\s|\\n)*\[(?:\s|\\n)*\](?![\s\S]*?(?:findings\.json","(?:content|old_string)":"|>\s*[^"\s]*findings\.json))'
+target: trace
+weight: 2
+---

package/skills/bug-hunter/evals/correct-code-has-no-bugs/graders/written-with-write-tool.md

@@ -0,0 +1,13 @@
+---
+type: regex
+# SKILL.md writes intent.md, findings.json and report.md with the Write tool,
+# never from a shell command. The content graders read a heredoc's literal JSON
+# as readily as a Write call's, so this is what holds the rule: it fails a Bash
+# command that redirects or tees into one of the three under .bug-hunter/. A
+# script that opens the file itself (python3, node -e) is not caught here; the
+# content graders fail that, since the trace then holds no literal JSON.
+pattern: '"command":"(?:\\.|[^"\\])*?(?:>|\btee\b(?:\\.|[^"\\|;&])*?)\s*[^"\s]*\.bug-hunter/[^"\s]*?(?:intent\.md|findings\.json|report\.md)'
+match: not_contains
+target: trace
+weight: 2
+---

package/skills/bug-hunter/evals/correct-code-has-no-bugs/prompt.md

@@ -0,0 +1,11 @@
+---
+name: correct-code-has-no-bugs
+description: A module that already meets its spec at every stated limit. Pins the skill reporting no bugs rather than turning a passing hunt into a finding.
+tags: [false-positive, boundary, scaffold, write]
+max_turns: 80
+timeout_seconds: 2400
+allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
+expected_outcome: findings.json under .bug-hunter/ has an empty bugs array; the boundary scenarios land in covered.
+---
+
+What will QA find in `src/paginate.js`? It should do exactly what `SPEC.md` describes.

package/skills/bug-hunter/evals/impossible-value-is-a-bug/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: impossible-value-is-a-bug
+context:
+  scaffold_script: fixture.sh

package/skills/bug-hunter/evals/impossible-value-is-a-bug/fixture.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# A duration formatter whose spec gives the two display forms and never says
+# how a fractional second rounds. The code floors the minutes and then rounds
+# the remaining seconds, so 59.6 shows as "60s" and 119.7 as "1m 60s". The
+# spec's silence on rounding is a real question, but no answer to it permits
+# a seconds field of 60: flooring gives "59s" and "1m 59s", rounding gives
+# "1m 0s" and "2m 0s". A result wrong under every reading of the silence is a
+# bug, not an ambiguity. This is the shape of the pilot's formatDuration
+# finding on workings-0c1703a05, which triage confirmed real.
+set -euo pipefail
+
+mkdir -p src tests
+
+cat > package.json <<'JSON'
+{
+  "name": "video-summary",
+  "version": "1.6.0",
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\""
+  }
+}
+JSON
+
+cat > SPEC.md <<'MD'
+# Video summary
+
+`formatDuration(secs)` in `src/duration.js` gives a video's length for the
+summary line. `secs` is the length in seconds, 0 or more. It is the frame
+count over the frame rate, so it is often fractional.
+
+- Under a minute, the length shows as `Ns`, for example `42s`.
+- From a minute up, it shows as `Mm Ss`, for example `3m 5s`.
+MD
+
+git init --quiet
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: video length on the summary line"
+
+cat > src/duration.js <<'JS'
+export function formatDuration(secs) {
+  const mins = Math.floor(secs / 60);
+  const remainingSecs = Math.round(secs % 60);
+  if (mins === 0) return `${remainingSecs}s`;
+  return `${mins}m ${remainingSecs}s`;
+}
+JS
+
+cat > tests/duration.test.js <<'JS'
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { formatDuration } from "../src/duration.js";
+
+test("under a minute shows seconds only", () => {
+  assert.equal(formatDuration(42), "42s");
+});
+
+test("a minute or more shows minutes and seconds", () => {
+  assert.equal(formatDuration(185), "3m 5s");
+});
+
+test("exactly a minute shows as 1m 0s", () => {
+  assert.equal(formatDuration(60), "1m 0s");
+});
+JS
+
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: format a video's length"

package/skills/bug-hunter/evals/impossible-value-is-a-bug/graders/bug-for-sixty-seconds.md

@@ -0,0 +1,37 @@
+---
+type: llm
+# As in proves-a-spec-boundary-bug: the judge cannot run the repro, so "fails
+# in the fixture" is judged against the fixture's code, quoted below.
+focus: trace
+weight: 3
+---
+
+The agent was asked what QA will find in `src/duration.js`. `SPEC.md` gives
+the two display forms (`Ns` under a minute, `Mm Ss` from a minute up) and
+says lengths are often fractional, but never says how a fractional second
+rounds. The fixture's code is:
+
+```js
+const mins = Math.floor(secs / 60);
+const remainingSecs = Math.round(secs % 60);
+if (mins === 0) return `${remainingSecs}s`;
+return `${mins}m ${remainingSecs}s`;
+```
+
+So `formatDuration(59.6)` returns `"60s"` and `formatDuration(119.7)` returns
+`"1m 60s"`. No rounding rule gives a seconds field of 60: flooring gives
+`"59s"` and `"1m 59s"`, rounding gives `"1m 0s"` and `"2m 0s"`.
+
+Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
+appears in a Write tool call or a shell command in the trace). If it was
+written more than once, judge only the last write, with any later Edit to it
+applied.
+
+PASS if its `bugs` array has an entry for a fractional length whose seconds
+show as 60, and whose `repro` asserts a result that the code above does not
+give for that input (for example `"1m 0s"` for 59.6, or a seconds field below
+60), so that it fails against the code. An ambiguity asking which way
+fractional seconds round may also be present; it does not affect the verdict.
+FAIL if no findings.json content is visible in the trace, if the seconds-of-60
+case appears only in `ambiguities`, or if the repro asserts something the code
+above already does.

package/skills/bug-hunter/evals/impossible-value-is-a-bug/graders/bug-hunter-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
+---

package/skills/bug-hunter/evals/impossible-value-is-a-bug/prompt.md

@@ -0,0 +1,11 @@
+---
+name: impossible-value-is-a-bug
+description: The spec is silent on how fractional seconds round, and the code shows a seconds field of 60, which no rounding rule gives. Pins a confirmed bug rather than an ambiguity about rounding.
+tags: [bug, arithmetic, scaffold, write]
+max_turns: 80
+timeout_seconds: 2400
+allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
+expected_outcome: findings.json under .bug-hunter/ has a bug for a fractional length whose seconds show as 60 (such as 59.6 giving "60s" or 119.7 giving "1m 60s"), with a repro that fails against the fixture.
+---
+
+QA picks up the video summary line next. What will they find in `src/duration.js`? It's meant to do what `SPEC.md` says.

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: proves-a-spec-boundary-bug
+context:
+  scaffold_script: fixture.sh

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/fixture.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# A username check with one bug, at the limit SPEC.md states: the spec says 3
+# to 20 characters inclusive and the code rejects a name of exactly 20. The
+# existing tests pass because none of them sits on the upper edge, which is
+# the gap a scenario written from the spec should land on. Everything else the
+# spec says, the code does, so the boundary is the only true finding.
+set -euo pipefail
+
+mkdir -p src tests
+
+cat > package.json <<'JSON'
+{
+  "name": "signup",
+  "version": "0.3.0",
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\""
+  }
+}
+JSON
+
+cat > SPEC.md <<'MD'
+# Signup
+
+## Usernames
+
+`isValidUsername(name)` in `src/username.js` decides whether the signup form
+accepts a username.
+
+- A username is 3 to 20 characters long, inclusive.
+- It contains only ASCII letters, digits and underscores.
+- Anything that is not a string is rejected.
+- The function returns `true` or `false`; it never throws.
+MD
+
+git init --quiet
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "spec: signup usernames"
+
+cat > src/username.js <<'JS'
+const ALLOWED = /^[A-Za-z0-9_]+$/;
+
+export function isValidUsername(name) {
+  if (typeof name !== "string") {
+    return false;
+  }
+
+  if (name.length < 3 || name.length >= 20) {
+    return false;
+  }
+
+  return ALLOWED.test(name);
+}
+JS
+
+cat > tests/username.test.js <<'JS'
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { isValidUsername } from "../src/username.js";
+
+test("an ordinary name is accepted", () => {
+  assert.equal(isValidUsername("alice_01"), true);
+});
+
+test("a two-character name is rejected", () => {
+  assert.equal(isValidUsername("ab"), false);
+});
+
+test("a name with a space is rejected", () => {
+  assert.equal(isValidUsername("bad name"), false);
+});
+
+test("a number is rejected", () => {
+  assert.equal(isValidUsername(42), false);
+});
+JS
+
+git add -A
+git -c user.email=fixture@example.com -c user.name=fixture commit --quiet -m "feat: validate usernames at signup"

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/bug-cites-the-limit.md

@@ -0,0 +1,31 @@
+---
+type: llm
+# A grader cannot run a command, so "the repro fails in the fixture" is judged
+# against the fixture's code, which is fixed and quoted below: a repro that
+# asserts a 20-character name is accepted fails against it, and one that
+# asserts anything the code already does passes. The judge reads the trace
+# because findings.json sits under a slug no file focus can name, and the
+# judge sees only a head-and-tail window of a long trace.
+focus: trace
+weight: 3
+---
+
+The agent was asked what QA will find in `src/username.js`. `SPEC.md` says a
+username is 3 to 20 characters long, inclusive. The fixture's code rejects a
+name when `name.length < 3 || name.length >= 20`, so a name of exactly 20
+characters is wrongly rejected. Nothing else in the code departs from the
+spec.
+
+Find the `findings.json` the agent wrote under `.bug-hunter/` (its content
+appears in a Write tool call or a shell command in the trace). If it was
+written more than once, judge only the last write, with any later Edit to it
+applied: an earlier draft that a later write replaced does not count.
+
+PASS if its `bugs` array has an entry whose `claim` is the intent claim for
+the 3 to 20 inclusive length limit (the claim id must resolve, through the
+`intent` array, to that limit), and whose `repro` asserts that a username of
+exactly 20 characters is accepted, so that it fails when run against the code
+above.
+FAIL if no findings.json content is visible in the trace, if `bugs` is empty,
+if the only bug cites some other claim, or if the repro asserts something the
+code above already does (so it would pass, not fail).

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/bug-hunter-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:[\w-]+:)?bug-hunter"'
+---

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/graders/findings-written.md

@@ -0,0 +1,10 @@
+---
+type: regex
+# The run directory is .bug-hunter/<YYYY-MM-DD>-<slug>/ and the slug is the
+# skill's choice. A file_exists glob would accept any directory name, so this
+# reads the harness's files target, one created path per line, which holds
+# the date prefix to the shape SKILL.md states.
+pattern: '^\.bug-hunter/\d{4}-\d{2}-\d{2}-[^/]+/findings\.json$'
+flags: m
+target: files
+---

package/skills/bug-hunter/evals/proves-a-spec-boundary-bug/prompt.md

@@ -0,0 +1,11 @@
+---
+name: proves-a-spec-boundary-bug
+description: An off-by-one at a limit the spec states is the bug the skill exists to prove. Pins a confirmed bug in findings.json that cites the limit and carries a repro that fails against the fixture.
+tags: [bug, boundary, scaffold, write]
+max_turns: 80
+timeout_seconds: 2400
+allowed_tools: [Read, Glob, Grep, Skill, Write, Edit, Bash, Agent]
+expected_outcome: findings.json under .bug-hunter/ has one bug citing the 3 to 20 inclusive claim, with a repro asserting a 20-character name is accepted.
+---
+
+QA starts on signup tomorrow. What are they going to find in `src/username.js`? `SPEC.md` is what it's meant to do.