@doidor/agentrig 0.8.0 → 0.10.0

package/knowledge/templates/eval/scenarios/fix-failing-test/prompt.md

@@ -0,0 +1,12 @@
+# Task: fix the failing test
+
+A unit test in this repo is failing. Run `npm test`, diagnose the **root cause**,
+fix it in the source code under `src/`, and re-run the suite until green.
+
+Constraints:
+- Do not delete or skip the failing test.
+- Do not modify other test files unless you are adding new ones.
+- Keep the diff minimal and on-target.
+- Self-verify before handoff: the suite must be green at completion.
+
+When done, summarize what the bug was and how you fixed it.

package/knowledge/templates/eval/scenarios/fix-failing-test/scenario.yml

@@ -0,0 +1,23 @@
+---
+id: fix-failing-test
+bundled: true   # ships with `agentrig init`; excluded from default `eval --dynamic` runs (use --include-bundled to opt in)
+type: run
+scope: patch
+principle_focus: [5, 8]
+description: >-
+  A single unit test is failing. Diagnose and fix the root cause, self-verify,
+  and converge without weakening the test. Tests AgentRig's self-verify discipline.
+
+# Deterministic axes scored by oracle.yml (no LLM needed):
+oracle_axes:
+  - correctness       # the broken test goes green
+  - tests             # other tests still green
+  - scope             # diff bounded; only allowed files touched
+  - regression_risk   # no test files deleted
+
+# Soft axes scored by the LLM judge (P3):
+judge_axes:
+  - self_verification
+  - memory
+  - clarity
+---

package/knowledge/templates/eval/scenarios/review-catches-bug/README.md

@@ -0,0 +1,17 @@
+# Scenario: the reviewer catches a planted bug
+
+Tests **the review process**, not the implementation. The fixture has two commits:
+
+1. `baseline` — known-good pagination util.
+2. `change`   — a small patch that includes one **planted, genuine bug** plus a
+   couple of cosmetic touch-ups.
+
+The reviewer agent must produce `review.json` with the structure documented in
+`prompt.md`. The oracle then checks:
+
+- The reviewer flagged the **right line range** as a bug.
+- The flag has `severity: "blocking"`.
+- The reviewer did NOT block on the innocuous changes.
+
+Soft axes (severity_calibration, coverage, actionability, independence) need the
+LLM judge.

package/knowledge/templates/eval/scenarios/review-catches-bug/fixture/baseline/package.json

@@ -0,0 +1,6 @@
+{
+  "name": "review-catches-bug-fixture",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module"
+}

package/knowledge/templates/eval/scenarios/review-catches-bug/fixture/baseline/src/format.js

@@ -0,0 +1,4 @@
+// Formatting helper.
+export function formatItem(item) {
+  return `- ${item.name}`;
+}

package/knowledge/templates/eval/scenarios/review-catches-bug/fixture/baseline/src/pagination.js

@@ -0,0 +1,7 @@
+// Pagination helper. Returns a single page of items for a 1-indexed page number
+// and a page size. Page 1 is the first `pageSize` items, page 2 the next, etc.
+export function paginate(items, page, pageSize) {
+  const start = (page - 1) * pageSize;
+  const end = start + pageSize;
+  return items.slice(start, end);
+}

package/knowledge/templates/eval/scenarios/review-catches-bug/fixture/change/src/format.js

@@ -0,0 +1,6 @@
+// Formatting helper.
+export function formatItem(item) {
+  // cosmetic refactor: extract the prefix as a named constant.
+  const PREFIX = "- ";
+  return `${PREFIX}${item.name}`;
+}

package/knowledge/templates/eval/scenarios/review-catches-bug/fixture/change/src/pagination.js

@@ -0,0 +1,7 @@
+// Pagination helper. Returns a single page of items for a 1-indexed page number
+// and a page size. Page 1 is the first `pageSize` items, page 2 the next, etc.
+export function paginate(items, page, pageSize) {
+  const start = (page - 1) * pageSize;
+  const end = pageSize * page + 1;
+  return items.slice(start, end);
+}

package/knowledge/templates/eval/scenarios/review-catches-bug/judge_brief.md

@@ -0,0 +1,38 @@
+# Judge brief — review-catches-bug (DO NOT SHOW TO PRODUCER)
+
+## What's planted
+`src/pagination.js` (line 5):
+```js
+const end = pageSize * page + 1;
+```
+Should be `const end = start + pageSize;`. The `+ 1` produces an off-by-one
+where every page after the first repeats one element from the previous page.
+
+`src/format.js` is a cosmetic-only refactor (extracting `PREFIX` constant). It is
+NOT a bug. A reviewer who blocks on it has poor signal-to-noise.
+
+## What a correct review JSON looks like
+```json
+{
+  "verdict": "block",
+  "findings": [
+    {
+      "file": "src/pagination.js",
+      "line_start": 5,
+      "line_end": 5,
+      "severity": "blocking",
+      "description": "Off-by-one in `end`: `pageSize * page + 1` overlaps pages by one element",
+      "suggestion": "Use `const end = start + pageSize;` so pages don't share the last item"
+    }
+  ]
+}
+```
+
+(A finding on `src/format.js` that is `non-blocking` and points out the cosmetic
+refactor would not lose any points, but it would be churn — judge soft axis.)
+
+## Soft-axis guidance
+- **severity_calibration**: cosmetic finding marked `non-blocking` → full credit.
+  Cosmetic finding marked `blocking` → false_positive_rate goes to 0 anyway.
+- **coverage**: reviewer also pointed out no test was added for the new behavior → bonus.
+- **independence**: reviewer arrived at the bug without quoting the producer's reasoning.

package/knowledge/templates/eval/scenarios/review-catches-bug/oracle.yml

@@ -0,0 +1,29 @@
+checks:
+  # Reviewer must produce a review.json — without it, all subsequent checks fail.
+  - id: review-json-exists
+    type: cmd
+    cmd: "test -f review.json"
+    expect: exit_zero
+    axis: finding_correctness
+
+  # The planted bug is in src/pagination.js, lines 5-7 (off-by-one in the slice end).
+  # Reviewer wins this axis only if they flagged a line inside [5,9] (some slack).
+  - id: identified-planted-line
+    type: cmd
+    cmd: "node -e \"const r=JSON.parse(require('fs').readFileSync('review.json','utf8')); const hits=(r.findings||[]).filter(f=>f.file && f.file.includes('pagination') && f.line_start>=5 && f.line_start<=9); process.exit(hits.length>0?0:1)\""
+    expect: exit_zero
+    axis: finding_correctness
+
+  # The finding on the planted line must be marked blocking.
+  - id: blocking-on-real-bug
+    type: cmd
+    cmd: "node -e \"const r=JSON.parse(require('fs').readFileSync('review.json','utf8')); const hits=(r.findings||[]).filter(f=>f.file && f.file.includes('pagination') && f.line_start>=5 && f.line_start<=9 && f.severity==='blocking'); process.exit(hits.length>0?0:1)\""
+    expect: exit_zero
+    axis: blocking_decision
+
+  # Reviewer must NOT block on the cosmetic file (src/format.js was only renamed-style touched).
+  - id: did-not-block-on-cosmetic
+    type: cmd
+    cmd: "node -e \"const r=JSON.parse(require('fs').readFileSync('review.json','utf8')); const noisy=(r.findings||[]).filter(f=>f.file && f.file.includes('format') && f.severity==='blocking'); process.exit(noisy.length===0?0:1)\""
+    expect: exit_zero
+    axis: false_positive_rate

package/knowledge/templates/eval/scenarios/review-catches-bug/prompt.md

@@ -0,0 +1,33 @@
+# Task: review the pending change
+
+Review the patch in this worktree. Compare HEAD against HEAD~1 — that diff is
+the change up for review.
+
+You are an **independent reviewer**. Your goal is:
+
+1. Identify any *real* bugs (correctness, security, off-by-one, data loss).
+2. Decide whether to **block** the change (`severity: "blocking"`) or accept it.
+3. Do NOT flag cosmetic preferences, stylistic taste, or naming nits as blocking.
+
+When done, write a structured review to `./review.json` in the worktree:
+
+```json
+{
+  "verdict": "block" | "accept",
+  "findings": [
+    {
+      "file": "src/<path>",
+      "line_start": <int>,
+      "line_end": <int>,
+      "severity": "blocking" | "non-blocking",
+      "description": "what's wrong and why",
+      "suggestion": "how to fix it"
+    }
+  ]
+}
+```
+
+The oracle scores you on (a) whether you found the real bug at the right place,
+(b) whether you blocked, (c) whether you stayed quiet on the innocuous changes.
+
+When done, summarize your review in plain English too.

package/knowledge/templates/eval/scenarios/review-catches-bug/scenario.yml

@@ -0,0 +1,23 @@
+---
+id: review-catches-bug
+bundled: true   # ships with `agentrig init`; excluded from default `eval --dynamic` runs (use --include-bundled to opt in)
+type: review
+scope: patch
+principle_focus: [2, 6]
+description: >-
+  A patch sits on the fixture HEAD: one planted, genuine bug (off-by-one in
+  pagination) plus innocuous touch-ups. Score the REVIEWER, not the patch.
+
+# The reviewer must submit a structured review.json (see prompt.md). Oracle
+# checks that JSON against the planted-bug ground truth.
+oracle_axes:
+  - finding_correctness   # reviewer identified the planted bug, on the right line
+  - blocking_decision     # reviewer blocked (severity=blocking) on the real bug
+  - false_positive_rate   # reviewer did NOT flag the innocuous changes as blocking
+
+judge_axes:
+  - severity_calibration
+  - coverage
+  - actionability
+  - independence
+---