@slowdini/slow-powers-opencode 0.4.4 → 0.5.0

package/README.md

@@ -69,7 +69,7 @@ opencode plugin @slowdini/slow-powers-opencode -g
 Slow-powers provides a set of highly focused skills that ensure your agent operates with maximum discipline:
 
 1. **`hardening-plans`** — Instructs the agent to re-review any plans before it hands them back to you, looking for hallucinations, logical inconsistencies, and other common plan mistakes.
-2. **`systematic-debugging`** — Guides the agent to locate the root cause of failures via scientific hypothesis testing, avoiding "guess-and-check" thrashing.
+2. **`investigating-bugs`** — Guides the agent to locate the root cause of failures via scientific hypothesis testing, avoiding "guess-and-check" thrashing.
 3. **`working-in-isolation`** — Establishes an isolated workspace (worktree or branch) so new work doesn't collide with existing or in-progress work, keeping protected branches like `main` clean.
 4. **`test-driven-development`** — Enforces a strict RED-GREEN-REFACTOR cycle, ensuring all code is backed by failing test verification first.
 5. **`verifying-development-work`** — Requires running actual test/build commands and presenting concrete evidence before any success claim, with a final review pass over the change, code AND comments, before work is handed back.
@@ -82,7 +82,7 @@ The skills declare prerequisite / next-step gates so the agent follows an intend
 
 **Plan mode:** plan mode → `hardening-plans` → `working-in-isolation` → `test-driven-development` → `verifying-development-work`
 
-**Debugging:** (`working-in-isolation`) → `systematic-debugging` → `verifying-development-work`
+**Debugging:** (`working-in-isolation`) → `investigating-bugs` → `verifying-development-work`
 
 ## Philosophy
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@slowdini/slow-powers-opencode",
-  "version": "0.4.4",
+  "version": "0.5.0",
   "description": "Slow-powers — structured development workflows for coding agents (TDD, debugging, verification, git hygiene)",
   "type": "module",
   "main": "./opencode/plugins/slow-powers.js",
@@ -36,19 +36,19 @@
   },
   "scripts": {
     "test": "bun test --path-ignore-patterns='skills-workspace/**'",
-    "evals": "skill-eval run --skill-dir ./skills --bootstrap ./bootstrap.md",
-    "evals:snapshot": "skill-eval snapshot --skill-dir ./skills",
-    "evals:validate": "skill-eval validate --skill-dir ./skills",
-    "evals:ingest": "skill-eval ingest --skill-dir ./skills",
-    "evals:finalize": "skill-eval finalize --skill-dir ./skills",
-    "evals:record-runs": "skill-eval record-runs --skill-dir ./skills",
-    "evals:fill-transcripts": "skill-eval fill-transcripts --skill-dir ./skills",
-    "evals:detect-stray-writes": "skill-eval detect-stray-writes --skill-dir ./skills",
-    "evals:teardown-guard": "skill-eval teardown-guard --skill-dir ./skills",
-    "evals:teardown": "skill-eval teardown --skill-dir ./skills",
-    "evals:grade": "skill-eval grade --skill-dir ./skills",
-    "evals:aggregate": "skill-eval aggregate --skill-dir ./skills",
-    "evals:promote-baseline": "skill-eval promote-baseline --skill-dir ./skills",
+    "evals": "eval-magic run --skill-dir ./skills --bootstrap ./bootstrap.md",
+    "evals:snapshot": "eval-magic snapshot --skill-dir ./skills",
+    "evals:validate": "eval-magic validate --skill-dir ./skills",
+    "evals:ingest": "eval-magic ingest --skill-dir ./skills",
+    "evals:finalize": "eval-magic finalize --skill-dir ./skills",
+    "evals:record-runs": "eval-magic record-runs --skill-dir ./skills",
+    "evals:fill-transcripts": "eval-magic fill-transcripts --skill-dir ./skills",
+    "evals:detect-stray-writes": "eval-magic detect-stray-writes --skill-dir ./skills",
+    "evals:teardown-guard": "eval-magic teardown-guard --skill-dir ./skills",
+    "evals:teardown": "eval-magic teardown --skill-dir ./skills",
+    "evals:grade": "eval-magic grade --skill-dir ./skills",
+    "evals:aggregate": "eval-magic aggregate --skill-dir ./skills",
+    "evals:promote-baseline": "eval-magic promote-baseline --skill-dir ./skills",
     "check": "biome check --write .",
     "check:ci": "biome check --error-on-warnings .",
     "typecheck": "tsc --noEmit",

package/skills/evaluating-skills/SKILL.md

@@ -5,7 +5,7 @@ description: Use when testing whether a new skill improves agent behavior, or wh
 
 # Evaluating Skills
 
-Skill development has two phases: **drafting** (`slow-powers:writing-skills`) and **evaluation** (this skill). This skill owns the *craft* of evaluation — deciding whether a change needs measuring, designing test cases, devising pressure-testing scenarios, writing assertions, and reading results. The *mechanics* of actually running an eval — building the workspace, staging skills, dispatching subagents, grading, aggregating — are owned by a dedicated tool, **[eval-magic](https://github.com/slowdini/eval-magic)**, which ships as a dependency-less prebuilt binary you invoke as `skill-eval`. See [Running the eval](#running-the-eval) for the hand-off.
+Skill development has two phases: **drafting** (`slow-powers:writing-skills`) and **evaluation** (this skill). This skill owns the *craft* of evaluation — deciding whether a change needs measuring, designing test cases, devising pressure-testing scenarios, writing assertions, and reading results. The *mechanics* of actually running an eval — building the workspace, staging skills, dispatching subagents, grading, aggregating — are owned by a dedicated tool, **[eval-magic](https://github.com/slowdini/eval-magic)**, which ships as a dependency-less prebuilt binary you invoke as `eval-magic`. See [Running the eval](#running-the-eval) for the hand-off.
 
 ## Overview
 
@@ -95,7 +95,7 @@ A test case has these parts:
 - **files** (optional): fixture files the prompt references
 - **skill_should_trigger** (optional, default `true`): set `false` for a *negative* eval where correct behavior is the skill **not** firing (e.g. an over-trigger guard — a feature request that shouldn't launch a debugging investigation). Negative evals are excluded from the skill-invocation rate, so a correct non-invocation isn't mistaken for the skill failing to fire.
 
-Cases live in `<skill>/evals/evals.json`. For the file shape, see the author-template example in the eval-magic README and validate against the bundled schema with `skill-eval validate`; for worked, maintained examples, read the live suites in this repo — e.g. `skills/verifying-development-work/evals/evals.json` and `skills/hardening-plans/evals/evals.json`.
+Cases live in `<skill>/evals/evals.json`. For the file shape, see the author-template example in the eval-magic README and validate against the bundled schema with `eval-magic validate`; for worked, maintained examples, read the live suites in this repo — e.g. `skills/verifying-development-work/evals/evals.json` and `skills/hardening-plans/evals/evals.json`.
 
 Tips for writing good prompts:
 
@@ -142,7 +142,7 @@ Keep the seeded turns short and concrete; the point is to establish momentum, no
 
 **The ceiling — state it plainly.** A seed is *text the subagent reads*, not a state it operates under. It cannot place the agent in a harness-injected mode — a real plan mode, an enforced multi-phase workflow, genuine context-window pressure — it can only *describe* one. So when the wild failure you're chasing was *caused* by such a mode (the documented case: an agent in plan mode that invoked **zero** skills because the mode's own procedure made loading them feel redundant), a text seed cannot fully reproduce it — the causal layer is exactly the one a prompt string can't inject. A seeded **pass is therefore necessary but not sufficient** — it under-estimates real-session difficulty — and a seed that *fails* to reproduce a known wild failure is usually hitting this ceiling, not testing a bad seed. Treat seeded results as a stronger-than-cold signal, not as ground truth, and don't let downstream work over-trust them.
 
-**Narrowing the gap — `--plan-mode`.** For the documented plan-mode case, the runner offers the highest-fidelity in-runner approximation: its `--plan-mode` flag injects the harness's *verbatim* plan-mode procedure into every dispatch as an operating-context layer the subagent is told it is operating under, rather than a paraphrase the agent merely reads in the seed prose. This narrows the gap (verbatim procedure > paraphrase) but does **not** close it: it is still text the agent reads, not an injected mode, so the necessary-not-sufficient ceiling above stands unchanged. Use it as the strongest in-runner signal and pair it with a paraphrase-seed arm. See `skill-eval run --help` for the flag and the per-harness profiles it depends on.
+**Narrowing the gap — `--plan-mode`.** For the documented plan-mode case, the runner offers the highest-fidelity in-runner approximation: its `--plan-mode` flag injects the harness's *verbatim* plan-mode procedure into every dispatch as an operating-context layer the subagent is told it is operating under, rather than a paraphrase the agent merely reads in the seed prose. This narrows the gap (verbatim procedure > paraphrase) but does **not** close it: it is still text the agent reads, not an injected mode, so the necessary-not-sufficient ceiling above stands unchanged. Use it as the strongest in-runner signal and pair it with a paraphrase-seed arm. See `eval-magic run --help` for the flag and the per-harness profiles it depends on.
 
 ## Writing assertions
 
@@ -186,12 +186,12 @@ Once a run is graded and aggregated, the headline is the **delta**: what the ski
 
 ## Running the eval
 
-The mechanics of executing a run live in **[eval-magic](https://github.com/slowdini/eval-magic)** — the `skill-eval` binary. eval-magic's README is the complete operating guide, and every flag is documented in the tool's own help.
+The mechanics of executing a run live in **[eval-magic](https://github.com/slowdini/eval-magic)** — the `eval-magic` binary. eval-magic's README is the complete operating guide, and every flag is documented in the tool's own help.
 
 | Need | Where |
 |------|-------|
 | Quickstart, install, the two modes end-to-end | the eval-magic README |
-| Every subcommand and flag; the `--skill-dir` model; workspace layout | `skill-eval --help` and `skill-eval <subcommand> --help` |
+| Every subcommand and flag; the `--skill-dir` model; workspace layout | `eval-magic --help` and `eval-magic <subcommand> --help` |
 | Full run mechanics: dispatch loop, transcript access, grading, aggregating, baselines | the eval-magic README |
 | Claude Code & Codex harness specifics — isolating from installed plugins, the guard, judging | the README's Harnesses section |
 | What a harness needs to reach Claude-Code-tier support | `docs/harness-parity.md` |
@@ -200,5 +200,5 @@ The mechanics of executing a run live in **[eval-magic](https://github.com/slowd
 
 - `slow-powers:writing-skills` — drafting a skill (Phase 1)
 - `pressure-scenarios.md` — pressure-scenario taxonomy for authoring prompts that stress discipline-enforcing skills
-- eval-magic (the `skill-eval` tool) — runs the evals this skill teaches you to author
+- eval-magic (the `eval-magic` tool) — runs the evals this skill teaches you to author
 - agentskills.io/skill-creation/evaluating-skills — the methodology this skill is derived from

package/skills/evaluating-skills/evals/baseline/BASELINE.md

@@ -1,9 +1,9 @@
 # Baseline — evaluating-skills
 
 Committed reference output from a canonical eval run. Regenerate with
-`skill-eval promote-baseline --skill evaluating-skills --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
+`eval-magic promote-baseline --skill evaluating-skills --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
 dispatch files, produced outputs) stays gitignored under `skills-workspace/`
-and is reclaimable by `skill-eval teardown` once promoted (this commit's marker).
+and is reclaimable by `eval-magic teardown` once promoted (this commit's marker).
 
 | Field | Value |
 |-------|-------|
@@ -20,4 +20,3 @@ and is reclaimable by `skill-eval teardown` once promoted (this commit's marker)
 Files:
 - `benchmark.json` — aggregate pass-rate / duration / token deltas.
 - `grading/<eval-id>__<condition>.json` — per-run assertion results and judge rationales.
-

package/skills/hardening-plans/evals/baseline/BASELINE.md

@@ -1,9 +1,9 @@
 # Baseline — hardening-plans
 
 Committed reference output from a canonical eval run. Regenerate with
-`skill-eval promote-baseline --skill hardening-plans --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
+`eval-magic promote-baseline --skill hardening-plans --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
 dispatch files, produced outputs) stays gitignored under `skills-workspace/`
-and is reclaimable by `skill-eval teardown` once promoted (this commit's marker).
+and is reclaimable by `eval-magic teardown` once promoted (this commit's marker).
 
 | Field | Value |
 |-------|-------|
@@ -24,4 +24,3 @@ the named-hand-off requirement). `new_skill` = the working tree at promotion
 Files:
 - `benchmark.json` — aggregate pass-rate / duration / token deltas.
 - `grading/<eval-id>__<condition>.json` — per-run assertion results and judge rationales.
-

package/skills/{systematic-debugging → investigating-bugs}/SKILL.md

@@ -1,9 +1,9 @@
 ---
-name: systematic-debugging
+name: investigating-bugs
 description: Use when encountering any bug, test failure, build error, or unexpected behavior.
 ---
 
-# Systematic Debugging
+# Investigating Bugs
 
 Avoid "guess-and-check" coding. Always identify the root cause before making changes.
 
@@ -22,11 +22,13 @@ Avoid "guess-and-check" coding. Always identify the root cause before making cha
 Before changing any code:
 1. **Read Error Messages and Stack Traces:** Read every line of the error. Note the exact file, line number, and error codes.
 2. **Reproduce Consistently:** Identify the exact steps, inputs, or environment needed to trigger the bug. If it cannot be reproduced, gather more logs instead of guessing.
+   * For flaky tests that pass sometimes and fail under load, the cause is usually arbitrary `sleep`/timeout delays. Wait on the actual condition, not a guessed duration — see `condition-based-waiting.md` in this directory.
 3. **Check Recent Changes:** Run a git diff. Analyze recent commits, dependency additions, or config changes.
 4. **Gather Evidence (Multi-Component Systems):**
    * Log inputs and outputs at every component boundary.
    * Instrument the layers step-by-step (e.g., Workflow -> Build Script -> Runtime -> DB) to pinpoint exactly where the state breaks.
 5. **Trace Data Flow:** Trace variables backward from the failure point to their source. Fix the bug at the source, not the symptom.
+   * When manual tracing dead-ends, instrument the suspect operation: log the key inputs, relevant environment, and a captured stack trace (`new Error().stack`) *just before* it runs. In tests, write to stderr — a logger may be suppressed. Read the captured stack to find the original caller, then remove the instrumentation.
 
 ---
 
@@ -56,14 +58,12 @@ Before changing any code:
 4. **The Three-Fix Limit (Architectural Check):**
    * If you attempt **three separate fixes** and the bug remains: **STOP.**
    * This is a strong signal that the issue is architectural (e.g., wrong model assumptions, coupled state, race conditions).
-   * Re-evaluate the system architecture and discuss the approach with your human partner before attempting a fourth patch.
+   * Re-evaluate the system architecture and discuss the approach with the user before attempting a fourth patch.
 
 ---
 
 ## Common Rationalizations
 
-> **Note:** The rationalizations below are prospective — they represent likely excuses an agent might produce under pressure, but they have not yet been validated through actual eval runs. After running pressure-test evals, replace or augment these with verbatim quotes from failed runs.
-
 | Excuse | Reality |
 |--------|---------|
 | "This is an emergency, we don't have time" | 5 minutes of investigation beats 5 hours of chasing symptoms. |
@@ -77,8 +77,6 @@ Before changing any code:
 
 ## Red Flags — STOP and Reset
 
-> **Note:** The red flags below are prospective — they represent likely warning signs, but they have not yet been validated through actual eval runs.
-
 - Writing a fix before reproducing the bug or reading the full stack trace
 - "Let's just try changing X to see if it works"
 - Stacking multiple speculative fixes on top of each other

package/skills/{systematic-debugging → investigating-bugs}/condition-based-waiting-example.ts

@@ -1,6 +1,6 @@
-// Complete implementation of condition-based waiting utilities
-// From: Lace test infrastructure improvements (2025-10-03)
-// Context: Fixed 15 flaky tests by replacing arbitrary timeouts
+// Complete implementation of condition-based waiting utilities.
+// Domain-specific example (event-driven thread manager) showing how the generic
+// waitFor pattern specializes into reusable helpers for a real codebase.
 
 import type { ThreadManager } from "~/threads/thread-manager";
 import type { LaceEvent, LaceEventType } from "~/threads/types";

package/skills/{systematic-debugging → investigating-bugs}/condition-based-waiting.md

@@ -78,7 +78,7 @@ async function waitFor<T>(
 }
 ```
 
-See `condition-based-waiting-example.ts` in this directory for complete implementation with domain-specific helpers (`waitForEvent`, `waitForEventCount`, `waitForEventMatch`) from actual debugging session.
+See `condition-based-waiting-example.ts` in this directory for a complete implementation with domain-specific helpers (`waitForEvent`, `waitForEventCount`, `waitForEventMatch`).
 
 ## Common Mistakes
 
@@ -104,11 +104,3 @@ await new Promise(r => setTimeout(r, 200));   // Then: wait for timed behavior
 1. First wait for triggering condition
 2. Based on known timing (not guessing)
 3. Comment explaining WHY
-
-## Real-World Impact
-
-From debugging session (2025-10-03):
-- Fixed 15 flaky tests across 3 files
-- Pass rate: 60% → 100%
-- Execution time: 40% faster
-- No more race conditions

package/skills/investigating-bugs/evals/baseline/BASELINE.md

@@ -0,0 +1,23 @@
+# Baseline — investigating-bugs
+
+Committed reference output from a canonical eval run. Regenerate with
+`eval-magic promote-baseline --skill investigating-bugs --iteration <N>` after aggregating. The ephemeral workspace (run records, timing,
+dispatch files, produced outputs) stays gitignored under `skills-workspace/`
+and is reclaimable by `eval-magic teardown` once promoted (this commit's marker).
+
+| Field | Value |
+|-------|-------|
+| Mode | new-skill |
+| Iteration | iteration-1 |
+| Harness | claude-code |
+| Agent model | unspecified |
+| Judge model | unspecified |
+| Conditions | with_skill, without_skill |
+| Run timestamp | 2026-06-11T04:22:46.590Z |
+| Label | issue-207 rename + pressure validation (Mode A, sonnet-4-6) |
+| Promoted from commit | 37289e4 |
+
+Files:
+- `benchmark.json` — aggregate pass-rate / duration / token deltas.
+- `grading/<eval-id>__<condition>.json` — per-run assertion results and judge rationales.
+

package/skills/investigating-bugs/evals/baseline/benchmark.json

@@ -0,0 +1,51 @@
+{
+  "generated": "2026-06-11T04:28:59.764Z",
+  "mode": "new-skill",
+  "conditions_compared": ["with_skill", "without_skill"],
+  "missing_gradings": 0,
+  "validity_warnings": [],
+  "run_summary": {
+    "with_skill": {
+      "pass_rate": {
+        "mean": 1.0,
+        "stddev": 0.0,
+        "n": 4
+      },
+      "duration_ms": {
+        "mean": 33639.0,
+        "stddev": 10185.0,
+        "n": 4
+      },
+      "total_tokens": {
+        "mean": 87928.0,
+        "stddev": 26986.0,
+        "n": 4
+      },
+      "skill_invocation_n": 3,
+      "skill_invocation_rate": 1.0
+    },
+    "without_skill": {
+      "pass_rate": {
+        "mean": 0.917,
+        "stddev": 0.144,
+        "n": 4
+      },
+      "duration_ms": {
+        "mean": 24224.0,
+        "stddev": 4179.0,
+        "n": 4
+      },
+      "total_tokens": {
+        "mean": 72100.0,
+        "stddev": 10863.0,
+        "n": 4
+      }
+    }
+  },
+  "delta": {
+    "direction": "with_skill - without_skill",
+    "pass_rate": 0.083,
+    "duration_ms": 9415.0,
+    "total_tokens": 15828.0
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/feature-request-no-debugging__with_skill.json

@@ -0,0 +1,17 @@
+{
+  "assertion_results": [
+    {
+      "id": "did_not_overtrigger_debugging",
+      "passed": true,
+      "evidence": "The agent responded with a direct feature implementation plan ('Here's how to add a `--verbose` flag to your CLI that prints each file as it's processed') covering argument parsing, threading the flag through processing logic, and wiring it up at the entry point — with no mention of bug reproduction, stack traces, root cause investigation, or any debugging phases.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 1,
+    "failed": 0,
+    "total": 1,
+    "pass_rate": 1.0
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/feature-request-no-debugging__without_skill.json

@@ -0,0 +1,17 @@
+{
+  "assertion_results": [
+    {
+      "id": "did_not_overtrigger_debugging",
+      "passed": true,
+      "evidence": "The agent responded with straightforward implementation guidance: 'To add a `--verbose` flag that prints each file as it's processed, here's what needs to change' — covering argument parsing, threading the flag into the processing loop, and updating help text. No debugging phases, bug reproduction, stack trace analysis, or failure hypotheses appear anywhere in the response.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 1,
+    "failed": 0,
+    "total": 1,
+    "pass_rate": 1.0
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/null-id-crash-investigate-first__with_skill.json

@@ -0,0 +1,46 @@
+{
+  "assertion_results": [
+    {
+      "id": "inspected_before_patching",
+      "passed": true,
+      "evidence": "matched ordinal 1: Read {\"file_path\":\"/Users/maximilianhaarhaus/Projects/slow-powers/skills-workspace/investigating-bugs/iteration-1/eval-null-id-crash-investigate-first/with_skill/inputs/orderHandler.ts\"}",
+      "confidence": 1.0,
+      "grader": "transcript_check"
+    },
+    {
+      "id": "root_cause_before_fix",
+      "passed": true,
+      "evidence": "Tool invocations show the agent read orderHandler.ts (ordinal 1) and repro.ts (ordinal 2) before writing any fix (ordinal 5). The final message explicitly states 'Before touching anything, I investigated the root cause' and explains WHY customer is undefined: 'The Order type declares customer as optional (customer?:), so guest-checkout orders arrive without it. When they do, order.customer is undefined and accessing .id throws.'",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "fix_targets_source_not_symptom",
+      "passed": true,
+      "evidence": "The agent explicitly named the root cause: 'The Order type declares customer as optional (customer?:), so guest-checkout orders arrive without it. When they do, order.customer is undefined and accessing .id throws.' It also distinguished source from symptom: 'This is deterministic — every guest checkout triggers it. The intermittency you see in production is just the rate of guest-checkout orders, not non-determinism in the code.' The fix (a guard in describeOrder itself) follows directly from this diagnosis, and the agent noted 'The fix lives at the root cause (the function's own missing guard), not at every call site.'",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 3,
+    "failed": 0,
+    "total": 3,
+    "pass_rate": 1.0
+  },
+  "meta_results": [
+    {
+      "id": "__skill_invoked",
+      "passed": true,
+      "evidence": "The agent's final_message opens with 'Before touching anything, I investigated the root cause' — a direct echo of the skill's Iron Law ('NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST'). It then follows the skill's Phase 1 (tracing data flow back to the optional `customer?:` field), Phase 3 (formulating a single hypothesis: 'every guest checkout triggers it'), and Phase 4 (writing a failing test before implementing the fix, then verifying no regressions). The claim 'The intermittency you see in production is just the rate of guest-checkout orders, not non-determinism in the code' mirrors the skill's Phase 1 Step 2 instruction to 'Reproduce Consistently' and explain apparent flakiness.",
+      "confidence": 0.97,
+      "grader": "llm_judge"
+    }
+  ],
+  "meta_summary": {
+    "passed": 1,
+    "failed": 0,
+    "total": 1,
+    "skill_invoked": true
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/null-id-crash-investigate-first__without_skill.json

@@ -0,0 +1,31 @@
+{
+  "assertion_results": [
+    {
+      "id": "inspected_before_patching",
+      "passed": true,
+      "evidence": "matched ordinal 1: Read {\"file_path\":\"/Users/maximilianhaarhaus/Projects/slow-powers/skills-workspace/investigating-bugs/iteration-1/eval-null-id-crash-investigate-first/without_skill/inputs/orderHandler.ts\"}",
+      "confidence": 1.0,
+      "grader": "transcript_check"
+    },
+    {
+      "id": "root_cause_before_fix",
+      "passed": false,
+      "evidence": "Tool invocations show the agent read orderHandler.ts (ordinal 1) and repro.ts (ordinal 2) and then immediately wrote the patched file (ordinal 4) without ever running repro.ts or executing any code to reproduce the crash. No Bash invocation ran the repro script or constructed a customer-less order; the agent inferred the root cause from reading alone and applied the fix in the same pass.",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "fix_targets_source_not_symptom",
+      "passed": true,
+      "evidence": "\"Guest-checkout orders arrive without a `customer`, which triggers the `TypeError` intermittently.\" — the agent explicitly named the root cause (guest-checkout orders legitimately have no customer) and identified the code's wrong assumption before applying the guard.",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 2,
+    "failed": 1,
+    "total": 3,
+    "pass_rate": 0.6666666666666666
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/seeded-stacked-guess-investigate-first__with_skill.json

@@ -0,0 +1,46 @@
+{
+  "assertion_results": [
+    {
+      "id": "investigated_before_second_patch",
+      "passed": true,
+      "evidence": "matched ordinal 1: Read {\"file_path\":\"/Users/maximilianhaarhaus/Projects/slow-powers/skills-workspace/investigating-bugs/iteration-1/eval-seeded-stacked-guess-investigate-first/with_skill/inputs/orderHandler.ts\"}",
+      "confidence": 1.0,
+      "grader": "transcript_check"
+    },
+    {
+      "id": "rejects_stacked_guess",
+      "passed": true,
+      "evidence": "Agent explicitly refused: 'I will not add another guard on top of the existing patch. That would be a third speculative change on an uninvestigated root cause.' It then investigated the root cause, identifying that customer is optional in the type and guest-checkout orders legitimately arrive with no customer field, before proposing a targeted fix.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "root_cause_named",
+      "passed": true,
+      "evidence": "The root cause is a **missing branch for guest-checkout orders in `describeOrder`**. The type correctly marks `customer` as optional (these orders are valid and intentional — guest checkouts), but the function was written as if a customer is always present.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 3,
+    "failed": 0,
+    "total": 3,
+    "pass_rate": 1.0
+  },
+  "meta_results": [
+    {
+      "id": "__skill_invoked",
+      "passed": true,
+      "evidence": "The agent explicitly invokes multiple skill fingerprints: it names the 'Red Flag pattern' verbatim ('each \"fix\" only shifts the bug to a new location'), quotes the skill's reset instruction ('STOP. Revert changes. Return to Phase 1.'), labels its investigation section 'Root cause investigation (Phase 1)' and its fix section 'Phase 4', and cites the skill's iron-law framing by refusing to add a third speculative change — all language and structure drawn directly from the skill content.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "meta_summary": {
+    "passed": 1,
+    "failed": 0,
+    "total": 1,
+    "skill_invoked": true
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/seeded-stacked-guess-investigate-first__without_skill.json

@@ -0,0 +1,31 @@
+{
+  "assertion_results": [
+    {
+      "id": "investigated_before_second_patch",
+      "passed": true,
+      "evidence": "matched ordinal 1: Read {\"file_path\":\"/Users/maximilianhaarhaus/Projects/slow-powers/skills-workspace/investigating-bugs/iteration-1/eval-seeded-stacked-guess-investigate-first/without_skill/inputs/orderHandler.ts\"}",
+      "confidence": 1.0,
+      "grader": "transcript_check"
+    },
+    {
+      "id": "rejects_stacked_guess",
+      "passed": true,
+      "evidence": "The agent explicitly declined to add the guard without investigation: 'we've now applied two patches in a row based on guesses, and each one revealed a deeper problem. Before we add a third guard, it's worth spending two minutes understanding what's actually happening.' It then read the source files and correctly identified that describeOrder is a logging function, not a save path, and asked to see the order-save and invoice-generation paths before changing any code.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "root_cause_named",
+      "passed": true,
+      "evidence": "The repro script shows that guest-checkout orders ({ id: \"A-1002\" }) legitimately have no customer — that's expected behaviour for guest checkouts.",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 3,
+    "failed": 0,
+    "total": 3,
+    "pass_rate": 1.0
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/seeded-three-fix-limit-stop__with_skill.json

@@ -0,0 +1,39 @@
+{
+  "assertion_results": [
+    {
+      "id": "stops_at_three_fixes",
+      "passed": true,
+      "evidence": "\"I can't ship a fourth patch. Three separate fixes failed, which is the signal to stop — not to keep stacking guesses. [...] The Three-Fix Limit has triggered.\" The agent explicitly declines the fourth timeout/retry patch and redirects to root-cause investigation (reproduce deterministically, read failure output, check git history).",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "names_architectural_cause",
+      "passed": true,
+      "evidence": "\"That pattern means we're chasing a symptom, not fixing a root cause\" and \"whether this is a timing issue, a data issue, or a race condition\" — the agent explicitly names race condition as a candidate root cause and frames the problem as architectural rather than a matter of waiting longer.",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 2,
+    "failed": 0,
+    "total": 2,
+    "pass_rate": 1.0
+  },
+  "meta_results": [
+    {
+      "id": "__skill_invoked",
+      "passed": true,
+      "evidence": "The agent explicitly names the skill's rule verbatim: '**The Three-Fix Limit has triggered.**' — a direct quote of the Phase 4 heading 'The Three-Fix Limit (Architectural Check)' — and structures its refusal around that rule, matching the skill's prescribed behavior of stopping after three failed fixes and redirecting to root-cause investigation.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    }
+  ],
+  "meta_summary": {
+    "passed": 1,
+    "failed": 0,
+    "total": 1,
+    "skill_invoked": true
+  }
+}

package/skills/investigating-bugs/evals/baseline/grading/seeded-three-fix-limit-stop__without_skill.json

@@ -0,0 +1,24 @@
+{
+  "assertion_results": [
+    {
+      "id": "stops_at_three_fixes",
+      "passed": true,
+      "evidence": "Agent explicitly declines the fourth patch: 'three fixes in a row haven't worked, and adding more timeouts and retries on top of them is very likely to make things worse or just mask the problem until it resurfaces in production.' It then pivots to root-cause diagnosis, proposing to check whether the batch insert completes for all 3 records and whether the test is using the correct DB connection.",
+      "confidence": 1.0,
+      "grader": "llm_judge"
+    },
+    {
+      "id": "names_architectural_cause",
+      "passed": true,
+      "evidence": "Agent names two architectural root causes: 'Are we looking at the right DB connection? If the test and the batch handler are using different connections or the read is hitting a read replica, the transaction commit won’t help.' and 'Is the batch insert itself completing for all 3 records? If the insert is silently failing or swallowing an error for one record, no amount of waiting will fix it.' Both go well beyond 'wait longer' framing.",
+      "confidence": 0.95,
+      "grader": "llm_judge"
+    }
+  ],
+  "summary": {
+    "passed": 2,
+    "failed": 0,
+    "total": 2,
+    "pass_rate": 1.0
+  }
+}