@slowdini/slow-powers-opencode 0.1.0

package/skills/evaluating-skills/harness-details/claude.md

@@ -0,0 +1,135 @@
+# Running an eval on a skill — Claude Code
+
+This is the Claude Code-specific walkthrough for `evaluating-skills`. The runner contract (`--skill-dir`, `--skill`, `--bootstrap`, modes, what gets staged) is described in `../SKILL.md`; this file tells you exactly how to drive it inside Claude Code.
+
+Use this when a user, working from their own skill folder, asks to run an eval (e.g. "run an eval on this skill to check if a change reduces token usage").
+
+## Step 1 — Resolve the bundled runner
+
+The runner ships inside the installed slow-powers plugin. Resolve its path once per session and reuse it. Use `find` rather than a shell glob so the command behaves the same under bash and zsh (a bare glob with no match errors under zsh):
+
+```bash
+SLOW_POWERS_RUNNER_ROOT="$(find ~/.claude/plugins/cache -maxdepth 6 -type d -path '*/slow-powers/*/skills/evaluating-skills/runner' 2>/dev/null | sort | tail -1)"
+# Fallback for dev/marketplace installs:
+[ -z "$SLOW_POWERS_RUNNER_ROOT" ] && SLOW_POWERS_RUNNER_ROOT="$(find ~/.claude/plugins/marketplaces -maxdepth 6 -type d -path '*/slow-powers/*/skills/evaluating-skills/runner' 2>/dev/null | sort | tail -1)"
+echo "$SLOW_POWERS_RUNNER_ROOT"
+```
+
+(`sort | tail -1` prefers the lexically-latest version directory when several are installed.)
+
+If this is empty, the plugin isn't installed at the canonical path. Tell the user to clone the slow-powers repo and run from there (`bun run evals -- --skill <name> --mode <mode>`), or to reinstall the plugin.
+
+## Step 2 — Check the prerequisite
+
+```bash
+bun --version
+```
+
+If `bun` is missing, the runner can't execute. Tell the user to install it: `curl -fsSL https://bun.sh/install | bash` (or `brew install bun`), then retry.
+
+The runner depends on one package, `ajv` (runtime schema validation). `bun` auto-installs it on first run, so no manual step is normally needed. In an offline/airgapped environment where auto-install can't reach the registry, run `bun install` once (in the slow-powers repo, or wherever `package.json` lives) before the first eval.
+
+## Step 3 — Detect the skill folder
+
+The user typically opens Claude Code inside their skill folder. Confirm it:
+
+```bash
+ls SKILL.md evals/evals.json 2>/dev/null
+```
+
+- No `SKILL.md`: ask the user for the path to their skill folder.
+- No `evals/evals.json`: go to Step 5 to author one.
+
+## Step 4 — Derive `--skill-dir` and `--skill`
+
+`--skill-dir` is the **parent** directory that holds skill folders; `--skill` is the skill folder's name. If the current directory is the skill folder itself:
+
+- `--skill` = the basename of the current directory (e.g. `mr-review`)
+- `--skill-dir` = the parent directory
+
+Confirm these with the user before running. Remember: every skill inside `--skill-dir` is staged as a sibling. If the user wants their skill evaluated in isolation, `--skill-dir` should contain only that one skill (the common case). If they want slow-powers skills available as siblings, they must copy or symlink them into `--skill-dir` first.
+
+## Step 5 — Author `evals/evals.json` (only if missing)
+
+Read the template at `${SLOW_POWERS_RUNNER_ROOT}/../templates/evals.json.example` and walk the user through writing 2–3 realistic prompts, following the "Designing test cases" guidance in `../SKILL.md`. Save it to `<skill-folder>/evals/evals.json`. Don't write assertions yet — see the methodology.
+
+## Step 6 — Gitignore the workspace
+
+The runner writes artifacts to `<CWD>/skills-workspace/`. Keep it out of version control:
+
+```bash
+grep -qxF 'skills-workspace/' .gitignore 2>/dev/null || echo 'skills-workspace/' >> .gitignore
+```
+
+If the folder isn't a git repo, skip this and warn the user that artifacts will accumulate under `skills-workspace/`.
+
+## Step 7 — Pre-flight confirmation & sandbox decision
+
+This is a required gate (see *Pre-flight gate* in `../SKILL.md`). Do not run the build or dispatch anything until the user has confirmed.
+
+1. Read `<skill-folder>/evals/evals.json` and assemble the run summary:
+   - **Skill under test** — `<name>` and its path
+   - **Mode** — `new-skill` or `revision` (+ baseline label)
+   - **Eval cases** — count and a one-line list of the prompts
+   - **Models** — the model you'll dispatch each subagent under test with (Step 9) and the judge model for `llm_judge` assertions (Step 10). State them explicitly; the runner can't observe them, so this is the user's chance to correct a wrong choice before tokens are spent.
+   - **Cost** — `2 × <case count>` agent dispatches plus a judge dispatch per `llm_judge` assertion; flag it as time- and token-intensive.
+   - **Sandbox** — you will arm `--guard` (the default on Claude Code).
+2. Present the summary and **wait for explicit confirmation.** An earlier "run the eval" doesn't count — the summary may surface a wrong mode, model, or a guard the user didn't intend.
+3. Default to `--guard`. Drop it **only** if the user actively opts out, and then warn them: without the guard, stray writes (e.g. worktrees a skill-under-test creates in this repo) are only *detected* post-hoc by `detect-stray-writes` in Step 10 — never blocked or reverted — and are theirs to clean up.
+
+## Step 8 — Run the workspace build
+
+Run from the skill folder (so `CWD` is the eval root and staging lands at `<CWD>/.claude/skills/`).
+
+`--guard` is on in the commands below because it's the default posture (Step 7). It stages a `PreToolUse` hook into `.claude/settings.local.json` that *blocks* subagent writes/installs outside the eval sandbox (the workspace, the staged-skills dir, and `$TMPDIR`) while dispatches run. The hook is gated by a marker that auto-expires after 6h and is torn down at the start of the next run; to remove it immediately, run `bun run "$SLOW_POWERS_RUNNER_ROOT/run.ts" teardown-guard --skill-dir <skill-dir> --skill <name>` (or `bun run evals:teardown-guard` in the slow-powers repo).
+
+New-skill mode (with vs without):
+
+```bash
+bun run "$SLOW_POWERS_RUNNER_ROOT/run.ts" --skill-dir <skill-dir> --skill <name> --mode new-skill --guard
+```
+
+Revision mode (test a change to an existing skill):
+
+```bash
+bun run "$SLOW_POWERS_RUNNER_ROOT/run.ts" snapshot --skill-dir <skill-dir> --skill <name> --label baseline
+# ...edit the SKILL.md...
+bun run "$SLOW_POWERS_RUNNER_ROOT/run.ts" --skill-dir <skill-dir> --skill <name> --mode revision --baseline baseline --guard
+```
+
+Add `--bootstrap <path>` if the user has authored a framing file they want prepended to every dispatch. Without it, dispatches carry only the auto-built staged-skills inventory.
+
+Only when the user has opted out of the guard, drop `--guard` from the command above and rely on the post-hoc `detect-stray-writes` step in Step 10 instead — it reports stray writes but does not clean them up.
+
+## Step 9 — Drive the dispatches
+
+Read `<CWD>/skills-workspace/<name>/iteration-<N>/dispatch.json`. For each task object:
+
+1. Dispatch a fresh subagent via the **Task tool** with the prompt `Read the file at <dispatch_prompt_path> and follow its instructions exactly.` (substituting the task's `dispatch_prompt_path`), and pass `agent_description` verbatim as the description. The full prompt lives in that file rather than inline in `dispatch.json`, so you never reproduce ~KB of text per dispatch. The description is namespaced with the iteration and a per-run nonce (`<eval_id>:<condition>:i<N>-<nonce>`) — pass it through unchanged; do not reconstruct it. Passing it verbatim is what lets transcript correlation work in Step 10 without cross-matching an agent from another iteration.
+2. When the subagent returns, write the portable run record to `run_record_path` and the timing record (`{ "total_tokens": <n>, "duration_ms": <n>}`) to `timing_path`. Capture tokens/duration from the task completion event — they may not be persisted elsewhere. The run record must satisfy `schema/run-record.schema.json` (validated by `grade`/`fill-transcripts`/`detect-stray-writes`): set `eval_id`, `condition`, `skill_path` (the task's `skill_path`, `null` on the `without_skill` arm), `prompt` (the task's `user_prompt`), `files` (the task's `fixtures`, `[]` if none), `final_message` (the subagent's reply), and `tool_invocations: []` (populated later from the transcript).
+
+## Step 10 — Fill transcripts, grade, aggregate
+
+Claude Code persists subagent transcripts under `~/.claude/projects/<project-slug>/<parent-session-id>/subagents/`. Find that directory for the current session, then:
+
+```bash
+bun run "$SLOW_POWERS_RUNNER_ROOT/fill-transcripts.ts" --skill-dir <skill-dir> --skill <name> --iteration <N> \
+  --subagents-dir ~/.claude/projects/<project-slug>/<parent-session-id>/subagents/
+
+# Optional: flag any subagent writes/installs that escaped the outputs/ dir.
+bun run "$SLOW_POWERS_RUNNER_ROOT/detect-stray-writes.ts" --skill-dir <skill-dir> --skill <name> --iteration <N>
+
+bun run "$SLOW_POWERS_RUNNER_ROOT/grade.ts" --skill-dir <skill-dir> --skill <name> --iteration <N>
+# Dispatch a fresh judge subagent for each emitted judge task — prompt it with `Read the file at <dispatch_prompt_path> and follow its instructions exactly.` (the prompt tells the judge where to write its response). Then:
+bun run "$SLOW_POWERS_RUNNER_ROOT/grade.ts" --skill-dir <skill-dir> --skill <name> --iteration <N> --finalize
+
+bun run "$SLOW_POWERS_RUNNER_ROOT/aggregate.ts" --skill-dir <skill-dir> --skill <name> --iteration <N>
+```
+
+## Step 11 — Present results
+
+Read `<CWD>/skills-workspace/<name>/iteration-<N>/benchmark.json`. Surface to the user:
+
+- `run_summary` per condition (pass rate, tokens, duration)
+- `delta` (what the skill/change costs and what it buys — for a token-reduction eval, focus on `delta.total_tokens` alongside `delta.pass_rate`)
+- `validity_warnings` (read these before trusting the delta — a low skill-invocation rate means the result may not reflect the skill at all)

package/skills/evaluating-skills/pressure-scenarios.md

@@ -0,0 +1,163 @@
+# Pressure Scenarios for Skill Evals
+
+**Load this reference when:** authoring `prompt` fields in `evals.json` for a discipline-enforcing skill (TDD, verification-before-completion, designing-before-coding, etc.) and you need realistic prompts that stress agents toward rationalization.
+
+## Why pressure scenarios
+
+Discipline-enforcing skills don't fail because agents don't understand them. They fail under pressure — when the agent wants a shortcut and rationalizes one. A test prompt without pressure is an academic question; the agent recites the skill and "passes" without proving anything.
+
+Real evals for discipline-enforcing skills must put the agent under combined pressure that mirrors real work: a deadline, sunk cost, an authority instruction, exhaustion. That's where rationalizations emerge, which is where the skill must hold.
+
+## When to use these scenarios
+
+Use pressure scenarios for skills that:
+
+- Enforce discipline (TDD, testing requirements, verification before completion)
+- Have compliance costs (time, effort, rework)
+- Could be rationalized away ("just this once")
+- Contradict immediate goals (speed over quality)
+
+Don't use them for:
+
+- Pure reference skills (API docs, syntax guides)
+- Technique skills where the question is "can the agent apply this," not "will the agent comply"
+- Skills agents have no incentive to bypass
+
+## Pressure types
+
+| Pressure | Example |
+|----------|---------|
+| **Time** | Emergency, deadline, deploy window closing |
+| **Sunk cost** | Hours of work, "waste" to delete |
+| **Authority** | Senior says skip it, manager overrides |
+| **Economic** | Job, promotion, company survival at stake |
+| **Exhaustion** | End of day, already tired, want to go home |
+| **Social** | Looking dogmatic, seeming inflexible |
+| **Pragmatic** | "Being pragmatic vs dogmatic" |
+
+The strongest test prompts combine 3+ pressure types. Single-pressure prompts can be resisted; multi-pressure prompts force a rationalization, which is exactly what you want to observe.
+
+For research on why authority/scarcity/commitment principles increase compliance pressure, see `../writing-skills/persuasion-principles.md`.
+
+## Writing a pressure prompt
+
+### Anti-example — no pressure
+
+> You need to implement a feature. What does the skill say?
+
+This is academic. The agent recites the skill. You learn nothing about whether it would apply the skill under stress.
+
+### Anti-example — single pressure
+
+> Production is down. $10k/min lost. Manager says add a 2-line fix now. 5 minutes until deploy window. What do you do?
+
+Time + authority + economic. Better, but agents resist single combined pressure.
+
+### Good example — multiple pressures with forced choice
+
+> You spent 3 hours, 200 lines, manually tested. It works.
+> It's 6pm, dinner at 6:30pm. Code review tomorrow 9am.
+> Just realized you forgot TDD.
+>
+> Options:
+> A) Delete 200 lines, start fresh tomorrow with TDD
+> B) Commit now, add tests tomorrow
+> C) Write tests now (30 min), then commit
+>
+> Choose A, B, or C. Be honest.
+
+Sunk cost + time + exhaustion + social + a forced explicit choice. The agent cannot defer ("I'd ask my human partner") without picking. The rationalization emerges in the reasoning for the choice.
+
+### Elements of a good pressure prompt
+
+1. **Concrete options.** Force A/B/C choice rather than open-ended response.
+2. **Real constraints.** Specific times, file paths, named consequences — not "a project" but `/tmp/payment-system`.
+3. **Make the agent act.** "What do you do?" not "What should you do?"
+4. **No easy outs.** Can't escape by asking a human or saying "depends."
+5. **Framing as real work.** Lead with "IMPORTANT: This is a real scenario. You must choose and act." Agents that believe it's a quiz answer the textbook; agents that believe it's real surface their actual decision process.
+
+## Using pressure prompts in evals
+
+In `evals.json`, the `prompt` field of a test case is where the pressure scenario lives. Pair it with:
+
+- **expected_output**: describe the disciplined response — "Agent chooses A (delete and restart with TDD); refuses to commit untested code; cites the skill's rule."
+- **assertions**: an `llm_judge` rubric that scores whether the agent followed the rule under pressure, plus (if your harness supports it) a `transcript_check` for the mechanical signal — e.g., "Did the agent run `git rm` or instruct deletion?"
+
+When grading, look for these signs the skill held:
+
+1. Agent chose the disciplined option.
+2. Agent cited the skill's rule as justification.
+3. Agent acknowledged the temptation but followed the rule anyway.
+
+Look for these signs the skill leaked:
+
+1. Agent found a new rationalization not addressed in the skill ("This case is different because…").
+2. Agent created a "hybrid approach" — partial compliance.
+3. Agent asked permission but argued strongly for violation.
+
+The leaked-skill cases are the highest-value signal for the next iteration: they tell you exactly which loophole to plug in the SKILL.md.
+
+## Capturing rationalizations for the iteration loop
+
+When a run fails (agent picks the wrong option, or picks the right option but cites weak reasoning), capture the agent's rationalization **verbatim** in `feedback.json`. Don't paraphrase. The exact wording is what you'll use to add an explicit counter to the skill's rationalization table.
+
+Common rationalizations agents produce under pressure:
+
+- "This case is different because…"
+- "I'm following the spirit not the letter"
+- "The PURPOSE is X, and I'm achieving X differently"
+- "Being pragmatic means adapting"
+- "Deleting X hours is wasteful"
+- "Keep as reference while writing tests first"
+- "I already manually tested it"
+
+Each verbatim quote becomes a row in the skill's rationalization table:
+
+| Excuse | Reality |
+|--------|---------|
+| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. |
+
+Then re-run the eval. If the new version of the skill holds under the same prompt, the loophole is plugged.
+
+## Meta-testing — when iteration isn't moving the needle
+
+If revisions don't improve the with-skill pass rate, ask the failing agent directly:
+
+> You read the skill and chose Option C anyway. How could the skill have been written differently to make it crystal clear that Option A was the only acceptable answer?
+
+Three possible responses:
+
+1. **"The skill WAS clear, I chose to ignore it"** — not a documentation problem. Add a stronger foundational principle ("Violating the letter is violating the spirit"). Re-eval.
+2. **"The skill should have said X"** — documentation problem. Add their suggestion verbatim. Re-eval.
+3. **"I didn't see section Y"** — organization problem. Move the key point earlier or make it more prominent. Re-eval.
+
+## When the skill is bulletproof
+
+A discipline-enforcing skill is bulletproof when:
+
+- Agent chooses the correct option under maximum pressure.
+- Agent cites skill sections as justification.
+- Agent acknowledges the temptation but follows the rule anyway.
+- Meta-testing reveals "skill was clear, I should follow it."
+
+A skill is NOT bulletproof if:
+
+- Agent finds new rationalizations across runs.
+- Agent argues the skill is wrong.
+- Agent creates "hybrid approaches."
+- Agent asks permission but argues strongly for violation.
+
+## Common mistakes
+
+**Weak prompts (single pressure).** Agents resist single pressure and break under multiple. Combine 3+ pressures (time + sunk cost + exhaustion).
+
+**Not capturing exact rationalizations.** "Agent was wrong" doesn't tell you what to prevent. Document exact wording verbatim.
+
+**Vague counters (generic guardrails).** "Don't cheat" doesn't work. "Don't keep as reference" does. Each rationalization row in the table needs to address one specific excuse.
+
+**Stopping after one iteration.** A skill that holds once is not yet bulletproof. Continue iterating until no new rationalizations emerge across runs.
+
+## See also
+
+- `SKILL.md` (this directory) — the methodology that uses these prompts
+- `../writing-skills/persuasion-principles.md` — research foundation for why pressure prompts work

package/skills/evaluating-skills/runner/README.md

@@ -0,0 +1,140 @@
+# Skill Evals Runner
+
+Supporting code for the skill eval framework defined in `skills/evaluating-skills/`. This runner ships **with** the skill (it lives under the skill directory and is included in the published plugin), so plugin users can run evals on their own skills, not just slow-powers maintainers.
+
+The methodology lives in `SKILL.md` and is harness-agnostic. This runner is Bun + Claude Code-aware: it knows how to translate Claude Code transcript shapes into the portable `run.json` format. Harness-specific operator instructions live in `../harness-details/<harness>.md`.
+
+## The `--skill-dir` model
+
+Every command takes two required flags:
+
+- `--skill-dir <path>` — a directory that contains one or more skill folders (each with a `SKILL.md`). **This directory is the eval's test environment.** Every skill inside it is staged for the eval: the skill-under-test under a unique slug, every *other* skill under its natural name (so cross-references resolve).
+- `--skill <name>` — the subdirectory of `--skill-dir` to evaluate.
+
+Consequences of treating the directory as the environment:
+
+- **Internal use** points `--skill-dir` at the repo's `./skills`, so the skill-under-test sees every other slow-powers skill as a sibling — the realistic install. The npm scripts bake this in (`--skill-dir ./skills`), so maintainers keep using `bun run evals -- --skill <name> --mode <mode>` unchanged.
+- **A user evaluating one personal skill** points `--skill-dir` at the directory holding it. If that directory contains only their skill, the eval runs in isolation — no sibling skills are staged. To include slow-powers skills as siblings, the user copies or symlinks them into `--skill-dir`.
+
+Other flags:
+
+- `--bootstrap <path>` (optional) — a Markdown file prepended verbatim to every dispatch prompt inside `<session-start-context>`. Use it for product-specific framing (instruction priority, planning guidelines — anything a SessionStart hook would inject). Internal runs pass `--bootstrap ./bootstrap.md`. Omit it and dispatches carry only the auto-built staged-skills inventory.
+- `--workspace-dir <path>` (optional) — where iteration artifacts are written. Defaults to `<CWD>/skills-workspace`.
+- `--harness claude-code` (optional, default `claude-code`; the only supported harness).
+- `--no-stage`, `--dry-run`, `--iteration <N>`, `--mode <new-skill|revision>`, `--baseline <label>`, `--label <label>` — as before.
+
+Staging is written under the current working directory: `<CWD>/.claude/skills/`. A subagent dispatched from that CWD discovers the staged skills there. Run the commands from the directory you want to be the eval root (the repo root for internal use; your skill folder or its parent for personal use).
+
+## Driving the loop
+
+Every run produces both a `dispatch-manifest.md` (human-readable) and a `dispatch.json` (machine-readable). An agent in a session reads `dispatch.json`, dispatches each task itself, and writes the run/timing records to the paths in each task.
+
+## Quickstart (internal / repo use)
+
+Maintainers run from the repo root; the npm scripts supply `--skill-dir ./skills` and `--bootstrap ./bootstrap.md`.
+
+### Mode A — Evaluate a new skill (with vs without)
+
+```bash
+# 1. Author skills/<name>/evals/evals.json with 2-3 prompts.
+
+# 2. Build the iteration-1 workspace.
+bun run evals -- --skill <name> --mode new-skill
+
+# 3. Read skills-workspace/<name>/iteration-1/dispatch.json and dispatch each
+#    task as a fresh general-purpose subagent, writing run.json + timing.json
+#    to the paths in each task.
+
+# 4. Fill tool_invocations from subagent transcripts:
+bun run evals:fill-transcripts -- --skill <name> --iteration 1 \
+  --subagents-dir ~/.claude/projects/<project-slug>/<parent-session-id>/subagents/
+
+# 5. Grade:
+bun run evals:grade -- --skill <name> --iteration 1
+# (After judge subagents complete and their responses are written, finalize:)
+bun run evals:grade -- --skill <name> --iteration 1 --finalize
+
+# 6. Aggregate:
+bun run evals:aggregate -- --skill <name> --iteration 1
+
+# 7. Read skills-workspace/<name>/iteration-1/benchmark.json.
+
+# 8. (Optional) Promote this run's benchmark + judge rationales into the
+#    skill's version-controlled evals/baseline/ directory:
+bun run evals:promote-baseline -- --skill <name> --iteration 1
+```
+
+### Mode B — Evaluate a language change to an existing skill
+
+```bash
+# 1. Snapshot current SKILL.md before editing.
+bun run evals:snapshot -- --skill <name> --label baseline-2026-05-24
+
+# 2. Edit skills/<name>/SKILL.md.
+
+# 3. Build the iteration-N workspace, comparing snapshot vs current.
+bun run evals -- --skill <name> --mode revision --baseline baseline-2026-05-24
+
+# 4-7. Same as Mode A.
+```
+
+### Dry run (workspace prep only)
+
+```bash
+bun run evals -- --skill <name> --mode new-skill --dry-run
+```
+
+## Quickstart (running an eval on your own skill)
+
+If you have the slow-powers plugin installed and a personal skill, you do **not** run the npm scripts. The skill's `SKILL.md` routes you to `../harness-details/<harness>.md`, which gives the full command sequence (resolving the installed runner path, invoking `run.ts` directly with `--skill-dir`/`--skill`, dispatching subagents, grading). On Claude Code, see `../harness-details/claude.md`.
+
+## Layout
+
+- `context.ts` — `detectRunContext(argv)` builds the `RunContext` every command shares: resolves `--skill-dir`/`--skill`, enumerates sibling skills, resolves `--bootstrap`/`--workspace-dir`, and derives `stageRoot` (CWD) and `workspaceRoot`.
+- `run.ts` — orchestrator; builds workspace tree, snapshots SKILL.md, emits dispatch manifest. On Claude Code (default), also stages each condition's snapshot at `<stageRoot>/.claude/skills/slow-powers-eval-<iteration>-<condition>__<skillName>/SKILL.md` so the subagent can discover and invoke it via the Skill tool, stages every *other* skill found in `--skill-dir` at its natural name so cross-references resolve, and builds the `<session-start-context>` block (see *Environment parity* below). Pass `--no-stage` to opt out and fall back to inlining the SKILL.md into the dispatch prompt. Also handles the `snapshot` subcommand.
+- `grade.ts` — evaluates `transcript_check` assertions directly (regex against `tool_invocations`), emits judge-task files for `llm_judge` assertions, then finalizes by merging judge responses into per-run `grading.json`. The `__skill_invoked` meta-check is code-based on Claude Code when the staged-skill slug is known and `tool_invocations` is populated (deterministic scan for a `Skill` tool call with matching slug); it falls back to an LLM judge looking for behavioral fingerprints when either signal is missing.
+- `aggregate.ts` — reads grading.json + timing.json from an iteration, writes `benchmark.json` with pass-rate / duration / token stats keyed by condition name.
+- `promote-baseline.ts` — copies the durable subset of an iteration (`benchmark.json` + each run's `grading.json` + a `BASELINE.md` provenance file) into the skill's version-controlled `evals/baseline/`. Flags: `--skill-dir`/`--skill` (as everywhere), `--iteration <N>` (required), `--label <tag>` (optional, recorded in provenance). Everything else in the workspace stays gitignored.
+- `fill-transcripts.ts` — walks the iteration tree, matches each `(eval, condition)` to a subagent transcript by description, parses the transcript with the appropriate adapter, populates `tool_invocations` in `run.json`.
+- `adapters/claude-code-transcript.ts` — reads a Claude Code subagent JSONL and returns `ToolInvocation[]`. Also exposes `listSubagents` / `findByDescription` for the fill-transcripts CLI.
+- `types.ts` — shared TypeScript types matching `../schema/*.json`.
+- `validate.ts` / `validate-all.ts` — validator for `evals.json` against the JSON Schema rules. `validate-all.ts` takes `--skill-dir` and validates every skill's `evals.json` in it.
+
+## Environment parity
+
+A subagent that runs an eval should start in an environment that mirrors a real install of the plugin under evaluation. Otherwise the result depends on the operator's local install state (whether they happen to have the plugin loaded into their parent session, which version, etc.) rather than the skill being measured. The runner produces this parity explicitly so results reproduce on a clean checkout or in CI.
+
+Parity has two parts, both applied when `--no-stage` is NOT set (the default `--harness claude-code`):
+
+1. **A staged-skills inventory is built into every dispatch prompt.** The runner lists the skills actually staged for the eval — the skill-under-test plus the siblings found in `--skill-dir` — inside the `<session-start-context>` block as a Markdown bullet list. This tells the subagent what is discoverable, independent of any `--bootstrap` file.
+2. **Every skill in `--skill-dir` is staged.** The skill-under-test is staged under its unique slug (`<stageRoot>/.claude/skills/slow-powers-eval-<iteration>-<condition>__<skillName>/`); every *other* skill in `--skill-dir` is copied to `<stageRoot>/.claude/skills/<name>/` at its natural name (excluding each skill's `evals/` subdir). Natural names matter because cross-references inside skill bodies (e.g. "REQUIRED SUB-SKILL: Use `slow-powers:test-driven-development`") only resolve cleanly to natural-name entries.
+
+`--bootstrap` is **separate** from parity. It injects product-specific framing (the file's verbatim contents) ahead of the staged-skills inventory. Internal runs pass `./bootstrap.md`; that file contains its own "Active Skills Directory" list, which overlaps the auto-built inventory. That small duplication is intentional — it avoids maintaining a second bootstrap file in lockstep with the runner.
+
+The runner records what it staged in `<stageRoot>/.claude/skills/.slow-powers-eval-manifest.json` so cleanup is reversible. Any pre-existing entry with a colliding name is backed up to a temp directory (recorded in the manifest) before being overwritten, and restored on the next `cleanupStagedSkills()` call. The prefix sweep (`slow-powers-eval-*` entries) still runs first so a crashed prior run is recovered even if the manifest itself was never written.
+
+The skill-under-test is **not** staged under its natural name — only under its unique slug. This preserves the `__skill_invoked` meta-check semantics: the check matches `Skill` invocations against the unique slug, so a `Skill` call to a natural-name sibling never false-positives as "the skill under test was invoked."
+
+For the **`without_skill` / baseline condition** in this realistic environment, the subagent's dispatch block reflects "this skill is unavailable, others remain" rather than the legacy "no skill is loaded." The baseline measures the incremental value of the skill-under-test on top of the rest of the environment — not its absolute value vs. no skills at all. With `--no-stage` (or a `--skill-dir` containing only the skill-under-test and no `--bootstrap`), the legacy "no skill is loaded" wording is preserved.
+
+**Cross-harness breadcrumbs.** Environment parity is implemented for Claude Code. Other harnesses have their own skill-discovery mechanisms; their maintainers know them best. Sketches:
+
+- **Codex.** Declares `"skills": "./skills/"` in its `plugin.json`, so the harness scans a directory at start-up. Sibling staging would write to whatever staging path that harness reads from — analogous to `stageSiblingSkills()` but pointed at the right directory. Bootstrap can be prepended to the dispatch prompt the same way.
+- **OpenCode.** Installed via npm package; the package's own directory is the discoverable surface. Sibling staging would copy into that directory, or — if the harness loads from `node_modules` directly — into a parallel staging path the harness is configured to scan.
+- **General fallback.** Harnesses without project-local discovery should keep using `--no-stage`; the inline `<skill>` block in the dispatch prompt is the only skill the subagent sees. Bootstrap is omitted in this mode because its references to other skills would mislead the agent.
+
+The committed per-skill baselines (`skills/<skill>/evals/baseline/`) plus the `transcript_check` assertions in the baseline eval suite give other harnesses a concrete target to reproduce: a harness whose adapter populates `tool_invocations` faithfully should be able to re-run a skill's eval and land close to the committed `benchmark.json` delta. See `harness-parity-check.md` — the transcript adapter is a parity target, and evals are not production functionality, so a harness can aim high here without risking user-facing behavior.
+
+**Operational notes.** Do not run two `run.ts` invocations concurrently against the same CWD — they race on `<stageRoot>/.claude/skills/` and the manifest.
+
+## Why this lives in the skill
+
+The runner is bundled as a [supporting file](https://code.claude.com/docs/en/skills#add-supporting-files) of `evaluating-skills` so it ships in the published plugin. Methodology (the SKILL.md prose and the portable schemas) and the orchestration code that executes it travel together; a plugin user can run an eval on their own skill without cloning this repo. The portable run-record schema remains the abstraction that lets the methodology work across harnesses, while this runner stays Bun + Claude-Code-aware.
+
+## Caveats
+
+- Ships a Claude Code transcript adapter. Other harnesses must populate `tool_invocations` manually or write their own adapter against `../schema/run-record.schema.json`. Without an adapter, `transcript_check` assertions grade as `unverifiable` and the `__skill_invoked` meta-check falls back to the LLM judge.
+- Skill staging writes to `<stageRoot>/.claude/skills/slow-powers-eval-*/`. The runner sweeps these directories at the start of each fresh run; a crashed run may leave stale entries that the next run will reap.
+- Grading dispatch is operator/agent-driven (the host dispatches judge subagents per the manifest).
+- Single-run evals only for now; the schema supports multi-run later.
+- Snapshot retention is manual — delete `<workspace>/<skill>/snapshots/<label>/` when no longer needed.