@slowdini/slow-powers-opencode 0.3.0 → 0.4.0

package/skills/evaluating-skills/harness-parity.md

@@ -1,155 +0,0 @@
-# Eval-Runner Harness Parity Check
-
-You are an agent running inside one of Slow-powers's supported harnesses. This file walks you through auditing **which eval-runner features are wired up for your harness** and prepping to close one gap. Claude Code is the reference implementation; other harnesses adapt its patterns using their own native conventions.
-
-This file covers the **skill-eval runner only** — the infrastructure under `skills/evaluating-skills/` that dispatches, records, and grades skill evals. Plugin-distribution parity (manifests, hooks, bootstrap injection, skill discovery) is audited separately by the root-level `harness-parity-check.md`. The eval runner is slated to move into its own project; this doc lives alongside it so it travels with the extraction.
-
-Read the file end-to-end before acting. The categories in Step 4 are the source of truth for what "eval-runner parity" means today — when a new feature is added to the runner, that table is updated and this file stays evergreen.
-
----
-
-## Step 1 — Identify your harness
-
-Name the harness you are running in. You almost certainly already know — confirm by checking:
-
-- Your invocation context and working directory
-- The tool names available to you in this session
-- Any session-start context block injected at the top of the conversation
-- Top-level files or directories matching your harness (e.g. `.<harness>-plugin/`, `<harness>-instructions.md`)
-
-The intended supported harnesses are: **Claude Code, Codex CLI, OpenCode**.
-
-If the harness you are running in is not in that list, stop and ask the user before continuing.
-
----
-
-## Step 2 — Read the reference materials
-
-Read these files in order. Each one teaches you something specific you will need in Step 3. Paths are relative to the repository root.
-
-| File | What to look for |
-|------|------------------|
-| `AGENTS.md` (or `CLAUDE.md`, which symlinks to it) | The Cross-Harness Compatibility rule, the canonical list of supported harnesses, the PR-scoping rule |
-| `skills/evaluating-skills/runner/README.md` | Contains explicit **Cross-harness breadcrumbs** — sketches of how Codex and OpenCode would implement environment parity. Treat these as starting points, not specifications |
-| `skills/evaluating-skills/runner/adapters/claude-code-transcript.ts` | The reference transcript adapter. A second harness would add its own adapter alongside this, translating that harness's transcript shape into the same `ToolInvocation[]` format |
-| `skills/evaluating-skills/harness-details/claude.md` | The reference per-harness operator walkthrough. Other harnesses would each get their own file alongside this |
-
-Do not skim. The parity report you produce in Step 4 is only as good as the reference you internalized here.
-
----
-
-## Step 3 — Discover your harness's existing surface area
-
-Enumerate, using ordinary file search, what already exists in the eval runner for your harness. Do not rely on memory or assumptions — search the working tree. Useful heuristics:
-
-- The harness name anywhere inside `skills/evaluating-skills/runner/` (especially `context.ts`, `adapters/`, `profiles/`)
-- A per-harness operator guide in `skills/evaluating-skills/harness-details/`
-- Tests under `tests/` exercising the runner for the harness
-
-Record every path you find. You will reference them in Step 4.
-
----
-
-## Step 4 — Produce a parity report
-
-For each category below, compare what Claude Code has against what your harness has. Categories are described as "what Claude does (reference)" so they survive renames — when something changes, this row of the table is updated and the rest of the file still applies.
-
-| Category | What Claude Code does (reference) |
-|----------|-----------------------------------|
-| Skill-eval transcript adapter | `skills/evaluating-skills/runner/adapters/claude-code-transcript.ts` |
-| Skill-eval auto-record (run/timing assembly) | `runner/record-runs.ts` assembles each task's `run.json` + `timing.json` from disk after dispatches: carry-over fields from `dispatch.json`, `final_message` from `outputs/final-message.md`, `tool_invocations`/tokens/duration from the persisted transcript (`parseTranscriptFull` — usage deduped by message id). Leans on transcript access, so it's a Claude-Code-tier acceleration like `fill-transcripts`; the portable contract (hand-authored records, `run-record.schema.json`) is unchanged. A harness closes this gap by extending its transcript adapter to supply the same three sources (final message, tool invocations, usage/timing) the recorder consumes |
-| Realistic eval environment (skill staging) | `runner/run.ts` stages skills under `<stageRoot>/.claude/skills/`, wraps any `--bootstrap` content in a `<session-start-context>` block, and emits a separate available-skills block. That block is rendered in the harness's **native** skill-list presentation — Claude Code's lives in `runner/adapters/claude-code-session.ts` (`The following skills are available for use with the Skill tool:` / `- name: description`). Another harness adds its own renderer there so its dispatches read like a real session in that harness, not an eval |
-| Eval subagent write enforcement | Opt-in `--guard` stages a `PreToolUse` hook (`runner/guard/`) that *denies* subagent writes/installs outside the eval sandbox while dispatches run. Portable fallback for every harness: the `evals:detect-stray-writes` post-pass (`runner/detect-stray-writes.ts`) flags out-of-bounds writes from the parsed transcript after the fact |
-| Eval plan-mode operating context | Opt-in `--plan-mode` injects a harness-specific plan-mode procedure profile (`runner/profiles/<harness>/plan-mode.md`) as a `<system-reminder>` operating-context layer in every dispatch, rendered by `renderPlanModeContext` in the harness session adapter (`runner/adapters/claude-code-session.ts`). Only `profiles/claude-code/plan-mode.md` exists today; a harness adds its own profile (its native plan/research-mode procedure) + renderer alongside the Claude ones. A harness with no profile has no `--plan-mode` and an unchanged dispatch contract |
-| Harness-details operator guide | `skills/evaluating-skills/harness-details/claude.md` |
-
-**Note on the transcript adapter (raised bar).** Slow-powers's baseline eval suite
-now uses `transcript_check` assertions — deterministic regex checks against a
-run's tool invocations (e.g. "a test command ran", "the sibling skill was
-loaded"). These only grade when a transcript adapter exists for your harness.
-A harness without one still functions: those assertions grade as *unverifiable*
-and the `llm_judge` assertions carry the substantive measurement, the same way
-Codex/OpenCode work today. But adapter richness is now an explicit parity
-target, not optional polish — a harness that adds or extends an adapter under
-`skills/evaluating-skills/runner/adapters/` lets more of the baseline suite grade
-mechanically. Treat the transcript-adapter row above as a goal to aim at, not a
-box already checked.
-
-**Note on write enforcement (parity goal).** Eval subagents are instructed to
-write only inside their `outputs/` dir, but nothing in the portable contract
-*enforces* it — a misbehaving subagent can edit the real repo or install
-packages, silently tainting the run. Two layers address this: the portable
-`detect-stray-writes` post-pass (available to every harness, since it works off
-the same parsed transcript the adapters already produce) and, on Claude Code, an
-opt-in `--guard` that stages a native `PreToolUse` hook to *block* the write
-before it happens. **Harness-level tool enforcement — denying out-of-bounds
-subagent writes using the harness's own permission/hook primitive — is an
-explicit parity goal, not optional polish.** A harness that can express a
-pre-tool guard (a hook, a permission rule, a sandboxed cwd) should wire one up so
-its eval runs are as self-contained as Claude Code's; until then, the
-`detect-stray-writes` report is the honest fallback. Treat the write-enforcement
-row above as a goal to aim at, with detection as the baseline every harness meets.
-
-**Note on plan-mode fidelity (residual parity goal).** `--plan-mode` injects a
-harness's *verbatim* plan-mode procedure as operating context, which is the
-closest a harness's eval runner can get to reproducing the wild failure where a
-real plan mode makes loading a skill feel redundant. It is **not** the real mode:
-it is still text the dispatched subagent reads, not a state the harness places it
-under, so a pass remains necessary-not-sufficient (see *Seeding conversation
-context (and its ceiling)* in `skills/evaluating-skills/SKILL.md`). A harness that
-can actually dispatch an eval subagent *into* its own plan/research mode — not
-merely describe it — would close this gap; that real-mode injection is the
-residual parity goal, with `--plan-mode` (a profile + renderer) as the approximation
-every harness can reach in the meantime.
-
-Surface your findings inline using this template:
-
-```
-## Eval-Runner Parity Report: <harness>
-Reference: Claude Code
-
-- **Skill-eval transcript adapter** — ✅ Implemented / ⚠️ Partial / ❌ Missing / N/A
-  - Where: <path or "would live at <path>">
-  - Gap: <one sentence, only if Partial/Missing>
-
-(... one block per category ...)
-
-## Summary
-- Strongest area: <category>
-- Highest-leverage gap: <category> — <why>
-- Suggested next gap to close this session: <category>
-```
-
-Status meanings:
-
-- **✅ Implemented** — fully wired up; feature works the same way Claude's does (using whatever native primitive the harness provides)
-- **⚠️ Partial** — some scaffolding exists but the feature isn't end-to-end functional
-- **❌ Missing** — no implementation; users of this harness do not get this feature
-- **N/A** — the category doesn't translate. State why
-
-The agent reports inline by default. If the user asks for a persistent artifact, write the report to `docs/parity-reports/<harness>-evals.md` (create the directory if missing).
-
----
-
-## Step 5 — Pick a gap and prep to close it
-
-Surface the report to the user and propose **one or two** gaps worth closing this session. Bias toward the smallest gap with the highest user impact — typically a transcript adapter or a harness-details guide, not a wholesale runner rework.
-
-Once the user picks a gap:
-
-1. Re-read Claude's reference implementation for that specific feature in detail. Note the *shape* of what it does — inputs, outputs, side effects — separately from the *Claude-specific mechanism* it uses.
-2. **Consult your harness's own documentation, MCP servers, or built-in references** before proposing harness-specific changes. Do not guess at hook schemas, transcript formats, or native tool names. If a `context7` or equivalent docs-fetch server is available, prefer it over your training data — assume your knowledge of the harness may be stale.
-3. Propose an adaptation that copies Claude's shape while using your harness's native conventions. State explicitly what you are copying and what you are adapting.
-4. Confirm with the user before writing code.
-5. If your gap involves creating or modifying a skill, load `slow-powers:writing-skills` first.
-
----
-
-## Guardrails
-
-- **Cross-Harness Compatibility is enforced.** A change for your harness MUST NOT break or degrade any other harness. Re-read the Cross-Harness Compatibility section of `AGENTS.md`.
-- **One problem per PR.** Per `AGENTS.md`, do not bundle unrelated changes. A parity-closing PR should add one feature for one harness.
-- **Do not edit `bootstrap.md` or shared skills as part of parity work.** Those are cross-cutting; changes need their own PRs with their own evidence.
-- **Do not fabricate features that don't exist in any harness yet.** Parity means "catch up to Claude," not "invent something new."
-- **Do not guess at harness-specific details.** If your harness's docs don't confirm something, ask the user before proceeding.
-- **Keep this file evergreen.** If you add a new feature category to the eval runner, add a row to the Step 4 table here and to the eval-runner tier table in `README.md` in the same PR. Distribution-side categories (manifests, hooks, bootstrap, docs) belong in the root `harness-parity-check.md` instead.

package/skills/evaluating-skills/runner/README.md

@@ -1,163 +0,0 @@
-# Skill Evals Runner
-
-Supporting code for the skill eval framework defined in `skills/evaluating-skills/`. This runner ships **with** the skill (it lives under the skill directory and is included in the published plugin), so plugin users can run evals on their own skills, not just slow-powers maintainers.
-
-The methodology lives in `SKILL.md` and is harness-agnostic. This runner is Bun + Claude Code-aware: it knows how to translate Claude Code transcript shapes into the portable `run.json` format. Harness-specific operator instructions live in `../harness-details/<harness>.md`.
-
-## The `--skill-dir` model
-
-Every command takes two required flags:
-
-- `--skill-dir <path>` — a directory that contains one or more skill folders (each with a `SKILL.md`). **This directory is the eval's test environment.** Every skill inside it is staged for the eval: the skill-under-test under a unique slug, every *other* skill under its natural name (so cross-references resolve).
-- `--skill <name>` — the subdirectory of `--skill-dir` to evaluate.
-
-Consequences of treating the directory as the environment:
-
-- **Internal use** points `--skill-dir` at the repo's `./skills`, so the skill-under-test sees every other slow-powers skill as a sibling — the realistic install. The npm scripts bake this in (`--skill-dir ./skills`), so maintainers keep using `bun run evals -- --skill <name> --mode <mode>` unchanged.
-- **A user evaluating one personal skill** points `--skill-dir` at the directory holding it. If that directory contains only their skill, the eval runs in isolation — no sibling skills are staged. To include slow-powers skills as siblings, the user copies or symlinks them into `--skill-dir`.
-
-Other flags:
-
-- `--bootstrap <path>` (optional) — a Markdown file prepended verbatim to every dispatch prompt inside `<session-start-context>`. Use it for product-specific framing (instruction priority, planning guidelines — anything a SessionStart hook would inject). Internal runs pass `--bootstrap ./bootstrap.md`. Omit it and dispatches carry only the auto-built staged-skills inventory.
-- `--workspace-dir <path>` (optional) — where iteration artifacts are written. Defaults to `<CWD>/skills-workspace`.
-- `--harness claude-code` (optional, default `claude-code`; the only supported harness).
-- `--no-stage`, `--dry-run`, `--iteration <N>`, `--mode <new-skill|revision>`, `--baseline <label>`, `--label <label>` — as before.
-- `--ref <git-ref>` (optional, `snapshot` only) — snapshot the skill (SKILL.md + sibling assets, excluding `evals/`) as it existed at a git ref, read straight from git without touching the working tree. Use it for the common edit-first Mode B order: edit the skill, *then* snapshot the old version with `--ref HEAD` (or any commit/tag/branch) as the baseline. Without `--ref`, `snapshot` reads the working tree as before.
-- `--only <id,id,...>` / `--skip <id,id,...>` (optional) — run only, or all-but, the named eval ids from `evals.json`. The two are mutually exclusive, and every named id must exist (the run aborts with the available ids listed otherwise). Use this for a cost-conscious reduced-set run instead of temporarily editing `evals.json` down. The pre-flight summary and the `N evals × 2 conditions` count reflect the filtered set.
-- `--plan-mode` (optional, Claude Code) — inject the harness's verbatim plan-mode procedure as an operating-context layer. When set, the runner reads `profiles/<harness>/plan-mode.md` and emits it (via the session adapter's `renderPlanModeContext`) as a `<system-reminder>` block in every dispatch, after the available-skills block and before the user request. It is identical across the with/without-skill arms and recorded as `plan_mode` in `dispatch.json`. This is issue #142's highest-fidelity in-runner approximation of a real plan mode — still text the agent reads, so a pass is necessary-not-sufficient; see *Seeding conversation context (and its ceiling)* in `../SKILL.md`. Opt-in, and meant only for plan-mode-relevant skills; a harness with no profile aborts the run, leaving the portable dispatch contract unchanged.
-
-Staging is written under the current working directory: `<CWD>/.claude/skills/`. A subagent dispatched from that CWD discovers the staged skills there. Run the commands from the directory you want to be the eval root (the repo root for internal use; your skill folder or its parent for personal use).
-
-## Driving the loop
-
-Every run produces both a `dispatch-manifest.md` (human-readable) and a `dispatch.json` (machine-readable). An agent in a session reads `dispatch.json` and dispatches each task itself. On Claude Code the rest is two fixed-order commands around the judge dispatches — `ingest` (record-runs → fill-transcripts → detect-stray-writes → grade) and `finalize` (grade --finalize → aggregate) — so the whole loop is three runner calls and two dispatch batches. On harnesses without persisted transcripts, the agent writes the records to the paths in each task by hand and runs the chained steps individually (the portable path).
-
-## Quickstart (internal / repo use)
-
-Maintainers run from the repo root; the npm scripts supply `--skill-dir ./skills` and `--bootstrap ./bootstrap.md`.
-
-### Mode A — Evaluate a new skill (with vs without)
-
-```bash
-# 1. Author skills/<name>/evals/evals.json with 2-3 prompts.
-
-# 2. Build the iteration-1 workspace.
-bun run evals -- --skill <name> --mode new-skill
-
-# 3. Read skills-workspace/<name>/iteration-1/dispatch.json and dispatch each
-#    task as a fresh general-purpose subagent (each writes its own
-#    outputs/final-message.md).
-
-# 4. Ingest — record-runs → fill-transcripts → detect-stray-writes → grade,
-#    in fixed order (assembles run.json + timing.json from dispatch.json,
-#    final-message.md, and the persisted transcripts, then emits judge tasks):
-bun run evals:ingest -- --skill <name> --iteration 1 \
-  --subagents-dir ~/.claude/projects/<project-slug>/<parent-session-id>/subagents/
-
-# 5. Dispatch each judge task ingest listed, writing responses to their
-#    response_path.
-
-# 6. Finalize — grade --finalize → aggregate:
-bun run evals:finalize -- --skill <name> --iteration 1
-
-# 7. Read skills-workspace/<name>/iteration-1/benchmark.json.
-
-# 8. (Optional) Promote this run's benchmark + judge rationales into the
-#    skill's version-controlled evals/baseline/ directory:
-bun run evals:promote-baseline -- --skill <name> --iteration 1
-```
-
-### Mode B — Evaluate a language change to an existing skill
-
-The common case is edit-first: you've already changed the skill, then decide to eval.
-Snapshot the *old* version from git — no working-tree dance:
-
-```bash
-# 1. Edit skills/<name>/SKILL.md (the "new" version is now in the working tree).
-
-# 2. Snapshot the old version straight from git as the baseline.
-bun run evals:snapshot -- --skill <name> --label baseline-2026-05-24 --ref HEAD
-
-# 3. Build the iteration-N workspace, comparing baseline (old) vs current (new).
-bun run evals -- --skill <name> --mode revision --baseline baseline-2026-05-24
-
-# 4-7. Same as Mode A.
-```
-
-If you snapshot *before* editing instead, drop `--ref HEAD` from step 2 (it reads the
-working tree) and run it before step 1.
-
-### Dry run (workspace prep only)
-
-```bash
-bun run evals -- --skill <name> --mode new-skill --dry-run
-```
-
-### Reduced-set run (cost-conscious subset)
-
-```bash
-# Run just two of the defined evals, leaving evals.json untouched.
-bun run evals -- --skill <name> --mode new-skill --only case-a,case-b
-# Or run everything except a slow case.
-bun run evals -- --skill <name> --mode new-skill --skip slow-case
-```
-
-## Quickstart (running an eval on your own skill)
-
-If you have the slow-powers plugin installed and a personal skill, you do **not** run the npm scripts. The skill's `SKILL.md` routes you to `../harness-details/<harness>.md`, which gives the full command sequence (resolving the installed runner path, invoking `run.ts` directly with `--skill-dir`/`--skill`, dispatching subagents, grading). On Claude Code, see `../harness-details/claude.md`.
-
-## Layout
-
-- `context.ts` — `detectRunContext(argv)` builds the `RunContext` every command shares: resolves `--skill-dir`/`--skill`, enumerates sibling skills, resolves `--bootstrap`/`--workspace-dir`, and derives `stageRoot` (CWD) and `workspaceRoot`.
-- `run.ts` — orchestrator; builds workspace tree, snapshots SKILL.md, emits dispatch manifest. On Claude Code (default), also stages each condition's snapshot at `<stageRoot>/.claude/skills/slow-powers-eval-<iteration>-<condition>__<skillName>/SKILL.md` so the subagent can discover and invoke it via the Skill tool, stages every *other* skill found in `--skill-dir` at its natural name so cross-references resolve, and builds the `<session-start-context>` block (see *Environment parity* below). Pass `--no-stage` to opt out and fall back to inlining the SKILL.md into the dispatch prompt. Pass `--stage-name <name>` to stage under a verbatim name instead of the eval slug (issue #144 name-confound experiments; single-staging-condition modes only, refuses to clobber an existing dir, registered for next-run cleanup). Also handles the `snapshot`, `ingest`/`finalize` (fixed-order post-dispatch chains over the sibling commands), and `teardown` subcommands.
-- `grade.ts` — evaluates `transcript_check` assertions directly (regex against `tool_invocations`), emits judge-task files for `llm_judge` assertions, then finalizes by merging judge responses into per-run `grading.json`. The `__skill_invoked` meta-check is code-based on Claude Code when the staged-skill slug is known and `tool_invocations` is populated (deterministic scan for a `Skill` tool call with matching slug); it falls back to an LLM judge looking for behavioral fingerprints when either signal is missing.
-- `aggregate.ts` — reads grading.json + timing.json from an iteration, writes `benchmark.json` with pass-rate / duration / token stats keyed by condition name.
-- `promote-baseline.ts` — copies the durable subset of an iteration (`benchmark.json` + each run's `grading.json` + a `BASELINE.md` provenance file) into the skill's version-controlled `evals/baseline/`. Flags: `--skill-dir`/`--skill` (as everywhere), `--iteration <N>` (required), `--label <tag>` (optional, recorded in provenance). Everything else in the workspace stays gitignored.
-- `record-runs.ts` — assembles a schema-valid `run.json` and backfills `timing.json` for every task in a runner-built iteration, from `dispatch.json` (carry-over fields) + `outputs/final-message.md` (`final_message`, transcript fallback) + the persisted transcript (`tool_invocations`, tokens, duration). Never clobbers existing records without `--overwrite`; transcript-derived timing carries `"source": "transcript"`. Claude-Code-tier, like `fill-transcripts` — transcript-less harnesses keep authoring records manually (the portable path).
-- `fill-transcripts.ts` — walks the iteration tree, matches each `(eval, condition)` to a subagent transcript by description, parses the transcript with the appropriate adapter, populates `tool_invocations` in `run.json`. Subsumed by `record-runs` for runner-built iterations; still the tool for filling a pre-existing (hand- or agent-written) `run.json`.
-- `detect-stray-writes.ts` — scans each run's `tool_invocations` for sandbox breaches and writes `stray-writes.json`: write tools targeting paths outside the run's outputs dir (violations), mutating Bash heuristics (warnings), and **live-source reads** — a read tool or Bash command accessing the live skill-under-test directory instead of its staged copy, the signature of the staged-slug resolution race (skills staged mid-session aren't guaranteed resolvable by the Skill tool, whose registry is built at session start; an agent that hits "Unknown skill" improvises and reads the live source, contaminating its arm). `aggregate` lifts all three into `benchmark.json`'s `validity_warnings`.
-- `adapters/claude-code-transcript.ts` — reads a Claude Code subagent JSONL and returns `ToolInvocation[]` (`parseTranscript`), or the full summary with usage tokens deduped by message id, wall-clock duration, and the last assistant text (`parseTranscriptFull`). Also exposes `listSubagents` / `findByDescription` for transcript correlation.
-- `types.ts` — shared TypeScript types matching `../schema/*.json`.
-- `validate.ts` / `validate-all.ts` — validator for `evals.json` against the JSON Schema rules. `validate-all.ts` takes `--skill-dir` and validates every skill's `evals.json` in it.
-
-## Environment parity
-
-A subagent that runs an eval should start in an environment that mirrors a real install of the plugin under evaluation. Otherwise the result depends on the operator's local install state (whether they happen to have the plugin loaded into their parent session, which version, etc.) rather than the skill being measured. The runner produces this parity explicitly so results reproduce on a clean checkout or in CI.
-
-**Caveat — parity is only as clean as the operator's session.** Staging controls what the runner *adds* (the skills below), not what the operator's session already *loaded*. Subagents are dispatched in-process and share the parent session's plugins, so if that session has the plugin-under-evaluation — or any plugin exposing a same-named skill — enabled, the subagent discovers that copy too. That is exactly the "operator's local install state" dependency this section warns against, and the unique staging slug does not prevent it (it stops an on-disk collision, not runtime discovery). The runner can't unload a live plugin; on Claude Code it emits a build-time *plugin-shadow* warning (also surfaced in `benchmark.json`'s `validity_warnings`) so the contamination is visible. Closing it is a launch-time step: run the eval from a plugin-isolated session — see `../harness-details/claude.md` → *Isolating from installed plugins*.
-
-Parity has two parts, both applied when `--no-stage` is NOT set (the default `--harness claude-code`):
-
-1. **An available-skills block is built into every dispatch prompt.** The runner lists the skills actually staged for the eval — the skill-under-test plus the siblings found in `--skill-dir` — as its **own block**, rendered the way the harness surfaces discoverable skills to a real session rather than in an eval-specific format. On Claude Code that is `The following skills are available for use with the Skill tool:` followed by `- name: description` bullets. This rendering is **harness-specific** and lives in `adapters/claude-code-session.ts` (a new harness adds its own renderer alongside it). The block is emitted *after*, and separate from, the `<session-start-context>` block — mirroring how a real session delivers the SessionStart hook and the skill list as two distinct surfaces. It tells the subagent what is discoverable, independent of any `--bootstrap` file.
-2. **Every skill in `--skill-dir` is staged.** The skill-under-test is staged under its unique slug (`<stageRoot>/.claude/skills/slow-powers-eval-<iteration>-<condition>__<skillName>/`); every *other* skill in `--skill-dir` is copied to `<stageRoot>/.claude/skills/<name>/` at its natural name (excluding each skill's `evals/` subdir). Natural names matter because cross-references inside skill bodies (e.g. "REQUIRED SUB-SKILL: Use `slow-powers:test-driven-development`") only resolve cleanly to natural-name entries.
-
-`--bootstrap` is **separate** from parity. It injects product-specific framing (the file's verbatim contents) inside the `<session-start-context>` block, ahead of the available-skills block. Internal runs pass `./bootstrap.md`. That file does **not** enumerate skills — the available-skills block is the single source of the skill list, so there is no duplication to keep in lockstep. (A *user-supplied* `--bootstrap` that does enumerate skills is handled defensively by `redactSkillFromBootstrap`, which strips the skill-under-test from the bootstrap prose on the `without_skill` arm so it can't leak into the control condition.)
-
-The runner records what it staged in `<stageRoot>/.claude/skills/.slow-powers-eval-manifest.json` so cleanup is reversible. Any pre-existing entry with a colliding name is backed up to a temp directory (recorded in the manifest) before being overwritten, and restored on the next `cleanupStagedSkills()` call. The prefix sweep (`slow-powers-eval-*` entries) still runs first so a crashed prior run is recovered even if the manifest itself was never written.
-
-The skill-under-test is **not** staged under its natural name — only under its unique slug. This preserves the `__skill_invoked` meta-check semantics: the check matches `Skill` invocations against the unique slug, so a `Skill` call to a natural-name sibling never false-positives as "the skill under test was invoked."
-
-For the **`without_skill` / baseline condition** in this realistic environment, the subagent's dispatch block reflects "this skill is unavailable, others remain" rather than the legacy "no skill is loaded." The baseline measures the incremental value of the skill-under-test on top of the rest of the environment — not its absolute value vs. no skills at all. With `--no-stage` (or a `--skill-dir` containing only the skill-under-test and no `--bootstrap`), the legacy "no skill is loaded" wording is preserved.
-
-**Cross-harness breadcrumbs.** Environment parity is implemented for Claude Code. Other harnesses have their own skill-discovery mechanisms; their maintainers know them best. Sketches:
-
-- **Codex.** Declares `"skills": "./skills/"` in its `plugin.json`, so the harness scans a directory at start-up. Sibling staging would write to whatever staging path that harness reads from — analogous to `stageSiblingSkills()` but pointed at the right directory. Bootstrap can be prepended to the dispatch prompt the same way.
-- **OpenCode.** Installed via npm package; the package's own directory is the discoverable surface. Sibling staging would copy into that directory, or — if the harness loads from `node_modules` directly — into a parallel staging path the harness is configured to scan.
-- **General fallback.** Harnesses without project-local discovery should keep using `--no-stage`; the inline `<skill>` block in the dispatch prompt is the only skill the subagent sees. Bootstrap is omitted in this mode because its references to other skills would mislead the agent.
-- **Plan-mode profiles (`--plan-mode`).** The plan-mode operating-context layer is also a harness-specific surface. The procedure text lives in `profiles/<harness>/plan-mode.md` and is wrapped by a `renderPlanModeContext` in that harness's session adapter (`adapters/<harness>-session.ts`), exactly mirroring how `renderAvailableSkillsBlock` is harness-specific. Only `profiles/claude-code/plan-mode.md` exists today; a harness that wants this fidelity layer adds its own profile file (its native plan/research mode procedure) plus a renderer alongside the Claude ones. A harness with no profile simply has no `--plan-mode`, and the portable dispatch contract is unchanged.
-
-The committed per-skill baselines (`skills/<skill>/evals/baseline/`) plus the `transcript_check` assertions in the baseline eval suite give other harnesses a concrete target to reproduce: a harness whose adapter populates `tool_invocations` faithfully should be able to re-run a skill's eval and land close to the committed `benchmark.json` delta. See `../harness-parity.md` — the transcript adapter is a parity target, and evals are not production functionality, so a harness can aim high here without risking user-facing behavior.
-
-**Operational notes.** Do not run two `run.ts` invocations concurrently against the same CWD — they race on `<stageRoot>/.claude/skills/` and the manifest.
-
-## Why this lives in the skill
-
-The runner is bundled as a [supporting file](https://code.claude.com/docs/en/skills#add-supporting-files) of `evaluating-skills` so it ships in the published plugin. Methodology (the SKILL.md prose and the portable schemas) and the orchestration code that executes it travel together; a plugin user can run an eval on their own skill without cloning this repo. The portable run-record schema remains the abstraction that lets the methodology work across harnesses, while this runner stays Bun + Claude-Code-aware.
-
-## Caveats
-
-- Ships a Claude Code transcript adapter. Other harnesses must populate `tool_invocations` manually or write their own adapter against `../schema/run-record.schema.json`. Without an adapter, `transcript_check` assertions grade as `unverifiable` and the `__skill_invoked` meta-check falls back to the LLM judge.
-- Skill staging writes to `<stageRoot>/.claude/skills/slow-powers-eval-*/`. The runner sweeps these directories at the start of each fresh run; a crashed run may leave stale entries that the next run will reap.
-- Grading dispatch is operator/agent-driven (the host dispatches judge subagents per the manifest).
-- Single-run evals only for now; the schema supports multi-run later.
-- Snapshot retention is manual — delete `<workspace>/<skill>/snapshots/<label>/` when no longer needed.

package/skills/evaluating-skills/runner/adapters/claude-code-session.test.ts

@@ -1,56 +0,0 @@
-import { describe, expect, test } from "bun:test";
-import type { AvailableSkill } from "../types";
-import {
-  renderAvailableSkillsBlock,
-  renderPlanModeContext,
-} from "./claude-code-session";
-
-const skill = (name: string, description: string): AvailableSkill => ({
-  name,
-  path: `/x/${name}/SKILL.md`,
-  description,
-});
-
-describe("renderAvailableSkillsBlock", () => {
-  test("uses the harness-native header and one `- name: description` bullet per skill", () => {
-    const block = renderAvailableSkillsBlock([skill("foo", "the foo skill")]);
-    expect(block).toContain(
-      "The following skills are available for use with the Skill tool:",
-    );
-    expect(block).toContain("- foo: the foo skill");
-    // The eval-flavored wording and custom format must be gone.
-    expect(block).not.toContain("staged and discoverable");
-    expect(block).not.toContain("*Trigger:*");
-  });
-
-  test("sorts skills by name", () => {
-    const block = renderAvailableSkillsBlock([
-      skill("zebra", "z"),
-      skill("alpha", "a"),
-    ]);
-    expect(block.indexOf("- alpha:")).toBeLessThan(block.indexOf("- zebra:"));
-  });
-
-  test("returns an empty string for an empty list", () => {
-    expect(renderAvailableSkillsBlock([])).toBe("");
-  });
-});
-
-describe("renderPlanModeContext", () => {
-  test("wraps the profile text in a harness-native system-reminder block", () => {
-    const block = renderPlanModeContext("Plan mode is active. Do not edit.");
-    expect(block).toContain("<system-reminder>");
-    expect(block).toContain("</system-reminder>");
-    expect(block).toContain("Plan mode is active. Do not edit.");
-  });
-
-  test("trims surrounding whitespace from the profile text", () => {
-    const block = renderPlanModeContext("\n\n  PROFILE-BODY  \n\n");
-    expect(block).toBe("<system-reminder>\nPROFILE-BODY\n</system-reminder>");
-  });
-
-  test("returns an empty string for empty or whitespace-only input", () => {
-    expect(renderPlanModeContext("")).toBe("");
-    expect(renderPlanModeContext("   \n  ")).toBe("");
-  });
-});

package/skills/evaluating-skills/runner/adapters/claude-code-session.ts

@@ -1,43 +0,0 @@
-// Claude Code-specific rendering of session-start context.
-//
-// The available-skills reminder is a *harness-specific* surface: Claude Code
-// presents discoverable skills to an agent as "The following skills are
-// available for use with the Skill tool:" followed by `- name: description`
-// bullets. Other harnesses (Codex, OpenCode) surface their skills differently,
-// so this rendering lives in an adapter rather than inline in the harness-
-// agnostic orchestrator. A new harness adds its own renderer alongside this one
-// (see ../../harness-parity.md).
-
-import type { AvailableSkill } from "../types";
-
-/**
- * Render the list of discoverable skills the way a real Claude Code session
- * surfaces them, so an eval dispatch mirrors a genuine session rather than
- * announcing itself as an eval. Returns an empty string when no skills are
- * staged (the caller omits the block entirely in that case).
- */
-export function renderAvailableSkillsBlock(skills: AvailableSkill[]): string {
-  if (skills.length === 0) return "";
-  const sorted = [...skills].sort((a, b) => a.name.localeCompare(b.name));
-  const lines = sorted.map((s) => `- ${s.name}: ${s.description}`);
-  return [
-    "The following skills are available for use with the Skill tool:",
-    "",
-    ...lines,
-  ].join("\n");
-}
-
-/**
- * Render a plan-mode profile the way Claude Code injects an operating mode into
- * a live session: as a `<system-reminder>` block the agent is told it is
- * operating under, not prose it merely reads. The profile text (the verbatim
- * plan-mode procedure) lives in `../profiles/claude-code/plan-mode.md`; this
- * adapter owns only the harness-native framing, so a new harness adds its own
- * renderer + profile alongside this one (see ../../harness-parity.md). Returns
- * an empty string for empty input so the caller can omit the section entirely.
- */
-export function renderPlanModeContext(profileText: string): string {
-  const trimmed = profileText.trim();
-  if (!trimmed) return "";
-  return ["<system-reminder>", trimmed, "</system-reminder>"].join("\n");
-}