@givolt/i3-composer-agent 0.0.0-stage → 0.6.12

package/docs/opencode-setup.md

@@ -0,0 +1,259 @@
+# OpenCode Setup
+
+This guide explains how to use i3 Composer Agent with OpenCode. The reusable assets are the markdown skills in the `skills/` directory; the root `AGENTS.md` file in this repository is repo-scoped and should not be copied into other projects.
+
+## Overview
+
+OpenCode discovers skills from several locations. i3 Composer Agent provides two optional usage styles:
+
+- **Agent-driven workflow:** skills are selected automatically via the built-in `skill` tool and a project-local `AGENTS.md` that you write for your own repository.
+- **Command-driven workflow:** manually invoke lifecycle commands with `.opencode/commands/` (optional).
+
+## Installation
+
+There are two ways to get the skills into your project:
+
+1. Install with the `skills` CLI (fastest).
+2. Prepare the plugin bundle and copy the skill directories manually.
+
+After either step, create your own project-local `AGENTS.md` and, if you want them, copy the `.opencode/commands/*.md` files.
+
+### Option 1: Install with `npx skills`
+
+The fastest path is the open [`skills` CLI](https://github.com/vercel-labs/skills):
+
+```bash
+npx skills add /path/to/i3-composer-agent            # install selected skills
+npx skills add /path/to/i3-composer-agent --list     # browse before installing
+```
+
+Install a single skill:
+
+```bash
+npx skills add /path/to/i3-composer-agent --skill spec-driven-development
+```
+
+By default `npx skills` installs into a tool-specific directory (often `.claude/skills/` or a shared location). OpenCode will discover skills placed there because it reads `.claude/skills/<name>/SKILL.md` and the generic `.agents/skills/<name>/SKILL.md` paths.
+
+If the skills land somewhere OpenCode does not scan, copy or symlink them into one of the discovery paths listed below, for example:
+
+```bash
+mkdir -p .opencode/skills
+cp -r .claude/skills/<skill-name> .opencode/skills/
+```
+
+> **Note:** Per-skill installs copy only the skill directory itself. If a skill references shared files under `references/`, copy those into the installed skill directory or install the whole pack. See [upstream agent-skills #361](https://github.com/addyosmani/agent-skills/issues/361) for background.
+
+### Option 2: Prepare the plugin bundle
+
+1. Prepare the plugin bundle:
+
+```bash
+npx @givolt/i3-composer-agent --extract --dir ./i3-composer-agent
+```
+
+2. Copy the desired skills into one of the OpenCode skill discovery paths.
+
+#### Project-local installation
+
+```bash
+mkdir -p .opencode/skills
+cp -r /path/to/i3-composer-agent/skills/<skill-name> .opencode/skills/
+```
+
+For example, to install `spec-driven-development` and `incremental-implementation`:
+
+```bash
+mkdir -p .opencode/skills
+cp -r /path/to/i3-composer-agent/skills/spec-driven-development .opencode/skills/
+cp -r /path/to/i3-composer-agent/skills/incremental-implementation .opencode/skills/
+```
+
+#### Global installation
+
+```bash
+mkdir -p ~/.config/opencode/skills
+cp -r /path/to/i3-composer-agent/skills/<skill-name> ~/.config/opencode/skills/
+```
+
+#### Cross-compatible paths
+
+OpenCode also discovers skills placed in Claude-compatible or generic agent paths:
+
+- `.claude/skills/<name>/SKILL.md`
+- `~/.claude/skills/<name>/SKILL.md`
+- `.agents/skills/<name>/SKILL.md`
+- `~/.agents/skills/<name>/SKILL.md`
+
+If you already share skills across Claude Code and OpenCode, any of these locations work.
+
+### What to copy
+
+Copy the directories under `skills/` (for example `skills/spec-driven-development/`). Each directory must contain a `SKILL.md` file. Do not copy the repository's root `AGENTS.md` or `CLAUDE.md`; those files configure development of this repository itself.
+
+## Project `AGENTS.md`
+
+Create an `AGENTS.md` in **your own project** root. This is the system prompt that tells OpenCode when and how to invoke the installed skills. Unlike the repo-scoped `AGENTS.md` in `addyosmani/agent-skills`, this file belongs to your project and should be adapted to your stack.
+
+Below is a template you can paste into your project's `AGENTS.md`:
+
+```markdown
+# i3 Composer Agent (OpenCode)
+
+This project uses skills installed under `.opencode/skills/` (or a compatible path).
+
+## Core Rules
+
+- If a task matches a skill, invoke it with the `skill` tool before acting.
+- Skills are located in `.opencode/skills/<skill-name>/SKILL.md`.
+- Follow the skill workflow strictly; do not partially apply it.
+- Never skip required steps such as spec, plan, or test when a skill demands them.
+
+## Intent → Skill Mapping
+
+Map the user's intent to the matching skill automatically:
+
+- Feature / new functionality → `spec-driven-development`, then `incremental-implementation` and `test-driven-development`
+- Planning / breakdown → `planning-and-task-breakdown`
+- Bug / failure / unexpected behavior → `debugging-and-error-recovery`
+- Code review → `code-review-and-quality`
+- Refactoring / simplification → `code-simplification`
+- API or interface design → `api-and-interface-design`
+- UI work → `frontend-ui-engineering`
+
+## Execution Model
+
+For every request:
+
+1. Determine if any skill applies (even a small chance).
+2. Load the skill with `skill({ name: "<skill-name>" })`.
+3. Follow the skill workflow exactly.
+4. Only proceed to implementation once required steps are complete.
+```
+
+Save this as `AGENTS.md` in your project root. OpenCode will load it automatically.
+
+> **Note:** The root `AGENTS.md` inside the `addyosmani/agent-skills` repository is intended for contributors working on this repository and should not be copied into other projects. See [CONTRIBUTING.md](../CONTRIBUTING.md#repo-scoped-files).
+
+## How It Works
+
+### 1. Skill Discovery
+
+OpenCode walks the following paths (project-local first, then global):
+
+- `.opencode/skills/<name>/SKILL.md`
+- `~/.config/opencode/skills/<name>/SKILL.md`
+- `.claude/skills/<name>/SKILL.md`
+- `~/.claude/skills/<name>/SKILL.md`
+- `.agents/skills/<name>/SKILL.md`
+- `~/.agents/skills/<name>/SKILL.md`
+
+Each skill must contain a `SKILL.md` file with a valid `name` and `description` in its frontmatter.
+
+### 2. Automatic Skill Invocation
+
+When your project's `AGENTS.md` instructs the agent to use skills, the agent evaluates every request and maps it to the appropriate skill.
+
+Examples:
+
+- "build a feature" → `incremental-implementation` + `test-driven-development`
+- "design a system" → `spec-driven-development`
+- "fix a bug" → `debugging-and-error-recovery`
+- "review this code" → `code-review-and-quality`
+
+### 3. Lifecycle Mapping (Implicit Commands)
+
+OpenCode does not require slash commands, but if you prefer them see the next section. In agent-driven mode the lifecycle is mapped implicitly:
+
+- DEFINE → `spec-driven-development`
+- PLAN → `planning-and-task-breakdown`
+- BUILD → `incremental-implementation` + `test-driven-development`
+- VERIFY → `debugging-and-error-recovery`
+- REVIEW → `code-review-and-quality`
+- SHIP → `shipping-and-launch`
+
+### Copy the optional slash commands
+
+If you prefer explicit commands, copy the example command files from this repository into your project and adjust them to invoke the skills you installed:
+
+```bash
+mkdir -p .opencode/commands
+cp /path/to/i3-composer-agent/.opencode/commands/*.md .opencode/commands/
+```
+
+> **Note:** The repository currently does not include `.opencode/commands/*.md` on `main`. You can create your own command files or watch PR #200, which proposes adding them. Once they exist, the pattern above applies.
+
+A typical command file looks like:
+
+```markdown
+---
+description: Break work into small verifiable tasks
+---
+
+Invoke the planning-and-task-breakdown skill. Read the spec and create tasks with acceptance criteria.
+```
+
+Save it as `.opencode/commands/plan.md` to enable `/plan` in OpenCode.
+
+## Usage Examples
+
+### Example 1: Feature Development
+
+User:
+```
+Add authentication to this app
+```
+
+Agent behavior:
+- Detects feature work
+- Invokes `spec-driven-development`
+- Produces a spec before writing code
+- Moves to planning and implementation skills
+
+### Example 2: Bug Fix
+
+User:
+```
+This endpoint is returning 500 errors
+```
+
+Agent behavior:
+- Invokes `debugging-and-error-recovery`
+- Reproduces → localizes → fixes → adds guards
+
+### Example 3: Code Review
+
+User:
+```
+Review this PR
+```
+
+Agent behavior:
+- Invokes `code-review-and-quality`
+- Applies structured review (correctness, design, readability, etc.)
+
+## Agent Expectations
+
+For OpenCode to work correctly, the agent should:
+
+- Always check if a skill applies before acting
+- Use the `skill` tool to load the skill when it applies
+- Never skip required workflows (spec, plan, test, etc.)
+- Not jump directly to implementation
+
+These rules are enforced by your project's `AGENTS.md`, not by the copy of the skill itself.
+
+## Limitations
+
+- Skill invocation depends on model compliance.
+- OpenCode does not install skills automatically; copy or install the directories you need.
+- If a skill references files under `references/`, you may need to copy those as well when installing manually.
+
+## Summary
+
+1. Install the skills you need, either with `npx skills add /path/to/i3-composer-agent` or by copying them from a clone of this repository into `.opencode/skills/` (project), `~/.config/opencode/skills/` (global), or a cross-compatible path such as `.claude/skills/` / `.agents/skills/`.
+2. Create your own project-local `AGENTS.md` with the rules and intent mapping above.
+3. OpenCode discovers the skills and your `AGENTS.md` guides the agent to invoke them.
+4. Optionally add `.opencode/commands/*.md` for explicit slash commands.
+
+This keeps the reusable assets (skills) separate from the repository-specific configuration (the `addyosmani/agent-skills` root `AGENTS.md`).

package/docs/other-hosts.md

@@ -0,0 +1,7 @@
+# Other hosts
+
+These hosts install the pack but are not in the README's install section, because no maintainer has run the current release on them. Each entry is one line: the host and its install command. To add one, see [Adding a Host Guide](../CONTRIBUTING.md#adding-a-host-guide).
+
+Skills are plain `SKILL.md` folders, so any agent that reads the i3 Composer Agent layout can load them. See [getting-started.md](getting-started.md).
+
+- **Oh My Pi**: `omp plugin marketplace add /path/to/i3-composer-agent`, then `omp plugin install i3-composer-agent@i3-composer-agent`

package/docs/skill-anatomy.md

@@ -0,0 +1,203 @@
+# Skill Anatomy
+
+This document describes the structure and format of i3-composer-agent skill files. Use this as a guide when contributing new skills or understanding existing ones.
+
+## File Location
+
+Every skill lives in its own directory under `skills/`:
+
+```
+skills/
+  skill-name/
+    SKILL.md           # Required: The skill definition
+    scripts/           # Optional: Runnable helpers used by the skill workflow
+    references/        # Optional: Skill-specific reference documentation
+    supporting-file.md # Optional: Reference material loaded on demand
+```
+
+`SKILL.md` is the only required file. Add `scripts/` or `references/` only when the skill actually needs them, and omit them entirely for simpler skills.
+
+## SKILL.md Format
+
+### Frontmatter (Required)
+
+```yaml
+---
+name: skill-name-with-hyphens
+description: Guides agents through [task/workflow]. Use when [specific trigger conditions].
+---
+```
+
+**Rules:**
+- `name`: Lowercase, hyphen-separated. Must match the directory name.
+- `description`: Start with what the skill does in third person, then include one or more clear "Use when" trigger conditions. Include both *what* and *when*. Maximum 1024 characters.
+- Top-level keys are limited to the [Agent Skills specification](https://agentskills.io) fields: `name`, `description`, `license`, `compatibility`, `metadata`, and `allowed-tools`. Vendor or runtime controls such as `model`, `tools`, `max_turns`, or `context` go under `metadata` or in a per-agent adapter file; see [docs/advanced-per-agent-configuration.md](advanced-per-agent-configuration.md). The validator rejects any other top-level key.
+
+Published names are compatibility identifiers. In particular,
+`browser-testing-with-devtools` is the stable upstream name because other
+skills refer to it directly. A downstream catalog may rename it, but that
+catalog owns the alias or migration mapping; downstream-only aliases are not
+maintained in this repository.
+
+**Why this matters:** Agents discover skills by reading descriptions. The description is injected into the system prompt, so it must tell the agent both what the skill provides and when to activate it. Do not summarize the workflow — if the description contains process steps, the agent may follow the summary instead of reading the full skill.
+
+### Standard Sections (Recommended Pattern)
+
+The frontmatter contract above is required. The section layout below is a recommended pattern, not a rigid template: equivalent headings are acceptable when they serve the same purpose clearly.
+
+```markdown
+# Skill Title
+
+## Overview
+One-two sentences explaining what this skill does and why it matters.
+
+## When to Use
+- Bullet list of triggering conditions (symptoms, task types)
+- When NOT to use (exclusions)
+
+## [Core Process / The Workflow / Steps]
+The main workflow, broken into numbered steps or phases.
+Include code examples where they help.
+Use flowcharts (ASCII) where decision points exist.
+
+## [Specific Techniques / Patterns]
+Detailed guidance for specific scenarios.
+Code examples, templates, configuration.
+
+## Common Rationalizations
+| Rationalization | Reality |
+|---|---|
+| Excuse agents use to skip steps | Why the excuse is wrong |
+
+## Red Flags
+- Behavioral patterns indicating the skill is being violated
+- Things to watch for during review
+
+## Verification
+After completing the skill's process, confirm:
+- [ ] Checklist of exit criteria
+- [ ] Evidence requirements
+```
+
+## Section Purposes
+
+### Overview
+The "elevator pitch" for the skill. Should answer: What does this skill do, and why should an agent follow it?
+
+### When to Use
+Helps agents and humans decide if this skill applies to the current task. Include both positive triggers ("Use when X") and negative exclusions ("NOT for Y").
+
+### Core Process
+The heart of the skill. This is the step-by-step workflow the agent follows. Must be specific and actionable — not vague advice.
+
+**Good:** "Run `npm test` and verify all tests pass"
+**Bad:** "Make sure the tests work"
+
+### Common Rationalizations
+The most distinctive feature of well-crafted skills. These are excuses agents use to skip important steps, paired with rebuttals. They prevent the agent from rationalizing its way out of following the process.
+
+Think of every time an agent has said "I'll add tests later" or "This is simple enough to skip the spec" — those go here with a factual counter-argument.
+
+### Red Flags
+Observable signs that the skill is being violated. Useful during code review and self-monitoring.
+
+### Verification
+The exit criteria. A checklist the agent uses to confirm the skill's process is complete. Every checkbox should be verifiable with evidence (test output, build result, screenshot, etc.).
+
+## Supporting Files
+
+Create supporting files only when:
+- Reference material exceeds 100 lines (keep the main SKILL.md focused)
+- Code tools or scripts are needed
+- Checklists are long enough to justify separate files
+
+Keep patterns and principles inline when under 50 lines.
+
+If a skill does not need runnable helpers, do not create an empty `scripts/` directory just to mirror other skills. Empty directories add noise without changing how the skill works.
+
+## Shared References
+
+Checklists used by more than one skill — testing, security, performance, accessibility, definition-of-done — live in `references/` at the repository root, deliberately *not* inside any skill directory.
+
+This is a pack-level design choice. The Agent Skills spec describes a skill as a self-contained directory, but several skills here point at the same checklists. Colocating those would force one of two options: copy the checklist into every skill that uses it, or pick one skill to "own" it and have the others reach into that directory. Both drift over time. A single repo-root copy stays the source of truth.
+
+The tradeoff is portability: a whole-repo install (such as the Claude Code marketplace plugin) carries `references/` along, but a per-skill install that copies only `skills/<name>/` leaves the repo-root sibling behind, and those links resolve to nothing. That gap is tracked in [upstream agent-skills #361](https://github.com/addyosmani/agent-skills/issues/361).
+
+Current convention: material used by exactly one skill is a supporting file inside that skill's directory; material shared across skills goes in `references/`.
+
+## Context Efficiency
+
+Skills load on demand: only the skill name and description sit in context at startup. The full `SKILL.md` loads only when an agent decides the skill is relevant. To keep that load cheap:
+
+- **Keep `SKILL.md` under 500 lines.** Move detailed reference material into supporting files.
+- **Write specific descriptions.** A precise description helps the agent activate the skill at the right moment and skip it otherwise.
+- **Use progressive disclosure.** Reference supporting files that are read only when the workflow reaches them.
+- **Prefer scripts over inline code.** Executing a script consumes no context; only its output does. Inline code blocks are paid for on every load.
+- **Keep file references one level deep.** Link directly from `SKILL.md` to supporting files rather than chaining through intermediate documents.
+
+## Script Requirements
+
+When a skill ships runnable helpers under `scripts/`, each script follows these conventions:
+
+- Use a `#!/bin/bash` shebang.
+- Use `set -e` for fail-fast behavior.
+- Write status messages to stderr: `echo "Message" >&2`.
+- Write machine-readable output (JSON) to stdout.
+- Include a cleanup trap for temporary files.
+- Reference the script path as `skills/<skill-name>/scripts/<script>.sh` (repo-relative).
+
+## Writing Principles
+
+1. **Process over knowledge.** Skills are workflows, not reference docs. Steps, not facts.
+2. **Specific over general.** "Run `npm test`" beats "verify the tests".
+3. **Evidence over assumption.** Every verification checkbox requires proof.
+4. **Anti-rationalization.** Every skip-worthy step needs a counter-argument in the rationalizations table.
+5. **Progressive disclosure.** Main SKILL.md is the entry point. Supporting files are loaded only when needed.
+6. **Token-conscious.** Every section must justify its inclusion. If removing it wouldn't change agent behavior, remove it.
+7. **Model-neutral.** Write the procedure, not the workaround. See [Write the Procedure, Not the Workaround](#write-the-procedure-not-the-workaround).
+
+## Write the Procedure, Not the Workaround
+
+Skills in this pack run on many agents and model generations. A step that exists because one model gets a specific call wrong is a liability on every other model: it constrains agents that would have solved the task directly, and it spends turns that stronger models need for the actual work. Cross-model transfer measurements ([WikiSkill, arXiv:2608.27454](https://arxiv.org/abs/2608.27454)) show skills authored against one model's execution failures can leave a stronger model performing *worse than with no skill at all*.
+
+**The rule:** if a step cannot be justified without naming a model, a model version, or one agent's private tool name, it belongs in an issue, not in a skill. Describe the capability ("run the focused test command", "write the file"), not the mechanism one runtime happens to expose (`run_command`, `write_to_file`).
+
+Two smells to check for before proposing skill content:
+
+- **Over-specified mechanics.** Prescribing an exact command form or serialization detail where stating the goal alone would do. If the justification is "model X gets this wrong otherwise", it's a workaround, not a procedure.
+- **Fragmented diagnostics.** A multi-step inspection sequence where one step would confirm the same thing. Every extra step spends turns, and turn budget is finite.
+
+This is about skill *content*; the portability of `references/` *paths* is a separate concern (see [Shared References](#shared-references)).
+
+## Naming Conventions
+
+- Skill directories: `lowercase-hyphen-separated`
+- Skill files: `SKILL.md` (always uppercase)
+- Supporting files: `lowercase-hyphen-separated.md`
+- Shared references: stored in the root `references/` directory, not inside skill directories (see [Shared References](#shared-references) for why).
+- Skill-specific references: a single supporting doc can stay as a loose file in the skill directory (the `Supporting files` entry above); when several related docs travel with the skill, the emerging convention for self-contained, distributable skills is to group them in a `references/` directory inside the skill directory, so the skill carries its own supporting docs.
+
+## Cross-Skill References
+
+Reference other skills by name:
+
+```markdown
+Follow the `test-driven-development` skill for writing tests.
+If the build breaks, use the `debugging-and-error-recovery` skill.
+```
+
+Don't duplicate content between skills — reference and link instead.
+
+## Required vs Recommended
+
+Required:
+
+- A `skills/<skill-name>/SKILL.md` file
+- Valid YAML frontmatter with `name` and `description`
+- A description that includes both what the skill does and when to use it
+
+Recommended:
+
+- The standard section flow shown above
+- Equivalent headings such as `How It Works`, `Core Process`, or `Workflow` when they read more naturally for the skill
+- Supporting files only when they keep the main `SKILL.md` focused

package/docs/windsurf-setup.md

@@ -0,0 +1,48 @@
+# Using i3-composer-agent with Windsurf
+
+## Setup
+
+### Project Rules
+
+Windsurf uses `.windsurfrules` for project-specific agent instructions:
+
+```bash
+# Create a combined rules file from your most important skills
+cat /path/to/i3-composer-agent/skills/test-driven-development/SKILL.md > .windsurfrules
+echo "\n---\n" >> .windsurfrules
+cat /path/to/i3-composer-agent/skills/incremental-implementation/SKILL.md >> .windsurfrules
+echo "\n---\n" >> .windsurfrules
+cat /path/to/i3-composer-agent/skills/code-review-and-quality/SKILL.md >> .windsurfrules
+```
+
+### Global Rules
+
+For skills you want across all projects, add them to Windsurf's global rules:
+
+1. Open Windsurf → Settings → AI → Global Rules
+2. Paste the content of your most-used skills
+
+## Recommended Configuration
+
+Keep `.windsurfrules` focused on 2-3 essential skills to stay within context limits:
+
+```
+# .windsurfrules
+# Essential i3-composer-agent for this project
+
+[Paste test-driven-development SKILL.md]
+
+---
+
+[Paste incremental-implementation SKILL.md]
+
+---
+
+[Paste code-review-and-quality SKILL.md]
+```
+
+## Usage Tips
+
+1. **Be selective** — Windsurf's context is limited. Choose skills that address your biggest quality gaps.
+2. **Reference in conversation** — Paste additional skill content into the chat when working on specific phases (e.g., paste `security-and-hardening` when building auth).
+3. **Use references as checklists** — Paste `references/security-checklist.md` and ask Windsurf to verify each item.

package/evals/README.md

@@ -0,0 +1,101 @@
+# Skill Evals
+
+How this repo measures whether its skills actually work: that they **trigger** when they should, **stay distinct** from each other, and **change agent behavior** the way each skill promises.
+
+## Prior art (and what we adopted)
+
+There is no single settled community standard for evaluating `SKILL.md` skills, but two approaches lead:
+
+- **Anthropic's skill-creator v2** defines a per-skill `evals.json` (prompt + `expectations[]`, graded from the transcript) plus trigger-accuracy testing of descriptions against sample prompts. We adopt its [`evals.json` schema](https://github.com/anthropics/skills/tree/main/skills/skill-creator) for our behavioral tier and add one optional `kind` field to select the artifact being graded.
+- **Superpowers** (obra) tests skills with bash + `claude -p` + prompt fixtures and grader scripts. Our behavioral runner follows the same headless-`claude` pattern, with the grading rubric drawn from `expectations[]`.
+
+What neither provides is a **deterministic, CI-safe** check for a multi-skill *catalog* — does each skill's description carry the vocabulary users actually say, and do two skills' descriptions collide? That's Tier 2 below, and it's this repo's addition.
+
+## The three tiers
+
+| Tier | What it checks | Runs | Cost |
+|---|---|---|---|
+| 1. Structural | Frontmatter, naming, required sections, command parity | CI (`validate-skills.js`, `validate-commands.js`) | Free |
+| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js`) | Free |
+| 3. Behavioral | An agent following the skill satisfies its `expectations[]` | On demand (`run-evals.js --behavioral`) | Tokens |
+
+Tier 2 is a **lexical approximation** of routing (stemmed TF-IDF over descriptions). It cannot judge semantics — that's Tier 3's job — but it catches the two failure modes that dominate real trigger bugs: a description missing the vocabulary users say (false negative), and an over-broad description that outranks the right skill (false positive). A Tier-2 failure usually means *fix the description*, not the eval.
+
+## Running
+
+```bash
+# Tier 2 — deterministic, runs in CI
+node scripts/run-evals.js
+node scripts/run-evals.js --min-rank1 95  # enforce the current routing floor
+
+# Tier 3 — behavioral, runs each eval through headless claude, then grades it
+node scripts/run-evals.js --behavioral test-driven-development            # spends tokens
+node scripts/run-evals.js --behavioral test-driven-development --dry-run  # prints the plan only
+```
+
+Tier 3 supports two behavioral artifact kinds. `execution` is the default: each eval runs in a throwaway git repository, real project inputs from `files[]` are materialized out of `evals/fixtures/` and committed as the baseline, and the grader judges the full `--output-format stream-json --verbose` execution trace, including tool calls. `dialogue` is reserved for skills whose deliverable is the conversation itself; it needs no fixture, and the grader judges the assistant's conversational turns without requiring file edits or commands. Claiming `dialogue` is a human-reviewed exemption, not a general escape hatch for execution skills.
+
+The executor runs with an explicit permission mode (`--permission-mode acceptEdits` plus a pre-approved tool list) so execution evals can genuinely edit files, run commands, inspect diffs, and make commits rather than being denied and narrating instead. Traces are fenced as untrusted data in the grader prompt and piped to the grader over stdin (they can be megabytes; argv would hit the OS argument-size limit), executor and grader calls carry timeouts, and grader output is validated as JSON before being written to `evals/results/` (gitignored) in skill-creator's `grading.json` shape. Discipline skills also include pressure cases for time pressure, sunk cost, and authority pressure; these verify that the workflow still holds when the prompt argues for skipping it.
+
+## Plugin evals (Claude Code)
+
+`claude plugin eval` (Claude Code 2.1.269 or later) loads the whole plugin, every skill at once, runs the cases under `evals/plugin/`, and scores each case with and without the plugin. It complements Tier 2 rather than replacing it: Tier 2 checks that a description carries the vocabulary users say; this checks that Claude Code's own router picks the skill on a natural prompt, and what the skill adds to the reply (the `Δ` column).
+
+```bash
+claude plugin eval . --no-publish                                          # every case, both arms, 3 runs each
+claude plugin eval . --case code-review-fires --runs 1 --ablation none     # one cheap iteration on one case
+```
+
+A case is a directory holding `prompt.md` (frontmatter: turn and tool limits; body: the prompt, sent verbatim) and one grader per file under `graders/`. Reports land in `evals/plugin/results/<timestamp>/`, already gitignored. Every run and every `llm` grader is a real model call on your account, so this runs on demand and never in CI, like Tier 3.
+
+Two rules keep it honest. The manifest's `experimental.evals` key scopes the scan to `evals/plugin/`, so a fixture elsewhere under `evals/` is never read as a case. And a `tool_used: Skill` grader only says whether the skill fired; in a two-arm run it is excluded from the score, so pair it with a grader on the reply itself. `code-review-fires` shows the pattern: the fired indicator, a free `regex` grader that looks for the skill's severity taxonomy (Critical, Required, Nit, Optional, Consider, FYI) used as a heading or as a label, and one `llm` judge on the planted off-by-one. `code-review-stays-quiet` is the precision check: the review skill must not fire on a test-first request, and its `Δ` is expected to be 0; a second, unscored indicator records whether test-driven-development fires instead.
+
+What the first runs taught (Claude Code 2.1.278, claude-opus-5). Skill invocation is stochastic: the review skill fired in 5 of 7 runs on the phrasing the case uses, and test-driven-development fired in 2 of 7 runs on the test-first prompt, so read `Δ` and the report rather than the exit code, or pass `--threshold 0.8` as the CI example in the docs does. In one run the review skill did not fire and the reply still used its severity taxonomy as headings, which no baseline run did in nine; the loaded skill descriptions may shape the reply even without an invocation. Phrasing matters, and the description is the lever. With the original description, "Review this diff before I merge it", "Can you do a proper code review of this change before I merge it?" and "Review this pull request for me before it goes into main" rarely fired the skill; the model reviewed the diff on its own, competently but without the verdict, the severity tiers or the five-axis pass. Describing what the skill adds did not help, and neither did naming diffs and pull requests on their own; what made the difference was saying that the skill applies even when the diff is pasted inline, which is the case the model was treating as "I can just do this myself". On claude-opus-5-5 with the model pinned, that one clause took the four review phrasings from 7 of 27 fires to 21 of 27, left the nine adjacent must-not-fire prompts at 0 of 27 both ways, and moved the plainest phrasing from 0 of 9 to 3 of 9, so it is better, not solved. `code-review-stays-quiet-on-commit-message` guards that clause: a pasted diff with a commit-message request must not fire the review skill. Two practical rules from the same day: pin `--model` and record the Claude Code version whenever you report numbers, because the CLI updates itself and the default model changed twice between morning and evening; and give a case a turn budget of 10 or more, since the model may reach a skill through its slash command (`agent-skills:review`, then `code-review-and-quality`), which costs two turns.
+
+## Eval case format
+
+One file per skill: `evals/cases/<skill-name>.json`.
+
+```json
+{
+  "skill_name": "test-driven-development",
+  "trigger": {
+    "positive": [
+      { "prompt": "Write a failing test for this bug before fixing it", "top_k": 3 }
+    ],
+    "negative": [
+      { "prompt": "Update the architecture diagram in the docs", "owner": "documentation-and-adrs" }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "kind": "execution",
+      "prompt": "Finance filed the reconciliation bug written up in BUG.md. Fix it.",
+      "expected_output": "A failing reproduction test for the lost-cent case, a fix preserving both README invariants (exact sum, earliest-shares fairness), the fairness invariant covered by its own test, full suite passing",
+      "files": [
+        "test-driven-development"
+      ],
+      "expectations": [
+        "A test reproducing the lost-cent case from BUG.md is added and shown failing before src/split.js is modified",
+        "The final implementation satisfies the full README fairness invariant (leftover cents go to the earliest shares): splitCents(10000, 3) returns [3334, 3333, 3333] as BUG.md expects and splitCents(100, 7) returns [15, 15, 14, 14, 14, 14, 14] as the README example shows; dumping the whole remainder on a single share would violate both",
+        "The fairness invariant from the README has its own test case in the suite on an input with remainder of at least 2 (such as splitCents(100, 7)), where dumping the whole remainder on one share would fail it, beyond the reported lost-cent case",
+        "The full suite is run with the repository's own command after the fix"
+      ]
+    }
+  ]
+}
+```
+
+- `evals[]` uses skill-creator's core schema (`id`, `prompt`, `expected_output`, optional `files[]`, `expectations[]`) plus this repository's optional `kind`. `kind` must be `execution` or `dialogue` and defaults to `execution` for compatibility. Execution evals require non-empty `files[]`; paths are relative to `evals/fixtures/` and may name a file or project directory. Dialogue evals may omit `files[]` because the transcript is the artifact. Expectations are verifiable statements a grader checks against the relevant artifact — behaviors, not phrasings.
+- `trigger` is this repo's extension. `positive` prompts are realistic user asks that should route here (`top_k` defaults to 3; tighten to 1 for a skill's signature ask). `negative` prompts belong to a *different* skill; this skill must not rank first for them. Declare that skill in `owner` where you can: the runner then asserts the owner **outranks** this skill, turning the negative into a real pairwise routing test instead of one that can pass vacuously when the prompt matches nothing.
+
+**Writing good trigger prompts:** paraphrase how users actually talk; don't copy the description (that's gaming the eval). If a realistic prompt can't rank because the description lacks its vocabulary, that is a real finding — improve the description.
+
+## Adding a skill
+
+Every skill ships with an eval file. When you add `skills/<name>/`, add `evals/cases/<name>.json` with at least 3 positive triggers, 2 negative triggers, and 1 behavioral eval. Execution evals must be backed by `evals/fixtures/<name>/`; use `kind: "dialogue"` only when the skill's deliverable is genuinely the conversation itself. Missing case files, incomplete case counts, unknown kinds, invalid fixture paths, and absent required fixtures are CI errors.
+
+## Metrics to watch
+
+The Tier-2 run prints a **trigger rank-1 rate** (share of positive prompts that rank their skill first, not merely top-k). CI runs with `--min-rank1 95`, leaving useful headroom below the checked-in 100% baseline so an unrelated description edit does not immediately turn CI red. Raise the floor as routing improves; never lower it to make a regression pass. Falling numbers mean descriptions are drifting toward each other. The collision check errors at ≥75% pairwise description similarity and warns at ≥50%. When these evals surface a description-vocabulary gap (see [#351](https://github.com/addyosmani/agent-skills/issues/351) for the original examples), fix the description, not the prompt.