@givolt/i3-composer-agent 1.0.1 → 2.1.0

package/evals/README.md

@@ -1,101 +0,0 @@
-# Skill Evals
-
-How this repo measures whether its skills actually work: that they **trigger** when they should, **stay distinct** from each other, and **change agent behavior** the way each skill promises.
-
-## Prior art (and what we adopted)
-
-There is no single settled community standard for evaluating `SKILL.md` skills, but two approaches lead:
-
-- **Anthropic's skill-creator v2** defines a per-skill `evals.json` (prompt + `expectations[]`, graded from the transcript) plus trigger-accuracy testing of descriptions against sample prompts. We adopt its [`evals.json` schema](https://github.com/anthropics/skills/tree/main/skills/skill-creator) for our behavioral tier and add one optional `kind` field to select the artifact being graded.
-- **Superpowers** (obra) tests skills with bash + `claude -p` + prompt fixtures and grader scripts. Our behavioral runner follows the same headless-`claude` pattern, with the grading rubric drawn from `expectations[]`.
-
-What neither provides is a **deterministic, CI-safe** check for a multi-skill *catalog* — does each skill's description carry the vocabulary users actually say, and do two skills' descriptions collide? That's Tier 2 below, and it's this repo's addition.
-
-## The three tiers
-
-| Tier | What it checks | Runs | Cost |
-|---|---|---|---|
-| 1. Structural | Frontmatter, naming, required sections, command parity | CI (`validate-skills.js`, `validate-commands.js`) | Free |
-| 2. Trigger & routing | Positive prompts rank their skill top-k; negative prompts don't; no two descriptions near-collide | CI (`run-evals.js`) | Free |
-| 3. Behavioral | An agent following the skill satisfies its `expectations[]` | On demand (`run-evals.js --behavioral`) | Tokens |
-
-Tier 2 is a **lexical approximation** of routing (stemmed TF-IDF over descriptions). It cannot judge semantics — that's Tier 3's job — but it catches the two failure modes that dominate real trigger bugs: a description missing the vocabulary users say (false negative), and an over-broad description that outranks the right skill (false positive). A Tier-2 failure usually means *fix the description*, not the eval.
-
-## Running
-
-```bash
-# Tier 2 — deterministic, runs in CI
-node scripts/run-evals.js
-node scripts/run-evals.js --min-rank1 95  # enforce the current routing floor
-
-# Tier 3 — behavioral, runs each eval through headless claude, then grades it
-node scripts/run-evals.js --behavioral test-driven-development            # spends tokens
-node scripts/run-evals.js --behavioral test-driven-development --dry-run  # prints the plan only
-```
-
-Tier 3 supports two behavioral artifact kinds. `execution` is the default: each eval runs in a throwaway git repository, real project inputs from `files[]` are materialized out of `evals/fixtures/` and committed as the baseline, and the grader judges the full `--output-format stream-json --verbose` execution trace, including tool calls. `dialogue` is reserved for skills whose deliverable is the conversation itself; it needs no fixture, and the grader judges the assistant's conversational turns without requiring file edits or commands. Claiming `dialogue` is a human-reviewed exemption, not a general escape hatch for execution skills.
-
-The executor runs with an explicit permission mode (`--permission-mode acceptEdits` plus a pre-approved tool list) so execution evals can genuinely edit files, run commands, inspect diffs, and make commits rather than being denied and narrating instead. Traces are fenced as untrusted data in the grader prompt and piped to the grader over stdin (they can be megabytes; argv would hit the OS argument-size limit), executor and grader calls carry timeouts, and grader output is validated as JSON before being written to `evals/results/` (gitignored) in skill-creator's `grading.json` shape. Discipline skills also include pressure cases for time pressure, sunk cost, and authority pressure; these verify that the workflow still holds when the prompt argues for skipping it.
-
-## Plugin evals (Claude Code)
-
-`claude plugin eval` (Claude Code 2.1.269 or later) loads the whole plugin, every skill at once, runs the cases under `evals/plugin/`, and scores each case with and without the plugin. It complements Tier 2 rather than replacing it: Tier 2 checks that a description carries the vocabulary users say; this checks that Claude Code's own router picks the skill on a natural prompt, and what the skill adds to the reply (the `Δ` column).
-
-```bash
-claude plugin eval . --no-publish                                          # every case, both arms, 3 runs each
-claude plugin eval . --case code-review-fires --runs 1 --ablation none     # one cheap iteration on one case
-```
-
-A case is a directory holding `prompt.md` (frontmatter: turn and tool limits; body: the prompt, sent verbatim) and one grader per file under `graders/`. Reports land in `evals/plugin/results/<timestamp>/`, already gitignored. Every run and every `llm` grader is a real model call on your account, so this runs on demand and never in CI, like Tier 3.
-
-Two rules keep it honest. The manifest's `experimental.evals` key scopes the scan to `evals/plugin/`, so a fixture elsewhere under `evals/` is never read as a case. And a `tool_used: Skill` grader only says whether the skill fired; in a two-arm run it is excluded from the score, so pair it with a grader on the reply itself. `code-review-fires` shows the pattern: the fired indicator, a free `regex` grader that looks for the skill's severity taxonomy (Critical, Required, Nit, Optional, Consider, FYI) used as a heading or as a label, and one `llm` judge on the planted off-by-one. `code-review-stays-quiet` is the precision check: the review skill must not fire on a test-first request, and its `Δ` is expected to be 0; a second, unscored indicator records whether test-driven-development fires instead.
-
-What the first runs taught (Claude Code 2.1.278, claude-opus-5). Skill invocation is stochastic: the review skill fired in 5 of 7 runs on the phrasing the case uses, and test-driven-development fired in 2 of 7 runs on the test-first prompt, so read `Δ` and the report rather than the exit code, or pass `--threshold 0.8` as the CI example in the docs does. In one run the review skill did not fire and the reply still used its severity taxonomy as headings, which no baseline run did in nine; the loaded skill descriptions may shape the reply even without an invocation. Phrasing matters, and the description is the lever. With the original description, "Review this diff before I merge it", "Can you do a proper code review of this change before I merge it?" and "Review this pull request for me before it goes into main" rarely fired the skill; the model reviewed the diff on its own, competently but without the verdict, the severity tiers or the five-axis pass. Describing what the skill adds did not help, and neither did naming diffs and pull requests on their own; what made the difference was saying that the skill applies even when the diff is pasted inline, which is the case the model was treating as "I can just do this myself". On claude-opus-5-5 with the model pinned, that one clause took the four review phrasings from 7 of 27 fires to 21 of 27, left the nine adjacent must-not-fire prompts at 0 of 27 both ways, and moved the plainest phrasing from 0 of 9 to 3 of 9, so it is better, not solved. `code-review-stays-quiet-on-commit-message` guards that clause: a pasted diff with a commit-message request must not fire the review skill. Two practical rules from the same day: pin `--model` and record the Claude Code version whenever you report numbers, because the CLI updates itself and the default model changed twice between morning and evening; and give a case a turn budget of 10 or more, since the model may reach a skill through its slash command (`agent-skills:review`, then `code-review-and-quality`), which costs two turns.
-
-## Eval case format
-
-One file per skill: `evals/cases/<skill-name>.json`.
-
-```json
-{
-  "skill_name": "test-driven-development",
-  "trigger": {
-    "positive": [
-      { "prompt": "Write a failing test for this bug before fixing it", "top_k": 3 }
-    ],
-    "negative": [
-      { "prompt": "Update the architecture diagram in the docs", "owner": "documentation-and-adrs" }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "kind": "execution",
-      "prompt": "Finance filed the reconciliation bug written up in BUG.md. Fix it.",
-      "expected_output": "A failing reproduction test for the lost-cent case, a fix preserving both README invariants (exact sum, earliest-shares fairness), the fairness invariant covered by its own test, full suite passing",
-      "files": [
-        "test-driven-development"
-      ],
-      "expectations": [
-        "A test reproducing the lost-cent case from BUG.md is added and shown failing before src/split.js is modified",
-        "The final implementation satisfies the full README fairness invariant (leftover cents go to the earliest shares): splitCents(10000, 3) returns [3334, 3333, 3333] as BUG.md expects and splitCents(100, 7) returns [15, 15, 14, 14, 14, 14, 14] as the README example shows; dumping the whole remainder on a single share would violate both",
-        "The fairness invariant from the README has its own test case in the suite on an input with remainder of at least 2 (such as splitCents(100, 7)), where dumping the whole remainder on one share would fail it, beyond the reported lost-cent case",
-        "The full suite is run with the repository's own command after the fix"
-      ]
-    }
-  ]
-}
-```
-
-- `evals[]` uses skill-creator's core schema (`id`, `prompt`, `expected_output`, optional `files[]`, `expectations[]`) plus this repository's optional `kind`. `kind` must be `execution` or `dialogue` and defaults to `execution` for compatibility. Execution evals require non-empty `files[]`; paths are relative to `evals/fixtures/` and may name a file or project directory. Dialogue evals may omit `files[]` because the transcript is the artifact. Expectations are verifiable statements a grader checks against the relevant artifact — behaviors, not phrasings.
-- `trigger` is this repo's extension. `positive` prompts are realistic user asks that should route here (`top_k` defaults to 3; tighten to 1 for a skill's signature ask). `negative` prompts belong to a *different* skill; this skill must not rank first for them. Declare that skill in `owner` where you can: the runner then asserts the owner **outranks** this skill, turning the negative into a real pairwise routing test instead of one that can pass vacuously when the prompt matches nothing.
-
-**Writing good trigger prompts:** paraphrase how users actually talk; don't copy the description (that's gaming the eval). If a realistic prompt can't rank because the description lacks its vocabulary, that is a real finding — improve the description.
-
-## Adding a skill
-
-Every skill ships with an eval file. When you add `skills/<name>/`, add `evals/cases/<name>.json` with at least 3 positive triggers, 2 negative triggers, and 1 behavioral eval. Execution evals must be backed by `evals/fixtures/<name>/`; use `kind: "dialogue"` only when the skill's deliverable is genuinely the conversation itself. Missing case files, incomplete case counts, unknown kinds, invalid fixture paths, and absent required fixtures are CI errors.
-
-## Metrics to watch
-
-The Tier-2 run prints a **trigger rank-1 rate** (share of positive prompts that rank their skill first, not merely top-k). CI runs with `--min-rank1 95`, leaving useful headroom below the checked-in 100% baseline so an unrelated description edit does not immediately turn CI red. Raise the floor as routing improves; never lower it to make a regression pass. Falling numbers mean descriptions are drifting toward each other. The collision check errors at ≥75% pairwise description similarity and warns at ≥50%. When these evals surface a description-vocabulary gap (see [#351](https://github.com/addyosmani/agent-skills/issues/351) for the original examples), fix the description, not the prompt.

package/evals/cases/api-and-interface-design.json

@@ -1,45 +0,0 @@
-{
-  "skill_name": "api-and-interface-design",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Design a REST endpoint for creating invoices, including error responses and versioning",
-        "top_k": 3
-      },
-      {
-        "prompt": "What should the public interface of this payments module expose to other teams?",
-        "top_k": 3
-      },
-      {
-        "prompt": "Help me define the contract between the frontend and the orders service",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "My unit test is failing with a null pointer exception",
-        "owner": "debugging-and-error-recovery"
-      },
-      {
-        "prompt": "Make this landing page responsive on mobile",
-        "owner": "frontend-ui-engineering"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Design the public API for a URL-shortening service: create, resolve, stats. Produce the endpoint contracts.",
-      "expected_output": "Endpoint contracts with methods, paths, request/response shapes, and explicit error semantics",
-      "files": [
-        "api-and-interface-design"
-      ],
-      "expectations": [
-        "Error responses are specified with status codes and a consistent error shape, not just happy paths",
-        "Input validation at the boundary is addressed for user-supplied URLs",
-        "Versioning or compatibility strategy is stated",
-        "The response does not silently invent unstated requirements"
-      ]
-    }
-  ]
-}

package/evals/cases/browser-testing-with-devtools.json

@@ -1,44 +0,0 @@
-{
-  "skill_name": "browser-testing-with-devtools",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Use Chrome DevTools to figure out why the checkout button click does nothing",
-        "top_k": 3
-      },
-      {
-        "prompt": "Inspect the network requests the dashboard makes when it first loads",
-        "top_k": 3
-      },
-      {
-        "prompt": "Verify in a real browser that the form submits and the data persists in localStorage",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Write a spec for the new billing feature before we start",
-        "owner": "spec-driven-development"
-      },
-      {
-        "prompt": "Break this PRD into ordered implementation tasks",
-        "owner": "planning-and-task-breakdown"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "The signup form renders but submitting it appears to do nothing. Verify the real behavior in the browser and report findings.",
-      "expected_output": "Runtime evidence from the browser: console errors, network activity, DOM state, and a diagnosis",
-      "files": [
-        "browser-testing-with-devtools"
-      ],
-      "expectations": [
-        "Findings are grounded in observed runtime data (console, network, DOM), not static code reading alone",
-        "The report distinguishes what was observed from what is inferred",
-        "A concrete next step or fix hypothesis is provided"
-      ]
-    }
-  ]
-}

package/evals/cases/ci-cd-and-automation.json

@@ -1,45 +0,0 @@
-{
-  "skill_name": "ci-cd-and-automation",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Set up a GitHub Actions pipeline that runs the tests on every pull request",
-        "top_k": 3
-      },
-      {
-        "prompt": "Add a deploy stage to the pipeline with a manual approval gate",
-        "top_k": 3
-      },
-      {
-        "prompt": "Automate the quality gates so nothing merges without passing checks",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Refactor this function so it is easier to read",
-        "owner": "code-simplification"
-      },
-      {
-        "prompt": "Add structured logging to the checkout service",
-        "owner": "observability-and-instrumentation"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Create a CI pipeline for a Node project: install, lint, test on every PR, and block merge on failure.",
-      "expected_output": "A working workflow definition with correct triggers, steps, and failure behavior",
-      "files": [
-        "ci-cd-and-automation"
-      ],
-      "expectations": [
-        "The workflow triggers on pull requests",
-        "Failure of any quality gate fails the pipeline run",
-        "Steps are ordered logically and cache or setup steps are sane",
-        "No secrets are hardcoded in the workflow"
-      ]
-    }
-  ]
-}

package/evals/cases/code-review-and-quality.json

@@ -1,49 +0,0 @@
-{
-  "skill_name": "code-review-and-quality",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Review this pull request before I merge it",
-        "top_k": 3
-      },
-      {
-        "prompt": "Do a quality pass on this diff for correctness and readability",
-        "top_k": 3
-      },
-      {
-        "prompt": "Can you review the changes another model just wrote for me?",
-        "top_k": 3
-      },
-      {
-        "prompt": "Review this diff before I merge it. It adds paging to the orders endpoint.",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Deploy this to production now",
-        "owner": "shipping-and-launch"
-      },
-      {
-        "prompt": "Write a failing test for the bug before fixing it",
-        "owner": "test-driven-development"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Review the provided diff that adds a user-search endpoint. Deliver a structured review.",
-      "expected_output": "A multi-axis review with severity-labelled findings and file:line references",
-      "files": [
-        "code-review-and-quality"
-      ],
-      "expectations": [
-        "Findings cover more than one axis (correctness, readability, architecture, security, performance)",
-        "Every finding carries a severity label from the skill's taxonomy",
-        "Security of user input is explicitly considered for the new endpoint",
-        "The review leads with high-leverage findings rather than nits"
-      ]
-    }
-  ]
-}

package/evals/cases/code-simplification.json

@@ -1,45 +0,0 @@
-{
-  "skill_name": "code-simplification",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "This function works but it is way too clever, simplify it without changing behavior",
-        "top_k": 3
-      },
-      {
-        "prompt": "Reduce the complexity of this module so juniors can maintain it",
-        "top_k": 3
-      },
-      {
-        "prompt": "Clean up this working code, it has grown hard to follow",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Add a feature flag system to the app",
-        "owner": "incremental-implementation"
-      },
-      {
-        "prompt": "Diagnose why the build broke overnight",
-        "owner": "debugging-and-error-recovery"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Simplify the provided 80-line function that parses config files, preserving exact behavior.",
-      "expected_output": "A simpler implementation with identical behavior and a summary of what was removed and why",
-      "files": [
-        "code-simplification"
-      ],
-      "expectations": [
-        "Behavior is preserved (tests unchanged and passing, or equivalence argued concretely)",
-        "Complexity is reduced rather than relocated",
-        "The response explains what was removed and why it was safe",
-        "No new features are added during the simplification"
-      ]
-    }
-  ]
-}

package/evals/cases/constraint-driven-development.json

@@ -1,79 +0,0 @@
-{
-  "skill_name": "constraint-driven-development",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Set up quality constraints for this project — I want to define our bar before the agent writes more code",
-        "top_k": 3
-      },
-      {
-        "prompt": "We have no written standards. Ask me what I care about and write it down with sane defaults",
-        "top_k": 3
-      },
-      {
-        "prompt": "The agent keeps adding @ts-ignore and skipping tests to get green. How do I stop that?",
-        "top_k": 3
-      },
-      {
-        "prompt": "What coverage threshold should we enforce, and where should it run so it doesn't slow us down?",
-        "top_k": 3
-      },
-      {
-        "prompt": "I care about accessibility and web performance — set those up as enforced constraints",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Review this pull request across correctness and security",
-        "owner": "code-review-and-quality"
-      },
-      {
-        "prompt": "Add a deploy stage to our pipeline with a manual approval gate",
-        "owner": "ci-cd-and-automation"
-      },
-      {
-        "prompt": "Write a failing test for this bug before we fix it",
-        "owner": "test-driven-development"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "kind": "dialogue",
-      "prompt": "Set up constraints for this repo. I don't really know what numbers to pick.",
-      "expected_output": "A short interview with defaults offered, followed by a CONSTRAINTS.md the user can explain",
-      "expectations": [
-        "The agent inspects the repo for stack, test runner, and existing linters before asking anything",
-        "No more than four questions are asked, and each one carries a usable default",
-        "When the user has no target number, the agent proposes recording today's value and holding it rather than inventing a threshold",
-        "The resulting constraints say where each check runs, keeping fast checks in the edit loop and slow checks later",
-        "Every number written down is accompanied by a reason"
-      ]
-    },
-    {
-      "id": 2,
-      "kind": "dialogue",
-      "prompt": "Our coverage is 62% and I want to enforce 85% starting today.",
-      "expected_output": "Pushback on setting an unmeetable budget, with the ratchet alternative offered",
-      "expectations": [
-        "The agent points out that a threshold the codebase fails today produces a permanently red build",
-        "It offers to record 62% and require that it not fall, as an adoptable alternative",
-        "It does not silently accept 85% without noting the consequence"
-      ]
-    },
-    {
-      "id": 3,
-      "kind": "dialogue",
-      "prompt": "Accessibility matters to us. Add it as a constraint.",
-      "expected_output": "A working accessibility check, not just a threshold written down",
-      "expectations": [
-        "The agent names axe-core and gives the install and run commands rather than only stating a violation threshold",
-        "It notes that axe needs a running URL and asks for one, or drops the dimension if the project has no URL to hit",
-        "The command that produces the verdict is recorded alongside the rule in CONSTRAINTS.md",
-        "The check is placed at the preview-deploy stage rather than in the per-edit loop"
-      ]
-    }
-  ]
-}

package/evals/cases/context-engineering.json

@@ -1,44 +0,0 @@
-{
-  "skill_name": "context-engineering",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "What context should I load into the agent before starting this refactor?",
-        "top_k": 3
-      },
-      {
-        "prompt": "The model's output quality dropped mid-session, how do I fix its context?",
-        "top_k": 3
-      },
-      {
-        "prompt": "Set up rules files and context packing for my repository",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Write unit tests for the date parser",
-        "owner": "test-driven-development"
-      },
-      {
-        "prompt": "Harden this endpoint against SQL injection",
-        "owner": "security-and-hardening"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "An agent session has degraded: responses are generic and it keeps forgetting project conventions. Fix the context setup.",
-      "expected_output": "A diagnosis of the context problem and a concrete loading strategy (rules files, packing, task-scoped context)",
-      "files": [
-        "context-engineering"
-      ],
-      "expectations": [
-        "The response diagnoses probable context causes rather than blaming the model generally",
-        "A specific context-loading plan is produced (what to load, when, and why)",
-        "Context is scoped to the task instead of loading everything"
-      ]
-    }
-  ]
-}

package/evals/cases/debugging-and-error-recovery.json

@@ -1,58 +0,0 @@
-{
-  "skill_name": "debugging-and-error-recovery",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "The app crashes with a TypeError when I click save, help me find the root cause",
-        "top_k": 3
-      },
-      {
-        "prompt": "The API is throwing intermittent 500 errors in production, help me find the root cause",
-        "top_k": 3
-      },
-      {
-        "prompt": "This test passed yesterday and fails today, figure out what broke",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Write the API documentation for the orders service",
-        "owner": "documentation-and-adrs"
-      },
-      {
-        "prompt": "Simplify this working function for readability",
-        "owner": "code-simplification"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "A test that passed yesterday now fails with an off-by-one error in pagination. Find and fix the root cause.",
-      "expected_output": "A reproduced failure, an identified root cause, a minimal fix, and a guard against regression",
-      "files": [
-        "debugging-and-error-recovery"
-      ],
-      "expectations": [
-        "The failure is reproduced before any fix is attempted",
-        "The root cause is identified and stated, not just the symptom patched",
-        "A regression test exists after the fix",
-        "The fix is minimal and scoped to the cause"
-      ]
-    },
-    {
-      "id": 2,
-      "prompt": "Production checkout is failing and a stakeholder demands an immediate pagination patch without reproduction. Respond under time pressure.",
-      "expected_output": "Evidence preserved, the failure reproduced, and a root-cause fix guarded by a regression test",
-      "files": [
-        "debugging-and-error-recovery"
-      ],
-      "expectations": [
-        "The time pressure does not cause reproduction or evidence gathering to be skipped",
-        "The requested speculative patch is not applied before the cause is localized",
-        "The final fix is tied to a reproduced cause and a regression test"
-      ]
-    }
-  ]
-}

package/evals/cases/deprecation-and-migration.json

@@ -1,52 +0,0 @@
-{
-  "skill_name": "deprecation-and-migration",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "We are sunsetting the v1 API, plan the deprecation",
-        "top_k": 3
-      },
-      {
-        "prompt": "Migrate users off the legacy auth system without breaking them",
-        "top_k": 3
-      },
-      {
-        "prompt": "How do I remove this old feature safely for existing users?",
-        "top_k": 3
-      },
-      {
-        "prompt": "Rename a database column without downtime",
-        "top_k": 3
-      },
-      {
-        "prompt": "How do I drop a column safely in production?",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Add dark mode to the settings page",
-        "owner": "frontend-ui-engineering"
-      },
-      {
-        "prompt": "Why is this test flaky on CI?",
-        "owner": "ci-cd-and-automation"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Plan the deprecation of a public v1 REST API with 200 external consumers, replaced by v2.",
-      "expected_output": "A staged deprecation plan: comms, timeline, compatibility window, monitoring, and removal criteria",
-      "files": [
-        "deprecation-and-migration"
-      ],
-      "expectations": [
-        "Consumers are notified with a timeline before any breaking change",
-        "A compatibility or migration window exists with monitoring of remaining usage",
-        "Removal is gated on measured migration, not a calendar date alone"
-      ]
-    }
-  ]
-}

package/evals/cases/documentation-and-adrs.json

@@ -1,44 +0,0 @@
-{
-  "skill_name": "documentation-and-adrs",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "Write an ADR for choosing Postgres over MongoDB",
-        "top_k": 3
-      },
-      {
-        "prompt": "Document the architecture decision behind the queue design",
-        "top_k": 3
-      },
-      {
-        "prompt": "Update the documentation for the new setup flow and record why it changed",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Find the root cause of the memory leak in the worker",
-        "owner": "debugging-and-error-recovery"
-      },
-      {
-        "prompt": "Harden the login endpoint against brute force attacks",
-        "owner": "security-and-hardening"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Record the decision to adopt event sourcing for the orders service as an ADR.",
-      "expected_output": "An ADR capturing context, decision, alternatives considered, and consequences",
-      "files": [
-        "documentation-and-adrs"
-      ],
-      "expectations": [
-        "The ADR states context, decision, alternatives, and consequences distinctly",
-        "Trade-offs and rejected options are recorded, not just the winning choice",
-        "The ADR records a status from the skill's lifecycle (Proposed, Accepted, Superseded or Deprecated) and a date"
-      ]
-    }
-  ]
-}

package/evals/cases/doubt-driven-development.json

@@ -1,43 +0,0 @@
-{
-  "skill_name": "doubt-driven-development",
-  "trigger": {
-    "positive": [
-      {
-        "prompt": "This change touches production auth, cross-examine every assumption before we proceed",
-        "top_k": 3
-      },
-      {
-        "prompt": "I am not confident in this approach, adversarially review my reasoning in a fresh context",
-        "top_k": 3
-      },
-      {
-        "prompt": "High-stakes migration tomorrow, stress the plan for hidden failure modes",
-        "top_k": 3
-      }
-    ],
-    "negative": [
-      {
-        "prompt": "Format this file with prettier"
-      },
-      {
-        "prompt": "Write the changelog entry for the release",
-        "owner": "git-workflow-and-versioning"
-      }
-    ]
-  },
-  "evals": [
-    {
-      "id": 1,
-      "prompt": "Before running an irreversible data migration, subject the migration plan to adversarial review.",
-      "expected_output": "Claims extracted, doubts raised against each, reconciliation, and a go or stop verdict",
-      "files": [
-        "doubt-driven-development"
-      ],
-      "expectations": [
-        "Non-trivial claims in the plan are extracted and challenged individually",
-        "At least one assumption is tested rather than accepted",
-        "The verdict distinguishes verified claims from surviving doubts"
-      ]
-    }
-  ]
-}