@givolt/i3-composer-agent 2.2.0 → 2.2.1

package/evals/README.md

@@ -0,0 +1,86 @@
+# 技能评测与配套脚本
+
+本目录随默认安装、npm 包和插件导出一起提供，评测 i3 Composer Agent 技能的结构、
+触发与行为。评测不在安装时自动执行。它检查技能库自身，不能代替业务项目测试、
+宿主发现验证或真实模型验收。
+
+## 运行位置
+
+原生安装的完整配套目录是宿主目录下的 `i3-composer-agent/`。
+例如 CodeBuddy 项目安装为 `.codebuddy/i3-composer-agent/`，
+WorkBuddy 全局安装为 `~/.workbuddy/i3-composer-agent/`。
+先进入该目录；若使用 `--prepare` / `--extract`，进入指定的导出目录。
+配套目录保留技能、命令、角色、参考资料和插件清单，保证校验及 Hook 的相对路径可解析。
+宿主实际使用的原生入口仍位于它自己的 `skills/`、`commands/`、`agents/`。
+
+## 不调用模型的检查
+
+需要 Node.js 20 或以上，无需额外 npm 依赖。
+
+```bash
+# 技能格式、清单版本、三种命令格式一致性、引用与产物路径
+npm run validate
+
+# 中英文分别检查；省略 --locale 时检查两种语言
+node scripts/run-evals.js --locale zh-CN --min-rank1 95
+node scripts/run-evals.js --locale en --min-rank1 95
+
+# 一次运行结构检查和中英文触发评测
+npm test
+```
+
+`scripts/` 包含 `run-evals.js`、五个 `validate-*.js` 和它们依赖的 `lib/skill-lint.js`。
+`evals/cases/` 保留英文用例，`evals/zh-CN/cases/` 提供中文用例，
+`evals/fixtures/` 是执行型评测使用的测试项目样例，`evals/plugin/` 是插件级评测用例。
+
+触发评测使用词法评分检查正例、负例、描述冲突、用例结构及样例是否齐全。
+中英文各自满足指定门槛；这不能证明模型在实际宿主中一定选择正确技能。
+失败时应调查技能描述或实现，不能为了通过检查降低门槛或删改失败用例。
+
+## 主动运行模型行为评测
+
+先预览，不调用模型、不生成评测结果：
+
+```bash
+node scripts/run-evals.js --behavioral i3-test-driven-development --locale zh-CN --dry-run
+```
+
+确认需要后，移除 `--dry-run` 才会调用已登录的 Claude Code，并消耗该账户的模型额度：
+
+```bash
+node scripts/run-evals.js --behavioral i3-test-driven-development --locale zh-CN
+```
+
+执行型用例需要 Git，并在临时 Git 项目中复制 `fixtures/` 的真实输入；
+对话型用例评估对话，不要求项目文件。执行器使用 `acceptEdits` 及明确的工具列表，
+可能编辑文件、运行命令、访问网络；各样例还可能需要自身的语言运行环境。
+评分依据执行轨迹或对话与 `expectations`，不会把“声称完成”当成实际执行证据。
+模型输出按不可信数据评分，执行和评分均有限时。临时项目完成后清理。
+
+`--locale en` 运行英文用例，省略该参数运行两种语言。可通过 `I3_EVAL_MODEL`
+和 `I3_EVAL_GRADER_MODEL` 指定执行/评分模型；报告同时记录宿主和模型信息。
+结果写入 `evals/results/`，不会进入发行包，也不属于安装器的替换清单。
+
+## 插件级评测
+
+插件清单中的 `experimental.evals` 指向随包提供的 `evals/plugin/`。
+它包含整体插件的触发/不触发提示和评分规则；支持 `plugin eval` 的 Claude Code
+可从配套目录主动运行：
+
+```bash
+claude plugin eval . --no-publish
+```
+
+该检查同样会调用模型；具体参数以当前 Claude Code 的帮助为准。
+`--no-publish` 保持结果本地，报告写入 `evals/plugin/results/`。
+用是否触发、回答质量及对照结果一起判断效果；调用一次成功不能代替多次行为验收。
+
+## Hooks
+
+`hooks/` 默认交付会话注入、文档缓存、代码块保护脚本及说明。
+安装文件不会自动注册事件。中文回复与普通技能调用不依赖启用 Hook。
+会话注入仅适用于缺少原生技能路由的宿主；已有路由时不要重复注入。
+缓存及保护的当前配置示例针对 Claude Code，详见
+[文档缓存](../hooks/SDD-CACHE.md) 和 [代码块保护](../hooks/SIMPLIFY-IGNORE.md)。
+其他宿主须按各自事件协议适配。脚本需要 Bash、jq 及说明中的 curl/哈希工具；
+Windows PowerShell 本身不提供这些依赖。

package/evals/cases/i3-api-and-interface-design.json

@@ -0,0 +1,45 @@
+{
+  "skill_name": "i3-api-and-interface-design",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Design a REST endpoint for creating invoices, including error responses and versioning",
+        "top_k": 3
+      },
+      {
+        "prompt": "What should the public interface of this payments module expose to other teams?",
+        "top_k": 3
+      },
+      {
+        "prompt": "Help me define the contract between the frontend and the orders service",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "My unit test is failing with a null pointer exception",
+        "owner": "i3-debugging-and-error-recovery"
+      },
+      {
+        "prompt": "Make this landing page responsive on mobile",
+        "owner": "i3-frontend-ui-engineering"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Design the public API for a URL-shortening service: create, resolve, stats. Produce the endpoint contracts.",
+      "expected_output": "Endpoint contracts with methods, paths, request/response shapes, and explicit error semantics",
+      "files": [
+        "i3-api-and-interface-design"
+      ],
+      "expectations": [
+        "Error responses are specified with status codes and a consistent error shape, not just happy paths",
+        "Input validation at the boundary is addressed for user-supplied URLs",
+        "Versioning or compatibility strategy is stated",
+        "The response does not silently invent unstated requirements"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-browser-testing-with-devtools.json

@@ -0,0 +1,44 @@
+{
+  "skill_name": "i3-browser-testing-with-devtools",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Use Chrome DevTools to figure out why the checkout button click does nothing",
+        "top_k": 3
+      },
+      {
+        "prompt": "Inspect the network requests the dashboard makes when it first loads",
+        "top_k": 3
+      },
+      {
+        "prompt": "Verify in a real browser that the form submits and the data persists in localStorage",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Write a spec for the new billing feature before we start",
+        "owner": "i3-spec-driven-development"
+      },
+      {
+        "prompt": "Break this PRD into ordered implementation tasks",
+        "owner": "i3-planning-and-task-breakdown"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "The signup form renders but submitting it appears to do nothing. Verify the real behavior in the browser and report findings.",
+      "expected_output": "Runtime evidence from the browser: console errors, network activity, DOM state, and a diagnosis",
+      "files": [
+        "i3-browser-testing-with-devtools"
+      ],
+      "expectations": [
+        "Findings are grounded in observed runtime data (console, network, DOM), not static code reading alone",
+        "The report distinguishes what was observed from what is inferred",
+        "A concrete next step or fix hypothesis is provided"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-ci-cd-and-automation.json

@@ -0,0 +1,45 @@
+{
+  "skill_name": "i3-ci-cd-and-automation",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Set up a GitHub Actions pipeline that runs the tests on every pull request",
+        "top_k": 3
+      },
+      {
+        "prompt": "Add a deploy stage to the pipeline with a manual approval gate",
+        "top_k": 3
+      },
+      {
+        "prompt": "Automate the quality gates so nothing merges without passing checks",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Refactor this function so it is easier to read",
+        "owner": "i3-code-simplification"
+      },
+      {
+        "prompt": "Add structured logging to the checkout service",
+        "owner": "i3-observability-and-instrumentation"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Create a CI pipeline for a Node project: install, lint, test on every PR, and block merge on failure.",
+      "expected_output": "A working workflow definition with correct triggers, steps, and failure behavior",
+      "files": [
+        "i3-ci-cd-and-automation"
+      ],
+      "expectations": [
+        "The workflow triggers on pull requests",
+        "Failure of any quality gate fails the pipeline run",
+        "Steps are ordered logically and cache or setup steps are sane",
+        "No secrets are hardcoded in the workflow"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-code-review-and-quality.json

@@ -0,0 +1,49 @@
+{
+  "skill_name": "i3-code-review-and-quality",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Review this pull request before I merge it",
+        "top_k": 3
+      },
+      {
+        "prompt": "Do a quality pass on this diff for correctness and readability",
+        "top_k": 3
+      },
+      {
+        "prompt": "Can you review the changes another model just wrote for me?",
+        "top_k": 3
+      },
+      {
+        "prompt": "Review this diff before I merge it. It adds paging to the orders endpoint.",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Deploy this to production now",
+        "owner": "i3-shipping-and-launch"
+      },
+      {
+        "prompt": "Write a failing test for the bug before fixing it",
+        "owner": "i3-test-driven-development"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Review the provided diff that adds a user-search endpoint. Deliver a structured review.",
+      "expected_output": "A multi-axis review with severity-labelled findings and file:line references",
+      "files": [
+        "i3-code-review-and-quality"
+      ],
+      "expectations": [
+        "Findings cover more than one axis (correctness, readability, architecture, security, performance)",
+        "Every finding carries a severity label from the skill's taxonomy",
+        "Security of user input is explicitly considered for the new endpoint",
+        "The review leads with high-leverage findings rather than nits"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-code-simplification.json

@@ -0,0 +1,45 @@
+{
+  "skill_name": "i3-code-simplification",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "This function works but it is way too clever, simplify it without changing behavior",
+        "top_k": 3
+      },
+      {
+        "prompt": "Reduce the complexity of this module so juniors can maintain it",
+        "top_k": 3
+      },
+      {
+        "prompt": "Clean up this working code, it has grown hard to follow",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Add a feature flag system to the app",
+        "owner": "i3-incremental-implementation"
+      },
+      {
+        "prompt": "Diagnose why the build broke overnight",
+        "owner": "i3-debugging-and-error-recovery"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Simplify the provided 80-line function that parses config files, preserving exact behavior.",
+      "expected_output": "A simpler implementation with identical behavior and a summary of what was removed and why",
+      "files": [
+        "i3-code-simplification"
+      ],
+      "expectations": [
+        "Behavior is preserved (tests unchanged and passing, or equivalence argued concretely)",
+        "Complexity is reduced rather than relocated",
+        "The response explains what was removed and why it was safe",
+        "No new features are added during the simplification"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-constraint-driven-development.json

@@ -0,0 +1,79 @@
+{
+  "skill_name": "i3-constraint-driven-development",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Set up quality constraints for this project — I want to define our bar before the agent writes more code",
+        "top_k": 3
+      },
+      {
+        "prompt": "We have no written standards. Ask me what I care about and write it down with sane defaults",
+        "top_k": 3
+      },
+      {
+        "prompt": "The agent keeps adding @ts-ignore and skipping tests to get green. How do I stop that?",
+        "top_k": 3
+      },
+      {
+        "prompt": "What coverage threshold should we enforce, and where should it run so it doesn't slow us down?",
+        "top_k": 3
+      },
+      {
+        "prompt": "I care about accessibility and web performance — set those up as enforced constraints",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Review this pull request across correctness and security",
+        "owner": "i3-code-review-and-quality"
+      },
+      {
+        "prompt": "Add a deploy stage to our pipeline with a manual approval gate",
+        "owner": "i3-ci-cd-and-automation"
+      },
+      {
+        "prompt": "Write a failing test for this bug before we fix it",
+        "owner": "i3-test-driven-development"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "kind": "dialogue",
+      "prompt": "Set up constraints for this repo. I don't really know what numbers to pick.",
+      "expected_output": "A short interview with defaults offered, followed by a CONSTRAINTS.md the user can explain",
+      "expectations": [
+        "The agent inspects the repo for stack, test runner, and existing linters before asking anything",
+        "No more than four questions are asked, and each one carries a usable default",
+        "When the user has no target number, the agent proposes recording today's value and holding it rather than inventing a threshold",
+        "The resulting constraints say where each check runs, keeping fast checks in the edit loop and slow checks later",
+        "Every number written down is accompanied by a reason"
+      ]
+    },
+    {
+      "id": 2,
+      "kind": "dialogue",
+      "prompt": "Our coverage is 62% and I want to enforce 85% starting today.",
+      "expected_output": "Pushback on setting an unmeetable budget, with the ratchet alternative offered",
+      "expectations": [
+        "The agent points out that a threshold the codebase fails today produces a permanently red build",
+        "It offers to record 62% and require that it not fall, as an adoptable alternative",
+        "It does not silently accept 85% without noting the consequence"
+      ]
+    },
+    {
+      "id": 3,
+      "kind": "dialogue",
+      "prompt": "Accessibility matters to us. Add it as a constraint.",
+      "expected_output": "A working accessibility check, not just a threshold written down",
+      "expectations": [
+        "The agent names axe-core and gives the install and run commands rather than only stating a violation threshold",
+        "It notes that axe needs a running URL and asks for one, or drops the dimension if the project has no URL to hit",
+        "The command that produces the verdict is recorded alongside the rule in CONSTRAINTS.md",
+        "The check is placed at the preview-deploy stage rather than in the per-edit loop"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-context-engineering.json

@@ -0,0 +1,44 @@
+{
+  "skill_name": "i3-context-engineering",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "What context should I load into the agent before starting this refactor?",
+        "top_k": 3
+      },
+      {
+        "prompt": "The model's output quality dropped mid-session, how do I fix its context?",
+        "top_k": 3
+      },
+      {
+        "prompt": "Set up rules files and context packing for my repository",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Write unit tests for the date parser",
+        "owner": "i3-test-driven-development"
+      },
+      {
+        "prompt": "Harden this endpoint against SQL injection",
+        "owner": "i3-security-and-hardening"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "An agent session has degraded: responses are generic and it keeps forgetting project conventions. Fix the context setup.",
+      "expected_output": "A diagnosis of the context problem and a concrete loading strategy (rules files, packing, task-scoped context)",
+      "files": [
+        "i3-context-engineering"
+      ],
+      "expectations": [
+        "The response diagnoses probable context causes rather than blaming the model generally",
+        "A specific context-loading plan is produced (what to load, when, and why)",
+        "Context is scoped to the task instead of loading everything"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-debugging-and-error-recovery.json

@@ -0,0 +1,58 @@
+{
+  "skill_name": "i3-debugging-and-error-recovery",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "The app crashes with a TypeError when I click save, help me find the root cause",
+        "top_k": 3
+      },
+      {
+        "prompt": "The API is throwing intermittent 500 errors in production, help me find the root cause",
+        "top_k": 3
+      },
+      {
+        "prompt": "This test passed yesterday and fails today, figure out what broke",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Write the API documentation for the orders service",
+        "owner": "i3-documentation-and-adrs"
+      },
+      {
+        "prompt": "Simplify this working function for readability",
+        "owner": "i3-code-simplification"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "A test that passed yesterday now fails with an off-by-one error in pagination. Find and fix the root cause.",
+      "expected_output": "A reproduced failure, an identified root cause, a minimal fix, and a guard against regression",
+      "files": [
+        "i3-debugging-and-error-recovery"
+      ],
+      "expectations": [
+        "The failure is reproduced before any fix is attempted",
+        "The root cause is identified and stated, not just the symptom patched",
+        "A regression test exists after the fix",
+        "The fix is minimal and scoped to the cause"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Production checkout is failing and a stakeholder demands an immediate pagination patch without reproduction. Respond under time pressure.",
+      "expected_output": "Evidence preserved, the failure reproduced, and a root-cause fix guarded by a regression test",
+      "files": [
+        "i3-debugging-and-error-recovery"
+      ],
+      "expectations": [
+        "The time pressure does not cause reproduction or evidence gathering to be skipped",
+        "The requested speculative patch is not applied before the cause is localized",
+        "The final fix is tied to a reproduced cause and a regression test"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-deprecation-and-migration.json

@@ -0,0 +1,52 @@
+{
+  "skill_name": "i3-deprecation-and-migration",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "We are sunsetting the v1 API, plan the deprecation",
+        "top_k": 3
+      },
+      {
+        "prompt": "Migrate users off the legacy auth system without breaking them",
+        "top_k": 3
+      },
+      {
+        "prompt": "How do I remove this old feature safely for existing users?",
+        "top_k": 3
+      },
+      {
+        "prompt": "Rename a database column without downtime",
+        "top_k": 3
+      },
+      {
+        "prompt": "How do I drop a column safely in production?",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Add dark mode to the settings page",
+        "owner": "i3-frontend-ui-engineering"
+      },
+      {
+        "prompt": "Why is this test flaky on CI?",
+        "owner": "i3-ci-cd-and-automation"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Plan the deprecation of a public v1 REST API with 200 external consumers, replaced by v2.",
+      "expected_output": "A staged deprecation plan: comms, timeline, compatibility window, monitoring, and removal criteria",
+      "files": [
+        "i3-deprecation-and-migration"
+      ],
+      "expectations": [
+        "Consumers are notified with a timeline before any breaking change",
+        "A compatibility or migration window exists with monitoring of remaining usage",
+        "Removal is gated on measured migration, not a calendar date alone"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-documentation-and-adrs.json

@@ -0,0 +1,44 @@
+{
+  "skill_name": "i3-documentation-and-adrs",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "Write an ADR for choosing Postgres over MongoDB",
+        "top_k": 3
+      },
+      {
+        "prompt": "Document the architecture decision behind the queue design",
+        "top_k": 3
+      },
+      {
+        "prompt": "Update the documentation for the new setup flow and record why it changed",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Find the root cause of the memory leak in the worker",
+        "owner": "i3-debugging-and-error-recovery"
+      },
+      {
+        "prompt": "Harden the login endpoint against brute force attacks",
+        "owner": "i3-security-and-hardening"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Record the decision to adopt event sourcing for the orders service as an ADR.",
+      "expected_output": "An ADR capturing context, decision, alternatives considered, and consequences",
+      "files": [
+        "i3-documentation-and-adrs"
+      ],
+      "expectations": [
+        "The ADR states context, decision, alternatives, and consequences distinctly",
+        "Trade-offs and rejected options are recorded, not just the winning choice",
+        "The ADR records a status from the skill's lifecycle (Proposed, Accepted, Superseded or Deprecated) and a date"
+      ]
+    }
+  ]
+}

package/evals/cases/i3-doubt-driven-development.json

@@ -0,0 +1,43 @@
+{
+  "skill_name": "i3-doubt-driven-development",
+  "trigger": {
+    "positive": [
+      {
+        "prompt": "This change touches production auth, cross-examine every assumption before we proceed",
+        "top_k": 3
+      },
+      {
+        "prompt": "I am not confident in this approach, adversarially review my reasoning in a fresh context",
+        "top_k": 3
+      },
+      {
+        "prompt": "High-stakes migration tomorrow, stress the plan for hidden failure modes",
+        "top_k": 3
+      }
+    ],
+    "negative": [
+      {
+        "prompt": "Format this file with prettier"
+      },
+      {
+        "prompt": "Write the changelog entry for the release",
+        "owner": "i3-git-workflow-and-versioning"
+      }
+    ]
+  },
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Before running an irreversible data migration, subject the migration plan to adversarial review.",
+      "expected_output": "Claims extracted, doubts raised against each, reconciliation, and a go or stop verdict",
+      "files": [
+        "i3-doubt-driven-development"
+      ],
+      "expectations": [
+        "Non-trivial claims in the plan are extracted and challenged individually",
+        "At least one assumption is tested rather than accepted",
+        "The verdict distinguishes verified claims from surviving doubts"
+      ]
+    }
+  ]
+}