@kolisachint/hoocode-agent 0.5.29 → 0.5.31

package/CHANGELOG.md

@@ -1,5 +1,132 @@
 # Changelog
 
+## [0.5.31] - 2026-08-23
+
+### Added
+
+- `/settings` has an **External tools** category. hoocode ships five optional
+  Rust binaries — `rg`, `fd`, `embsearch`, `webtools`, `voicetools` — that
+  expand what it can do, and nothing in the product ever said so. hoocode works
+  without every one of them, which is exactly why they stayed invisible: search
+  silently got slower, semantic ranking silently never happened, and web and
+  voice were features nobody knew were there. Each row shows live status
+  (installed / found on `PATH` / env override / not installed), and opens a
+  detail view naming what it enables, what hoocode does instead when it is
+  missing, where the release comes from, and the env vars that steer it.
+
+- hoocode can ship skills of its own. It read them from `~/.agents/skills`,
+  `.hoocode/skills`, `.claude/skills` and installed packages — every source
+  except itself — which is why it shipped three subagents and zero skills while
+  telling users skills are the extension unit. Built-ins are catalogued in
+  `core/builtin-skills.ts` and load at lowest precedence, so a skill of the same
+  name from anywhere else wins and the collision is reported.
+
+  A skill costs its description on every turn, so each built-in can be gated on
+  the feature it serves rather than on everyone's token budget.
+
+- The first one: `plugin-authoring`, the craft half of `ProposePlugin`/
+  `UpdatePlugin` — when a capability is worth extracting, naming and describing
+  it so it triggers again, portability rules, and the hook trap where a changed
+  command adds a second hook instead of replacing one. Gated on
+  `enablePluginTools`, which is off by default, so a default session pays
+  nothing for it.
+
+  Built-ins are materialized to a content-addressed cache under
+  `~/.hoocode/cache/builtin-skills/`. A skill is loaded by reading its file, so
+  its location has to be a real path, and the compiled standalone binary has no
+  install directory to read from; materializing the same embedded copy
+  everywhere keeps the skill set identical across npm, pnpm, source and the
+  binary. If the cache cannot be written the built-ins are absent and everything
+  else runs normally. `--no-skills` and `--light` suppress them.
+
+- An agent-selection eval: `bun run agent-eval` scores the built-in agent roster
+  against a gold set of real tasks, reporting how often each agent is chosen
+  when it should be, how often the parent correctly keeps work inline, and a
+  confusion matrix naming which agent loses to which. It reuses the plugin G4
+  trigger harness rather than adding a second one, and describes each agent with
+  the summarized text `<available_agents>` actually emits.
+
+  This exists to settle whether `plan` and `explore` are two agents or one:
+  they ship with the same tools, the same isolation and the same background
+  flag, and `complexity` on the Task tool already expresses the only other
+  difference. That was being argued from intuition; it is now measurable.
+
+### Fixed
+
+- The G4 plugin trigger gate never had a judge. `trigger-eval.ts` takes its
+  model call as a parameter so scoring stays testable, and nothing in the tree
+  ever passed one — so every G4 run has reported `not-run` since it was written.
+  `createLlmTriggerJudge` is that judge, shared with the agent-selection eval.
+
+- `/new-skill`, `/new-agent` and `/new-command` scaffolded different content
+  depending on whether `--platform` was set. Each command has two write paths —
+  the per-vendor emitters and the plain `.hoocode/` writer — and each carried
+  its own copy of the body, which had drifted in both directions: the
+  `.hoocode/` command body documented the `${@:N}` / `${@:N:L}` slice
+  placeholders that the platform path silently omitted, and the `.hoocode/`
+  agent body identified the agent as running inside hoocode where the platform
+  one did not. Both paths read one definition now, and the richer text won in
+  each case.
+
+### Changed
+
+- The four built-in mode prompts have one home. `templates/modes/<mode>/system.md`
+  is now the only copy: `core/mode-prompts.ts` re-exports the embedded copy as
+  `DEFAULT_MODE_PROMPTS` instead of carrying a second, hand-written set. The two
+  had already drifted — `/init` scaffolds the template text into a project, so a
+  user who ran it and a user who did not were getting differently worded mode
+  rules from the same version.
+
+- The `/grill` phases moved out of TypeScript into `templates/prompts/*.md`.
+  They are prose the runtime injects verbatim — no interpolation, no branching —
+  so they are edited as prose now. The message `/grill` produces is unchanged.
+
+- The Task delegation appendix moved to `templates/prompts/task-main.md` and its
+  two `task-background-*` variants. At ~600 tok/turn it is the largest block of
+  always-on text hoocode emits once the Task tool is on, and it had exactly one
+  interpolation slot in otherwise static prose, so a string constant bought
+  nothing. The rendered prompt is byte-identical on both the with- and
+  without-background-agents branches.
+
+- `ProposePlugin` and `UpdatePlugin` shed ~55% of their always-on prompt
+  guidance (~626 to ~301 tok/turn when the plugin system is enabled). What was
+  removed was how-to-author-well guidance, which is not a tool contract and now
+  lives in the `plugin-authoring` skill; what stayed is the trigger, the
+  transparency rule for autonomous authoring, the hook trap, and the two hard
+  prohibitions. Several removed lines also restated the tools' own descriptions
+  or each other.
+
+- Settings rows that are inert without one of those binaries now say so. The
+  `web` and `semantic search` tool-group switches, the web tools timeout and the
+  voice silence window carry a `needs <binary>` marker and explain the fallback
+  and whether hoocode will fetch the binary. The rows stay settable — the
+  setting is what makes hoocode fetch the binary in the first place, and hiding
+  a row when its dependency is missing would recreate the same silence.
+
+## [0.5.30] - 2026-08-23
+
+### Added
+
+- `/settings` prices what it is about to cost you. A line under the pane reports
+  the fixed per-turn surface — system prompt plus active tool schemas, the text
+  re-sent on every request — and each tool carries its own schema price beside
+  its switch, including the tools that are off, where the price is what turning
+  one back on will cost. The numbers are the same measurement
+  `--print-token-surface` prints, re-measured after every change, so a toggle
+  that moves the surface shows what it moved.
+
+- The `light` preset is editable from `/settings`. It was the one low-token
+  control with no home in the pane, which is backwards for the setting whose
+  entire purpose is the token budget. It reads at startup, so it applies on the
+  next session.
+
+- `/settings` grouping: a **Context** category now holds auto-compact, context
+  GC and the light preset. Those three decide the token budget and were in three
+  different places — auto-compact alone at the top level, context GC filed under
+  tool settings, light nowhere. Tool output display joins the truncation caps it
+  belongs with under one **Tool output** row, and every top-level row now opens
+  something rather than cycling in place.
+
 ## [0.5.29] - 2026-08-23
 
 ### Added

package/dist/core/agent-selection-eval.d.ts

@@ -0,0 +1,91 @@
+/**
+ * Does the agent roster steer a dispatch to the right agent?
+ *
+ * The `Task` tool asks the model to pick a `subagent_type` from
+ * `<available_agents>`, and that block contains nothing but each agent's
+ * summarized description. So agent selection is the same question G4 asks about
+ * skills — "given these descriptions and this situation, which one fires?" —
+ * and it reuses the same harness rather than growing a second one.
+ *
+ * What this exists to decide: `plan` and `explore` ship with the same tools, the
+ * same isolation and the same `background` flag, differing only in model tier
+ * and output contract — and `complexity` on the Task tool already expresses the
+ * tier. Whether they are two agents or one is a question about whether the model
+ * can actually tell them apart from their descriptions, which is measurable and
+ * was previously being argued from intuition.
+ *
+ * The candidates use `summarizeAgentDescription`, not the raw frontmatter: the
+ * summary is what the system prompt actually emits, and evaluating the full
+ * description would score text the model never sees.
+ */
+import { type TriggerCandidate, type TriggerCase, type TriggerEvalOutcome, type TriggerJudge } from "./extensions/plugins/trigger-eval.js";
+/** How often one agent was chosen where another was expected. */
+export interface ConfusionEntry {
+    expected: string;
+    actual: string;
+    count: number;
+}
+export interface AgentSelectionReport {
+    corpusHash: string;
+    agents: string[];
+    caseCount: number;
+    /** Cases where the expected agent was chosen, over cases expecting any agent. */
+    accuracy?: number;
+    /** Cases correctly left to the parent, over cases expecting no delegation. */
+    inlineAccuracy?: number;
+    /** Every wrong pick, most frequent first. The pairs here are the finding. */
+    confusion: ConfusionEntry[];
+    /** Per-agent recall: chosen / expected. An agent nobody picks is dead weight. */
+    perAgent: Array<{
+        agent: string;
+        expected: number;
+        chosen: number;
+        recall: number;
+    }>;
+}
+/**
+ * The built-in agents as judge candidates, described exactly as the system
+ * prompt describes them.
+ *
+ * `own: true` for all of them: unlike a plugin eval there is no foreign roster
+ * to discriminate against, so every case is scored against the same closed set.
+ * The `expect: null` cases carry the discriminative half instead — they ask
+ * whether the model declines to delegate work it should keep.
+ */
+export declare function agentCandidates(cwd?: string): TriggerCandidate[];
+/** Load a gold set: `{ "cases": [{ "prompt": "...", "expect": "explore" | null }] }`. */
+export declare function loadAgentCases(file: string): TriggerCase[] | undefined;
+/**
+ * Every expected agent in the gold set must exist in the roster.
+ *
+ * A typo'd or removed agent name would otherwise score as a permanent miss and
+ * read as a description problem, which is the most expensive way to be wrong
+ * about an eval.
+ */
+export declare function validateAgentCases(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[]): string[];
+/**
+ * Turn a scored run into the report that answers the design question.
+ *
+ * `runTriggerEval`'s recall/specificity are the right numbers for a plugin
+ * defending itself against a foreign roster. Here the roster is closed, so the
+ * useful shape is a confusion matrix: which agent loses to which, and how often.
+ * "explore and plan are interchangeable" is a claim about one cell.
+ */
+export declare function summarizeAgentSelection(outcome: Extract<TriggerEvalOutcome, {
+    status: "ran";
+}>, candidates: readonly TriggerCandidate[]): AgentSelectionReport;
+export type AgentSelectionOutcome = {
+    status: "not-run";
+    reason: string;
+} | {
+    status: "ran";
+    report: AgentSelectionReport;
+    outcome: Extract<TriggerEvalOutcome, {
+        status: "ran";
+    }>;
+};
+/** Score the roster against a gold set. Never throws; a missing model is `not-run`. */
+export declare function runAgentSelectionEval(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[] | undefined, judge: TriggerJudge | undefined): Promise<AgentSelectionOutcome>;
+/** Human-readable report for the CLI. */
+export declare function formatAgentSelectionReport(report: AgentSelectionReport): string;
+//# sourceMappingURL=agent-selection-eval.d.ts.map

package/dist/core/agent-selection-eval.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"agent-selection-eval.d.ts","sourceRoot":"","sources":["../../src/core/agent-selection-eval.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAKH,OAAO,EAEN,KAAK,gBAAgB,EACrB,KAAK,WAAW,EAChB,KAAK,kBAAkB,EACvB,KAAK,YAAY,EACjB,MAAM,sCAAsC,CAAC;AAE9C,iEAAiE;AACjE,MAAM,WAAW,cAAc;IAC9B,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;CACd;AAED,MAAM,WAAW,oBAAoB;IACpC,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,SAAS,EAAE,MAAM,CAAC;IAClB,iFAAiF;IACjF,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,8EAA8E;IAC9E,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,6EAA6E;IAC7E,SAAS,EAAE,cAAc,EAAE,CAAC;IAC5B,iFAAiF;IACjF,QAAQ,EAAE,KAAK,CAAC;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;CACrF;AAED;;;;;;;;GAQG;AACH,wBAAgB,eAAe,CAAC,GAAG,GAAE,MAAsB,GAAG,gBAAgB,EAAE,CAU/E;AAED,yFAAyF;AACzF,wBAAgB,cAAc,CAAC,IAAI,EAAE,MAAM,GAAG,WAAW,EAAE,GAAG,SAAS,CAiBtE;AAED;;;;;;GAMG;AACH,wBAAgB,kBAAkB,CAAC,UAAU,EAAE,SAAS,gBAAgB,EAAE,EAAE,KAAK,EAAE,SAAS,WAAW,EAAE,GAAG,MAAM,EAAE,CASnH;AASD;;;;;;;GAOG;AACH,wBAAgB,uBAAuB,CACtC,OAAO,EAAE,OAAO,CAAC,kBAAkB,EAAE;IAAE,MAAM,EAAE,KAAK,CAAA;CAAE,CAAC,EACvD,UAAU,EAAE,SAAS,gBAAgB,EAAE,GACrC,oBAAoB,CAiDtB;AAED,MAAM,MAAM,qBAAqB,GAC9B;IAAE,MAAM,EAAE,SAAS,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,GACrC;IAAE,MAAM,EAAE,KAAK,CAAC;IAAC,MAAM,EAAE,oBAAoB,CAAC;IAAC,OAAO,EAAE,OAAO,CAAC,kBAAkB,EAAE;QAAE,MAAM,EAAE,KAAK,CAAA;KAAE,CAAC,CAAA;CAAE,CAAC;AAE5G,uFAAuF;AACvF,wBAAsB,qBAAqB,CAC1C,UAAU,EAAE,SAAS,gBAAgB,EAAE,EACvC,KAAK,EAAE,SAAS,WAAW,EAAE,GAAG,SAAS,EACzC,KAAK,EAAE,YAAY,GAAG,SAAS,GAC7B,OAAO,CAAC,qBAAqB,CAAC,CAmBhC;AAED,yCAAyC;AACzC,wBAAgB,0BAA0B,CAAC,MAAM,EAAE,oBAAoB,GAAG,MAAM,CAkB/E","sourcesContent":["/**\n * Does the agent roster steer a dispatch to the right agent?\n *\n * The `Task` tool asks the model to pick a `subagent_type` from\n * `<available_agents>`, and that block contains nothing but each agent's\n * summarized description. So agent selection is the same question G4 asks about\n * skills — \"given these descriptions and this situation, which one fires?\" —\n * and it reuses the same harness rather than growing a second one.\n *\n * What this exists to decide: `plan` and `explore` ship with the same tools, the\n * same isolation and the same `background` flag, differing only in model tier\n * and output contract — and `complexity` on the Task tool already expresses the\n * tier. Whether they are two agents or one is a question about whether the model\n * can actually tell them apart from their descriptions, which is measurable and\n * was previously being argued from intuition.\n *\n * The candidates use `summarizeAgentDescription`, not the raw frontmatter: the\n * summary is what the system prompt actually emits, and evaluating the full\n * description would score text the model never sees.\n */\n\nimport { createHash } from \"node:crypto\";\nimport { existsSync, readFileSync } from \"node:fs\";\nimport { loadAgentRegistry, summarizeAgentDescription } from \"./agent-registry.js\";\nimport {\n\trunTriggerEval,\n\ttype TriggerCandidate,\n\ttype TriggerCase,\n\ttype TriggerEvalOutcome,\n\ttype TriggerJudge,\n} from \"./extensions/plugins/trigger-eval.js\";\n\n/** How often one agent was chosen where another was expected. */\nexport interface ConfusionEntry {\n\texpected: string;\n\tactual: string;\n\tcount: number;\n}\n\nexport interface AgentSelectionReport {\n\tcorpusHash: string;\n\tagents: string[];\n\tcaseCount: number;\n\t/** Cases where the expected agent was chosen, over cases expecting any agent. */\n\taccuracy?: number;\n\t/** Cases correctly left to the parent, over cases expecting no delegation. */\n\tinlineAccuracy?: number;\n\t/** Every wrong pick, most frequent first. The pairs here are the finding. */\n\tconfusion: ConfusionEntry[];\n\t/** Per-agent recall: chosen / expected. An agent nobody picks is dead weight. */\n\tperAgent: Array<{ agent: string; expected: number; chosen: number; recall: number }>;\n}\n\n/**\n * The built-in agents as judge candidates, described exactly as the system\n * prompt describes them.\n *\n * `own: true` for all of them: unlike a plugin eval there is no foreign roster\n * to discriminate against, so every case is scored against the same closed set.\n * The `expect: null` cases carry the discriminative half instead — they ask\n * whether the model declines to delegate work it should keep.\n */\nexport function agentCandidates(cwd: string = process.cwd()): TriggerCandidate[] {\n\tconst registry = loadAgentRegistry({ cwd, includeBuiltins: true, includeClaude: false });\n\treturn registry\n\t\t.list()\n\t\t.map((agent) => ({\n\t\t\tname: agent.name,\n\t\t\tdescription: summarizeAgentDescription(agent.description ?? \"\"),\n\t\t\town: true,\n\t\t}))\n\t\t.sort((a, b) => a.name.localeCompare(b.name));\n}\n\n/** Load a gold set: `{ \"cases\": [{ \"prompt\": \"...\", \"expect\": \"explore\" | null }] }`. */\nexport function loadAgentCases(file: string): TriggerCase[] | undefined {\n\tif (!existsSync(file)) return undefined;\n\ttry {\n\t\tconst raw = JSON.parse(readFileSync(file, \"utf-8\")) as { cases?: unknown };\n\t\tif (!Array.isArray(raw.cases)) return undefined;\n\t\tconst cases = raw.cases.filter(\n\t\t\t(c): c is TriggerCase =>\n\t\t\t\t!!c &&\n\t\t\t\ttypeof c === \"object\" &&\n\t\t\t\ttypeof (c as TriggerCase).prompt === \"string\" &&\n\t\t\t\t(c as TriggerCase).prompt.trim().length > 0 &&\n\t\t\t\t((c as TriggerCase).expect === null || typeof (c as TriggerCase).expect === \"string\"),\n\t\t);\n\t\treturn cases.length > 0 ? cases : undefined;\n\t} catch {\n\t\treturn undefined;\n\t}\n}\n\n/**\n * Every expected agent in the gold set must exist in the roster.\n *\n * A typo'd or removed agent name would otherwise score as a permanent miss and\n * read as a description problem, which is the most expensive way to be wrong\n * about an eval.\n */\nexport function validateAgentCases(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[]): string[] {\n\tconst known = new Set(candidates.map((c) => c.name));\n\tconst problems: string[] = [];\n\tfor (const [i, testCase] of cases.entries()) {\n\t\tif (testCase.expect !== null && !known.has(testCase.expect)) {\n\t\t\tproblems.push(`case ${i} expects \"${testCase.expect}\", which is not in the roster`);\n\t\t}\n\t}\n\treturn problems;\n}\n\nfunction hashCorpus(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[]): string {\n\tconst h = createHash(\"sha256\");\n\tfor (const c of candidates) h.update(`${c.name} ${c.description} `);\n\tfor (const c of cases) h.update(`${c.prompt} ${c.expect ?? \"\"} `);\n\treturn h.digest(\"hex\").slice(0, 16);\n}\n\n/**\n * Turn a scored run into the report that answers the design question.\n *\n * `runTriggerEval`'s recall/specificity are the right numbers for a plugin\n * defending itself against a foreign roster. Here the roster is closed, so the\n * useful shape is a confusion matrix: which agent loses to which, and how often.\n * \"explore and plan are interchangeable\" is a claim about one cell.\n */\nexport function summarizeAgentSelection(\n\toutcome: Extract<TriggerEvalOutcome, { status: \"ran\" }>,\n\tcandidates: readonly TriggerCandidate[],\n): AgentSelectionReport {\n\tconst confusion = new Map<string, ConfusionEntry>();\n\tconst expectedCounts = new Map<string, number>();\n\tconst chosenCounts = new Map<string, number>();\n\tlet delegated = 0;\n\tlet delegatedCorrect = 0;\n\tlet inline = 0;\n\tlet inlineCorrect = 0;\n\n\tfor (const result of outcome.record.results) {\n\t\tif (result.expected === null) {\n\t\t\tinline++;\n\t\t\tif (result.actual === null) inlineCorrect++;\n\t\t\tcontinue;\n\t\t}\n\t\tdelegated++;\n\t\texpectedCounts.set(result.expected, (expectedCounts.get(result.expected) ?? 0) + 1);\n\t\tif (result.correct) {\n\t\t\tdelegatedCorrect++;\n\t\t\tchosenCounts.set(result.expected, (chosenCounts.get(result.expected) ?? 0) + 1);\n\t\t\tcontinue;\n\t\t}\n\t\t// \"(none)\" is a real outcome, not a missing value: declining to delegate\n\t\t// work that should have been delegated is a different failure from picking\n\t\t// the wrong agent, and collapsing them would hide which one is happening.\n\t\tconst actual = result.actual ?? \"(none)\";\n\t\tconst key = `${result.expected} ${actual}`;\n\t\tconst entry = confusion.get(key) ?? { expected: result.expected, actual, count: 0 };\n\t\tentry.count++;\n\t\tconfusion.set(key, entry);\n\t}\n\n\tconst perAgent = candidates\n\t\t.map((candidate) => {\n\t\t\tconst expected = expectedCounts.get(candidate.name) ?? 0;\n\t\t\tconst chosen = chosenCounts.get(candidate.name) ?? 0;\n\t\t\treturn { agent: candidate.name, expected, chosen, recall: expected > 0 ? chosen / expected : 0 };\n\t\t})\n\t\t.sort((a, b) => a.recall - b.recall);\n\n\treturn {\n\t\tcorpusHash: outcome.record.corpusHash,\n\t\tagents: candidates.map((c) => c.name),\n\t\tcaseCount: outcome.record.caseCount,\n\t\taccuracy: delegated > 0 ? delegatedCorrect / delegated : undefined,\n\t\tinlineAccuracy: inline > 0 ? inlineCorrect / inline : undefined,\n\t\tconfusion: [...confusion.values()].sort((a, b) => b.count - a.count),\n\t\tperAgent,\n\t};\n}\n\nexport type AgentSelectionOutcome =\n\t| { status: \"not-run\"; reason: string }\n\t| { status: \"ran\"; report: AgentSelectionReport; outcome: Extract<TriggerEvalOutcome, { status: \"ran\" }> };\n\n/** Score the roster against a gold set. Never throws; a missing model is `not-run`. */\nexport async function runAgentSelectionEval(\n\tcandidates: readonly TriggerCandidate[],\n\tcases: readonly TriggerCase[] | undefined,\n\tjudge: TriggerJudge | undefined,\n): Promise<AgentSelectionOutcome> {\n\tif (cases && cases.length > 0) {\n\t\tconst problems = validateAgentCases(candidates, cases);\n\t\tif (problems.length > 0) return { status: \"not-run\", reason: `invalid gold set: ${problems.join(\"; \")}` };\n\t}\n\n\tconst outcome = await runTriggerEval(\"agent-selection\", candidates, cases, judge);\n\tif (outcome.status === \"not-run\") return outcome;\n\n\treturn {\n\t\tstatus: \"ran\",\n\t\treport: {\n\t\t\t...summarizeAgentSelection(outcome, candidates),\n\t\t\t// The shared harness hashes its own way; restate it over exactly what\n\t\t\t// this eval judged so two reports are comparable on their own terms.\n\t\t\tcorpusHash: hashCorpus(candidates, cases ?? []),\n\t\t},\n\t\toutcome,\n\t};\n}\n\n/** Human-readable report for the CLI. */\nexport function formatAgentSelectionReport(report: AgentSelectionReport): string {\n\tconst pct = (v: number | undefined) => (v === undefined ? \"n/a\" : `${Math.round(v * 100)}%`);\n\tconst lines = [\n\t\t`corpus ${report.corpusHash} - ${report.caseCount} case(s) over ${report.agents.length} agent(s): ${report.agents.join(\", \")}`,\n\t\t`delegation accuracy ${pct(report.accuracy)}   inline accuracy ${pct(report.inlineAccuracy)}`,\n\t\t\"\",\n\t\t\"per agent (lowest recall first):\",\n\t];\n\tfor (const row of report.perAgent) {\n\t\tlines.push(`  ${row.agent.padEnd(18)} ${row.chosen}/${row.expected} chosen  (${pct(row.recall)})`);\n\t}\n\tif (report.confusion.length > 0) {\n\t\tlines.push(\"\", \"confusions (expected then actual):\");\n\t\tfor (const entry of report.confusion) {\n\t\t\tlines.push(`  ${entry.expected} => ${entry.actual}   x${entry.count}`);\n\t\t}\n\t}\n\treturn lines.join(\"\\n\");\n}\n"]}

package/dist/core/agent-selection-eval.js

@@ -0,0 +1,186 @@
+/**
+ * Does the agent roster steer a dispatch to the right agent?
+ *
+ * The `Task` tool asks the model to pick a `subagent_type` from
+ * `<available_agents>`, and that block contains nothing but each agent's
+ * summarized description. So agent selection is the same question G4 asks about
+ * skills — "given these descriptions and this situation, which one fires?" —
+ * and it reuses the same harness rather than growing a second one.
+ *
+ * What this exists to decide: `plan` and `explore` ship with the same tools, the
+ * same isolation and the same `background` flag, differing only in model tier
+ * and output contract — and `complexity` on the Task tool already expresses the
+ * tier. Whether they are two agents or one is a question about whether the model
+ * can actually tell them apart from their descriptions, which is measurable and
+ * was previously being argued from intuition.
+ *
+ * The candidates use `summarizeAgentDescription`, not the raw frontmatter: the
+ * summary is what the system prompt actually emits, and evaluating the full
+ * description would score text the model never sees.
+ */
+import { createHash } from "node:crypto";
+import { existsSync, readFileSync } from "node:fs";
+import { loadAgentRegistry, summarizeAgentDescription } from "./agent-registry.js";
+import { runTriggerEval, } from "./extensions/plugins/trigger-eval.js";
+/**
+ * The built-in agents as judge candidates, described exactly as the system
+ * prompt describes them.
+ *
+ * `own: true` for all of them: unlike a plugin eval there is no foreign roster
+ * to discriminate against, so every case is scored against the same closed set.
+ * The `expect: null` cases carry the discriminative half instead — they ask
+ * whether the model declines to delegate work it should keep.
+ */
+export function agentCandidates(cwd = process.cwd()) {
+    const registry = loadAgentRegistry({ cwd, includeBuiltins: true, includeClaude: false });
+    return registry
+        .list()
+        .map((agent) => ({
+        name: agent.name,
+        description: summarizeAgentDescription(agent.description ?? ""),
+        own: true,
+    }))
+        .sort((a, b) => a.name.localeCompare(b.name));
+}
+/** Load a gold set: `{ "cases": [{ "prompt": "...", "expect": "explore" | null }] }`. */
+export function loadAgentCases(file) {
+    if (!existsSync(file))
+        return undefined;
+    try {
+        const raw = JSON.parse(readFileSync(file, "utf-8"));
+        if (!Array.isArray(raw.cases))
+            return undefined;
+        const cases = raw.cases.filter((c) => !!c &&
+            typeof c === "object" &&
+            typeof c.prompt === "string" &&
+            c.prompt.trim().length > 0 &&
+            (c.expect === null || typeof c.expect === "string"));
+        return cases.length > 0 ? cases : undefined;
+    }
+    catch {
+        return undefined;
+    }
+}
+/**
+ * Every expected agent in the gold set must exist in the roster.
+ *
+ * A typo'd or removed agent name would otherwise score as a permanent miss and
+ * read as a description problem, which is the most expensive way to be wrong
+ * about an eval.
+ */
+export function validateAgentCases(candidates, cases) {
+    const known = new Set(candidates.map((c) => c.name));
+    const problems = [];
+    for (const [i, testCase] of cases.entries()) {
+        if (testCase.expect !== null && !known.has(testCase.expect)) {
+            problems.push(`case ${i} expects "${testCase.expect}", which is not in the roster`);
+        }
+    }
+    return problems;
+}
+function hashCorpus(candidates, cases) {
+    const h = createHash("sha256");
+    for (const c of candidates)
+        h.update(`${c.name} ${c.description} `);
+    for (const c of cases)
+        h.update(`${c.prompt} ${c.expect ?? ""} `);
+    return h.digest("hex").slice(0, 16);
+}
+/**
+ * Turn a scored run into the report that answers the design question.
+ *
+ * `runTriggerEval`'s recall/specificity are the right numbers for a plugin
+ * defending itself against a foreign roster. Here the roster is closed, so the
+ * useful shape is a confusion matrix: which agent loses to which, and how often.
+ * "explore and plan are interchangeable" is a claim about one cell.
+ */
+export function summarizeAgentSelection(outcome, candidates) {
+    const confusion = new Map();
+    const expectedCounts = new Map();
+    const chosenCounts = new Map();
+    let delegated = 0;
+    let delegatedCorrect = 0;
+    let inline = 0;
+    let inlineCorrect = 0;
+    for (const result of outcome.record.results) {
+        if (result.expected === null) {
+            inline++;
+            if (result.actual === null)
+                inlineCorrect++;
+            continue;
+        }
+        delegated++;
+        expectedCounts.set(result.expected, (expectedCounts.get(result.expected) ?? 0) + 1);
+        if (result.correct) {
+            delegatedCorrect++;
+            chosenCounts.set(result.expected, (chosenCounts.get(result.expected) ?? 0) + 1);
+            continue;
+        }
+        // "(none)" is a real outcome, not a missing value: declining to delegate
+        // work that should have been delegated is a different failure from picking
+        // the wrong agent, and collapsing them would hide which one is happening.
+        const actual = result.actual ?? "(none)";
+        const key = `${result.expected} ${actual}`;
+        const entry = confusion.get(key) ?? { expected: result.expected, actual, count: 0 };
+        entry.count++;
+        confusion.set(key, entry);
+    }
+    const perAgent = candidates
+        .map((candidate) => {
+        const expected = expectedCounts.get(candidate.name) ?? 0;
+        const chosen = chosenCounts.get(candidate.name) ?? 0;
+        return { agent: candidate.name, expected, chosen, recall: expected > 0 ? chosen / expected : 0 };
+    })
+        .sort((a, b) => a.recall - b.recall);
+    return {
+        corpusHash: outcome.record.corpusHash,
+        agents: candidates.map((c) => c.name),
+        caseCount: outcome.record.caseCount,
+        accuracy: delegated > 0 ? delegatedCorrect / delegated : undefined,
+        inlineAccuracy: inline > 0 ? inlineCorrect / inline : undefined,
+        confusion: [...confusion.values()].sort((a, b) => b.count - a.count),
+        perAgent,
+    };
+}
+/** Score the roster against a gold set. Never throws; a missing model is `not-run`. */
+export async function runAgentSelectionEval(candidates, cases, judge) {
+    if (cases && cases.length > 0) {
+        const problems = validateAgentCases(candidates, cases);
+        if (problems.length > 0)
+            return { status: "not-run", reason: `invalid gold set: ${problems.join("; ")}` };
+    }
+    const outcome = await runTriggerEval("agent-selection", candidates, cases, judge);
+    if (outcome.status === "not-run")
+        return outcome;
+    return {
+        status: "ran",
+        report: {
+            ...summarizeAgentSelection(outcome, candidates),
+            // The shared harness hashes its own way; restate it over exactly what
+            // this eval judged so two reports are comparable on their own terms.
+            corpusHash: hashCorpus(candidates, cases ?? []),
+        },
+        outcome,
+    };
+}
+/** Human-readable report for the CLI. */
+export function formatAgentSelectionReport(report) {
+    const pct = (v) => (v === undefined ? "n/a" : `${Math.round(v * 100)}%`);
+    const lines = [
+        `corpus ${report.corpusHash} - ${report.caseCount} case(s) over ${report.agents.length} agent(s): ${report.agents.join(", ")}`,
+        `delegation accuracy ${pct(report.accuracy)}   inline accuracy ${pct(report.inlineAccuracy)}`,
+        "",
+        "per agent (lowest recall first):",
+    ];
+    for (const row of report.perAgent) {
+        lines.push(`  ${row.agent.padEnd(18)} ${row.chosen}/${row.expected} chosen  (${pct(row.recall)})`);
+    }
+    if (report.confusion.length > 0) {
+        lines.push("", "confusions (expected then actual):");
+        for (const entry of report.confusion) {
+            lines.push(`  ${entry.expected} => ${entry.actual}   x${entry.count}`);
+        }
+    }
+    return lines.join("\n");
+}
+//# sourceMappingURL=agent-selection-eval.js.map

package/dist/core/agent-selection-eval.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"agent-selection-eval.js","sourceRoot":"","sources":["../../src/core/agent-selection-eval.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;GAmBG;AAEH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,UAAU,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACnD,OAAO,EAAE,iBAAiB,EAAE,yBAAyB,EAAE,MAAM,qBAAqB,CAAC;AACnF,OAAO,EACN,cAAc,GAKd,MAAM,sCAAsC,CAAC;AAuB9C;;;;;;;;GAQG;AACH,MAAM,UAAU,eAAe,CAAC,GAAG,GAAW,OAAO,CAAC,GAAG,EAAE,EAAsB;IAChF,MAAM,QAAQ,GAAG,iBAAiB,CAAC,EAAE,GAAG,EAAE,eAAe,EAAE,IAAI,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC,CAAC;IACzF,OAAO,QAAQ;SACb,IAAI,EAAE;SACN,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;QAChB,IAAI,EAAE,KAAK,CAAC,IAAI;QAChB,WAAW,EAAE,yBAAyB,CAAC,KAAK,CAAC,WAAW,IAAI,EAAE,CAAC;QAC/D,GAAG,EAAE,IAAI;KACT,CAAC,CAAC;SACF,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;AAAA,CAC/C;AAED,yFAAyF;AACzF,MAAM,UAAU,cAAc,CAAC,IAAY,EAA6B;IACvE,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;QAAE,OAAO,SAAS,CAAC;IACxC,IAAI,CAAC;QACJ,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,IAAI,EAAE,OAAO,CAAC,CAAwB,CAAC;QAC3E,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC;YAAE,OAAO,SAAS,CAAC;QAChD,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,MAAM,CAC7B,CAAC,CAAC,EAAoB,EAAE,CACvB,CAAC,CAAC,CAAC;YACH,OAAO,CAAC,KAAK,QAAQ;YACrB,OAAQ,CAAiB,CAAC,MAAM,KAAK,QAAQ;YAC5C,CAAiB,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC;YAC3C,CAAE,CAAiB,CAAC,MAAM,KAAK,IAAI,IAAI,OAAQ,CAAiB,CAAC,MAAM,KAAK,QAAQ,CAAC,CACtF,CAAC;QACF,OAAO,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7C,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,SAAS,CAAC;IAClB,CAAC;AAAA,CACD;AAED;;;;;;GAMG;AACH,MAAM,UAAU,kBAAkB,CAAC,UAAuC,EAAE,KAA6B,EAAY;IACpH,MAAM,KAAK,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;IACrD,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,KAAK,MAAM,CAAC,CAAC,EAAE,QAAQ,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;QAC7C,IAAI,QAAQ,CAAC,MAAM,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC7D,QAAQ,CAAC,IAAI,CAAC,QAAQ,CAAC,aAAa,QAAQ,CAAC,MAAM,+BAA+B,CAAC,CAAC;QACrF,CAAC;IACF,CAAC;IACD,OAAO,QAAQ,CAAC;AAAA,CAChB;AAED,SAAS,UAAU,CAAC,UAAuC,EAAE,KAA6B,EAAU;IACnG,MAAM,CAAC,GAAG,UAAU,CAAC,QAAQ,CAAC,CAAC;IAC/B,KAAK,MAAM,CAAC,IAAI,UAAU;QAAE,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,WAAW,GAAG,CAAC,CAAC;IACpE,KAAK,MAAM,CAAC,IAAI,KAAK;QAAE,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,MAAM,IAAI,EAAE,GAAG,CAAC,CAAC;IAClE,OAAO,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AAAA,CACpC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,uBAAuB,CACtC,OAAuD,EACvD,UAAuC,EAChB;IACvB,MAAM,SAAS,GAAG,IAAI,GAAG,EAA0B,CAAC;IACpD,MAAM,cAAc,GAAG,IAAI,GAAG,EAAkB,CAAC;IACjD,MAAM,YAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;IAC/C,IAAI,SAAS,GAAG,CAAC,CAAC;IAClB,IAAI,gBAAgB,GAAG,CAAC,CAAC;IACzB,IAAI,MAAM,GAAG,CAAC,CAAC;IACf,IAAI,aAAa,GAAG,CAAC,CAAC;IAEtB,KAAK,MAAM,MAAM,IAAI,OAAO,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC;QAC7C,IAAI,MAAM,CAAC,QAAQ,KAAK,IAAI,EAAE,CAAC;YAC9B,MAAM,EAAE,CAAC;YACT,IAAI,MAAM,CAAC,MAAM,KAAK,IAAI;gBAAE,aAAa,EAAE,CAAC;YAC5C,SAAS;QACV,CAAC;QACD,SAAS,EAAE,CAAC;QACZ,cAAc,CAAC,GAAG,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,cAAc,CAAC,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QACpF,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;YACpB,gBAAgB,EAAE,CAAC;YACnB,YAAY,CAAC,GAAG,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC,YAAY,CAAC,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YAChF,SAAS;QACV,CAAC;QACD,yEAAyE;QACzE,2EAA2E;QAC3E,0EAA0E;QAC1E,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,IAAI,QAAQ,CAAC;QACzC,MAAM,GAAG,GAAG,GAAG,MAAM,CAAC,QAAQ,IAAI,MAAM,EAAE,CAAC;QAC3C,MAAM,KAAK,GAAG,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE,MAAM,CAAC,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,EAAE,CAAC;QACpF,KAAK,CAAC,KAAK,EAAE,CAAC;QACd,SAAS,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;IAC3B,CAAC;IAED,MAAM,QAAQ,GAAG,UAAU;SACzB,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC;QACnB,MAAM,QAAQ,GAAG,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACzD,MAAM,MAAM,GAAG,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACrD,OAAO,EAAE,KAAK,EAAE,SAAS,CAAC,IAAI,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,QAAQ,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;IAAA,CACjG,CAAC;SACD,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC;IAEtC,OAAO;QACN,UAAU,EAAE,OAAO,CAAC,MAAM,CAAC,UAAU;QACrC,MAAM,EAAE,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC;QACrC,SAAS,EAAE,OAAO,CAAC,MAAM,CAAC,SAAS;QACnC,QAAQ,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,gBAAgB,GAAG,SAAS,CAAC,CAAC,CAAC,SAAS;QAClE,cAAc,EAAE,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,aAAa,GAAG,MAAM,CAAC,CAAC,CAAC,SAAS;QAC/D,SAAS,EAAE,CAAC,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC;QACpE,QAAQ;KACR,CAAC;AAAA,CACF;AAMD,uFAAuF;AACvF,MAAM,CAAC,KAAK,UAAU,qBAAqB,CAC1C,UAAuC,EACvC,KAAyC,EACzC,KAA+B,EACE;IACjC,IAAI,KAAK,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC/B,MAAM,QAAQ,GAAG,kBAAkB,CAAC,UAAU,EAAE,KAAK,CAAC,CAAC;QACvD,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC;YAAE,OAAO,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,qBAAqB,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;IAC3G,CAAC;IAED,MAAM,OAAO,GAAG,MAAM,cAAc,CAAC,iBAAiB,EAAE,UAAU,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;IAClF,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS;QAAE,OAAO,OAAO,CAAC;IAEjD,OAAO;QACN,MAAM,EAAE,KAAK;QACb,MAAM,EAAE;YACP,GAAG,uBAAuB,CAAC,OAAO,EAAE,UAAU,CAAC;YAC/C,sEAAsE;YACtE,qEAAqE;YACrE,UAAU,EAAE,UAAU,CAAC,UAAU,EAAE,KAAK,IAAI,EAAE,CAAC;SAC/C;QACD,OAAO;KACP,CAAC;AAAA,CACF;AAED,yCAAyC;AACzC,MAAM,UAAU,0BAA0B,CAAC,MAA4B,EAAU;IAChF,MAAM,GAAG,GAAG,CAAC,CAAqB,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC;IAC7F,MAAM,KAAK,GAAG;QACb,UAAU,MAAM,CAAC,UAAU,MAAM,MAAM,CAAC,SAAS,iBAAiB,MAAM,CAAC,MAAM,CAAC,MAAM,cAAc,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE;QAC9H,uBAAuB,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,sBAAsB,GAAG,CAAC,MAAM,CAAC,cAAc,CAAC,EAAE;QAC7F,EAAE;QACF,kCAAkC;KAClC,CAAC;IACF,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,QAAQ,EAAE,CAAC;QACnC,KAAK,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,GAAG,CAAC,MAAM,IAAI,GAAG,CAAC,QAAQ,aAAa,GAAG,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;IACpG,CAAC;IACD,IAAI,MAAM,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACjC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,oCAAoC,CAAC,CAAC;QACrD,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;YACtC,KAAK,CAAC,IAAI,CAAC,KAAK,KAAK,CAAC,QAAQ,OAAO,KAAK,CAAC,MAAM,OAAO,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;QACxE,CAAC;IACF,CAAC;IACD,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAAA,CACxB","sourcesContent":["/**\n * Does the agent roster steer a dispatch to the right agent?\n *\n * The `Task` tool asks the model to pick a `subagent_type` from\n * `<available_agents>`, and that block contains nothing but each agent's\n * summarized description. So agent selection is the same question G4 asks about\n * skills — \"given these descriptions and this situation, which one fires?\" —\n * and it reuses the same harness rather than growing a second one.\n *\n * What this exists to decide: `plan` and `explore` ship with the same tools, the\n * same isolation and the same `background` flag, differing only in model tier\n * and output contract — and `complexity` on the Task tool already expresses the\n * tier. Whether they are two agents or one is a question about whether the model\n * can actually tell them apart from their descriptions, which is measurable and\n * was previously being argued from intuition.\n *\n * The candidates use `summarizeAgentDescription`, not the raw frontmatter: the\n * summary is what the system prompt actually emits, and evaluating the full\n * description would score text the model never sees.\n */\n\nimport { createHash } from \"node:crypto\";\nimport { existsSync, readFileSync } from \"node:fs\";\nimport { loadAgentRegistry, summarizeAgentDescription } from \"./agent-registry.js\";\nimport {\n\trunTriggerEval,\n\ttype TriggerCandidate,\n\ttype TriggerCase,\n\ttype TriggerEvalOutcome,\n\ttype TriggerJudge,\n} from \"./extensions/plugins/trigger-eval.js\";\n\n/** How often one agent was chosen where another was expected. */\nexport interface ConfusionEntry {\n\texpected: string;\n\tactual: string;\n\tcount: number;\n}\n\nexport interface AgentSelectionReport {\n\tcorpusHash: string;\n\tagents: string[];\n\tcaseCount: number;\n\t/** Cases where the expected agent was chosen, over cases expecting any agent. */\n\taccuracy?: number;\n\t/** Cases correctly left to the parent, over cases expecting no delegation. */\n\tinlineAccuracy?: number;\n\t/** Every wrong pick, most frequent first. The pairs here are the finding. */\n\tconfusion: ConfusionEntry[];\n\t/** Per-agent recall: chosen / expected. An agent nobody picks is dead weight. */\n\tperAgent: Array<{ agent: string; expected: number; chosen: number; recall: number }>;\n}\n\n/**\n * The built-in agents as judge candidates, described exactly as the system\n * prompt describes them.\n *\n * `own: true` for all of them: unlike a plugin eval there is no foreign roster\n * to discriminate against, so every case is scored against the same closed set.\n * The `expect: null` cases carry the discriminative half instead — they ask\n * whether the model declines to delegate work it should keep.\n */\nexport function agentCandidates(cwd: string = process.cwd()): TriggerCandidate[] {\n\tconst registry = loadAgentRegistry({ cwd, includeBuiltins: true, includeClaude: false });\n\treturn registry\n\t\t.list()\n\t\t.map((agent) => ({\n\t\t\tname: agent.name,\n\t\t\tdescription: summarizeAgentDescription(agent.description ?? \"\"),\n\t\t\town: true,\n\t\t}))\n\t\t.sort((a, b) => a.name.localeCompare(b.name));\n}\n\n/** Load a gold set: `{ \"cases\": [{ \"prompt\": \"...\", \"expect\": \"explore\" | null }] }`. */\nexport function loadAgentCases(file: string): TriggerCase[] | undefined {\n\tif (!existsSync(file)) return undefined;\n\ttry {\n\t\tconst raw = JSON.parse(readFileSync(file, \"utf-8\")) as { cases?: unknown };\n\t\tif (!Array.isArray(raw.cases)) return undefined;\n\t\tconst cases = raw.cases.filter(\n\t\t\t(c): c is TriggerCase =>\n\t\t\t\t!!c &&\n\t\t\t\ttypeof c === \"object\" &&\n\t\t\t\ttypeof (c as TriggerCase).prompt === \"string\" &&\n\t\t\t\t(c as TriggerCase).prompt.trim().length > 0 &&\n\t\t\t\t((c as TriggerCase).expect === null || typeof (c as TriggerCase).expect === \"string\"),\n\t\t);\n\t\treturn cases.length > 0 ? cases : undefined;\n\t} catch {\n\t\treturn undefined;\n\t}\n}\n\n/**\n * Every expected agent in the gold set must exist in the roster.\n *\n * A typo'd or removed agent name would otherwise score as a permanent miss and\n * read as a description problem, which is the most expensive way to be wrong\n * about an eval.\n */\nexport function validateAgentCases(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[]): string[] {\n\tconst known = new Set(candidates.map((c) => c.name));\n\tconst problems: string[] = [];\n\tfor (const [i, testCase] of cases.entries()) {\n\t\tif (testCase.expect !== null && !known.has(testCase.expect)) {\n\t\t\tproblems.push(`case ${i} expects \"${testCase.expect}\", which is not in the roster`);\n\t\t}\n\t}\n\treturn problems;\n}\n\nfunction hashCorpus(candidates: readonly TriggerCandidate[], cases: readonly TriggerCase[]): string {\n\tconst h = createHash(\"sha256\");\n\tfor (const c of candidates) h.update(`${c.name} ${c.description} `);\n\tfor (const c of cases) h.update(`${c.prompt} ${c.expect ?? \"\"} `);\n\treturn h.digest(\"hex\").slice(0, 16);\n}\n\n/**\n * Turn a scored run into the report that answers the design question.\n *\n * `runTriggerEval`'s recall/specificity are the right numbers for a plugin\n * defending itself against a foreign roster. Here the roster is closed, so the\n * useful shape is a confusion matrix: which agent loses to which, and how often.\n * \"explore and plan are interchangeable\" is a claim about one cell.\n */\nexport function summarizeAgentSelection(\n\toutcome: Extract<TriggerEvalOutcome, { status: \"ran\" }>,\n\tcandidates: readonly TriggerCandidate[],\n): AgentSelectionReport {\n\tconst confusion = new Map<string, ConfusionEntry>();\n\tconst expectedCounts = new Map<string, number>();\n\tconst chosenCounts = new Map<string, number>();\n\tlet delegated = 0;\n\tlet delegatedCorrect = 0;\n\tlet inline = 0;\n\tlet inlineCorrect = 0;\n\n\tfor (const result of outcome.record.results) {\n\t\tif (result.expected === null) {\n\t\t\tinline++;\n\t\t\tif (result.actual === null) inlineCorrect++;\n\t\t\tcontinue;\n\t\t}\n\t\tdelegated++;\n\t\texpectedCounts.set(result.expected, (expectedCounts.get(result.expected) ?? 0) + 1);\n\t\tif (result.correct) {\n\t\t\tdelegatedCorrect++;\n\t\t\tchosenCounts.set(result.expected, (chosenCounts.get(result.expected) ?? 0) + 1);\n\t\t\tcontinue;\n\t\t}\n\t\t// \"(none)\" is a real outcome, not a missing value: declining to delegate\n\t\t// work that should have been delegated is a different failure from picking\n\t\t// the wrong agent, and collapsing them would hide which one is happening.\n\t\tconst actual = result.actual ?? \"(none)\";\n\t\tconst key = `${result.expected} ${actual}`;\n\t\tconst entry = confusion.get(key) ?? { expected: result.expected, actual, count: 0 };\n\t\tentry.count++;\n\t\tconfusion.set(key, entry);\n\t}\n\n\tconst perAgent = candidates\n\t\t.map((candidate) => {\n\t\t\tconst expected = expectedCounts.get(candidate.name) ?? 0;\n\t\t\tconst chosen = chosenCounts.get(candidate.name) ?? 0;\n\t\t\treturn { agent: candidate.name, expected, chosen, recall: expected > 0 ? chosen / expected : 0 };\n\t\t})\n\t\t.sort((a, b) => a.recall - b.recall);\n\n\treturn {\n\t\tcorpusHash: outcome.record.corpusHash,\n\t\tagents: candidates.map((c) => c.name),\n\t\tcaseCount: outcome.record.caseCount,\n\t\taccuracy: delegated > 0 ? delegatedCorrect / delegated : undefined,\n\t\tinlineAccuracy: inline > 0 ? inlineCorrect / inline : undefined,\n\t\tconfusion: [...confusion.values()].sort((a, b) => b.count - a.count),\n\t\tperAgent,\n\t};\n}\n\nexport type AgentSelectionOutcome =\n\t| { status: \"not-run\"; reason: string }\n\t| { status: \"ran\"; report: AgentSelectionReport; outcome: Extract<TriggerEvalOutcome, { status: \"ran\" }> };\n\n/** Score the roster against a gold set. Never throws; a missing model is `not-run`. */\nexport async function runAgentSelectionEval(\n\tcandidates: readonly TriggerCandidate[],\n\tcases: readonly TriggerCase[] | undefined,\n\tjudge: TriggerJudge | undefined,\n): Promise<AgentSelectionOutcome> {\n\tif (cases && cases.length > 0) {\n\t\tconst problems = validateAgentCases(candidates, cases);\n\t\tif (problems.length > 0) return { status: \"not-run\", reason: `invalid gold set: ${problems.join(\"; \")}` };\n\t}\n\n\tconst outcome = await runTriggerEval(\"agent-selection\", candidates, cases, judge);\n\tif (outcome.status === \"not-run\") return outcome;\n\n\treturn {\n\t\tstatus: \"ran\",\n\t\treport: {\n\t\t\t...summarizeAgentSelection(outcome, candidates),\n\t\t\t// The shared harness hashes its own way; restate it over exactly what\n\t\t\t// this eval judged so two reports are comparable on their own terms.\n\t\t\tcorpusHash: hashCorpus(candidates, cases ?? []),\n\t\t},\n\t\toutcome,\n\t};\n}\n\n/** Human-readable report for the CLI. */\nexport function formatAgentSelectionReport(report: AgentSelectionReport): string {\n\tconst pct = (v: number | undefined) => (v === undefined ? \"n/a\" : `${Math.round(v * 100)}%`);\n\tconst lines = [\n\t\t`corpus ${report.corpusHash} - ${report.caseCount} case(s) over ${report.agents.length} agent(s): ${report.agents.join(\", \")}`,\n\t\t`delegation accuracy ${pct(report.accuracy)}   inline accuracy ${pct(report.inlineAccuracy)}`,\n\t\t\"\",\n\t\t\"per agent (lowest recall first):\",\n\t];\n\tfor (const row of report.perAgent) {\n\t\tlines.push(`  ${row.agent.padEnd(18)} ${row.chosen}/${row.expected} chosen  (${pct(row.recall)})`);\n\t}\n\tif (report.confusion.length > 0) {\n\t\tlines.push(\"\", \"confusions (expected then actual):\");\n\t\tfor (const entry of report.confusion) {\n\t\t\tlines.push(`  ${entry.expected} => ${entry.actual}   x${entry.count}`);\n\t\t}\n\t}\n\treturn lines.join(\"\\n\");\n}\n"]}

package/dist/core/builtin-skills.d.ts

@@ -0,0 +1,61 @@
+/**
+ * Skills hoocode itself ships.
+ *
+ * hoocode reads skills from `~/.agents/skills`, `.hoocode/skills`, `.claude/skills`
+ * and installed packages — every source except its own. That gap is why it
+ * shipped three subagents and zero skills while telling users skills are the
+ * extension unit: there was simply nowhere for a first-party skill to live.
+ *
+ * The obstacle is that a skill's `<location>` has to be a real readable path —
+ * the model loads a skill by `read`ing it — and the Bun-compiled binary has no
+ * `templates/` beside it. So rather than resolve the package directory (which
+ * differs across npm/pnpm/source/binary layouts and would give the compiled
+ * binary a silently degraded skill set), every install materializes the same
+ * embedded copy into a cache directory. One code path, same behaviour
+ * everywhere.
+ *
+ * The cache is keyed by content hash, so an upgrade writes a new directory and
+ * a dev build that changes a skill without changing the version still takes
+ * effect. It is a cache, not user-editable state: `~/.agents/skills` is where a
+ * user's own skills go, and nothing here ever writes there.
+ */
+/** What decides whether a built-in skill is registered this session. */
+export interface BuiltinSkillGate {
+    /** The `enablePluginTools` setting (the plugin system's master switch). */
+    enablePluginTools: boolean;
+}
+export interface BuiltinSkill {
+    /** Directory name under `templates/skills`, and the skill's own name. */
+    name: string;
+    /** Why hoocode ships it. Documentation, and the catalog test reads it. */
+    summary: string;
+    /**
+     * Registered only when this returns true. A skill costs its description on
+     * every turn, so one that only makes sense alongside a feature rides that
+     * feature's switch rather than the default user's token budget.
+     *
+     * Omit for a skill that should always be available.
+     */
+    gate?: (options: BuiltinSkillGate) => boolean;
+}
+export declare const BUILTIN_SKILLS: readonly BuiltinSkill[];
+/** Root of the materialized copy for the current content. */
+export declare function builtinSkillsCacheDir(agentDir?: string): string;
+/**
+ * Write the embedded skills to the cache directory if they are not already
+ * there, and return its path.
+ *
+ * Returns null when nothing could be written — a read-only home, a full disk.
+ * That is a degraded session, not a broken one: the caller contributes no skill
+ * paths and hoocode runs exactly as it did before these existed.
+ */
+export declare function materializeBuiltinSkills(agentDir?: string): string | null;
+/**
+ * The skill directories to load this session, after gating.
+ *
+ * Returns per-skill directories rather than the root so a gated-off skill is
+ * genuinely absent rather than loaded and filtered later — the load is what
+ * costs the description on every turn.
+ */
+export declare function builtinSkillPaths(gate: BuiltinSkillGate, agentDir?: string): string[];
+//# sourceMappingURL=builtin-skills.d.ts.map

package/dist/core/builtin-skills.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"builtin-skills.d.ts","sourceRoot":"","sources":["../../src/core/builtin-skills.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAQH,wEAAwE;AACxE,MAAM,WAAW,gBAAgB;IAChC,2EAA2E;IAC3E,iBAAiB,EAAE,OAAO,CAAC;CAC3B;AAED,MAAM,WAAW,YAAY;IAC5B,yEAAyE;IACzE,IAAI,EAAE,MAAM,CAAC;IACb,0EAA0E;IAC1E,OAAO,EAAE,MAAM,CAAC;IAChB;;;;;;OAMG;IACH,IAAI,CAAC,EAAE,CAAC,OAAO,EAAE,gBAAgB,KAAK,OAAO,CAAC;CAC9C;AAED,eAAO,MAAM,cAAc,EAAE,SAAS,YAAY,EASjD,CAAC;AAcF,6DAA6D;AAC7D,wBAAgB,qBAAqB,CAAC,QAAQ,GAAE,MAAsB,GAAG,MAAM,CAE9E;AAED;;;;;;;GAOG;AACH,wBAAgB,wBAAwB,CAAC,QAAQ,GAAE,MAAsB,GAAG,MAAM,GAAG,IAAI,CA2BxF;AAED;;;;;;GAMG;AACH,wBAAgB,iBAAiB,CAAC,IAAI,EAAE,gBAAgB,EAAE,QAAQ,GAAE,MAAsB,GAAG,MAAM,EAAE,CAQpG","sourcesContent":["/**\n * Skills hoocode itself ships.\n *\n * hoocode reads skills from `~/.agents/skills`, `.hoocode/skills`, `.claude/skills`\n * and installed packages — every source except its own. That gap is why it\n * shipped three subagents and zero skills while telling users skills are the\n * extension unit: there was simply nowhere for a first-party skill to live.\n *\n * The obstacle is that a skill's `<location>` has to be a real readable path —\n * the model loads a skill by `read`ing it — and the Bun-compiled binary has no\n * `templates/` beside it. So rather than resolve the package directory (which\n * differs across npm/pnpm/source/binary layouts and would give the compiled\n * binary a silently degraded skill set), every install materializes the same\n * embedded copy into a cache directory. One code path, same behaviour\n * everywhere.\n *\n * The cache is keyed by content hash, so an upgrade writes a new directory and\n * a dev build that changes a skill without changing the version still takes\n * effect. It is a cache, not user-editable state: `~/.agents/skills` is where a\n * user's own skills go, and nothing here ever writes there.\n */\n\nimport { createHash } from \"node:crypto\";\nimport { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from \"node:fs\";\nimport { dirname, join } from \"node:path\";\nimport { getAgentDir } from \"../config.js\";\nimport { EMBEDDED_SKILLS } from \"../init-templates.generated.js\";\n\n/** What decides whether a built-in skill is registered this session. */\nexport interface BuiltinSkillGate {\n\t/** The `enablePluginTools` setting (the plugin system's master switch). */\n\tenablePluginTools: boolean;\n}\n\nexport interface BuiltinSkill {\n\t/** Directory name under `templates/skills`, and the skill's own name. */\n\tname: string;\n\t/** Why hoocode ships it. Documentation, and the catalog test reads it. */\n\tsummary: string;\n\t/**\n\t * Registered only when this returns true. A skill costs its description on\n\t * every turn, so one that only makes sense alongside a feature rides that\n\t * feature's switch rather than the default user's token budget.\n\t *\n\t * Omit for a skill that should always be available.\n\t */\n\tgate?: (options: BuiltinSkillGate) => boolean;\n}\n\nexport const BUILTIN_SKILLS: readonly BuiltinSkill[] = [\n\t{\n\t\tname: \"plugin-authoring\",\n\t\tsummary:\n\t\t\t\"The craft half of ProposePlugin/UpdatePlugin: when a capability is worth extracting, naming it so it triggers again, portability, and the hook trap.\",\n\t\t// Useless without the tools it describes, and those are off by default,\n\t\t// so this costs nothing for a user who never enables the plugin system.\n\t\tgate: (options) => options.enablePluginTools,\n\t},\n];\n\n/** Stable short hash of the embedded skill tree; the cache directory's name. */\nfunction contentHash(): string {\n\tconst hash = createHash(\"sha256\");\n\tfor (const key of Object.keys(EMBEDDED_SKILLS).sort()) {\n\t\thash.update(key);\n\t\thash.update(\"\\0\");\n\t\thash.update(EMBEDDED_SKILLS[key] ?? \"\");\n\t\thash.update(\"\\0\");\n\t}\n\treturn hash.digest(\"hex\").slice(0, 12);\n}\n\n/** Root of the materialized copy for the current content. */\nexport function builtinSkillsCacheDir(agentDir: string = getAgentDir()): string {\n\treturn join(agentDir, \"cache\", \"builtin-skills\", contentHash());\n}\n\n/**\n * Write the embedded skills to the cache directory if they are not already\n * there, and return its path.\n *\n * Returns null when nothing could be written — a read-only home, a full disk.\n * That is a degraded session, not a broken one: the caller contributes no skill\n * paths and hoocode runs exactly as it did before these existed.\n */\nexport function materializeBuiltinSkills(agentDir: string = getAgentDir()): string | null {\n\tconst root = builtinSkillsCacheDir(agentDir);\n\ttry {\n\t\tfor (const [relativePath, content] of Object.entries(EMBEDDED_SKILLS)) {\n\t\t\tconst target = join(root, relativePath);\n\t\t\t// Content is hash-addressed, so an existing file with the right size is\n\t\t\t// already correct; re-reading beats re-writing on every startup.\n\t\t\tif (existsSync(target) && readFileSync(target, \"utf-8\") === content) continue;\n\t\t\tmkdirSync(dirname(target), { recursive: true });\n\t\t\t// Write-then-rename so a killed process never leaves a half-written\n\t\t\t// SKILL.md that would parse as a malformed skill on the next run.\n\t\t\tconst temp = `${target}.${process.pid}.tmp`;\n\t\t\twriteFileSync(temp, content, \"utf-8\");\n\t\t\trenameSync(temp, target);\n\t\t}\n\t\treturn root;\n\t} catch {\n\t\t// Drop a partial tree so the next run rebuilds it rather than loading a\n\t\t// half-written skill. The cleanup gets its own guard: `force` swallows\n\t\t// ENOENT but not ENOTDIR, and a cleanup that throws would turn the\n\t\t// degraded path back into a crash — which is the failure this whole\n\t\t// branch exists to prevent.\n\t\ttry {\n\t\t\trmSync(root, { recursive: true, force: true });\n\t\t} catch {}\n\t\treturn null;\n\t}\n}\n\n/**\n * The skill directories to load this session, after gating.\n *\n * Returns per-skill directories rather than the root so a gated-off skill is\n * genuinely absent rather than loaded and filtered later — the load is what\n * costs the description on every turn.\n */\nexport function builtinSkillPaths(gate: BuiltinSkillGate, agentDir: string = getAgentDir()): string[] {\n\tconst enabled = BUILTIN_SKILLS.filter((skill) => !skill.gate || skill.gate(gate));\n\tif (enabled.length === 0) return [];\n\n\tconst root = materializeBuiltinSkills(agentDir);\n\tif (!root) return [];\n\n\treturn enabled.map((skill) => join(root, skill.name)).filter((dir) => existsSync(dir));\n}\n"]}