oh-my-knowledge 0.32.0 → 0.33.0

package/README.md

@@ -9,7 +9,7 @@
 **English** | [简体中文](./README.zh.md)
 
 **Did your prompt actually get better?**
-A/B test your prompts and skills with statistical rigor — bootstrap CI, Krippendorff α, length-debias — all on by default.
+A/B test your prompts and skills with statistical rigor — bootstrap CI and length-debias on by default, Krippendorff α the moment you add a gold set.
 
 ![omk report — verdict pill "v2 is clearly better than v1 — ready to ship"](./assets/screenshots/report-overview.png)
 
@@ -23,6 +23,8 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
 
+> The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
+
 Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
 
 Deeper: [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors & artifact layout](docs/reference/executors.md)
@@ -62,7 +64,7 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 | | omk | promptfoo | DeepEval | LangSmith |
 |--|--|--|--|--|
 | Bootstrap CI | ✓ default | ✗ | ✗ | ✗ |
-| Krippendorff α (judge ↔ human) | ✓ default | ✗ | ✗ | ✗ |
+| Krippendorff α (judge ↔ human) | ✓ with gold set | ✗ | ✗ | ✗ |
 | Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ |
 | Saturation curve | ✓ | ✗ | ✗ | ✗ |
 | Three-layer scoring isolation | ✓ | ✗ | partial | ✗ |
@@ -70,7 +72,7 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 | Native Claude Code skill | ✓ | ✗ | ✗ | ✗ |
 | Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✓ |
 
-omk's moat is **default-on safety net** — Bootstrap CI, judge ↔ human α, and length-debias aren't advanced flags; they're the default. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
+omk's moat is **default-on safety net** — Bootstrap CI and length-debias aren't advanced flags; they're the default, and judge ↔ human α comes free the moment you add a gold set. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
 
 RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/reference/comparison.md](docs/reference/comparison.md).
 
@@ -80,9 +82,9 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 |---|---|
 | **One-line verdict** | `omk eval` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
 | **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
-| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / any custom command |
-| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS |
-| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/explanation/statistical-rigor.md) |
+| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / Anthropic API / any custom command |
+| **30+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS |
+| **Statistical rigor** | Bootstrap CI / length-debias / saturation curve on by default; Krippendorff α auto-computed with a gold set. [Details →](docs/explanation/statistical-rigor.md) |
 | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage |
 | **LLM health audit** | `omk doctor` grades 7 builtin dimensions; `--static-only` runs offline without an LLM |
 | **Production observability** | parse Claude Code session JSONL traces; measure per-skill failure rate / latency / cost / knowledge-gap signals |
@@ -100,7 +102,7 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 ## Documentation
 
 - **[How it works](docs/explanation/architecture.md)** — interleaved scheduling, variant resolution, dual-channel scoring, six-dim report
-- **[Eval sample format](docs/reference/eval-sample-format.md)** — sample schema, scoring formulas, 21+ assertion types, custom JS assertions
+- **[Eval sample format](docs/reference/eval-sample-format.md)** — sample schema, scoring formulas, 30+ assertion types, custom JS assertions
 - **[CLI reference](docs/reference/cli.md)** — all seven commands with bash examples and flag tables
 - **[Executors & artifact layout](docs/reference/executors.md)** — built-in / custom executors, agent evaluation, common model configs (Claude / OpenAI / GLM / Qwen / DeepSeek / Moonshot / Ollama)
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
@@ -117,9 +119,9 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 
 ## Requirements
 
-- Node.js >= 20
+- Node.js >= 22
 - `claude` CLI (for the default executor and LLM judge; see [Claude Code](https://claude.ai/code))
-  - not needed if you use other executors (openai / gemini) with `--no-judge`
+  - not needed if you use other executors (openai-api / anthropic-api / gemini) with `--no-judge`
 
 ## Security notice
 

package/README.zh.md

@@ -9,7 +9,7 @@
 [English](./README.md) | **简体中文**
 
 **你改完 prompt，真的变好了吗？**
-用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、Krippendorff α、长度去偏，全部默认开。
+用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、长度去偏默认开，配一份人工 gold 就自动算 Krippendorff α。
 
 ![omk 报告 — verdict pill「v2 明显优于 v1，可以发布」](./assets/screenshots/report-overview-zh.png)
 
@@ -23,6 +23,8 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例；`omk eval` 跑控制变量 A/B，5 分钟内出 HTML 报告 + 一行 verdict。
 
+> 命令行有新版本时会自动提示（每 20 小时最多一次）；想永久关闭该提醒，设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
+
 手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。
 
 深入：[CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测样本格式](docs/zh/reference/eval-sample-format.md) · [执行器与 artifact 布局](docs/zh/reference/executors.md)
@@ -62,7 +64,7 @@ omk sample skills/my-skill.md
 | | omk | promptfoo | DeepEval | LangSmith |
 |--|--|--|--|--|
 | Bootstrap 置信区间 | ✓ 默认 | ✗ | ✗ | ✗ |
-| Krippendorff α（评委 ↔ 人工） | ✓ 默认 | ✗ | ✗ | ✗ |
+| Krippendorff α（评委 ↔ 人工） | ✓ 加 gold 即开 | ✗ | ✗ | ✗ |
 | 长度去偏的评委 prompt | ✓ 默认 | ✗ | ✗ | ✗ |
 | 饱和曲线 | ✓ | ✗ | ✗ | ✗ |
 | 三层独立评分 | ✓ | ✗ | 部分 | ✗ |
@@ -70,7 +72,7 @@ omk sample skills/my-skill.md
 | 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ |
 | 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✓ |
 
-omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag，是默认行为。其他工具让你**手动**接置信区间；omk 让你**默认无法忽略**它。需要 SaaS 看板？选 LangSmith。要快速 prompt 迭代不要统计层？选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」？选 omk。**
+omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 长度去偏不是 advanced flag，是默认行为；评委 ↔ 人工 α 只要给一份 gold 集就自动算。其他工具让你**手动**接置信区间；omk 让你**默认无法忽略**它。需要 SaaS 看板？选 LangSmith。要快速 prompt 迭代不要统计层？选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」？选 omk。**
 
 RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比（7 个工具 × 25+ 维度）： [docs/zh/reference/comparison.md](docs/zh/reference/comparison.md)
 
@@ -80,9 +82,9 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 |------|------|
 | **Verdict 一行结论** | `omk eval` 六档判定 + ship 建议 + exit code 路由，与 HTML 报告 verdict pill 共享规则 |
 | **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
-| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini 及自定义命令 |
-| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
-| **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/explanation/statistical-rigor.md) |
+| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / Anthropic API 及自定义命令 |
+| **30+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
+| **统计严谨性** | Bootstrap CI / 长度去偏 / 饱和曲线默认开，Krippendorff α 提供 gold 集即自动计算。[详情 →](docs/zh/explanation/statistical-rigor.md) |
 | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
 | **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分；`--static-only` 可离线无 LLM 调用 |
 | **线上 session 观测** | 解析 Claude Code session JSONL，测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
@@ -100,7 +102,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 ## 文档
 
 - **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
-- **[评测样本格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、21+ 断言类型、自定义 JS 断言
+- **[评测样本格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、30+ 断言类型、自定义 JS 断言
 - **[CLI 参考](docs/zh/reference/cli.md)** —— 7 个命令的 bash 示例和 flag 表
 - **[执行器与 artifact 布局](docs/zh/reference/executors.md)** —— 内置 / 自定义执行器、Agent 评测、常见模型配置（Claude / OpenAI / GLM / 通义 / DeepSeek / Moonshot / Ollama）
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
@@ -117,9 +119,9 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 
 ## 系统要求
 
-- Node.js >= 20
+- Node.js >= 22
 - `claude` CLI（默认执行器和 LLM 评委需要，参考 [Claude Code](https://claude.ai/code)）
-  - 如果使用其它执行器（openai / gemini）+ `--no-judge` 则可不需要
+  - 如果使用其它执行器（openai-api / anthropic-api / gemini）+ `--no-judge` 则可不需要
 
 ## 安全说明
 

package/dist/analysis/saturation.d.ts

@@ -28,6 +28,14 @@
  * that look like convergence but aren't.
  */
 export type SaturationMethod = 'slope' | 'bootstrap-ci-width' | 'plateau-height';
+/** Default consecutive-window run length required before declaring saturation. */
+export declare const DEFAULT_SATURATION_WINDOW_SIZE = 3;
+/**
+ * Default `bootstrap-ci-width` cutoff: declare saturation when the relative
+ * CI-width shrink per checkpoint stays under 5%. Single source of truth for
+ * the documented threshold; guarded by `doc-constants-drift.test.ts`.
+ */
+export declare const DEFAULT_CI_WIDTH_SHRINK_THRESHOLD = 0.05;
 /**
  * One observation in a saturation curve. `n` is the cumulative sample
  * count at this checkpoint; `mean` and (optional) `ciWidth` come from

package/dist/analysis/saturation.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"saturation.d.ts","sourceRoot":"","sources":["../../src/analysis/saturation.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AAIH,MAAM,MAAM,gBAAgB,GAAG,OAAO,GAAG,oBAAoB,GAAG,gBAAgB,CAAC;AAEjF;;;;GAIG;AACH,MAAM,WAAW,oBAAoB;IACnC,CAAC,EAAE,MAAM,CAAC;IACV,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,gBAAgB;IAC/B,oFAAoF;IACpF,SAAS,EAAE,OAAO,CAAC;IACnB,kFAAkF;IAClF,GAAG,EAAE,MAAM,GAAG,IAAI,CAAC;IACnB,iFAAiF;IACjF,UAAU,EAAE,MAAM,GAAG,QAAQ,GAAG,KAAK,CAAC;IACtC,wCAAwC;IACxC,MAAM,EAAE,gBAAgB,CAAC;IACzB,sBAAsB;IACtB,SAAS,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,KAAK,EAAE,KAAK,CAAC;QAAE,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAC5C,0DAA0D;IAC1D,MAAM,EAAE,MAAM,CAAC;CAChB;AAED;;;;;;;;;;;;;;;GAeG;AACH,wBAAgB,mBAAmB,CACjC,gBAAgB,EAAE,MAAM,EAAE,EAAE,EAC5B,MAAM,GAAE,gBAAuC,EAC/C,SAAS,CAAC,EAAE,MAAM,EAClB,UAAU,SAAI,EACd,gBAAgB,SAAO,EACvB,IAAI,CAAC,EAAE,MAAM,GACZ,gBAAgB,CA4GlB;AAUD;;;;;;GAMG;AACH,wBAAgB,qBAAqB,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,GAAG,MAAM,EAAE,EAAE,CAQlE"}
+{"version":3,"file":"saturation.d.ts","sourceRoot":"","sources":["../../src/analysis/saturation.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AAIH,MAAM,MAAM,gBAAgB,GAAG,OAAO,GAAG,oBAAoB,GAAG,gBAAgB,CAAC;AAEjF,kFAAkF;AAClF,eAAO,MAAM,8BAA8B,IAAI,CAAC;AAEhD;;;;GAIG;AACH,eAAO,MAAM,iCAAiC,OAAO,CAAC;AAEtD;;;;GAIG;AACH,MAAM,WAAW,oBAAoB;IACnC,CAAC,EAAE,MAAM,CAAC;IACV,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,gBAAgB;IAC/B,oFAAoF;IACpF,SAAS,EAAE,OAAO,CAAC;IACnB,kFAAkF;IAClF,GAAG,EAAE,MAAM,GAAG,IAAI,CAAC;IACnB,iFAAiF;IACjF,UAAU,EAAE,MAAM,GAAG,QAAQ,GAAG,KAAK,CAAC;IACtC,wCAAwC;IACxC,MAAM,EAAE,gBAAgB,CAAC;IACzB,sBAAsB;IACtB,SAAS,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,KAAK,EAAE,KAAK,CAAC;QAAE,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAC5C,0DAA0D;IAC1D,MAAM,EAAE,MAAM,CAAC;CAChB;AAED;;;;;;;;;;;;;;;GAeG;AACH,wBAAgB,mBAAmB,CACjC,gBAAgB,EAAE,MAAM,EAAE,EAAE,EAC5B,MAAM,GAAE,gBAAuC,EAC/C,SAAS,CAAC,EAAE,MAAM,EAClB,UAAU,SAAiC,EAC3C,gBAAgB,SAA4B,EAC5C,IAAI,CAAC,EAAE,MAAM,GACZ,gBAAgB,CA4GlB;AAUD;;;;;;GAMG;AACH,wBAAgB,qBAAqB,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,GAAG,MAAM,EAAE,EAAE,CAQlE"}

package/dist/analysis/saturation.js

@@ -27,7 +27,15 @@
  * window run-of-success is required. This guards against random dips
  * that look like convergence but aren't.
  */
-import { bootstrapMeanCI } from '../eval-core/bootstrap.js';
+import { bootstrapMeanCI, DEFAULT_BOOTSTRAP_ALPHA, DEFAULT_BOOTSTRAP_SAMPLES } from '../eval-core/bootstrap.js';
+/** Default consecutive-window run length required before declaring saturation. */
+export const DEFAULT_SATURATION_WINDOW_SIZE = 3;
+/**
+ * Default `bootstrap-ci-width` cutoff: declare saturation when the relative
+ * CI-width shrink per checkpoint stays under 5%. Single source of truth for
+ * the documented threshold; guarded by `doc-constants-drift.test.ts`.
+ */
+export const DEFAULT_CI_WIDTH_SHRINK_THRESHOLD = 0.05;
 /**
  * Compute saturation from a sequence of cumulative score arrays.
  *
@@ -44,13 +52,13 @@ import { bootstrapMeanCI } from '../eval-core/bootstrap.js';
  * @param bootstrapSamples Used only by 'bootstrap-ci-width' (default 1000).
  * @param seed             Optional seed for reproducible CIs.
  */
-export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-width', threshold, windowSize = 3, bootstrapSamples = 1000, seed) {
+export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-width', threshold, windowSize = DEFAULT_SATURATION_WINDOW_SIZE, bootstrapSamples = DEFAULT_BOOTSTRAP_SAMPLES, seed) {
     const trace = [];
     const checkpoints = [];
     for (const scores of cumulativeScores) {
         if (scores.length === 0)
             continue;
-        const ci = bootstrapMeanCI(scores, 0.05, bootstrapSamples, seed);
+        const ci = bootstrapMeanCI(scores, DEFAULT_BOOTSTRAP_ALPHA, bootstrapSamples, seed);
         checkpoints.push({
             n: scores.length,
             mean: ci.estimate,
@@ -151,7 +159,7 @@ export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-wid
 function defaultThreshold(method) {
     switch (method) {
         case 'slope': return 0.005;
-        case 'bootstrap-ci-width': return 0.05;
+        case 'bootstrap-ci-width': return DEFAULT_CI_WIDTH_SHRINK_THRESHOLD;
         case 'plateau-height': return 0.1;
     }
 }

package/dist/analysis/saturation.js.map

@@ -1 +1 @@
-{"version":3,"file":"saturation.js","sourceRoot":"","sources":["../../src/analysis/saturation.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AAEH,OAAO,EAAE,eAAe,EAAE,MAAM,2BAA2B,CAAC;AAgC5D;;;;;;;;;;;;;;;GAeG;AACH,MAAM,UAAU,mBAAmB,CACjC,gBAA4B,EAC5B,SAA2B,oBAAoB,EAC/C,SAAkB,EAClB,UAAU,GAAG,CAAC,EACd,gBAAgB,GAAG,IAAI,EACvB,IAAa;IAEb,MAAM,KAAK,GAA8B,EAAE,CAAC;IAC5C,MAAM,WAAW,GAA2B,EAAE,CAAC;IAE/C,KAAK,MAAM,MAAM,IAAI,gBAAgB,EAAE,CAAC;QACtC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;YAAE,SAAS;QAClC,MAAM,EAAE,GAAG,eAAe,CAAC,MAAM,EAAE,IAAI,EAAE,gBAAgB,EAAE,IAAI,CAAC,CAAC;QACjE,WAAW,CAAC,IAAI,CAAC;YACf,CAAC,EAAE,MAAM,CAAC,MAAM;YAChB,IAAI,EAAE,EAAE,CAAC,QAAQ;YACjB,OAAO,EAAE,EAAE,CAAC,IAAI,GAAG,EAAE,CAAC,GAAG;SAC1B,CAAC,CAAC;IACL,CAAC;IAED,gEAAgE;IAChE,4EAA4E;IAC5E,sEAAsE;IACtE,iEAAiE;IACjE,MAAM,CAAC,GAAG,SAAS,IAAI,gBAAgB,CAAC,MAAM,CAAC,CAAC;IAChD,MAAM,KAAK,GAAG,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACjF,MAAM,UAAU,GACd,KAAK,IAAI,EAAE,CAAC,CAAC,CAAC,MAAM;QACpB,CAAC,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,CAAC,QAAQ;YACxB,CAAC,CAAC,KAAK,CAAC;IAEV,MAAM,SAAS,GAAG,CAAC,SAAoC,EAAoB,EAAE,CAAC,CAAC;QAC7E,SAAS,EAAE,KAAK;QAChB,GAAG,EAAE,IAAI;QACT,UAAU;QACV,MAAM;QACN,SAAS,EAAE,CAAC;QACZ,KAAK;QACL,MAAM,EAAE,EAAE;QACV,GAAG,SAAS;KACb,CAAC,CAAC;IAEH,IAAI,WAAW,CAAC,MAAM,GAAG,UAAU,GAAG,CAAC,EAAE,CAAC;QACxC,OAAO,SAAS,CAAC;YACf,MAAM,EAAE,WAAW,WAAW,CAAC,MAAM,WAAW,UAAU,GAAG,CAAC,IAAI;kBAC9D,gBAAgB,UAAU,GAAG;SAClC,CAAC,CAAC;IACL,CAAC;IAED,0EAA0E;IAC1E,sEAAsE;IACtE,0EAA0E;IAC1E,0DAA0D;IAC1D,MAAM,MAAM,GAAG,CAAC,CAAS,EAAU,EAAE,CACnC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,KAAK,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACzD,IAAI,YAAkD,CAAC;IACvD,IAAI,WAAmB,CAAC;IACxB,QAAQ,MAAM,EAAE,CAAC;QACf,KAAK,OAAO,CAAC,CAAC,CAAC;YACb,YAAY,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;gBAClD,CAAC,EAAE,EAAE,CAAC,CAAC;gBACP,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;aAC/F,CAAC,CAAC,CAAC;YACJ,WAAW,GAAG,sBAAsB,CAAC;YACrC,MAAM;QACR,CAAC;QACD,KAAK,oBAAoB,CAAC,CAAC,CAAC;YAC1B,YAAY,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE;gBAChD,MAAM,IAAI,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,OAAO,IAAI,CAAC,CAAC;gBACzC,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,IAAI,CAAC,CAAC;gBAC7B,oFAAoF;gBACpF,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;gBACxD,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YAC1C,CAAC,CAAC,CAAC;YACH,WAAW,GAAG,YAAY,CAAC;YAC3B,MAAM;QACR,CAAC;QACD,KAAK,gBAAgB,CAAC,CAAC,CAAC;YACtB,uEAAuE;YACvE,qEAAqE;YACrE,oDAAoD;YACpD,YAAY,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE;gBACvC,IAAI,CAAC,GAAG,UAAU;oBAAE,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,iBAAiB,EAAE,CAAC;gBACzE,MAAM,MAAM,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,GAAG,UAAU,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;gBAC3E,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,CAAC;gBACxD,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;YAC5C,CAAC,CAAC,CAAC;YACH,WAAW,GAAG,mBAAmB,CAAC;YAClC,MAAM;QACR,CAAC;IACH,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,GAAG,YAAY,CAAC,CAAC;IAE5B,yEAAyE;IACzE,yEAAyE;IACzE,KAAK,IAAI,CAAC,GAAG,UAAU,GAAG,CAAC,EAAE,CAAC,GAAG,YAAY,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1D,IAAI,QAAQ,GAAG,IAAI,CAAC;QACpB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,UAAU,EAAE,CAAC,EAAE,EAAE,CAAC;YACpC,IAAI,YAAY,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAAC,QAAQ,GAAG,KAAK,CAAC;gBAAC,MAAM;YAAC,CAAC;QAClE,CAAC;QACD,IAAI,QAAQ,EAAE,CAAC;YACb,MAAM,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YAC9B,OAAO,SAAS,CAAC;gBACf,SAAS,EAAE,IAAI;gBACf,GAAG;gBACH,MAAM,EAAE,MAAM,UAAU,QAAQ,WAAW,MAAM,CAAC,QAAQ,GAAG,OAAO;aACrE,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC;QACf,MAAM,EAAE,GAAG,WAAW,YAAY,UAAU,UAAU,CAAC,GAAG;KAC3D,CAAC,CAAC;AACL,CAAC;AAED,SAAS,gBAAgB,CAAC,MAAwB;IAChD,QAAQ,MAAM,EAAE,CAAC;QACf,KAAK,OAAO,CAAC,CAAC,OAAO,KAAK,CAAC;QAC3B,KAAK,oBAAoB,CAAC,CAAC,OAAO,IAAI,CAAC;QACvC,KAAK,gBAAgB,CAAC,CAAC,OAAO,GAAG,CAAC;IACpC,CAAC;AACH,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,qBAAqB,CAAC,IAAgB;IACpD,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,MAAM,GAAG,GAAe,EAAE,CAAC;IAC3B,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;QACf,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}
+{"version":3,"file":"saturation.js","sourceRoot":"","sources":["../../src/analysis/saturation.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA4BG;AAEH,OAAO,EAAE,eAAe,EAAE,uBAAuB,EAAE,yBAAyB,EAAE,MAAM,2BAA2B,CAAC;AAIhH,kFAAkF;AAClF,MAAM,CAAC,MAAM,8BAA8B,GAAG,CAAC,CAAC;AAEhD;;;;GAIG;AACH,MAAM,CAAC,MAAM,iCAAiC,GAAG,IAAI,CAAC;AA8BtD;;;;;;;;;;;;;;;GAeG;AACH,MAAM,UAAU,mBAAmB,CACjC,gBAA4B,EAC5B,SAA2B,oBAAoB,EAC/C,SAAkB,EAClB,UAAU,GAAG,8BAA8B,EAC3C,gBAAgB,GAAG,yBAAyB,EAC5C,IAAa;IAEb,MAAM,KAAK,GAA8B,EAAE,CAAC;IAC5C,MAAM,WAAW,GAA2B,EAAE,CAAC;IAE/C,KAAK,MAAM,MAAM,IAAI,gBAAgB,EAAE,CAAC;QACtC,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;YAAE,SAAS;QAClC,MAAM,EAAE,GAAG,eAAe,CAAC,MAAM,EAAE,uBAAuB,EAAE,gBAAgB,EAAE,IAAI,CAAC,CAAC;QACpF,WAAW,CAAC,IAAI,CAAC;YACf,CAAC,EAAE,MAAM,CAAC,MAAM;YAChB,IAAI,EAAE,EAAE,CAAC,QAAQ;YACjB,OAAO,EAAE,EAAE,CAAC,IAAI,GAAG,EAAE,CAAC,GAAG;SAC1B,CAAC,CAAC;IACL,CAAC;IAED,gEAAgE;IAChE,4EAA4E;IAC5E,sEAAsE;IACtE,iEAAiE;IACjE,MAAM,CAAC,GAAG,SAAS,IAAI,gBAAgB,CAAC,MAAM,CAAC,CAAC;IAChD,MAAM,KAAK,GAAG,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACjF,MAAM,UAAU,GACd,KAAK,IAAI,EAAE,CAAC,CAAC,CAAC,MAAM;QACpB,CAAC,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,CAAC,QAAQ;YACxB,CAAC,CAAC,KAAK,CAAC;IAEV,MAAM,SAAS,GAAG,CAAC,SAAoC,EAAoB,EAAE,CAAC,CAAC;QAC7E,SAAS,EAAE,KAAK;QAChB,GAAG,EAAE,IAAI;QACT,UAAU;QACV,MAAM;QACN,SAAS,EAAE,CAAC;QACZ,KAAK;QACL,MAAM,EAAE,EAAE;QACV,GAAG,SAAS;KACb,CAAC,CAAC;IAEH,IAAI,WAAW,CAAC,MAAM,GAAG,UAAU,GAAG,CAAC,EAAE,CAAC;QACxC,OAAO,SAAS,CAAC;YACf,MAAM,EAAE,WAAW,WAAW,CAAC,MAAM,WAAW,UAAU,GAAG,CAAC,IAAI;kBAC9D,gBAAgB,UAAU,GAAG;SAClC,CAAC,CAAC;IACL,CAAC;IAED,0EAA0E;IAC1E,sEAAsE;IACtE,0EAA0E;IAC1E,0DAA0D;IAC1D,MAAM,MAAM,GAAG,CAAC,CAAS,EAAU,EAAE,CACnC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,KAAK,CAAC,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACzD,IAAI,YAAkD,CAAC;IACvD,IAAI,WAAmB,CAAC;IACxB,QAAQ,MAAM,EAAE,CAAC;QACf,KAAK,OAAO,CAAC,CAAC,CAAC;YACb,YAAY,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;gBAClD,CAAC,EAAE,EAAE,CAAC,CAAC;gBACP,MAAM,EAAE,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;aAC/F,CAAC,CAAC,CAAC;YACJ,WAAW,GAAG,sBAAsB,CAAC;YACrC,MAAM;QACR,CAAC;QACD,KAAK,oBAAoB,CAAC,CAAC,CAAC;YAC1B,YAAY,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE;gBAChD,MAAM,IAAI,GAAG,WAAW,CAAC,CAAC,CAAC,CAAC,OAAO,IAAI,CAAC,CAAC;gBACzC,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,IAAI,CAAC,CAAC;gBAC7B,oFAAoF;gBACpF,MAAM,GAAG,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;gBACxD,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YAC1C,CAAC,CAAC,CAAC;YACH,WAAW,GAAG,YAAY,CAAC;YAC3B,MAAM;QACR,CAAC;QACD,KAAK,gBAAgB,CAAC,CAAC,CAAC;YACtB,uEAAuE;YACvE,qEAAqE;YACrE,oDAAoD;YACpD,YAAY,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE;gBACvC,IAAI,CAAC,GAAG,UAAU;oBAAE,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,iBAAiB,EAAE,CAAC;gBACzE,MAAM,MAAM,GAAG,WAAW,CAAC,KAAK,CAAC,CAAC,GAAG,UAAU,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;gBAC3E,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,CAAC;gBACxD,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;YAC5C,CAAC,CAAC,CAAC;YACH,WAAW,GAAG,mBAAmB,CAAC;YAClC,MAAM;QACR,CAAC;IACH,CAAC;IAED,KAAK,CAAC,IAAI,CAAC,GAAG,YAAY,CAAC,CAAC;IAE5B,yEAAyE;IACzE,yEAAyE;IACzE,KAAK,IAAI,CAAC,GAAG,UAAU,GAAG,CAAC,EAAE,CAAC,GAAG,YAAY,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1D,IAAI,QAAQ,GAAG,IAAI,CAAC;QACpB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,UAAU,EAAE,CAAC,EAAE,EAAE,CAAC;YACpC,IAAI,YAAY,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAAC,QAAQ,GAAG,KAAK,CAAC;gBAAC,MAAM;YAAC,CAAC;QAClE,CAAC;QACD,IAAI,QAAQ,EAAE,CAAC;YACb,MAAM,GAAG,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YAC9B,OAAO,SAAS,CAAC;gBACf,SAAS,EAAE,IAAI;gBACf,GAAG;gBACH,MAAM,EAAE,MAAM,UAAU,QAAQ,WAAW,MAAM,CAAC,QAAQ,GAAG,OAAO;aACrE,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,OAAO,SAAS,CAAC;QACf,MAAM,EAAE,GAAG,WAAW,YAAY,UAAU,UAAU,CAAC,GAAG;KAC3D,CAAC,CAAC;AACL,CAAC;AAED,SAAS,gBAAgB,CAAC,MAAwB;IAChD,QAAQ,MAAM,EAAE,CAAC;QACf,KAAK,OAAO,CAAC,CAAC,OAAO,KAAK,CAAC;QAC3B,KAAK,oBAAoB,CAAC,CAAC,OAAO,iCAAiC,CAAC;QACpE,KAAK,gBAAgB,CAAC,CAAC,OAAO,GAAG,CAAC;IACpC,CAAC;AACH,CAAC;AAED;;;;;;GAMG;AACH,MAAM,UAAU,qBAAqB,CAAC,IAAgB;IACpD,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,MAAM,GAAG,GAAe,EAAE,CAAC;IAC3B,KAAK,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;QACf,GAAG,CAAC,IAAI,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}

package/dist/authoring/evolver.d.ts

@@ -15,7 +15,28 @@ interface WeakSample {
         none?: string;
     };
 }
-export declare function extractWeakSamples(report: Report, variantKey: string, count?: number): WeakSample[];
+export declare function extractWeakSamples(report: Report, variantKey: string, count?: number, sampleIdFilter?: Set<string>): WeakSample[];
+/** A train / holdout partition of a sample set. */
+interface HoldoutSplit {
+    trainIds: Set<string>;
+    holdoutIds: Set<string>;
+}
+/**
+ * Deterministically split sample ids into train / holdout by `ratio` (fraction
+ * held out). Holdout members are picked at an even stride so the partition is
+ * representative of the ordering, and the split is stable across rounds and runs
+ * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
+ * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
+ */
+export declare function splitHoldout(sampleIds: string[], ratio: number): HoldoutSplit | null;
+/**
+ * A view of `report` whose results are restricted to `sampleIds`. Used to keep the
+ * holdout split out of the sample-fixer: under an active holdout, only training-split
+ * samples may enter the --auto-fix-samples prompt or be rewritten — otherwise the
+ * skill's samples get tuned to the very samples that decide acceptance, reintroducing
+ * the leak holdout exists to prevent.
+ */
+export declare function restrictReportToSamples(report: Report, sampleIds: Set<string>): Report;
 export declare function allNonTripwireAssertionsPass(report: Report, variantKey: string): boolean;
 export declare function buildImprovementPrompt(skillContent: string, score: number, weakSamples: WeakSample[]): string;
 /** @deprecated Use ProgressCallback from evaluation-core.ts */
@@ -61,15 +82,26 @@ interface EvolveOptions {
     noDiagnostic?: boolean;
     /** 跳过 doctor 健康检查门禁。默认 false。 */
     skipDoctor?: boolean;
+    /** Fraction of samples held out for the accept decision (0..1). Default 0 = off.
+     *  When > 0, a candidate is accepted on its **holdout** composite rather than the
+     *  training composite, and weak-sample extraction only sees the training split —
+     *  so the skill is never tuned to the samples that judge it. Too small a split
+     *  (either side < MIN_HOLDOUT_SUBSET) falls back to full-set scoring + a warning. */
+    holdoutRatio?: number;
     onProgress?: ProgressCallback | null;
     onRoundProgress?: ((progress: EvolveRoundProgressInfo) => void) | null;
 }
 interface TrajectoryEntry {
     round: number;
+    /** Accept-decision score: holdout composite when holdout is active, else full-set. */
     score: number;
     delta: number;
     accepted: boolean;
     costUSD: number;
+    /** Present when holdout is active: the training-split composite (improvement signal). */
+    trainScore?: number;
+    /** Present when holdout is active: the holdout-split composite (== score). */
+    holdoutScore?: number;
 }
 export interface EvolveResult {
     startScore: number;
@@ -86,6 +118,15 @@ export interface EvolveResult {
     reusedBaselineReportId?: string;
     /** False = 任一轮的 exec / judge 不报 cost → totalCostUSD 是 lower-bound 而非真值。 */
     costReported?: boolean;
+    /** Holdout split summary when `--holdout-ratio` > 0. `disabled` is true when the
+     *  split was too small and evolve fell back to full-set scoring (CLI formats the
+     *  user-facing message bilingually). */
+    holdout?: {
+        ratio: number;
+        trainCount: number;
+        holdoutCount: number;
+        disabled?: boolean;
+    };
     trajectory: TrajectoryEntry[];
     bestSkillPath: string;
     allVersions: string[];
@@ -97,6 +138,6 @@ export interface RoundReport {
     report: Report;
 }
 export declare function mergeEvolveReports(roundReports: RoundReport[], skillName: string, totalCostUSD: number, samples?: Sample[], skillPath?: string): Report;
-export declare function evolveSkill({ skillPath, samplesPath, rounds, target, stopOnAssertionsPass, autoFixSamples, sampleFixMaxAttempts, reuseLatestEval, model, judgeModels, improveModel, improveMode, executorName, concurrency, timeoutMs, skipConnectivity, effort, noDiagnostic, skipDoctor, onProgress, onRoundProgress, }: EvolveOptions): Promise<EvolveResult>;
+export declare function evolveSkill({ skillPath, samplesPath, rounds, target, stopOnAssertionsPass, autoFixSamples, sampleFixMaxAttempts, reuseLatestEval, model, judgeModels, improveModel, improveMode, executorName, concurrency, timeoutMs, skipConnectivity, effort, noDiagnostic, skipDoctor, holdoutRatio, onProgress, onRoundProgress, }: EvolveOptions): Promise<EvolveResult>;
 export {};
 //# sourceMappingURL=evolver.d.ts.map

package/dist/authoring/evolver.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"evolver.d.ts","sourceRoot":"","sources":["../../src/authoring/evolver.ts"],"names":[],"mappings":"AASA,OAAO,KAAK,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,EAAe,MAAM,EAAiB,MAAM,mBAAmB,CAAC;AA0BnH,UAAU,UAAU;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,gBAAgB,EAAE,MAAM,EAAE,CAAC;IAC3B,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAAC;IAC1C,UAAU,CAAC,EAAE;QACX,OAAO,CAAC,EAAE,MAAM,CAAC;QACjB,QAAQ,CAAC,EAAE,MAAM,CAAC;QAClB,MAAM,CAAC,EAAE,MAAM,CAAC;QAChB,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;QACrB,eAAe,CAAC,EAAE,MAAM,CAAC;QACzB,gBAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,IAAI,CAAC,EAAE,MAAM,CAAC;KACf,CAAC;CACH;AAiGD,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,KAAK,GAAE,MAAU,GAAG,UAAU,EAAE,CA8BtG;AAED,wBAAgB,4BAA4B,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAWxF;AA2ID,wBAAgB,sBAAsB,CAAC,YAAY,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,GAAG,MAAM,CA4B7G;AAuBD,+DAA+D;AAC/D,MAAM,MAAM,kBAAkB,GAAG,UAAU,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC;AAEjE,MAAM,WAAW,uBAAuB;IACtC,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;qCACiC;IACjC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,OAAO,CAAC;CAClB;AAED,UAAU,aAAa;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,oBAAoB,CAAC,EAAE,MAAM,CAAC;IAC9B,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;4DAEwD;IACxD,WAAW,CAAC,EAAE,WAAW,EAAE,CAAC;IAC5B,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,wEAAwE;IACxE,WAAW,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IAClC,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,+CAA+C;IAC/C,MAAM,CAAC,EAAE,KAAK,GAAG,QAAQ,GAAG,MAAM,GAAG,OAAO,GAAG,KAAK,CAAC;IACrD,qCAAqC;IACrC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,iCAAiC;IACjC,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,UAAU,CAAC,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACrC,eAAe,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,uBAAuB,KAAK,IAAI,CAAC,GAAG,IAAI,CAAC;CACxE;AAED,UAAU,eAAe;IACvB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,YAAY,EAAE,MAAM,CAAC;IACrB,UAAU,CAAC,EAAE,QAAQ,GAAG,iBAAiB,GAAG,qBAAqB,GAAG,QAAQ,CAAC;IAC7E,WAAW,CAAC,EAAE,KAAK,CAAC;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAC5E,sBAAsB,CAAC,EAAE,MAAM,CAAC;IAChC,2EAA2E;IAC3E,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,UAAU,EAAE,eAAe,EAAE,CAAC;IAC9B,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,WAAW;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,YAAY,EAAE,WAAW,EAAE,EAC3B,SAAS,EAAE,MAAM,EACjB,YAAY,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,MAAM,EAAE,EAClB,SAAS,CAAC,EAAE,MAAM,GACjB,MAAM,CAyER;AAED,wBAAsB,WAAW,CAAC,EAChC,SAAS,EACT,WAAW,EACX,MAAU,EACV,MAAa,EACb,oBAA4B,EAC5B,cAAsB,EACtB,oBAAwB,EACxB,eAAuB,EACvB,KAAqB,EACrB,WAAW,EACX,YAA4B,EAC5B,WAAqB,EACrB,YAAuB,EACvB,WAAe,EACf,SAAS,EACT,gBAAwB,EACxB,MAAM,EACN,YAAY,EACZ,UAAU,EACV,UAAiB,EACjB,eAAsB,GACvB,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC,CA6PvC"}
+{"version":3,"file":"evolver.d.ts","sourceRoot":"","sources":["../../src/authoring/evolver.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,EAAe,MAAM,EAAiB,MAAM,mBAAmB,CAAC;AA0BnH,UAAU,UAAU;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,gBAAgB,EAAE,MAAM,EAAE,CAAC;IAC3B,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAAC;IAC1C,UAAU,CAAC,EAAE;QACX,OAAO,CAAC,EAAE,MAAM,CAAC;QACjB,QAAQ,CAAC,EAAE,MAAM,CAAC;QAClB,MAAM,CAAC,EAAE,MAAM,CAAC;QAChB,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;QACrB,eAAe,CAAC,EAAE,MAAM,CAAC;QACzB,gBAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,IAAI,CAAC,EAAE,MAAM,CAAC;KACf,CAAC;CACH;AAiGD,wBAAgB,kBAAkB,CAChC,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,MAAM,EAClB,KAAK,GAAE,MAAU,EACjB,cAAc,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAC3B,UAAU,EAAE,CA+Bd;AAED,mDAAmD;AACnD,UAAU,YAAY;IACpB,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;IACtB,UAAU,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;CACzB;AAMD;;;;;;GAMG;AACH,wBAAgB,YAAY,CAAC,SAAS,EAAE,MAAM,EAAE,EAAE,KAAK,EAAE,MAAM,GAAG,YAAY,GAAG,IAAI,CAYpF;AAmBD;;;;;;GAMG;AACH,wBAAgB,uBAAuB,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,MAAM,CAEtF;AAED,wBAAgB,4BAA4B,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAWxF;AA2ID,wBAAgB,sBAAsB,CAAC,YAAY,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,GAAG,MAAM,CA4B7G;AAuBD,+DAA+D;AAC/D,MAAM,MAAM,kBAAkB,GAAG,UAAU,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC;AAEjE,MAAM,WAAW,uBAAuB;IACtC,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;qCACiC;IACjC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,CAAC,EAAE,OAAO,CAAC;CAClB;AAED,UAAU,aAAa;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,oBAAoB,CAAC,EAAE,MAAM,CAAC;IAC9B,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;4DAEwD;IACxD,WAAW,CAAC,EAAE,WAAW,EAAE,CAAC;IAC5B,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,wEAAwE;IACxE,WAAW,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IAClC,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,+CAA+C;IAC/C,MAAM,CAAC,EAAE,KAAK,GAAG,QAAQ,GAAG,MAAM,GAAG,OAAO,GAAG,KAAK,CAAC;IACrD,qCAAqC;IACrC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,iCAAiC;IACjC,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB;;;;yFAIqF;IACrF,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,UAAU,CAAC,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACrC,eAAe,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,uBAAuB,KAAK,IAAI,CAAC,GAAG,IAAI,CAAC;CACxE;AAED,UAAU,eAAe;IACvB,KAAK,EAAE,MAAM,CAAC;IACd,sFAAsF;IACtF,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,yFAAyF;IACzF,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8EAA8E;IAC9E,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,YAAY,EAAE,MAAM,CAAC;IACrB,UAAU,CAAC,EAAE,QAAQ,GAAG,iBAAiB,GAAG,qBAAqB,GAAG,QAAQ,CAAC;IAC7E,WAAW,CAAC,EAAE,KAAK,CAAC;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAC5E,sBAAsB,CAAC,EAAE,MAAM,CAAC;IAChC,2EAA2E;IAC3E,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB;;4CAEwC;IACxC,OAAO,CAAC,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,YAAY,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;KAAE,CAAC;IAC1F,UAAU,EAAE,eAAe,EAAE,CAAC;IAC9B,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,WAAW;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,YAAY,EAAE,WAAW,EAAE,EAC3B,SAAS,EAAE,MAAM,EACjB,YAAY,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,MAAM,EAAE,EAClB,SAAS,CAAC,EAAE,MAAM,GACjB,MAAM,CAyER;AAED,wBAAsB,WAAW,CAAC,EAChC,SAAS,EACT,WAAW,EACX,MAAU,EACV,MAAa,EACb,oBAA4B,EAC5B,cAAsB,EACtB,oBAAwB,EACxB,eAAuB,EACvB,KAAqB,EACrB,WAAW,EACX,YAA4B,EAC5B,WAAqB,EACrB,YAAuB,EACvB,WAAe,EACf,SAAS,EACT,gBAAwB,EACxB,MAAM,EACN,YAAY,EACZ,UAAU,EACV,YAAgB,EAChB,UAAiB,EACjB,eAAsB,GACvB,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC,CA4RvC"}

package/dist/authoring/evolver.js

@@ -6,6 +6,7 @@ import { persistReport, DEFAULT_OUTPUT_DIR, generateRunId, hashString } from '..
 import { createFileStore } from '../server/report-store.js';
 import { analyzeResults } from '../analysis/report-diagnostics.js';
 import { loadSamples } from '../inputs/load-samples.js';
+import { buildVariantSummary } from '../eval-core/schema.js';
 import { fixSamples } from './sample-fixer.js';
 const IMPROVE_SYSTEM_PROMPT = `你是一个 AI 提示词改进专家。你的任务是分析评测结果中的薄弱环节，针对性地改进 skill（系统提示词），使其在评测中获得更高的分数。
 
@@ -121,9 +122,11 @@ function readSkillName(skillPath) {
         return null;
     }
 }
-export function extractWeakSamples(report, variantKey, count = 5) {
+export function extractWeakSamples(report, variantKey, count = 5, sampleIdFilter) {
     const weakSamples = [];
     for (const r of report.results) {
+        if (sampleIdFilter && !sampleIdFilter.has(r.sample_id))
+            continue;
         const v = r.variants[variantKey];
         if (!v || typeof v.compositeScore !== 'number')
             continue;
@@ -153,6 +156,60 @@ export function extractWeakSamples(report, variantKey, count = 5) {
         .sort((a, b) => a.compositeScore - b.compositeScore)
         .slice(0, count);
 }
+/** Below this many samples on either side, a holdout split is too small to be
+ *  meaningful — evolve falls back to full-set scoring and warns. */
+const MIN_HOLDOUT_SUBSET = 3;
+/**
+ * Deterministically split sample ids into train / holdout by `ratio` (fraction
+ * held out). Holdout members are picked at an even stride so the partition is
+ * representative of the ordering, and the split is stable across rounds and runs
+ * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
+ * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
+ */
+export function splitHoldout(sampleIds, ratio) {
+    if (!(ratio > 0) || sampleIds.length === 0)
+        return null;
+    const holdoutCount = Math.round(sampleIds.length * ratio);
+    const trainCount = sampleIds.length - holdoutCount;
+    if (holdoutCount < MIN_HOLDOUT_SUBSET || trainCount < MIN_HOLDOUT_SUBSET)
+        return null;
+    const stride = sampleIds.length / holdoutCount;
+    const holdoutIds = new Set();
+    for (let k = 0; k < holdoutCount; k++) {
+        holdoutIds.add(sampleIds[Math.floor(k * stride)]);
+    }
+    const trainIds = new Set(sampleIds.filter((id) => !holdoutIds.has(id)));
+    return { trainIds, holdoutIds };
+}
+/**
+ * Mean composite over the subset of a report's results whose sample_id is in
+ * `ids`, using the same aggregation as the full-run summary
+ * (`buildVariantSummary`) so train / holdout scores stay comparable to the
+ * headline composite. Returns 0 when the subset has no scorable entries.
+ */
+function subsetCompositeScore(report, variantKey, ids) {
+    const entries = [];
+    for (const r of report.results) {
+        if (!ids.has(r.sample_id))
+            continue;
+        const v = r.variants[variantKey];
+        if (v)
+            entries.push(v);
+    }
+    if (entries.length === 0)
+        return 0;
+    return buildVariantSummary(entries).avgCompositeScore ?? 0;
+}
+/**
+ * A view of `report` whose results are restricted to `sampleIds`. Used to keep the
+ * holdout split out of the sample-fixer: under an active holdout, only training-split
+ * samples may enter the --auto-fix-samples prompt or be rewritten — otherwise the
+ * skill's samples get tuned to the very samples that decide acceptance, reintroducing
+ * the leak holdout exists to prevent.
+ */
+export function restrictReportToSamples(report, sampleIds) {
+    return { ...report, results: report.results.filter((r) => sampleIds.has(r.sample_id)) };
+}
 export function allNonTripwireAssertionsPass(report, variantKey) {
     for (const entry of report.results) {
         const variant = entry.variants[variantKey];
@@ -401,7 +458,7 @@ export function mergeEvolveReports(roundReports, skillName, totalCostUSD, sample
     report.analysis = analyzeResults(report, { samples });
     return report;
 }
-export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target = null, stopOnAssertionsPass = false, autoFixSamples = false, sampleFixMaxAttempts = 2, reuseLatestEval = false, model = DEFAULT_MODEL, judgeModels, improveModel = DEFAULT_MODEL, improveMode = 'agent', executorName = 'claude', concurrency = 1, timeoutMs, skipConnectivity = false, effort, noDiagnostic, skipDoctor, onProgress = null, onRoundProgress = null, }) {
+export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target = null, stopOnAssertionsPass = false, autoFixSamples = false, sampleFixMaxAttempts = 2, reuseLatestEval = false, model = DEFAULT_MODEL, judgeModels, improveModel = DEFAULT_MODEL, improveMode = 'agent', executorName = 'claude', concurrency = 1, timeoutMs, skipConnectivity = false, effort, noDiagnostic, skipDoctor, holdoutRatio = 0, onProgress = null, onRoundProgress = null, }) {
     if (judgeModels && judgeModels.length > 1) {
         throw new Error('evolveSkill does not support multi-judge ensemble (received '
             + `${judgeModels.length} judges). Pass a single-judge array, e.g. `
@@ -421,6 +478,28 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
     if (!existsSync(absSamplesPath))
         throw new Error(`samples file not found: ${absSamplesPath}`);
     mkdirSync(evolveDir, { recursive: true });
+    // Holdout split (opt-in). Computed once over the canonical sample order so it's
+    // stable across rounds. When active, accept decisions use the holdout composite
+    // and weak-sample extraction only sees the training split — the skill is never
+    // tuned to the samples that decide whether it's accepted.
+    const allSampleIds = loadSamples(absSamplesPath).samples.map((s) => s.sample_id);
+    const holdoutSplit = splitHoldout(allSampleIds, holdoutRatio);
+    const holdoutInfo = holdoutRatio > 0
+        ? {
+            ratio: holdoutRatio,
+            trainCount: holdoutSplit?.trainIds.size ?? allSampleIds.length,
+            holdoutCount: holdoutSplit?.holdoutIds.size ?? 0,
+            ...(holdoutSplit ? {} : { disabled: true }),
+        }
+        : undefined;
+    // Accept-decision score for a report's variant: holdout composite when the split
+    // is active, otherwise the full-set composite (legacy behavior).
+    const decisionScore = (report, key) => holdoutSplit
+        ? subsetCompositeScore(report, key, holdoutSplit.holdoutIds)
+        : (report.summary[key]?.avgCompositeScore ?? 0);
+    const trainScoreOf = (report, key) => holdoutSplit ? subsetCompositeScore(report, key, holdoutSplit.trainIds) : undefined;
+    // Per-round trajectory tail: train / holdout breakdown, only when split active.
+    const splitScores = (report, key, decision) => holdoutSplit ? { trainScore: trainScoreOf(report, key), holdoutScore: decision } : {};
     // Save original as r0
     let currentBest = readFileSync(absSkillPath, 'utf-8').trim();
     const r0Path = join(evolveDir, `${skillName}.r0.md`);
@@ -461,13 +540,13 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         });
     }
     const baselineVariantKey = Object.keys(baselineReport.summary)[0];
-    bestScore = baselineReport.summary[baselineVariantKey]?.avgCompositeScore ?? 0;
+    bestScore = decisionScore(baselineReport, baselineVariantKey);
     const baselineCost = baselineReused ? 0 : baselineReport.meta.totalCostUSD;
     totalCostUSD += baselineCost;
     const baselineCostReported = baselineReused || !reportHasUnreportedCost(baselineReport);
     if (!baselineCostReported)
         totalCostReported = false;
-    trajectory.push({ round: 0, score: bestScore, delta: 0, accepted: true, costUSD: baselineCost });
+    trajectory.push({ round: 0, score: bestScore, delta: 0, accepted: true, costUSD: baselineCost, ...splitScores(baselineReport, baselineVariantKey, bestScore) });
     roundReports.push({ round: 0, accepted: true, report: baselineReport });
     if (onRoundProgress)
         onRoundProgress({ round: 0, totalRounds: rounds, phase: 'baseline', score: bestScore, costUSD: baselineCost, costReported: baselineCostReported, reused: baselineReused });
@@ -487,6 +566,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
             ...(sampleFixes.length > 0 && { sampleFixes }),
             ...(reusedBaselineReportId && { reusedBaselineReportId }),
             ...(totalCostReported ? {} : { costReported: false }),
+            ...(holdoutInfo ? { holdout: holdoutInfo } : {}),
             trajectory,
             bestSkillPath: allVersions[bestRound],
             allVersions,
@@ -513,7 +593,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
                 totalCostReported = false;
         }
         const lastVariantKey = Object.keys(lastReport.summary)[0];
-        const weakSamples = extractWeakSamples(lastReport, lastVariantKey);
+        const weakSamples = extractWeakSamples(lastReport, lastVariantKey, 5, holdoutSplit?.trainIds);
         // Generate improvement
         const candidatePath = join(evolveDir, `${skillName}.r${round}.md`);
         const basePrompt = buildImprovementPrompt(currentBest, bestScore, weakSamples);
@@ -575,7 +655,9 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
             const sampleFix = await autoFixSamplesAfterSkillRound({
                 samplesPath: absSamplesPath,
                 skillContent: candidateContent,
-                report: lastReport,
+                // Under an active holdout, the sample-fixer may only see training-split samples —
+                // never the holdout samples that drive the accept decision (leak guard).
+                report: holdoutSplit ? restrictReportToSamples(lastReport, holdoutSplit.trainIds) : lastReport,
                 treatmentKey: lastVariantKey,
                 executorName,
                 model: improveModel,
@@ -593,7 +675,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
             samplesPath: absSamplesPath, skillDir, model, judgeModels: effectiveJudgeModels, executorName, concurrency, timeoutMs, skipConnectivity, effort, noDiagnostic, skipDoctor, onProgress,
         });
         const candidateVariantKey = Object.keys(candidateReport.summary)[0];
-        const candidateScore = candidateReport.summary[candidateVariantKey]?.avgCompositeScore ?? 0;
+        const candidateScore = decisionScore(candidateReport, candidateVariantKey);
         const roundCost = improveCostUSD + preEvalSampleFixCost + candidateReport.meta.totalCostUSD;
         const roundCostReported = improveCostReported && !reportHasUnreportedCost(candidateReport);
         if (!roundCostReported)
@@ -612,7 +694,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         if (accepted)
             roundReports.push({ round, accepted, report: candidateReport });
         const roundDelta = candidateScore - trajectory[trajectory.length - 1].score;
-        trajectory.push({ round, score: candidateScore, delta: roundDelta, accepted, costUSD: roundCost });
+        trajectory.push({ round, score: candidateScore, delta: roundDelta, accepted, costUSD: roundCost, ...splitScores(candidateReport, candidateVariantKey, candidateScore) });
         if (onRoundProgress)
             onRoundProgress({ round, totalRounds: rounds, phase: 'done', score: candidateScore, delta: roundDelta, accepted, costUSD: roundCost, costReported: roundCostReported });
         // Early stop
@@ -652,6 +734,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         ...(sampleFixes.length > 0 && { sampleFixes }),
         ...(reusedBaselineReportId && { reusedBaselineReportId }),
         ...(totalCostReported ? {} : { costReported: false }),
+        ...(holdoutInfo ? { holdout: holdoutInfo } : {}),
         trajectory,
         bestSkillPath: allVersions[bestRound],
         allVersions,