oh-my-knowledge 0.32.0 → 0.34.0

package/README.md

@@ -9,7 +9,9 @@
 **English** | [简体中文](./README.zh.md)
 
 **Did your prompt actually get better?**
-A/B test your prompts and skills with statistical rigor — bootstrap CI, Krippendorff α, length-debias — all on by default.
+A/B test your prompts and skills with statistical rigor — bootstrap CI and length-debias on by default, Krippendorff α the moment you add a gold set.
+
+📖 **Full documentation: [oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** (searchable, English / 简体中文)
 
 ![omk report — verdict pill "v2 is clearly better than v1 — ready to ship"](./assets/screenshots/report-overview.png)
 
@@ -23,12 +25,22 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
 
+> The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
+
 Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
 
-Deeper: [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors & artifact layout](docs/reference/executors.md)
+Deeper: [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
 ## Use inside AI Coding Agents
 
+Install the official omk Agent Skill to let your coding agent run omk workflows from natural language:
+
+```bash
+omk install omk-agent-skill
+```
+
+By default, omk installs only into detected local targets it explicitly supports: Codex/AGENTS when `~/.codex` or `~/.agents` exists, and Claude Code when `~/.claude` exists. Use `--to all` to force every target omk currently knows, or `--dest` for a custom skill root.
+
 ### Use inside Claude Code
 
 When the `omk` skill is available in Claude Code, you can invoke it directly:
@@ -62,15 +74,15 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 | | omk | promptfoo | DeepEval | LangSmith |
 |--|--|--|--|--|
 | Bootstrap CI | ✓ default | ✗ | ✗ | ✗ |
-| Krippendorff α (judge ↔ human) | ✓ default | ✗ | ✗ | ✗ |
+| Krippendorff α (judge ↔ human) | ✓ with gold set | ✗ | ✗ | ✗ |
 | Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ |
 | Saturation curve | ✓ | ✗ | ✗ | ✗ |
 | Three-layer scoring isolation | ✓ | ✗ | partial | ✗ |
 | Per-variant skill isolation (construct validity) | ✓ default | ✗ | ✗ | ✗ |
-| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ |
+| Native Agent Skill | ✓ | ✗ | ✗ | ✗ |
 | Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✓ |
 
-omk's moat is **default-on safety net** — Bootstrap CI, judge ↔ human α, and length-debias aren't advanced flags; they're the default. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
+omk's moat is **default-on safety net** — Bootstrap CI and length-debias aren't advanced flags; they're the default, and judge ↔ human α comes free the moment you add a gold set. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
 
 RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/reference/comparison.md](docs/reference/comparison.md).
 
@@ -80,9 +92,9 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 |---|---|
 | **One-line verdict** | `omk eval` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
 | **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
-| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / any custom command |
-| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS |
-| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/explanation/statistical-rigor.md) |
+| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / Anthropic API / any custom command |
+| **30+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS |
+| **Statistical rigor** | Bootstrap CI / length-debias / saturation curve on by default; Krippendorff α auto-computed with a gold set. [Details →](docs/explanation/statistical-rigor.md) |
 | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage |
 | **LLM health audit** | `omk doctor` grades 7 builtin dimensions; `--static-only` runs offline without an LLM |
 | **Production observability** | parse Claude Code session JSONL traces; measure per-skill failure rate / latency / cost / knowledge-gap signals |
@@ -99,14 +111,17 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 
 ## Documentation
 
+The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** — searchable, with an English / 简体中文 switcher. Key pages:
+
 - **[How it works](docs/explanation/architecture.md)** — interleaved scheduling, variant resolution, dual-channel scoring, six-dim report
-- **[Eval sample format](docs/reference/eval-sample-format.md)** — sample schema, scoring formulas, 21+ assertion types, custom JS assertions
-- **[CLI reference](docs/reference/cli.md)** — all seven commands with bash examples and flag tables
-- **[Executors & artifact layout](docs/reference/executors.md)** — built-in / custom executors, agent evaluation, common model configs (Claude / OpenAI / GLM / Qwen / DeepSeek / Moonshot / Ollama)
+- **[Eval sample format](docs/reference/eval-sample-format.md)** — sample schema, scoring formulas, 30+ assertion types, custom JS assertions
+- **[CLI reference](docs/reference/cli.md)** — all top-level commands with bash examples and flag tables
+- **[Executors](docs/reference/executors.md)** & **[artifact layout](docs/reference/artifact-layout.md)** — built-in / custom executors; how `variant` resolves to an artifact + runtime context
+- **[How-to guides](docs/guides/agent-eval.md)** — [evaluate an agent](docs/guides/agent-eval.md) (project runtime context) and [use non-Claude models](docs/guides/non-claude-models.md) (GLM / Qwen / DeepSeek / Moonshot / Ollama)
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
 - **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
 - **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
-- **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / LangSmith / Langfuse / Braintrust etc.
+- **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai
 
 ## Environment variables
 
@@ -117,9 +132,9 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 
 ## Requirements
 
-- Node.js >= 20
+- Node.js >= 22
 - `claude` CLI (for the default executor and LLM judge; see [Claude Code](https://claude.ai/code))
-  - not needed if you use other executors (openai / gemini) with `--no-judge`
+  - not needed if you use other executors (openai-api / anthropic-api / gemini) with `--no-judge`
 
 ## Security notice
 

package/README.zh.md

@@ -9,7 +9,9 @@
 [English](./README.md) | **简体中文**
 
 **你改完 prompt，真的变好了吗？**
-用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、Krippendorff α、长度去偏，全部默认开。
+用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、长度去偏默认开，配一份人工 gold 就自动算 Krippendorff α。
+
+📖 **完整文档：[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)**（可搜索，可切换英文）
 
 ![omk 报告 — verdict pill「v2 明显优于 v1，可以发布」](./assets/screenshots/report-overview-zh.png)
 
@@ -23,12 +25,22 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例；`omk eval` 跑控制变量 A/B，5 分钟内出 HTML 报告 + 一行 verdict。
 
+> 命令行有新版本时会自动提示（每 20 小时最多一次）；想永久关闭该提醒，设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
+
 手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。
 
-深入：[CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测样本格式](docs/zh/reference/eval-sample-format.md) · [执行器与 artifact 布局](docs/zh/reference/executors.md)
+深入：[CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
 
 ## 在 AI Coding Agent 中使用
 
+安装 omk 官方 Agent Skill 后，可以直接用自然语言让 coding agent 跑 omk 工作流：
+
+```bash
+omk install omk-agent-skill
+```
+
+默认只会安装到本机已检测到、且 omk 明确支持的目标：检测到 `~/.codex` 或 `~/.agents` 时写入 Codex/AGENTS，检测到 `~/.claude` 时写入 Claude Code。要强制写入当前 omk 已知的全部目标，用 `--to all`；要指定自定义 skill 根目录，用 `--dest`。
+
 ### 在 Claude Code 中使用
 
 当 `omk` skill 已在 Claude Code 中可用时，可以直接这样调用：
@@ -62,15 +74,15 @@ omk sample skills/my-skill.md
 | | omk | promptfoo | DeepEval | LangSmith |
 |--|--|--|--|--|
 | Bootstrap 置信区间 | ✓ 默认 | ✗ | ✗ | ✗ |
-| Krippendorff α（评委 ↔ 人工） | ✓ 默认 | ✗ | ✗ | ✗ |
+| Krippendorff α（评委 ↔ 人工） | ✓ 加 gold 即开 | ✗ | ✗ | ✗ |
 | 长度去偏的评委 prompt | ✓ 默认 | ✗ | ✗ | ✗ |
 | 饱和曲线 | ✓ | ✗ | ✗ | ✗ |
 | 三层独立评分 | ✓ | ✗ | 部分 | ✗ |
 | 用例隔离(construct validity) | ✓ 默认 | ✗ | ✗ | ✗ |
-| 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ |
+| 原生 Agent Skill | ✓ | ✗ | ✗ | ✗ |
 | 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✓ |
 
-omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag，是默认行为。其他工具让你**手动**接置信区间；omk 让你**默认无法忽略**它。需要 SaaS 看板？选 LangSmith。要快速 prompt 迭代不要统计层？选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」？选 omk。**
+omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 长度去偏不是 advanced flag，是默认行为；评委 ↔ 人工 α 只要给一份 gold 集就自动算。其他工具让你**手动**接置信区间；omk 让你**默认无法忽略**它。需要 SaaS 看板？选 LangSmith。要快速 prompt 迭代不要统计层？选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」？选 omk。**
 
 RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比（7 个工具 × 25+ 维度）： [docs/zh/reference/comparison.md](docs/zh/reference/comparison.md)
 
@@ -80,15 +92,15 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 |------|------|
 | **Verdict 一行结论** | `omk eval` 六档判定 + ship 建议 + exit code 路由，与 HTML 报告 verdict pill 共享规则 |
 | **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
-| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini 及自定义命令 |
-| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
-| **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/explanation/statistical-rigor.md) |
+| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / Anthropic API 及自定义命令 |
+| **30+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
+| **统计严谨性** | Bootstrap CI / 长度去偏 / 饱和曲线默认开，Krippendorff α 提供 gold 集即自动计算。[详情 →](docs/zh/explanation/statistical-rigor.md) |
 | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
 | **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分；`--static-only` 可离线无 LLM 调用 |
 | **线上 session 观测** | 解析 Claude Code session JSONL，测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
 | **知识缺口识别** | 严重度加权的信号量化风险敞口，不宣称完备性 |
 | **用例隔离 (construct validity)** | `--strict-baseline`（默认开）三堵 baseline 拿到被测 skill 的污染路径 |
-| **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段（HF Dataset Cards 风），studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
+| **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段（HF Dataset Cards 风），studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/zh/specs/sample-design-spec.md](docs/zh/specs/sample-design-spec.md) |
 | **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
 | **盲测 A/B** | `--blind` 隐藏变体名称，HTML 报告有揭晓按钮 |
 | **多轮方差分析** | `--repeat N` 重复 N 次，计算均值/标准差/置信区间/t 检验 |
@@ -99,14 +111,17 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 
 ## 文档
 
+完整文档已发布到 **[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)** —— 可搜索，可切换英文。重点页面：
+
 - **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
-- **[评测样本格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、21+ 断言类型、自定义 JS 断言
-- **[CLI 参考](docs/zh/reference/cli.md)** —— 7 个命令的 bash 示例和 flag 表
-- **[执行器与 artifact 布局](docs/zh/reference/executors.md)** —— 内置 / 自定义执行器、Agent 评测、常见模型配置（Claude / OpenAI / GLM / 通义 / DeepSeek / Moonshot / Ollama）
+- **[评测用例格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、30+ 断言类型、自定义 JS 断言
+- **[CLI 参考](docs/zh/reference/cli.md)** —— 顶层命令的 bash 示例和 flag 表
+- **[执行器](docs/zh/reference/executors.md)** & **[artifact 布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器；variant 如何解析为 artifact + runtime context
+- **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)（项目 runtime context）与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)（GLM / 通义 / DeepSeek / Moonshot / Ollama）
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
-- **[用例设计规范](docs/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射
+- **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射
 - **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
-- **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / LangSmith / Langfuse / Braintrust 等 25+ 维度横评
+- **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai 等 25+ 维度横评
 
 ## 环境变量
 
@@ -117,9 +132,9 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 
 ## 系统要求
 
-- Node.js >= 20
+- Node.js >= 22
 - `claude` CLI（默认执行器和 LLM 评委需要，参考 [Claude Code](https://claude.ai/code)）
-  - 如果使用其它执行器（openai / gemini）+ `--no-judge` 则可不需要
+  - 如果使用其它执行器（openai-api / anthropic-api / gemini）+ `--no-judge` 则可不需要
 
 ## 安全说明
 
@@ -128,7 +143,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 | 功能 | 风险 | 适用场景 |
 |------|------|----------|
 | **自定义断言**（`custom`） | 动态加载并执行用户指定的 `.mjs` 文件 | 仅使用自己编写或审查过的断言文件 |
-| **eval-samples.json** | 断言配置可引用外部文件路径 | 不要使用来源不明的样本文件 |
+| **eval-samples.json** | 断言配置可引用外部文件路径 | 不要使用来源不明的用例文件 |
 
 **建议：**
 

package/dist/analysis/coverage-analyzer.d.ts

@@ -54,4 +54,3 @@ export declare function computeCoverage(results: ResultEntry[], variant: string,
  * Returns a map of variant → CoverageReport.
  */
 export declare function computeReportCoverage(report: Report, artifactContents: Record<string, string | null>, cwds: Record<string, string | null>): Record<string, CoverageReport>;
-//# sourceMappingURL=coverage-analyzer.d.ts.map

package/dist/analysis/coverage-analyzer.js

@@ -259,4 +259,3 @@ export function computeReportCoverage(report, artifactContents, cwds) {
     }
     return coverageMap;
 }
-//# sourceMappingURL=coverage-analyzer.js.map

package/dist/analysis/failure-clusterer.d.ts

@@ -93,4 +93,3 @@ export interface FailureClusterReport {
 }
 export declare function clusterFailures(req: FailureClusterRequest): Promise<FailureClusterReport>;
 export declare function formatFailureClusterReport(r: FailureClusterReport): string;
-//# sourceMappingURL=failure-clusterer.d.ts.map

package/dist/analysis/failure-clusterer.js

@@ -295,4 +295,3 @@ export function formatFailureClusterReport(r) {
     lines.push('');
     return lines.join('\n');
 }
-//# sourceMappingURL=failure-clusterer.js.map

package/dist/analysis/gap-analyzer.d.ts

@@ -118,4 +118,3 @@ export declare function applyHedgingClassifierToReports(reports: Record<string,
     reports: Record<string, GapReport>;
     costUSD: number;
 }>;
-//# sourceMappingURL=gap-analyzer.d.ts.map

package/dist/analysis/gap-analyzer.js

@@ -468,4 +468,3 @@ export async function applyHedgingClassifierToReports(reports, executor, opts) {
     }
     return { reports: out, costUSD: totalCost };
 }
-//# sourceMappingURL=gap-analyzer.js.map

package/dist/analysis/hedging-classifier.d.ts

@@ -31,4 +31,3 @@ export declare function clearHedgingCache(): void;
  * 失败降级:单批 LLM 调用 / 解析失败 → 该 batch 全部 isUncertainty=true (保守保留)。
  */
 export declare function classifyHedgingCandidates(candidates: HedgingCandidate[], executor: ExecutorFn, opts?: ClassifyOptions): Promise<ClassifyResult>;
-//# sourceMappingURL=hedging-classifier.d.ts.map

package/dist/analysis/hedging-classifier.js

@@ -12,7 +12,7 @@
 import { createHash } from 'node:crypto';
 const DEFAULT_MAX_CANDIDATES = 50;
 const DEFAULT_BATCH_SIZE = 10;
-const DEFAULT_MODEL = 'claude-haiku-4-5-20251001';
+const DEFAULT_MODEL = 'claude-haiku-4-5';
 // in-memory cache,进程内复用。key = sha256(sentence)。
 const verdictCache = new Map();
 export function clearHedgingCache() {
@@ -141,4 +141,3 @@ export async function classifyHedgingCandidates(candidates, executor, opts) {
     }
     return { verdicts, costUSD, truncated };
 }
-//# sourceMappingURL=hedging-classifier.js.map

package/dist/analysis/report-diagnostics.d.ts

@@ -31,4 +31,3 @@ export declare function analyzeResults(report: Report, opts?: AnalyzeResultsOpti
  */
 export declare function buildSampleQualityAggregate(samples: Sample[]): SampleQualityAggregate;
 export declare function generateAnalysisSummary(report: Report, lang?: Lang): string | undefined;
-//# sourceMappingURL=report-diagnostics.d.ts.map

package/dist/analysis/report-diagnostics.js

@@ -750,4 +750,3 @@ function detectHighCost(results, variants, insights) {
         });
     }
 }
-//# sourceMappingURL=report-diagnostics.js.map

package/dist/analysis/sample-diagnostics.d.ts

@@ -28,7 +28,7 @@ export type SampleIssueKind = 'flat_scores' | 'all_pass' | 'all_fail' | 'near_du
 export interface SampleIssue {
     sample_id: string;
     severity: 'error' | 'warning' | 'info';
-    kind: SampleIssueKind;
+    issueKind: SampleIssueKind;
     /** Minimal evidence to make the issue actionable. */
     evidence: Record<string, unknown>;
 }
@@ -82,4 +82,3 @@ export declare function formatSampleDiagnostics(diag: SampleDiagnosticReport, op
     lang?: DiagnosticLang;
 }): string;
 export {};
-//# sourceMappingURL=sample-diagnostics.d.ts.map

package/dist/analysis/sample-diagnostics.js

@@ -69,19 +69,19 @@ export function diagnoseSamples(report, options = {}) {
             const min = Math.min(...scores);
             if (max === 5 && min === 5) {
                 issues.push({
-                    sample_id: entry.sample_id, severity: 'info', kind: 'all_pass',
+                    sample_id: entry.sample_id, severity: 'info', issueKind: 'all_pass',
                     evidence: { scores: scoresMap(entry, variants) },
                 });
             }
             else if (max === 1 && min === 1) {
                 issues.push({
-                    sample_id: entry.sample_id, severity: 'error', kind: 'all_fail',
+                    sample_id: entry.sample_id, severity: 'error', issueKind: 'all_fail',
                     evidence: { scores: scoresMap(entry, variants) },
                 });
             }
             else if (max - min < opt.flatThreshold) {
                 issues.push({
-                    sample_id: entry.sample_id, severity: 'warning', kind: 'flat_scores',
+                    sample_id: entry.sample_id, severity: 'warning', issueKind: 'flat_scores',
                     evidence: { scores: scoresMap(entry, variants), spread: Number((max - min).toFixed(2)), threshold: opt.flatThreshold },
                 });
             }
@@ -89,7 +89,7 @@ export function diagnoseSamples(report, options = {}) {
         // Errored on at least one variant — sample may be broken (env / executor / fixture).
         if (errors > 0) {
             issues.push({
-                sample_id: entry.sample_id, severity: errors === variants.length ? 'error' : 'warning', kind: 'error_prone',
+                sample_id: entry.sample_id, severity: errors === variants.length ? 'error' : 'warning', issueKind: 'error_prone',
                 evidence: { errorCount: errors, variantCount: variants.length },
             });
         }
@@ -97,7 +97,7 @@ export function diagnoseSamples(report, options = {}) {
         const maxStddev = judgeStddevs.length > 0 ? Math.max(...judgeStddevs) : 0;
         if (maxStddev >= opt.ambiguousStddev) {
             issues.push({
-                sample_id: entry.sample_id, severity: 'warning', kind: 'ambiguous_rubric',
+                sample_id: entry.sample_id, severity: 'warning', issueKind: 'ambiguous_rubric',
                 evidence: { maxStddev: Number(maxStddev.toFixed(2)), threshold: opt.ambiguousStddev, stddevs: judgeStddevs.map((s) => Number(s.toFixed(2))) },
             });
         }
@@ -110,13 +110,13 @@ export function diagnoseSamples(report, options = {}) {
     for (const s of sampleStats) {
         if (medianCost > 0 && s.cost >= opt.costOutlierK * medianCost) {
             issues.push({
-                sample_id: s.entry.sample_id, severity: 'info', kind: 'cost_outlier',
+                sample_id: s.entry.sample_id, severity: 'info', issueKind: 'cost_outlier',
                 evidence: { cost: Number(s.cost.toFixed(4)), medianCost: Number(medianCost.toFixed(4)), multiplier: opt.costOutlierK },
             });
         }
         if (medianLatency > 0 && s.latencyMs >= opt.latencyOutlierK * medianLatency) {
             issues.push({
-                sample_id: s.entry.sample_id, severity: 'info', kind: 'latency_outlier',
+                sample_id: s.entry.sample_id, severity: 'info', issueKind: 'latency_outlier',
                 evidence: { latencyMs: s.latencyMs, medianMs: medianLatency, multiplier: opt.latencyOutlierK },
             });
         }
@@ -145,7 +145,7 @@ export function diagnoseSamples(report, options = {}) {
                         continue;
                     seenPair.add(key);
                     issues.push({
-                        sample_id: prompts[i].id, severity: 'warning', kind: 'near_duplicate',
+                        sample_id: prompts[i].id, severity: 'warning', issueKind: 'near_duplicate',
                         evidence: { duplicateOf: prompts[j].id, rouge1: Number(score.toFixed(2)), threshold: opt.duplicateRouge },
                     });
                 }
@@ -170,7 +170,7 @@ export function diagnoseSamples(report, options = {}) {
             if (containsRubricGradeKeyword(rubric))
                 continue;
             issues.push({
-                sample_id: entry.sample_id, severity: 'info', kind: 'rubric_clarity_low',
+                sample_id: entry.sample_id, severity: 'info', issueKind: 'rubric_clarity_low',
                 evidence: { rubricLength: rubric.length, rubricSnippet: rubric.slice(0, 80) },
             });
         }
@@ -205,7 +205,7 @@ export function diagnoseSamples(report, options = {}) {
                 // 报警挂在该 capability 的第一个 sample 上(便于定位),其他在 evidence 里列。
                 const primarySampleId = info.sampleIds[0];
                 issues.push({
-                    sample_id: primarySampleId, severity: 'warning', kind: 'capability_thin',
+                    sample_id: primarySampleId, severity: 'warning', issueKind: 'capability_thin',
                     evidence: { capability: cap, sampleCount: info.count, threshold, sampleIds: info.sampleIds },
                 });
             }
@@ -215,10 +215,10 @@ export function diagnoseSamples(report, options = {}) {
     issues.sort((a, b) => SEVERITY_RANK[a.severity] - SEVERITY_RANK[b.severity] || a.sample_id.localeCompare(b.sample_id));
     const byKind = {};
     for (const i of issues) {
-        if (!byKind[i.kind])
-            byKind[i.kind] = [];
-        if (!byKind[i.kind].includes(i.sample_id))
-            byKind[i.kind].push(i.sample_id);
+        if (!byKind[i.issueKind])
+            byKind[i.issueKind] = [];
+        if (!byKind[i.issueKind].includes(i.sample_id))
+            byKind[i.issueKind].push(i.sample_id);
     }
     const totals = {
         samples: results.length,
@@ -301,7 +301,7 @@ function evidenceString(evidence, key, fallback = '') {
 }
 export function formatSampleIssue(issue, lang = 'zh') {
     const evidence = issue.evidence;
-    switch (issue.kind) {
+    switch (issue.issueKind) {
         case 'all_pass':
             return lang === 'zh'
                 ? '所有 variant 得分均为 5——用例可能太简单或断言过宽'
@@ -371,8 +371,8 @@ export function formatSampleIssue(issue, lang = 'zh') {
         }
         default:
             return lang === 'zh'
-                ? `结构化诊断：${issue.kind}`
-                : `Structured diagnostic: ${issue.kind}`;
+                ? `结构化诊断：${issue.issueKind}`
+                : `Structured diagnostic: ${issue.issueKind}`;
     }
 }
 /**
@@ -401,7 +401,7 @@ export function formatSampleDiagnostics(diag, options = {}) {
         if (ids.length === 0)
             continue;
         lines.push(`  [${kind}] ${ids.length} sample(s)`);
-        const matching = diag.issues.filter((i) => i.kind === kind);
+        const matching = diag.issues.filter((i) => i.issueKind === kind);
         const display = topN ? matching.slice(0, topN) : matching;
         for (const issue of display) {
             const sev = issue.severity === 'error' ? '✗' : issue.severity === 'warning' ? '⚠' : 'ℹ';
@@ -455,4 +455,3 @@ export function formatSampleDiagnostics(diag, options = {}) {
     lines.push('');
     return lines.join('\n');
 }
-//# sourceMappingURL=sample-diagnostics.js.map

package/dist/analysis/saturation.d.ts

@@ -28,6 +28,14 @@
  * that look like convergence but aren't.
  */
 export type SaturationMethod = 'slope' | 'bootstrap-ci-width' | 'plateau-height';
+/** Default consecutive-window run length required before declaring saturation. */
+export declare const DEFAULT_SATURATION_WINDOW_SIZE = 3;
+/**
+ * Default `bootstrap-ci-width` cutoff: declare saturation when the relative
+ * CI-width shrink per checkpoint stays under 5%. Single source of truth for
+ * the documented threshold; guarded by `doc-constants-drift.test.ts`.
+ */
+export declare const DEFAULT_CI_WIDTH_SHRINK_THRESHOLD = 0.05;
 /**
  * One observation in a saturation curve. `n` is the cumulative sample
  * count at this checkpoint; `mean` and (optional) `ciWidth` come from
@@ -82,4 +90,3 @@ export declare function findSaturationPoint(cumulativeScores: number[][], method
  * Use this when feeding `runMultiple` results to `findSaturationPoint`.
  */
 export declare function buildCumulativeScores(runs: number[][]): number[][];
-//# sourceMappingURL=saturation.d.ts.map

package/dist/analysis/saturation.js

@@ -27,7 +27,15 @@
  * window run-of-success is required. This guards against random dips
  * that look like convergence but aren't.
  */
-import { bootstrapMeanCI } from '../eval-core/bootstrap.js';
+import { bootstrapMeanCI, DEFAULT_BOOTSTRAP_ALPHA, DEFAULT_BOOTSTRAP_SAMPLES } from '../eval-core/bootstrap.js';
+/** Default consecutive-window run length required before declaring saturation. */
+export const DEFAULT_SATURATION_WINDOW_SIZE = 3;
+/**
+ * Default `bootstrap-ci-width` cutoff: declare saturation when the relative
+ * CI-width shrink per checkpoint stays under 5%. Single source of truth for
+ * the documented threshold; guarded by `doc-constants-drift.test.ts`.
+ */
+export const DEFAULT_CI_WIDTH_SHRINK_THRESHOLD = 0.05;
 /**
  * Compute saturation from a sequence of cumulative score arrays.
  *
@@ -44,13 +52,13 @@ import { bootstrapMeanCI } from '../eval-core/bootstrap.js';
  * @param bootstrapSamples Used only by 'bootstrap-ci-width' (default 1000).
  * @param seed             Optional seed for reproducible CIs.
  */
-export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-width', threshold, windowSize = 3, bootstrapSamples = 1000, seed) {
+export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-width', threshold, windowSize = DEFAULT_SATURATION_WINDOW_SIZE, bootstrapSamples = DEFAULT_BOOTSTRAP_SAMPLES, seed) {
     const trace = [];
     const checkpoints = [];
     for (const scores of cumulativeScores) {
         if (scores.length === 0)
             continue;
-        const ci = bootstrapMeanCI(scores, 0.05, bootstrapSamples, seed);
+        const ci = bootstrapMeanCI(scores, DEFAULT_BOOTSTRAP_ALPHA, bootstrapSamples, seed);
         checkpoints.push({
             n: scores.length,
             mean: ci.estimate,
@@ -151,7 +159,7 @@ export function findSaturationPoint(cumulativeScores, method = 'bootstrap-ci-wid
 function defaultThreshold(method) {
     switch (method) {
         case 'slope': return 0.005;
-        case 'bootstrap-ci-width': return 0.05;
+        case 'bootstrap-ci-width': return DEFAULT_CI_WIDTH_SHRINK_THRESHOLD;
         case 'plateau-height': return 0.1;
     }
 }
@@ -171,4 +179,3 @@ export function buildCumulativeScores(runs) {
     }
     return out;
 }
-//# sourceMappingURL=saturation.js.map