oh-my-knowledge 0.22.0 → 0.24.0

package/README.md

@@ -7,49 +7,15 @@
 
 **English** | [简体中文](./README.zh.md)
 
-**omk** — LLM evaluation framework with built-in statistical rigor. Bootstrap CI / Krippendorff α / length-debias / saturation curves out of the box. Native support for Claude Code skills, prompts, agents, and RAG.
+**omk** — The knowledge you give your LLM — what's it actually worth?
+omk answers with objective data, not gut feeling.
 
-**Fix the model, vary the knowledge artifact, let the data speak.**
+**Fix the model, vary the knowledge artifact.**
 
-## Why this tool
-
-Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
-
-## Key features
-
-- **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
-- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
-- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
-- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
-- **Pre-merge CI gate** — `omk bench gate` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
-- **One-line ship/no-ship verdict** — `omk bench verdict <reportId>` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
-
-### Statistical rigor
+<a id="statistical-rigor"></a>
+> Built-in: Bootstrap CI · Krippendorff α (judge ↔ human) · length-debias · saturation curves · construct-validity isolation. [Why these matter →](docs/statistical-rigor.md)
 
-The biggest LLM-eval failure mode is "confident bias" — narrow CIs around the wrong answer. omk's statistical layer ships four pieces so conclusions can be externally audited:
-
-- **Bootstrap CI** (`--bootstrap`) — distribution-free confidence intervals. The t-test breaks on ordinal LLM scores; bootstrap resamples raw observations and stays valid at small N (< 30) and on skewed data. Pairwise diff CI not crossing 0 = significant.
-- **Human Gold + Krippendorff α** (`--gold-dir`) — bring an external annotation as anchor. CI tells you "is the judge stable", α tells you "is the judge correct" — two complementary axes. omk warns when the gold annotator and the judge are the same model (would inflate α).
-- **Length-controlled judge prompt** (default ON) — research shows LLM judges over-weight verbosity. omk's judge prompt explicitly states "length is not a quality signal"; template hash is `v3-cot-length` so older reports (with the legacy hash) are visibly different. `omk bench debias-validate length <reportId>` re-judges with the opposite setting and reports the score shift.
-- **Saturation curve** — answers "have I run enough samples?". With `--repeat ≥ 5` we accumulate cumulative N → bootstrap CI; when CI shrink rate stays under 5% across 3 windows, more samples buy nothing. The HTML report inlines the SVG curve plus a verdict.
-
-## Why omk over alternatives
-
-| | omk | promptfoo | DeepEval | RAGAS | LangSmith |
-|--|--|--|--|--|--|
-| Bootstrap CI | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Krippendorff α (judge ↔ human) | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ | ✗ |
-| Saturation curve | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Three-layer scoring isolation | ✓ | ✗ | partial | ✗ | ✗ |
-| Per-variant skill isolation (construct validity) | ✓ default | ✗ | ✗ | ✗ | ✗ |
-| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Full Chinese docs | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✗ | ✓ |
-
-omk's moat is **statistical rigor** — every conclusion is auditable by a researcher. If you need a hosted SaaS dashboard, choose LangSmith. If you want quick local prompt iteration without statistics, choose promptfoo. **If you ship to production and someone will ask "why should I trust this number?", choose omk**.
-
-Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md)
+![omk report](./assets/screenshots/report-overview.png)
 
 ## Quick start
 
@@ -70,7 +36,7 @@ cd my-eval
 omk bench run --dry-run
 
 # run the evaluation (auto-discovers everything under skills/)
-omk bench run
+omk bench run    # → HTML report with verdict in 5 minutes
 
 # CLI output language: zh (default) / en — flag wins over env
 omk bench run --lang en
@@ -89,26 +55,57 @@ After installing omk, talk to it in natural language from Claude Code:
 
 You can also just say "compare v1 vs v2 for me" or "improve this artifact" — omk picks the right command.
 
+## Why this tool
+
+Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
+
+## Key features
+
+- **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
+- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
+- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
+- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
+- **Pre-merge CI gate** — `omk bench gate` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
+- **One-line ship/no-ship verdict** — `omk bench verdict <reportId>` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
+
+## Why omk over alternatives
+
+| | omk | promptfoo | DeepEval | LangSmith |
+|--|--|--|--|--|
+| Bootstrap CI | ✓ default | ✗ | ✗ | ✗ |
+| Krippendorff α (judge ↔ human) | ✓ default | ✗ | ✗ | ✗ |
+| Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ |
+| Saturation curve | ✓ | ✗ | ✗ | ✗ |
+| Three-layer scoring isolation | ✓ | ✗ | partial | ✗ |
+| Per-variant skill isolation (construct validity) | ✓ default | ✗ | ✗ | ✗ |
+| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ |
+| Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✓ |
+
+omk's moat is **default-on safety net** — Bootstrap CI, judge ↔ human α, and length-debias aren't advanced flags; they're the default. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
+
+RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md).
+
 ## Features
 
 | Feature | What it does |
 |---|---|
-| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
-| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
-| **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
-| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve |
-| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against: (1) SDK skill auto-discovery, (2) subagent Skill tool, (3) cwd file-system access via the `skills/<name>/` symlink that's normally there for the treatment variant. eval.yaml `allowedSkills` for per-variant whitelists |
 | **One-line verdict** | `omk bench verdict <id>` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
-| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
+| **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
+| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / any custom command |
+| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
+| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/statistical-rigor.md) |
 | **Sample diagnostics** | `omk bench diagnose <id>` — 7 issue kinds (low discrimination / duplicates / ambiguous rubric / cost outliers / etc.) + 0-100 healthScore |
 | **Failure clustering** | `omk bench failures <id>` — single LLM call clusters failed samples and emits per-cluster fixes |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
 | **Hard budget caps** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` — abort on total-cost overrun, flag per-sample overruns; partial report persisted |
-| **Multi-executor** | Claude CLI / Claude SDK / OpenAI / Gemini / any custom command |
+| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against: (1) SDK skill auto-discovery, (2) subagent Skill tool, (3) cwd file-system access via the `skills/<name>/` symlink that's normally there for the treatment variant. eval.yaml `allowedSkills` for per-variant whitelists |
+| **Sample design science** | Sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata fields (HF Dataset Cards style). `bench diagnose` shows coverage breakdown + flags `rubric_clarity_low` (short rubric without grading keywords) / `capability_thin` (capability supported by ≤ N×0.2 samples). `bench gen-samples` auto-stamps provenance. See [docs/sample-design-spec.md](docs/sample-design-spec.md) for the 8 industry-gap mapping |
 | **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
 | **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
-| **Parallel execution** | `--concurrency N` runs N tasks at once |
 | **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
+| **Parallel execution** | `--concurrency N` runs N tasks at once |
+| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
 | **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
 | **Traceability** | reports carry CLI version, Node version, artifact version fingerprint, judge prompt hash |
 | **EN / ZH switch** | one-click language toggle in the HTML report |
@@ -134,8 +131,8 @@ flowchart TD
     end
 
     subgraph Exec["④ Executor (fixed model)"]
-        E["claude / claude-sdk / openai / gemini<br/>anthropic-api / openai-api / custom"]
-        T["claude-sdk extracts<br/>turns / toolCalls trace"]
+        E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / custom"]
+        T["claude-sdk / codex extract<br/>turns / toolCalls trace"]
         E -.-> T
     end
 
@@ -372,13 +369,13 @@ options:
   --control <expr>       control-group variant expression (experiment role = control)
   --treatment <v1,v2>    treatment-group variant expressions, comma-separated
                          at least one of --control / --treatment is required
-                         (unless you use --config or --each)
+                         (unless you use --config or --batch)
                          special values: baseline (empty artifact), git:name (git HEAD),
                          git:ref:name (specific commit), path with "/" (read file directly)
   --config <path>        YAML/JSON config file (evaluation-as-code); declares
                          samples + variants + model + executor in one file; CLI
                          flags override config fields when both are provided
-  --model <name>         model under test (default: sonnet)
+  --model <name>         task execution model (default: sonnet)
   --judge-model <name>   judge model (default: haiku)
   --output-dir <path>    output dir (default: ~/.oh-my-knowledge/reports/)
   --no-judge             skip the LLM judge
@@ -394,7 +391,7 @@ options:
                          (default: .mcp.json in cwd)
   --no-serve             don't auto-start the report server after the run
   --verbose              print per-sample details (duration, tokens, output preview)
-  --each                 batch mode: evaluate each artifact independently vs baseline
+  --batch                batch mode: evaluate each artifact independently vs baseline
                          requires {name}.eval-samples.json paired with each artifact
   --judge-repeat <n>     run the LLM judge N times per (sample × dimension) and report stddev
   --judge-models <list>  multi-judge ensemble: "executor1:model1,executor2:model2"
@@ -419,9 +416,9 @@ options:
 
 **Difference from `cost_max` / `latency_max` assertions**: assertions are **per-sample scoring rules** (exceeding the cap fails that one assertion, the run continues); budget caps are **workflow-level hard limits** (`totalUSD` overrun aborts the run and persists a partial report; per-sample overruns fail the offending sample but the run continues). Assertions answer "is quality acceptable?"; budgets answer "are cost/time within the envelope?".
 
-### `omk bench run --each` (batch mode)
+### `omk bench run --batch` (batch mode)
 
-When `skills/` contains several **independent** artifacts, use `--each` to evaluate each one against baseline and produce a merged report.
+When `skills/` contains several **independent** artifacts, use `--batch` to evaluate each one against baseline and produce a BatchEvaluationReport with child EvaluationReports.
 
 ```
 skills/
@@ -441,8 +438,8 @@ Pairing rules:
 - artifacts without paired samples are skipped with a warning
 
 ```bash
-omk bench run --each
-omk bench run --each --dry-run
+omk bench run --batch
+omk bench run --batch --dry-run
 ```
 
 ### `omk bench gen-samples` (generate test cases)
@@ -454,7 +451,7 @@ Reads an artifact's content and uses an LLM to auto-generate eval-samples. Revie
 omk bench gen-samples skills/my-skill.md
 
 # batch-generate for every artifact under skills/ that lacks samples
-omk bench gen-samples --each
+omk bench gen-samples --batch
 
 # specify sample count
 omk bench gen-samples skills/my-skill.md --count 10
@@ -463,10 +460,10 @@ omk bench gen-samples skills/my-skill.md --count 10
 Options:
 
 ```
-  --each                 batch-generate for every artifact missing samples
+  --batch                batch-generate for every artifact missing samples
   --count <n>            samples per artifact (default: 5)
   --model <name>         model used for generation (default: sonnet)
-  --skill-dir <path>     artifact dir (default: skills), used with --each
+  --skill-dir <path>     artifact dir (default: skills), used with --batch
 ```
 
 ### `omk bench evolve` (self-iterating improvement)
@@ -670,13 +667,16 @@ The command writes `~/.oh-my-knowledge/analyses/<timestamp>-skill-health.json`.
 |---|---|---|
 | `claude` | default | invokes `claude -p` via Claude CLI |
 | `claude-sdk` | structured output | uses Claude Agent SDK — no stdout parsing, avoids buffer truncation |
-| `openai` | cross-vendor comparison | invokes `openai api` CLI |
+| `codex` | OpenAI agent CLI | invokes `codex exec --json` (`@openai/codex` npm); best-effort tool trace; **costUSD not reported** (codex CLI does not emit USD; check usage externally) |
+| `codex-sdk` | OpenAI agent SDK | uses `@openai/codex-sdk` with its bundled `@openai/codex` binary and streamed SDK events; **costUSD not reported** |
 | `gemini` | cross-vendor comparison | invokes `gemini` CLI |
 | `anthropic-api` | no CLI needed | calls Anthropic HTTP API directly (needs `ANTHROPIC_API_KEY`) |
 | `openai-api` | no CLI needed | calls OpenAI HTTP API directly (needs `OPENAI_API_KEY`) |
 
 API-direct executors support custom base URLs via env: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
 
+Codex construct-validity notes: (1) `codex` uses the `codex` binary on `PATH`; `codex-sdk` uses the bundled `@openai/codex` binary resolved by `@openai/codex-sdk`. Reports persist per-variant `meta.executorRuntimes` plus `meta.executorRuntime` / `meta.judgeRuntime` fingerprints (binary or SDK version + capability snapshot), and `bench diff` / `bench verdict` warn when strict comparability cannot be audited. If runtime fingerprints differ, treat results as an executor-runtime comparison, not only prompt/template behavior. (2) Both executors isolate user-level config: `codex` passes `--ephemeral` + `--ignore-user-config`; `codex-sdk` redirects `$CODEX_HOME` to a per-process tmp dir (auth.json symlinked through). User-level `~/.codex/config.toml` does not leak into eval runs in either case.
+
 ### Custom executor
 
 Any shell command can serve as an executor, communicating via stdin/stdout JSON:
@@ -720,7 +720,7 @@ skills/
 | `./path/to/file.md` | path with `/`: read the file directly as an artifact |
 | `variant@/path/to/project` | attach a run dir to any variant; supports `name@cwd`, `git:name@cwd`, `/file.md@cwd` |
 
-When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--each`. With `--each`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
+When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--batch`. With `--batch`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
 
 ```bash
 # explicit: one control, one or more treatments

package/README.zh.md

@@ -7,47 +7,15 @@
 
 [English](./README.md) | **简体中文**
 
-**omk** — 内置统计严谨性的 LLM 评测框架。Bootstrap CI / Krippendorff α / 长度偏差校正 / 饱和曲线开箱即用。原生支持 Claude Code skill、prompt、agent、RAG。
+**omk** — 你给 LLM 的知识,价值在哪里?
+omk 帮你用客观数据回答,而不是凭感觉。
 
-**固定模型，只变知识载体，数据说话。**
+**固定模型,只变知识载体。**
 
-## 为什么需要这个工具
-
-做知识工程的团队会产出大量知识载体（当前常见是 skill，也包括 prompt、agent、workflow 等）。当被问到"v2 比 v1 好在哪"时，需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题：相同模型、相同测试样本，只改变知识载体。
-
-## 核心能力
-
-- **控制变量离线评测** — 固定模型和样本，只变知识载体；兼容 Claude Code skill、CLAUDE.md prompt、RAG 知识库等任何 markdown 形式的指令
-- **六维独立打分** — Fact / Behavior / LLM-judge / Cost / Efficiency / Stability 分别出信号，单一维度的回退不会被其他维度的收益掩盖
-- **线上 session 观测** — 解析 Claude Code session JSONL，在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
-- **知识缺口识别** — 严重度加权的信号（显式标记 / 搜索失败 / hedging 用语 / 反复失败）量化风险敞口,不宣称完备性
-- **合并前 CI 门** — `omk bench gate` 强制三层 all-pass（fact + behavior + llm-judge），抓复合分掩盖的单层回退
-- **一行 ship/no-ship 结论** — `omk bench verdict <reportId>` 聚合 bootstrap CI / 三层 ci-gate / saturation / human α,给六档 verdict（PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO）+ 行动建议;exit code 反映是否可 ship
-
-### 统计严谨性
-LLM 评测最容易踩的坑是"自信的偏差"——CI 很窄但结论错。omk 的统计层做四件事让结论可被外部审计：
+<a id="statistical-rigor"></a>
+> 默认带:Bootstrap 置信区间 · Krippendorff α(评委 ↔ 人工)· 长度去偏 · 饱和曲线 · 用例隔离(construct validity)。[这些为什么重要 →](docs/zh/statistical-rigor.md)
 
-- **Bootstrap CI** (`--bootstrap`) — 不假设分布的置信区间。t 检验在 LLM 序数评分上失效，bootstrap 直接重采样原始数据，对小 N（< 30）和偏态分布都稳。pairwise diff CI 不含 0 = 显著差异。
-- **Human Gold + Krippendorff α** (`--gold-dir`) — 引入外部标注作为锚点。CI 解决"评委稳不稳"，α 解决"评委对不对"——两个维度互补。omk 自动检测污染（gold annotator 与 judge 同模型时警告）。
-- **Length-controlled judge prompt** (默认开启) — 研究证实 LLM 评委隐性偏向更长的回答。omk 的 judge prompt 加显式段落"长度不是质量信号"，template hash 为 v3-cot-length，跟旧版本的报告 hash 肉眼可辨。`omk bench debias-validate length <reportId>` 重判检测偏差幅度。
-- **Saturation curve** — 回答"我跑够样本了吗"。`--repeat ≥ 5` 时累积 N → 均值 + bootstrap CI 序列，CI 宽度衰减率 < 5% 持续 3 个窗口判定饱和——再多样本对结论无实质收益。HTML 报告内联 SVG 曲线 + verdict。
-
-## 为什么选 omk
-
-| | omk | promptfoo | DeepEval | RAGAS | LangSmith |
-|--|--|--|--|--|--|
-| Bootstrap CI | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Krippendorff α(评委 ↔ 人工锚点） | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Length-debias 评委 prompt | ✓ 默认 | ✗ | ✗ | ✗ | ✗ |
-| 饱和曲线 | ✓ | ✗ | ✗ | ✗ | ✗ |
-| 三层独立评分 | ✓ | ✗ | 部分 | ✗ | ✗ |
-| 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ | ✗ |
-| 完整中文文档 | ✓ | ✗ | ✗ | ✗ | ✗ |
-| 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✗ | ✓ |
-
-omk 的护城河是**统计严谨性** — 每条结论都能被研究者审计。需要托管 SaaS 看板?选 LangSmith。要本地快速 prompt 迭代不要统计层?选 promptfoo。**要 ship 到生产且会被问"为什么应该相信这个数字"?选 omk**。
-
-完整对比(7 个工具 × 25+ 维度): [docs/zh/comparison.md](docs/zh/comparison.md)
+![omk 报告](./assets/screenshots/report-overview-zh.png)
 
 ## 快速开始
 
@@ -68,7 +36,7 @@ cd my-eval
 omk bench run --dry-run
 
 # 运行评测（自动发现 skills/ 目录下的所有 artifact）
-omk bench run
+omk bench run    # → 5 分钟出 HTML 报告 + verdict
 
 # CLI 输出语言: zh (默认) / en — flag 优先级高于环境变量
 omk bench run --lang en
@@ -87,27 +55,58 @@ OMK_LANG=en omk bench report
 
 或直接说"帮我评测 v1 和 v2 的差异"、"改进一下这个 artifact"，omk 会自动理解意图并调用对应命令。
 
+## 为什么需要这个工具
+
+做知识工程的团队会产出大量知识载体（当前常见是 skill，也包括 prompt、agent、workflow 等）。当被问到"v2 比 v1 好在哪"时，需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题：相同模型、相同测试用例，只改变知识载体。
+
+## 核心能力
+
+- **控制变量离线评测** — 固定模型和用例，只变知识载体；兼容 Claude Code skill、CLAUDE.md prompt、RAG 知识库等任何 markdown 形式的指令
+- **六维独立打分** — Fact / Behavior / LLM-judge / Cost / Efficiency / Stability 分别出信号，单一维度的回退不会被其他维度的收益掩盖
+- **线上 session 观测** — 解析 Claude Code session JSONL，在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
+- **知识缺口识别** — 严重度加权的信号（显式标记 / 搜索失败 / hedging 用语 / 反复失败）量化风险敞口,不宣称完备性
+- **合并前 CI 门** — `omk bench gate` 强制三层 all-pass（fact + behavior + llm-judge），抓复合分掩盖的单层回退
+- **一行 ship/no-ship 结论** — `omk bench verdict <reportId>` 聚合 bootstrap CI / 三层 ci-gate / saturation / human α,给六档 verdict(PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO)+ 行动建议;exit code 反映是否可 ship
+
+## 为什么选 omk
+
+| | omk | promptfoo | DeepEval | LangSmith |
+|--|--|--|--|--|
+| Bootstrap 置信区间 | ✓ 默认 | ✗ | ✗ | ✗ |
+| Krippendorff α(评委 ↔ 人工) | ✓ 默认 | ✗ | ✗ | ✗ |
+| 长度去偏的评委 prompt | ✓ 默认 | ✗ | ✗ | ✗ |
+| 饱和曲线 | ✓ | ✗ | ✗ | ✗ |
+| 三层独立评分 | ✓ | ✗ | 部分 | ✗ |
+| 用例隔离(construct validity) | ✓ 默认 | ✗ | ✗ | ✗ |
+| 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ |
+| 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✓ |
+
+omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag,是默认行为。其他工具让你**手动**接置信区间;omk 让你**默认无法忽略**它。需要 SaaS 看板?选 LangSmith。要快速 prompt 迭代不要统计层?选 promptfoo。**要发到生产且会被问"为什么应该相信这个数字"?选 omk。**
+
+RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比(7 个工具 × 25+ 维度): [docs/zh/comparison.md](docs/zh/comparison.md)
+
 ## 特性
 
 | 特性 | 说明 |
 |------|------|
-| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
-| **断言取反 + 组合** | 通用 `not: true` 字段 + `assert-set` (any/all) 任意嵌套 |
+| **Verdict 一行结论** | `omk bench verdict <id>` 六档判定 + ship 建议 + exit code 路由,与 HTML 报告 verdict pill 共享规则 |
 | **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
-| **统计严谨性** | Bootstrap CI / Krippendorff α / Length-debias / Saturation curve |
+| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini 及自定义命令 |
+| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
+| **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/statistical-rigor.md) |
+| **用例质量诊断** | `omk bench diagnose <id>` 7 类 issue(区分度低 / 重复 / 歧义 / 成本异常 / 全 fail 等)+ healthScore 0-100 |
+| **失败聚类 + 根因** | `omk bench failures <id>` 单 LLM 调用聚类失败用例 + 每 cluster 给修复建议 |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖,自动继承长度去偏 |
+| **预算硬阈值** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` 总成本 + 单用例成本/耗时上限,超出中止保留 partial report |
 | **用例隔离 (construct validity)** | `--strict-baseline` (默认开) 三堵 baseline 拿到被测 skill 的污染路径:(1) SDK skill auto-discovery (2) subagent Skill 工具调用 (3) cwd 文件系统(避免 baseline 顺 `skills/<name>/` symlink 直接 Read 到 SKILL.md)。eval.yaml `allowedSkills` 支持 per-variant 白名单 |
-| **Verdict 一行结论** | `omk bench verdict <id>` 六档判定 + ship 建议 + exit code 路由,与 HTML 报告 verdict pill 共享规则 |
-| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖,自动继承 length-debias |
-| **样本质量诊断** | `omk bench diagnose <id>` 7 类 issue（区分度低 / 重复 / 歧义 / 成本异常 / 全 fail 等）+ healthScore 0-100 |
-| **失败聚类 + 根因** | `omk bench failures <id>` 单 LLM 调用聚类失败样本 + 每 cluster 给修复建议 |
-| **预算硬阈值** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` 总成本 + 单样本成本/耗时上限,超出中止保留 partial report |
-| **多执行器** | 支持 Claude CLI / Claude SDK / OpenAI / Gemini 及自定义命令 |
+| **用例设计科学性 (sample design science)** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段(HF Dataset Cards 风)。`bench diagnose` 输出 coverage 分桶 + 检测 `rubric_clarity_low` / `capability_thin` 两类新 issue。`bench gen-samples` 自动给生成的用例打 provenance。详见 [docs/sample-design-spec.md](docs/sample-design-spec.md),含 8 条行业 gap(HELM / MMLU-Pro / Construct Validity / IRT / Dataset Cards / Adversarial)的 omk v1 映射 |
 | **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
 | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容（SSO 保护的知识库等） |
 | **盲测 A/B** | `--blind` 隐藏变体名称，HTML 报告有揭晓按钮 |
-| **并行执行** | `--concurrency N` 并行 N 个任务 |
 | **多轮方差分析** | `--repeat N` 重复 N 次，计算均值/标准差/置信区间/t 检验 |
-| **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本样本 |
+| **并行执行** | `--concurrency N` 并行 N 个任务 |
+| **断言取反 + 组合** | 通用 `not: true` 字段 + `assert-set` (any/all) 任意嵌套 |
+| **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
 | **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹、judge prompt hash |
 | **中英切换** | HTML 报告右上角一键切换语言 |
 
@@ -132,7 +131,7 @@ flowchart TD
     end
 
     subgraph Exec["④ 执行器(固定模型)"]
-        E["claude / claude-sdk / openai / gemini<br/>anthropic-api / openai-api / 自定义命令"]
+        E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / 自定义命令"]
         T["claude-sdk 抽取<br/>turns / toolCalls trace"]
         E -.-> T
     end
@@ -369,7 +368,7 @@ omk bench run [选项]
   --skill-dir <路径>     artifact 目录（默认：skills）
   --control <expr>       对照组变体表达式（experiment role = control）
   --treatment <v1,v2>    实验组变体表达式,逗号分隔
-                         除非用 --config 或 --each,--control / --treatment 两者至少传一个
+                         除非用 --config 或 --batch,--control / --treatment 两者至少传一个
                          特殊值：baseline（空 artifact）、git:name（git 历史版本）、
                          git:ref:name（指定 commit）、含 / 的路径（直接读取文件）
   --config <路径>        YAML/JSON 配置文件（evaluation-as-code）;在一个文件里声明
@@ -390,7 +389,7 @@ omk bench run [选项]
                          （默认：当前目录的 .mcp.json）
   --no-serve             评测完成后不自动启动报告服务
   --verbose              打印每个样本的详细执行结果（耗时、tokens、输出预览）
-  --each                 批量评测：每个 artifact 独立和 baseline 对比
+  --batch                批量评测：每个 artifact 独立和 baseline 对比
                          需要每个 artifact 配对 {name}.eval-samples.json
   --judge-repeat <n>     每条 sample × dimension 跑 LLM 评委 N 次,输出 stddev (评委自一致性)
   --judge-models <list>  多评委 ensemble: "executor1:model1,executor2:model2"
@@ -412,9 +411,9 @@ omk bench run [选项]
 
 **和 `cost_max` / `latency_max` 断言的区别**: 断言是**单样本评分维度**(超出直接打 0 分,run 继续);budget 是**工作流级硬阈值**(`totalUSD` 超出整个 run abort 保留 partial report,per-sample 超出该样本失败但 run 继续)。一个回答"质量是否达标",一个回答"花钱/时间是否在预算内"。
 
-### `omk bench run --each`（批量评测）
+### `omk bench run --batch`（批量评测）
 
-当 skills/ 下放了多个**独立的** artifact 时，使用 `--each` 逐个评测，每个 artifact 独立和 baseline 对比，生成一份合并报告。
+当 skills/ 下放了多个**独立的** artifact 时，使用 `--batch` 逐个评测，每个 artifact 独立和 baseline 对比，生成一份 BatchEvaluationReport，内部索引多个 child EvaluationReport。
 
 ```
 skills/
@@ -434,8 +433,8 @@ skills/
 - 没有配对 eval-samples 的 artifact 会被跳过并打印警告
 
 ```bash
-omk bench run --each
-omk bench run --each --dry-run
+omk bench run --batch
+omk bench run --batch --dry-run
 ```
 
 ### `omk bench gen-samples`（生成测评用例）
@@ -447,7 +446,7 @@ omk bench run --each --dry-run
 omk bench gen-samples skills/my-skill.md
 
 # 为 skills/ 下所有缺少测试集的 artifact 批量生成
-omk bench gen-samples --each
+omk bench gen-samples --batch
 
 # 指定生成数量
 omk bench gen-samples skills/my-skill.md --count 10
@@ -456,10 +455,10 @@ omk bench gen-samples skills/my-skill.md --count 10
 选项：
 
 ```
-  --each                 为所有缺少 eval-samples 的 artifact 批量生成
+  --batch                为所有缺少 eval-samples 的 artifact 批量生成
   --count <n>            每个 artifact 生成的样本数（默认：5）
   --model <名称>         生成用的模型（默认：sonnet）
-  --skill-dir <路径>     artifact 目录（默认：skills），配合 --each 使用
+  --skill-dir <路径>     artifact 目录（默认：skills），配合 --batch 使用
 ```
 
 ### `omk bench evolve`（自我循环改进）
@@ -663,13 +662,16 @@ omk analyze ~/.claude/projects/my-project --kb /path/to/project
 |--------|----------|------|
 | `claude` | 默认 | 通过 `claude -p` 调用 Claude CLI |
 | `claude-sdk` | 结构化输出 | 通过 Claude Agent SDK 调用，无 stdout 解析，避免 buffer 截断 |
-| `openai` | 跨厂商对比 | 通过 `openai api` CLI 调用 |
+| `codex` | OpenAI agent CLI | 通过 `codex exec --json` 调用,需本地装好登录的 codex(`@openai/codex`);best-effort tool trace,**costUSD 不报**(codex 自身不输出 USD,需外部账单核算) |
+| `codex-sdk` | OpenAI agent SDK | 通过 `@openai/codex-sdk` 调用其自带的 `@openai/codex` binary 和 SDK 事件流;**costUSD 不报** |
 | `gemini` | 跨厂商对比 | 通过 `gemini` CLI 调用 |
 | `anthropic-api` | 无需 CLI | 直接调用 Anthropic HTTP API（需 `ANTHROPIC_API_KEY`） |
 | `openai-api` | 无需 CLI | 直接调用 OpenAI HTTP API（需 `OPENAI_API_KEY`） |
 
 API 直调执行器支持通过环境变量自定义 Base URL：`ANTHROPIC_BASE_URL`、`OPENAI_BASE_URL`。
 
+Codex construct-validity 说明:(1) `codex` 使用 `PATH` 上找到的 `codex` binary;`codex-sdk` 使用 `@openai/codex-sdk` 解析到的自带 `@openai/codex` binary。报告会持久化 per-variant `meta.executorRuntimes` 以及 `meta.executorRuntime` / `meta.judgeRuntime` 指纹(binary 或 SDK 版本 + 能力快照),`bench diff` / `bench verdict` 会在 strict comparability 无法审计时提示。runtime 指纹不一致时,结果应解释为 executor runtime 对比,而不只是 prompt/template 行为对比。(2) 两个 executor 都隔离用户级 config:`codex` 传 `--ephemeral` + `--ignore-user-config`,`codex-sdk` 把 `$CODEX_HOME` 重定向到 per-process tmp 目录(auth.json 通过 symlink 透传)。用户的 `~/.codex/config.toml` 不会渗入任意一个 executor 的 eval。
+
 ### 自定义执行器
 
 任何 shell 命令都可以作为执行器，通过 stdin/stdout JSON 协议通信：
@@ -713,7 +715,7 @@ skills/
 | `./path/to/file.md` | 含 `/` 的路径，直接读取文件作为 artifact |
 | `variant@/path/to/project` | 给任意变体附加运行目录，支持 `name@cwd`、`git:name@cwd`、`/file.md@cwd` |
 
-`--control` 和 `--treatment` 都不传时,用 `--config eval.yaml` 或 `--each`。`--each` 模式下会自动用 `baseline` 作对照组,每个被发现的 artifact 作实验组。
+`--control` 和 `--treatment` 都不传时,用 `--config eval.yaml` 或 `--batch`。`--batch` 模式下会自动用 `baseline` 作对照组,每个被发现的 artifact 作实验组。
 
 ```bash
 # 显式:一个 control,一个或多个 treatment

package/dist/src/analysis/report-diagnostics.d.ts

@@ -1,9 +1,34 @@
 /**
  * Auto-analysis: detect patterns and generate insights from evaluation results.
  */
-import type { Report, AnalysisResult } from '../types/index.js';
+import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Lang } from '../types/index.js';
+/** opts for `analyzeResults`. Optional because most older callers don't have
+ *  samples in scope; new callers (evaluation-pipeline / evolver) pass them in to
+ *  populate `analysis.sampleQuality`. */
+export interface AnalyzeResultsOptions {
+    /** Original Sample[] from eval-samples. Enables `analysis.sampleQuality` aggregation. */
+    samples?: Sample[];
+}
 /**
- * Analyze an evaluation report and produce insights + suggestions.
+ * Analyze an evaluation report and produce structured insights.
  */
-export declare function analyzeResults(report: Report): AnalysisResult;
+export declare function analyzeResults(report: Report, opts?: AnalyzeResultsOptions): AnalysisResult;
+/**
+ * Build sample design science aggregate from sample metadata.
+ *
+ * Pure function — no result/score data needed. Reads:
+ * - `Sample.capability` (string[], normalized case-insensitive + dash/camel/underscore stripped)
+ * - `Sample.difficulty` ('easy' | 'medium' | 'hard')
+ * - `Sample.construct` (free-form string)
+ * - `Sample.provenance` ('human' | 'llm-generated' | 'production-trace')
+ * - `Sample.rubric` (for avgRubricLength)
+ *
+ * Missing fields are bucketed under the `unspecified` key in the relevant
+ * distribution map, so users see "I have N samples without difficulty declared".
+ *
+ * Used by `bench diagnose` CLI to surface coverage gaps. Does NOT participate
+ * in grading / judge / verdict. See docs/sample-design-spec.md.
+ */
+export declare function buildSampleQualityAggregate(samples: Sample[]): SampleQualityAggregate;
+export declare function generateAnalysisSummary(report: Report, lang?: Lang): string | undefined;
 //# sourceMappingURL=report-diagnostics.d.ts.map

package/dist/src/analysis/report-diagnostics.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"report-diagnostics.d.ts","sourceRoot":"","sources":["../../../src/analysis/report-diagnostics.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,EAAE,MAAM,EAAwB,cAAc,EAAE,MAAM,mBAAmB,CAAC;AAEtF;;GAEG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,MAAM,GAAG,cAAc,CA2C7D"}
+{"version":3,"file":"report-diagnostics.d.ts","sourceRoot":"","sources":["../../../src/analysis/report-diagnostics.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,EAAE,MAAM,EAAwB,cAAc,EAAE,MAAM,EAAE,sBAAsB,EAAE,IAAI,EAAE,MAAM,mBAAmB,CAAC;AAG5H;;yCAEyC;AACzC,MAAM,WAAW,qBAAqB;IACpC,yFAAyF;IACzF,OAAO,CAAC,EAAE,MAAM,EAAE,CAAC;CACpB;AAED;;GAEG;AACH,wBAAgB,cAAc,CAAC,MAAM,EAAE,MAAM,EAAE,IAAI,GAAE,qBAA0B,GAAG,cAAc,CAkD/F;AAED;;;;;;;;;;;;;;;GAeG;AACH,wBAAgB,2BAA2B,CAAC,OAAO,EAAE,MAAM,EAAE,GAAG,sBAAsB,CAuErF;AAED,wBAAgB,uBAAuB,CAAC,MAAM,EAAE,MAAM,EAAE,IAAI,GAAE,IAAW,GAAG,MAAM,GAAG,SAAS,CA+O7F"}