oh-my-knowledge 0.19.0 → 0.20.0

package/README.md

@@ -7,7 +7,7 @@
 
 **English** | [简体中文](./README.zh.md)
 
-Knowledge-artifact evaluation toolkit — measure your artifact's quality with objective data.
+**omk** — LLM evaluation framework with built-in statistical rigor. Bootstrap CI / Krippendorff α / length-debias / saturation curves out of the box. Native support for Claude Code skills, prompts, agents, and RAG.
 
 **Fix the model, vary the knowledge artifact, let the data speak.**
 
@@ -22,6 +22,33 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 - **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
 - **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
 - **Pre-merge CI gate** — `omk bench ci` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
+- **One-line ship/no-ship verdict** — `omk bench verdict <reportId>` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
+
+### Statistical rigor
+
+The biggest LLM-eval failure mode is "confident bias" — narrow CIs around the wrong answer. omk's statistical layer ships four pieces so conclusions can be externally audited:
+
+- **Bootstrap CI** (`--bootstrap`) — distribution-free confidence intervals. The t-test breaks on ordinal LLM scores; bootstrap resamples raw observations and stays valid at small N (< 30) and on skewed data. Pairwise diff CI not crossing 0 = significant.
+- **Human Gold + Krippendorff α** (`--gold-dir`) — bring an external annotation as anchor. CI tells you "is the judge stable", α tells you "is the judge correct" — two complementary axes. omk warns when the gold annotator and the judge are the same model (would inflate α).
+- **Length-controlled judge prompt** (default ON) — research shows LLM judges over-weight verbosity. omk's judge prompt explicitly states "length is not a quality signal"; template hash is `v3-cot-length` so older reports (with the legacy hash) are visibly different. `omk bench debias-validate length <reportId>` re-judges with the opposite setting and reports the score shift.
+- **Saturation curve** — answers "have I run enough samples?". With `--repeat ≥ 5` we accumulate cumulative N → bootstrap CI; when CI shrink rate stays under 5% across 3 windows, more samples buy nothing. The HTML report inlines the SVG curve plus a verdict.
+
+## Why omk over alternatives
+
+| | omk | promptfoo | DeepEval | RAGAS | LangSmith |
+|--|--|--|--|--|--|
+| Bootstrap CI | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Krippendorff α (judge ↔ human) | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ | ✗ |
+| Saturation curve | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Three-layer scoring isolation | ✓ | ✗ | partial | ✗ | ✗ |
+| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Full Chinese docs | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✗ | ✓ |
+
+omk's moat is **statistical rigor** — every conclusion is auditable by a researcher. If you need a hosted SaaS dashboard, choose LangSmith. If you want quick local prompt iteration without statistics, choose promptfoo. **If you ship to production and someone will ask "why should I trust this number?", choose omk**.
+
+Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md)
 
 ## Quick start
 
@@ -61,15 +88,23 @@ You can also just say "compare v1 vs v2 for me" or "improve this artifact" — o
 
 | Feature | What it does |
 |---|---|
-| **18 assertion types** | substring, regex, JSON Schema, semantic similarity, custom JS function, and more |
+| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
+| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
 | **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
+| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve |
+| **One-line verdict** | `omk bench verdict <id>` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
+| **Sample diagnostics** | `omk bench diagnose <id>` — 7 issue kinds (low discrimination / duplicates / ambiguous rubric / cost outliers / etc.) + 0-100 healthScore |
+| **Failure clustering** | `omk bench failures <id>` — single LLM call clusters failed samples and emits per-cluster fixes |
+| **Hard budget caps** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` — abort on total-cost overrun, flag per-sample overruns; partial report persisted |
 | **Multi-executor** | Claude CLI / Claude SDK / OpenAI / Gemini / any custom command |
+| **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
 | **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
 | **Parallel execution** | `--concurrency N` runs N tasks at once |
 | **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
 | **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
-| **Traceability** | reports carry CLI version, Node version, artifact version fingerprint |
+| **Traceability** | reports carry CLI version, Node version, artifact version fingerprint, judge prompt hash |
 | **EN / ZH switch** | one-click language toggle in the HTML report |
 
 ## How it works
@@ -112,7 +147,7 @@ flowchart TD
     end
 
     subgraph Report["⑦ Report"]
-        R["Six dims: Fact / Behavior / LLM-judge / Cost / Efficiency / Stability<br/>JSON + HTML · blind reveal<br/>CLI/Node/version fingerprint traceable"]
+        R["Six dims: Fact / Behavior / LLM-judge / Cost / Efficiency / Stability<br/>JSON + HTML · top verdict pill · blind reveal<br/>CLI/Node/version fingerprint traceable"]
     end
 
     S --> U
@@ -246,7 +281,7 @@ The judge model (default `haiku`) scores 1–5 against the rubric. In `dimension
 
 ### Assertion types
 
-**Deterministic assertions (18 total):**
+**Deterministic assertions (21+ total):**
 
 | Type | Description |
 |---|---|
@@ -259,9 +294,43 @@ The judge model (default `haiku`) scores 1–5 against the rubric. In `dimension
 | `word_count_min` / `word_count_max` | word-count bounds |
 | `contains_all` / `contains_any` | multi-value match |
 | `cost_max` / `latency_max` | cost / latency caps |
-| `semantic_similarity` | LLM-based semantic similarity |
+| `tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max` | agent tool-call assertions |
+| `tool_output_contains` / `tool_input_contains` | match content of a tool's input or output |
+| `turns_min` / `turns_max` | conversation-turn bounds |
+| `rouge_n_min` | ROUGE-N recall ≥ threshold (`reference` field holds the gold text; `n` defaults to 1; `threshold` defaults to 0.5) |
+| `levenshtein_max` | edit distance ≤ value (for "output should be near-identical to reference") |
+| `bleu_min` | BLEU-4 ≥ threshold (unsmoothed; degenerates to 0 on short text) |
+| `faithfulness` | output stays grounded in `sample.context` (anti-hallucination); LLM judge 1-5; threshold defaults to 3 |
+| `answer_relevancy` | output directly answers `sample.prompt`; catches dodging, topic drift, verbosity; threshold defaults to 3 |
+| `context_recall` | gold facts in `sample.context` are actually used in the output; `reference` may explicitly enumerate gold facts; threshold defaults to 3 |
+| `semantic_similarity` | LLM-based holistic semantic similarity (complementary to the three RAG metrics above) |
 | `custom` | custom JS function (30 s timeout) |
 
+**Universal modifier:**
+
+Any assertion takes `not: true` to invert (replaces paired `not_contains` / `not_equals` etc; legacy types remain as aliases):
+
+```yaml
+- type: regex
+  pattern: "TODO|FIXME"
+  not: true              # output must NOT contain TODO/FIXME
+```
+
+**Composition (assert-set):**
+
+`assert-set` combines child assertions with `any` (OR) or `all` (AND) and supports nesting:
+
+```yaml
+- type: assert-set
+  mode: any              # at least one child must pass (mode: 'all' = all must pass)
+  children:
+    - { type: contains, value: "parameterized" }
+    - { type: contains, value: "prepared statement" }
+    - { type: regex, pattern: "bind\\(.*\\?" }
+```
+
+Children can independently use `not: true`; nested `assert-set`s can express any boolean shape.
+
 ### Custom assertion
 
 ```js
@@ -314,14 +383,35 @@ options:
   --timeout <sec>        per-task executor timeout (default: 120)
   --repeat <n>           repeat N times for variance analysis (default: 1)
   --executor <name>      executor (default: claude); supports custom commands
-  --skip-preflight       skip pre-evaluation model reachability check
+  --skip-preflight       skip evaluation model reachability check
   --mcp-config <path>    MCP config for fetching private-doc URLs via MCP Server
                          (default: .mcp.json in cwd)
   --no-serve             don't auto-start the report server after the run
   --verbose              print per-sample details (duration, tokens, output preview)
   --each                 batch mode: evaluate each artifact independently vs baseline
                          requires {name}.eval-samples.json paired with each artifact
-```
+  --judge-repeat <n>     run the LLM judge N times per (sample × dimension) and report stddev
+  --judge-models <list>  multi-judge ensemble: "executor1:model1,executor2:model2"
+                         ≥ 2 judges enables ensemble + inter-judge agreement output
+  --bootstrap            enable distribution-free CIs: bootstrap CI per variant +
+                         pairwise diff CI (CI containing 0 = not significant)
+  --bootstrap-samples N  bootstrap resample count (default 1000)
+  --gold-dir <path>      after the run, compare scores against the gold dataset
+                         (Krippendorff α / κ / Pearson). Result is written to
+                         report.meta.humanAgreement and shown in the HTML report
+  --no-debias-length     revert to legacy v2-cot judge prompt (no "length is not
+                         a quality signal" paragraph) — for byte-compat with
+                         legacy reports whose hash predates v3-cot-length
+  --budget-usd <num>             total cost cap (USD); on overrun the run aborts and
+                                 a partial report is persisted (`report.meta.budgetExhausted = true`)
+  --budget-per-sample-usd <num>  per-sample cost cap; offending samples fail individually,
+                                 the run continues
+  --budget-per-sample-ms <num>   per-sample latency cap (ms); same semantics as cost cap
+```
+
+**eval.yaml budget**: declare `budget: { totalUSD?, perSampleUSD?, perSampleMs? }` (all optional, must be ≥ 0). CLI flags of the same name override the config values.
+
+**Difference from `cost_max` / `latency_max` assertions**: assertions are **per-sample scoring rules** (exceeding the cap fails that one assertion, the run continues); budget caps are **workflow-level hard limits** (`totalUSD` overrun aborts the run and persists a partial report; per-sample overruns fail the offending sample but the run continues). Assertions answer "is quality acceptable?"; budgets answer "are cost/time within the envelope?".
 
 ### `omk bench run --each` (batch mode)
 
@@ -423,6 +513,113 @@ omk bench report [options]
 omk bench init [dir]     # scaffold an eval project
 ```
 
+### `omk bench gold` (human gold anchor)
+
+Bring a human (or stronger-model proxy) annotation as an external anchor and compute Krippendorff α / weighted κ / Pearson against the LLM judge. Answers "is the judge correct?", complementary to Bootstrap CI's "is the judge stable?".
+
+```bash
+omk bench gold init [--out <dir>] [--annotator <id>]   # scaffold a dataset template
+omk bench gold validate <dir>                          # check schema (annotator / date / version / score range)
+omk bench gold compare <reportId> --gold-dir <dir>     # compare against an existing report; prints α/κ/r + verdict
+```
+
+Dataset layout:
+
+```
+gold-dir/
+├── metadata.yaml      # annotator (must NOT match the omk judge model — would trigger contamination warning) + date + version
+└── annotations.yaml   # [{ sample_id, score, reason? }] concatenated by sample_id
+```
+
+α thresholds follow Krippendorff (2011): ≥ 0.80 strong agreement; [0.67, 0.80) acceptable; < 0.40 large divergence — investigate rubric / prompt.
+
+Full demo: [examples/gold-dataset/](examples/gold-dataset/)
+
+### `omk bench debias-validate length` (judge length-bias check)
+
+Re-judges every (sample × variant) of an existing report with the OPPOSITE length-debias setting (v3-cot-length ↔ v2-cot) and bootstraps the CI on the score difference. A significant shift = the judge is sensitive to the length-debias instruction (indirect evidence of length bias).
+
+```bash
+omk bench debias-validate length <reportId> [options]
+  --variant <name>            check a single variant only
+  --judge-model <id>          override the report's judge model
+  --bootstrap-samples N       bootstrap iterations (default 1000)
+  --seed N                    deterministic seed
+```
+
+Verdict bucket: none / weak / medium (|0.2-0.5|) / strong (≥ 0.5). Re-judge cost roughly doubles vs the original judge pass.
+
+### `omk bench saturation` (saturation curve)
+
+Answers "have I run enough samples?". Reads the saturation trace from an existing report (no re-run). Verdicts only emit when the original run used `--repeat ≥ 5`; below that, the curve is plotted but no verdict is computed.
+
+```bash
+omk bench saturation <reportId> [options]
+  --variant <name>             single-variant view
+  --method <m>                 slope | bootstrap-ci-width (default) | plateau-height
+  --threshold <num>            method-specific cutoff (defaults match the method)
+  --window <num>               consecutive windows that must satisfy the threshold (default 3)
+```
+
+The HTML report inlines an SVG curve (cumulative N on X, mean ± 95% CI shading on Y, one curve per variant) automatically.
+
+### `omk bench verdict` (one-line ship/no-ship verdict)
+
+Aggregates bootstrap CI / three-layer ci-gate / saturation / human α into one of six verdicts: **PROGRESS** (significant improvement, all three layers pass → exit 0), **CAUTIOUS** (real gain but with a warning — broken gate / trivially small / control regressed → exit 1), **REGRESS** (significant negative shift → exit 1), **NOISE** (CI spans 0, undecidable → exit 1), **UNDERPOWERED** (sample size too small → exit 1), **SOLO** (single variant; exit 0 only if its own three-layer gate passes).
+
+```bash
+omk bench verdict <reportId> [options]
+  --threshold <num>      three-layer gate threshold (default 3.5, matches `omk bench ci`)
+  --trivial-diff <num>   "practically tiny" cutoff (default 0.1)
+  --verbose              expand per-pair detail
+```
+
+Shares its rule module with the HTML report's verdict pill — CLI and UI cannot disagree.
+
+### `omk bench diagnose` (sample quality diagnostics)
+
+Answers "is the conclusion polluted by bad samples?". Diagnoses 7 sample-quality issues: `flat_scores` (low discrimination), `all_pass` (too easy), `all_fail` (broken — error severity), `near_duplicate` (prompt ROUGE-1 ≥ threshold), `ambiguous_rubric` (high judge stddev across `--judge-repeat ≥ 2`), `cost_outlier` (≥ k× median), `latency_outlier` (≥ k× median), `error_prone` (executor failure).
+
+```bash
+omk bench diagnose <reportId> [options]
+  --top <n>                   show top N per kind (default 10, 0 = all)
+  --duplicate-rouge <num>     near-duplicate ROUGE-1 threshold (default 0.7)
+  --ambiguous-stddev <num>    judge-stddev threshold (default 1.0)
+  --cost-k <num>              cost-outlier multiplier vs median (default 3)
+  --latency-k <num>           latency-outlier multiplier vs median (default 3)
+  --flat <num>                flat_scores spread threshold (default 0.5)
+```
+
+Output includes a healthScore (0-100, formula `100 - normalized × 20` where `normalized = (errors×8 + warnings×3 + infos×1) / N`). Exit code is 0 only when `healthScore ≥ 70` AND no error-severity issue — CI-friendly.
+
+### `omk bench failures` (failure case LLM clustering)
+
+When 14 of 50 samples failed, reading them one by one is slow. This command sends failed samples to a single LLM call, clusters them into ≤ N groups, and emits per-cluster root cause + fix. "Failed" = `compositeScore < threshold` OR `ok = false`.
+
+```bash
+omk bench failures <reportId> [options]
+  --judge-executor <name>     executor (default: claude)
+  --judge-model <id>          clustering model (default: from report.meta.judgeModel)
+  --max-clusters <n>          maximum clusters (default 5)
+  --threshold <num>           failure score threshold (default 3)
+  --max-feed <n>              max failures fed to LLM (default 50; takes the worst)
+```
+
+Tolerant: ```json``` markdown fences, `"sample_id@variant"` string member form, hallucinated members are dropped, single-failure case skips the LLM call, executor errors degrade to unclassified.
+
+### `omk bench diff` (report comparison — single / dual mode)
+
+**Single-arg mode** (within-report sample-level): `omk bench diff <reportId>` — within one report, drill down per-sample comparing `variants[0]` against `variants[1]` (or `--variant <name>`).
+
+**Dual-arg mode** (cross-report variant-level): `omk bench diff <reportId1> <reportId2>` — compare the same variant across two reports (legacy behavior preserved).
+
+```bash
+omk bench diff <reportId> [--variant <name>] [--regressions-only] [--threshold 0] [--top N]
+omk bench diff <reportId1> <reportId2> [--regressions-only] [--threshold 0]
+```
+
+Single-arg output sorts by |Δ| desc; rows where Δ < threshold are highlighted as regressions. `--top N` caps row count, `--regressions-only` shows only negative Δ samples.
+
 ## `omk analyze` — production observability
 
 `omk bench run` is **offline evaluation** (fixed controls, repeatable, scored). Production is different — no control group, no ground truth, no repetition, so scoring isn't valid there. `omk analyze` turns existing Claude Code session traces into **skill-health reports** (coverage, gap signals, execution stability, tokens/latency per skill). It gives you clues about **which skill is worth re-evaluating offline**, not a production score.

package/README.zh.md

@@ -7,7 +7,7 @@
 
 [English](./README.md) | **简体中文**
 
-知识载体评测工具 — 用客观数据衡量你的 artifact 质量。
+**omk** — 内置统计严谨性的 LLM 评测框架。Bootstrap CI / Krippendorff α / 长度偏差校正 / 饱和曲线开箱即用。原生支持 Claude Code skill、prompt、agent、RAG。
 
 **固定模型，只变知识载体，数据说话。**
 
@@ -22,6 +22,32 @@
 - **线上 session 观测** — 解析 Claude Code session JSONL，在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
 - **知识缺口识别** — 严重度加权的信号（显式标记 / 搜索失败 / hedging 用语 / 反复失败）量化风险敞口,不宣称完备性
 - **合并前 CI 门** — `omk bench ci` 强制三层 all-pass（fact + behavior + llm-judge），抓复合分掩盖的单层回退
+- **一行 ship/no-ship 结论** — `omk bench verdict <reportId>` 聚合 bootstrap CI / 三层 ci-gate / saturation / human α,给六档 verdict（PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO）+ 行动建议;exit code 反映是否可 ship
+
+### 统计严谨性
+LLM 评测最容易踩的坑是"自信的偏差"——CI 很窄但结论错。omk 的统计层做四件事让结论可被外部审计：
+
+- **Bootstrap CI** (`--bootstrap`) — 不假设分布的置信区间。t 检验在 LLM 序数评分上失效，bootstrap 直接重采样原始数据，对小 N（< 30）和偏态分布都稳。pairwise diff CI 不含 0 = 显著差异。
+- **Human Gold + Krippendorff α** (`--gold-dir`) — 引入外部标注作为锚点。CI 解决"评委稳不稳"，α 解决"评委对不对"——两个维度互补。omk 自动检测污染（gold annotator 与 judge 同模型时警告）。
+- **Length-controlled judge prompt** (默认开启) — 研究证实 LLM 评委隐性偏向更长的回答。omk 的 judge prompt 加显式段落"长度不是质量信号"，template hash 为 v3-cot-length，跟旧版本的报告 hash 肉眼可辨。`omk bench debias-validate length <reportId>` 重判检测偏差幅度。
+- **Saturation curve** — 回答"我跑够样本了吗"。`--repeat ≥ 5` 时累积 N → 均值 + bootstrap CI 序列，CI 宽度衰减率 < 5% 持续 3 个窗口判定饱和——再多样本对结论无实质收益。HTML 报告内联 SVG 曲线 + verdict。
+
+## 为什么选 omk
+
+| | omk | promptfoo | DeepEval | RAGAS | LangSmith |
+|--|--|--|--|--|--|
+| Bootstrap CI | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Krippendorff α(评委 ↔ 人工锚点） | ✓ | ✗ | ✗ | ✗ | ✗ |
+| Length-debias 评委 prompt | ✓ 默认 | ✗ | ✗ | ✗ | ✗ |
+| 饱和曲线 | ✓ | ✗ | ✗ | ✗ | ✗ |
+| 三层独立评分 | ✓ | ✗ | 部分 | ✗ | ✗ |
+| 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ | ✗ |
+| 完整中文文档 | ✓ | ✗ | ✗ | ✗ | ✗ |
+| 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✗ | ✓ |
+
+omk 的护城河是**统计严谨性** — 每条结论都能被研究者审计。需要托管 SaaS 看板?选 LangSmith。要本地快速 prompt 迭代不要统计层?选 promptfoo。**要 ship 到生产且会被问"为什么应该相信这个数字"?选 omk**。
+
+完整对比(7 个工具 × 25+ 维度): [docs/zh/comparison.md](docs/zh/comparison.md)
 
 ## 快速开始
 
@@ -61,15 +87,23 @@ omk bench run
 
 | 特性 | 说明 |
 |------|------|
-| **18 种断言** | 包含子串、正则、JSON Schema、语义相似度、自定义函数等 |
+| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
+| **断言取反 + 组合** | 通用 `not: true` 字段 + `assert-set` (any/all) 任意嵌套 |
 | **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
+| **统计严谨性** | Bootstrap CI / Krippendorff α / Length-debias / Saturation curve |
+| **Verdict 一行结论** | `omk bench verdict <id>` 六档判定 + ship 建议 + exit code 路由,与 HTML 报告 verdict pill 共享规则 |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖,自动继承 length-debias |
+| **样本质量诊断** | `omk bench diagnose <id>` 7 类 issue（区分度低 / 重复 / 歧义 / 成本异常 / 全 fail 等）+ healthScore 0-100 |
+| **失败聚类 + 根因** | `omk bench failures <id>` 单 LLM 调用聚类失败样本 + 每 cluster 给修复建议 |
+| **预算硬阈值** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` 总成本 + 单样本成本/耗时上限,超出中止保留 partial report |
 | **多执行器** | 支持 Claude CLI / Claude SDK / OpenAI / Gemini 及自定义命令 |
+| **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
 | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容（SSO 保护的知识库等） |
 | **盲测 A/B** | `--blind` 隐藏变体名称，HTML 报告有揭晓按钮 |
 | **并行执行** | `--concurrency N` 并行 N 个任务 |
 | **多轮方差分析** | `--repeat N` 重复 N 次，计算均值/标准差/置信区间/t 检验 |
 | **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本样本 |
-| **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹 |
+| **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹、judge prompt hash |
 | **中英切换** | HTML 报告右上角一键切换语言 |
 
 ## 工作原理
@@ -112,7 +146,7 @@ flowchart TD
     end
 
     subgraph Report["⑦ 报告"]
-        R["六维: 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性<br/>JSON + HTML · 盲测揭晓<br/>CLI/Node/版本指纹可追溯"]
+        R["六维: 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性<br/>JSON + HTML · 顶部 verdict pill · 盲测揭晓<br/>CLI/Node/版本指纹可追溯"]
     end
 
     S --> U
@@ -246,7 +280,7 @@ flowchart TD
 
 ### 断言类型
 
-**确定性断言（18 种）：**
+**确定性断言（21+ 种）：**
 
 | 类型 | 说明 |
 |------|------|
@@ -259,9 +293,43 @@ flowchart TD
 | `word_count_min` / `word_count_max` | 词数范围 |
 | `contains_all` / `contains_any` | 多值匹配 |
 | `cost_max` / `latency_max` | 成本/延迟限制 |
-| `semantic_similarity` | LLM 语义相似度 |
+| `tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max` | Agent 工具调用断言 |
+| `tool_output_contains` / `tool_input_contains` | 工具输入/输出内容匹配 |
+| `turns_min` / `turns_max` | 多轮对话轮数限制 |
+| `rouge_n_min` | ROUGE-N recall ≥ threshold（`reference` 字段填参考答案，`n` 默认 1，`threshold` 默认 0.5） |
+| `levenshtein_max` | 编辑距离 ≤ value（用于"输出跟参考几乎一致"场景） |
+| `bleu_min` | BLEU-4 ≥ threshold（unsmoothed，短文本会塌陷到 0） |
+| `faithfulness` | 输出是否被 `sample.context` 支持(反幻觉);LLM judge 1-5 评分,threshold 默认 3 |
+| `answer_relevancy` | 输出是否切题回答 `sample.prompt`;能抓住跑题、回避、冗余;threshold 默认 3 |
+| `context_recall` | `sample.context` 关键事实在输出中的覆盖率;`reference` 可显式指定 gold facts;threshold 默认 3 |
+| `semantic_similarity` | LLM 语义相似度（与 reference 的整体相似度,与 RAG 三 metric 互补） |
 | `custom` | 自定义 JS 函数（30s 超时） |
 
+**通用修饰：**
+
+任何断言加 `not: true` 即反向（替代 `not_contains` / `not_equals` 等成对类型；老类型保留作 alias）：
+
+```yaml
+- type: regex
+  pattern: "TODO|FIXME"
+  not: true              # 必须不含 TODO/FIXME
+```
+
+**断言组合（— assert-set）：**
+
+`assert-set` 类型让多个断言以 `any`（OR）或 `all`（AND）逻辑组合，可嵌套：
+
+```yaml
+- type: assert-set
+  mode: any              # 任一通过即过 (mode: 'all' 则需全部通过)
+  children:
+    - { type: contains, value: "参数化" }
+    - { type: contains, value: "prepared statement" }
+    - { type: regex, pattern: "bind\\(.*\\?" }
+```
+
+子断言可独立带 `not: true`；嵌套 assert-set 可表达任意布尔逻辑。
+
 ### 自定义断言
 
 ```js
@@ -319,7 +387,25 @@ omk bench run [选项]
   --verbose              打印每个样本的详细执行结果（耗时、tokens、输出预览）
   --each                 批量评测：每个 artifact 独立和 baseline 对比
                          需要每个 artifact 配对 {name}.eval-samples.json
-```
+  --judge-repeat <n>     每条 sample × dimension 跑 LLM 评委 N 次,输出 stddev (评委自一致性)
+  --judge-models <list>  多评委 ensemble: "executor1:model1,executor2:model2"
+                         ≥ 2 个 judge 触发 ensemble + inter-judge agreement 输出
+  --bootstrap            启用 distribution-free CI:每个 variant 加 bootstrap CI,
+                         pairwise diff CI 含 0 = 不显著
+  --bootstrap-samples N  bootstrap 重采样次数 (默认 1000)
+  --gold-dir <路径>      跑完自动对比 human gold 算 Krippendorff α / κ / Pearson,
+                         结果写入 report.meta.humanAgreement,HTML 报告显示「人工锚点」
+  --no-debias-length     退回 v2-cot 评委 prompt (不含"长度不是质量信号"段落),
+                         用于复现旧版本（v3-cot-length 之前）的报告 hash
+  --budget-usd <num>             总成本上限 (USD);超出中止评测,partial report 仍持久化
+                                 (`report.meta.budgetExhausted = true`)
+  --budget-per-sample-usd <num>  单样本成本上限;超出该样本失败但评测继续
+  --budget-per-sample-ms <num>   单样本耗时上限 (ms);超出该样本失败但评测继续
+```
+
+**eval.yaml 预算字段**: `budget: { totalUSD?, perSampleUSD?, perSampleMs? }`,所有字段可选且必须 ≥ 0。CLI 同名 flag 覆盖配置值。
+
+**和 `cost_max` / `latency_max` 断言的区别**: 断言是**单样本评分维度**(超出直接打 0 分,run 继续);budget 是**工作流级硬阈值**(`totalUSD` 超出整个 run abort 保留 partial report,per-sample 超出该样本失败但 run 继续)。一个回答"质量是否达标",一个回答"花钱/时间是否在预算内"。
 
 ### `omk bench run --each`（批量评测）
 
@@ -421,6 +507,113 @@ omk bench report [选项]
 omk bench init [目录]    # 生成评测项目脚手架
 ```
 
+### `omk bench gold`（人工锚点）
+
+人工标注（或更强模型代理）作为外部锚点，与 LLM 评委的分数对比 Krippendorff α / 加权 κ / Pearson。回答"评委对不对"，与 Bootstrap CI 的"评委稳不稳"互补。
+
+```bash
+omk bench gold init [--out <dir>] [--annotator <id>]   # 生成数据集模板
+omk bench gold validate <dir>                          # 校验 schema (annotator/时间/版本/score 范围)
+omk bench gold compare <reportId> --gold-dir <dir>     # 与已有 report 对比,输出 verdict + α/κ/r
+```
+
+dataset 目录结构：
+
+```
+gold-dir/
+├── metadata.yaml      # annotator (注意不要与 omk judge 同模型,会触发污染警告) + 时间 + 版本
+└── annotations.yaml   # [{ sample_id, score, reason? }] 按 sample_id 拼接
+```
+
+α 阈值参考 Krippendorff (2011)：≥ 0.80 高度一致；[0.67, 0.80) 可接受；< 0.40 偏差大需排查 rubric / prompt。
+
+完整 demo: [examples/gold-dataset/](examples/gold-dataset/)
+
+### `omk bench debias-validate length`（评委长度偏差检测）
+
+重判已有 report 的所有 (sample × variant)，用相反的 length-debias 设置（v3-cot-length ↔ v2-cot），bootstrap CI 算两次差值。差异显著 = 评委对长度敏感（length bias 间接证据）。
+
+```bash
+omk bench debias-validate length <reportId> [选项]
+  --variant <name>            只测一个 variant
+  --judge-model <id>          override report 的 judge model
+  --bootstrap-samples N       bootstrap 迭代数 (默认 1000)
+  --seed N                    确定性种子
+```
+
+verdict 分四档：未检测 / 弱 / 中（差值 |0.2-0.5|）/ 强（差值 ≥ 0.5）。重判 cost 大致翻倍。
+
+### `omk bench saturation`（饱和曲线）
+
+回答"我跑够样本了吗"。从已有 report 读取 saturation trace 输出判定，无需重跑评测。需要原 run 跑了 `--repeat ≥ 5` 才会有 verdict（低 repeat 只画曲线）。
+
+```bash
+omk bench saturation <reportId> [选项]
+  --variant <name>             只看一个 variant
+  --method <m>                 slope | bootstrap-ci-width (默认) | plateau-height
+  --threshold <num>            方法相关阈值 (默认随 method)
+  --window <num>               连续多少窗口满足才判饱和 (默认 3)
+```
+
+HTML 报告会内联 SVG 饱和曲线（横 N，纵 mean ± 95% CI 阴影带，per-variant 一条），自动渲染。
+
+### `omk bench verdict`（一行 ship/no-ship 结论）
+
+聚合 bootstrap CI / 三层 ci-gate / saturation / human α 给一行结论。Verdict 六档:**PROGRESS**（显著改进 + 三层全过 → exit 0）/ **CAUTIOUS**（改进真实但有警告:gate 破/幅度太小/控制组本身崩 → exit 1）/ **REGRESS**（显著回退 → exit 1）/ **NOISE**（CI 跨 0,无法判定 → exit 1）/ **UNDERPOWERED**（样本不足 → exit 1）/ **SOLO**（单变体,仅自身三层 gate 过才 exit 0）。
+
+```bash
+omk bench verdict <reportId> [选项]
+  --threshold <num>      三层 gate 阈值 (默认 3.5,匹配 omk bench ci)
+  --trivial-diff <num>   "幅度太小"阈值 (默认 0.1)
+  --verbose              展开 per-pair 详情
+```
+
+与 HTML 报告顶部的 verdict pill 共享规则模块,CLI 与 UI 不会矛盾。
+
+### `omk bench diagnose`（样本质量诊断）
+
+回答"测评结论是否被坏样本污染"。诊断 7 类样本质量问题:`flat_scores`（区分度低）/ `all_pass`（太简单）/ `all_fail`（broken,error 级）/ `near_duplicate`（prompt ROUGE-1 ≥ 阈值）/ `ambiguous_rubric`（judge stddev 大,需要 `--judge-repeat ≥ 2`）/ `cost_outlier`（≥ k× median）/ `latency_outlier`（≥ k× median）/ `error_prone`（执行失败）。
+
+```bash
+omk bench diagnose <reportId> [选项]
+  --top <n>                   每类显示前 N 个 (默认 10,0=全部)
+  --duplicate-rouge <num>     near-duplicate ROUGE-1 阈值 (默认 0.7)
+  --ambiguous-stddev <num>    歧义 judge stddev 阈值 (默认 1.0)
+  --cost-k <num>              成本异常倍数 vs median (默认 3)
+  --latency-k <num>           耗时异常倍数 vs median (默认 3)
+  --flat <num>                flat_scores 分差阈值 (默认 0.5)
+```
+
+输出含 healthScore（0-100,公式 `100 - normalized × 20`,其中 `normalized = (errors×8 + warnings×3 + infos×1) / N`）。exit code 0 仅当 `healthScore ≥ 70` 且无 error 级 issue,适合 CI 链。
+
+### `omk bench failures`（失败 case LLM 聚类）
+
+跑完 14 条失败,逐个看太慢。本命令把失败样本喂给单次 LLM 调用,自动聚到 ≤ N 个 cluster,每个 cluster 给根因 + 修复建议。失败定义:`compositeScore < threshold` 或 `ok = false`。
+
+```bash
+omk bench failures <reportId> [选项]
+  --judge-executor <name>     执行器 (默认 claude)
+  --judge-model <id>          聚类用 model (默认沿用 report.meta.judgeModel)
+  --max-clusters <n>          最多多少 cluster (默认 5)
+  --threshold <num>           算失败的分数阈值 (默认 3)
+  --max-feed <n>              最多喂给 LLM 多少条 (默认 50,超出取最差)
+```
+
+容错:tolerate ```json``` markdown fence、`"sample_id@variant"` 字符串成员形式、hallucinated 成员自动剔除、单条失败跳过 LLM 直接列出、executor 错误降级到 unclassified。
+
+### `omk bench diff`（报告对比 — 单参 / 双参双模式）
+
+**单参模式**(within-report sample-level 钻取): `omk bench diff <reportId>` — 在同一份报告内对比两个 variant 的逐样本得分,默认对比 `variants[0]` vs `variants[1]`。
+
+**双参模式**(cross-report variant-level): `omk bench diff <reportId1> <reportId2>` — 跨报告对比同一 variant 的整体均值漂移(向后兼容旧用法)。
+
+```bash
+omk bench diff <reportId> [--variant <name>] [--regressions-only] [--threshold 0] [--top N]
+omk bench diff <reportId1> <reportId2> [--regressions-only] [--threshold 0]
+```
+
+单参模式表格按 |Δ| 排序,Δ < threshold 高亮 regression。`--top N` 限制行数,`--regressions-only` 过滤到只看回退。
+
 ## `omk analyze` — 生产观测
 
 `omk bench run` 是**离线评测**(固定对照、可复现、可评分)。生产环境不一样 — 没对照组、没标准答案、没重复,所以评分在那里不成立。`omk analyze` 把已有的 Claude Code session trace 转成**skill 健康度报告**(按 skill 维度的覆盖率、缺口信号、执行稳定性、tokens/延迟)。它给的是"哪个 skill 值得拉回离线再测一遍"的线索,不是生产评分。

package/dist/src/analysis/failure-clusterer.d.ts

@@ -0,0 +1,96 @@
+/**
+ * Failure clustering — turn a list of "what failed" into "why it failed".
+ *
+ * Why this exists
+ * ---------------
+ * Senior-engineer pain point: 50 samples ran, 14 failed. Reading 14 failure
+ * cases one by one is tedious; the user wants the tool to say "8 of these
+ * are tool-call errors, 4 are refusals, 2 are formatting errors", with a
+ * suggested fix per cluster.
+ *
+ * This module:
+ *
+ *  1. Selects failed samples from a report (compositeScore < threshold OR
+ *     ok=false on at least one variant).
+ *  2. Builds a compact failure description per sample (failed assertion
+ *     types + judge reason snippet + tool failure pattern).
+ *  3. Sends the full set to a single LLM call asking it to (a) propose
+ *     N cluster labels, (b) assign each failure to a cluster, (c) suggest
+ *     a fix per cluster.
+ *  4. Returns the structured cluster report. Pure function — no I/O,
+ *     accepts the executor as an argument.
+ *
+ * Design choice: single-call LLM clustering rather than embedding-based
+ * k-means. Reasons:
+ *
+ *  - Failure descriptions are short (~50-200 tokens each), so 14 of them
+ *    fit comfortably in one prompt. A 50-failure batch is also fine.
+ *  - The LLM can produce HUMAN-READABLE cluster labels ("tool call error",
+ *    "premature refusal") rather than numeric centroids — exactly what
+ *    the user needs to act on.
+ *  - One call = one judge cost = predictable. Embedding + clustering
+ *    needs N+1 calls minimum.
+ *  - Acceptable inaccuracy: clusters are advisory, not load-bearing. We
+ *    document this clearly and surface raw failures alongside.
+ *
+ * The clustering prompt is structured to be JSON-only output for parsing
+ * stability. We do NOT include length-debias instructions — clustering is
+ * a categorization task, not a quality scoring task, so the length-debias
+ * directive is inapplicable.
+ */
+import type { ExecutorFn, Report } from '../types.js';
+export interface FailureClusterRequest {
+    report: Report;
+    /** Executor for the clustering LLM call. */
+    executor: ExecutorFn;
+    /** Model id to use for clustering. */
+    judgeModel: string;
+    /** Max number of clusters. Default 5; the LLM may produce fewer. */
+    maxClusters?: number;
+    /** compositeScore < this OR ok=false counts as failure. Default 3. */
+    failureThreshold?: number;
+    /** Cap on how many failures to feed the LLM (selects worst by score). Default 50. */
+    maxFailuresFed?: number;
+}
+export interface FailureCase {
+    sample_id: string;
+    variant: string;
+    /** Composite score (0 if errored). */
+    score: number;
+    /** Whether the executor itself errored (ok=false). */
+    errored: boolean;
+    /** Description summarizing what went wrong, fed to the LLM. */
+    description: string;
+}
+export interface FailureCluster {
+    /** Human-readable cluster label, in the report's primary language. */
+    label: string;
+    /** Free-text root cause analysis from the LLM. */
+    rootCause: string;
+    /** Suggested fix the user can apply. */
+    suggestedFix: string;
+    /** Sample IDs (with variant) belonging to this cluster. */
+    members: Array<{
+        sample_id: string;
+        variant: string;
+    }>;
+}
+export interface FailureClusterReport {
+    /** All failures considered (after threshold + cap). */
+    failures: FailureCase[];
+    /** Clusters returned by the LLM, sorted by member count desc. */
+    clusters: FailureCluster[];
+    /** Failures the LLM didn't put in any cluster (label = "other"). */
+    unclassified: Array<{
+        sample_id: string;
+        variant: string;
+    }>;
+    /** USD cost of the clustering call. */
+    clusterCostUSD: number;
+    /** Truncation flag — set when more failures existed than maxFailuresFed. */
+    truncated: boolean;
+    totalFailures: number;
+}
+export declare function clusterFailures(req: FailureClusterRequest): Promise<FailureClusterReport>;
+export declare function formatFailureClusterReport(r: FailureClusterReport): string;
+//# sourceMappingURL=failure-clusterer.d.ts.map

package/dist/src/analysis/failure-clusterer.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"failure-clusterer.d.ts","sourceRoot":"","sources":["../../../src/analysis/failure-clusterer.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAuCG;AAEH,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,EAA8B,MAAM,aAAa,CAAC;AAElF,MAAM,WAAW,qBAAqB;IACpC,MAAM,EAAE,MAAM,CAAC;IACf,4CAA4C;IAC5C,QAAQ,EAAE,UAAU,CAAC;IACrB,sCAAsC;IACtC,UAAU,EAAE,MAAM,CAAC;IACnB,oEAAoE;IACpE,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,sEAAsE;IACtE,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,qFAAqF;IACrF,cAAc,CAAC,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,WAAW;IAC1B,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,sCAAsC;IACtC,KAAK,EAAE,MAAM,CAAC;IACd,sDAAsD;IACtD,OAAO,EAAE,OAAO,CAAC;IACjB,+DAA+D;IAC/D,WAAW,EAAE,MAAM,CAAC;CACrB;AAED,MAAM,WAAW,cAAc;IAC7B,sEAAsE;IACtE,KAAK,EAAE,MAAM,CAAC;IACd,kDAAkD;IAClD,SAAS,EAAE,MAAM,CAAC;IAClB,wCAAwC;IACxC,YAAY,EAAE,MAAM,CAAC;IACrB,2DAA2D;IAC3D,OAAO,EAAE,KAAK,CAAC;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;CACxD;AAED,MAAM,WAAW,oBAAoB;IACnC,uDAAuD;IACvD,QAAQ,EAAE,WAAW,EAAE,CAAC;IACxB,iEAAiE;IACjE,QAAQ,EAAE,cAAc,EAAE,CAAC;IAC3B,oEAAoE;IACpE,YAAY,EAAE,KAAK,CAAC;QAAE,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IAC5D,uCAAuC;IACvC,cAAc,EAAE,MAAM,CAAC;IACvB,4EAA4E;IAC5E,SAAS,EAAE,OAAO,CAAC;IACnB,aAAa,EAAE,MAAM,CAAC;CACvB;AAeD,wBAAsB,eAAe,CAAC,GAAG,EAAE,qBAAqB,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAkF/F;AAwID,wBAAgB,0BAA0B,CAAC,CAAC,EAAE,oBAAoB,GAAG,MAAM,CAsC1E"}