oh-my-knowledge 0.23.0 → 0.25.0

package/README.md

@@ -1,55 +1,22 @@
 # oh-my-knowledge
 
 [![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
+[![npm weekly downloads](https://img.shields.io/npm/dw/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
 [![CI](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml/badge.svg)](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
 [![Node.js Version](https://img.shields.io/node/v/oh-my-knowledge.svg)](https://nodejs.org)
 
 **English** | [简体中文](./README.zh.md)
 
-**omk** — LLM evaluation framework with built-in statistical rigor. Bootstrap CI / Krippendorff α / length-debias / saturation curves out of the box. Native support for Claude Code skills, prompts, agents, and RAG.
+**omk** — The knowledge you give your LLM — what's it actually worth?
+omk answers with objective data, not gut feeling.
 
-**Fix the model, vary the knowledge artifact, let the data speak.**
+**Fix the model, vary the knowledge artifact.**
 
-## Why this tool
-
-Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
-
-## Key features
-
-- **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
-- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
-- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
-- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
-- **Pre-merge CI gate** — `omk bench gate` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
-- **One-line ship/no-ship verdict** — `omk bench verdict <reportId>` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
-
-### Statistical rigor
-
-The biggest LLM-eval failure mode is "confident bias" — narrow CIs around the wrong answer. omk's statistical layer ships four pieces so conclusions can be externally audited:
-
-- **Bootstrap CI** (`--bootstrap`) — distribution-free confidence intervals. The t-test breaks on ordinal LLM scores; bootstrap resamples raw observations and stays valid at small N (< 30) and on skewed data. Pairwise diff CI not crossing 0 = significant.
-- **Human Gold + Krippendorff α** (`--gold-dir`) — bring an external annotation as anchor. CI tells you "is the judge stable", α tells you "is the judge correct" — two complementary axes. omk warns when the gold annotator and the judge are the same model (would inflate α).
-- **Length-controlled judge prompt** (default ON) — research shows LLM judges over-weight verbosity. omk's judge prompt explicitly states "length is not a quality signal"; template hash is `v3-cot-length` so older reports (with the legacy hash) are visibly different. `omk bench debias-validate length <reportId>` re-judges with the opposite setting and reports the score shift.
-- **Saturation curve** — answers "have I run enough samples?". With `--repeat ≥ 5` we accumulate cumulative N → bootstrap CI; when CI shrink rate stays under 5% across 3 windows, more samples buy nothing. The HTML report inlines the SVG curve plus a verdict.
-
-## Why omk over alternatives
-
-| | omk | promptfoo | DeepEval | RAGAS | LangSmith |
-|--|--|--|--|--|--|
-| Bootstrap CI | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Krippendorff α (judge ↔ human) | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ | ✗ |
-| Saturation curve | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Three-layer scoring isolation | ✓ | ✗ | partial | ✗ | ✗ |
-| Per-variant skill isolation (construct validity) | ✓ default | ✗ | ✗ | ✗ | ✗ |
-| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Full Chinese docs | ✓ | ✗ | ✗ | ✗ | ✗ |
-| Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✗ | ✓ |
-
-omk's moat is **statistical rigor** — every conclusion is auditable by a researcher. If you need a hosted SaaS dashboard, choose LangSmith. If you want quick local prompt iteration without statistics, choose promptfoo. **If you ship to production and someone will ask "why should I trust this number?", choose omk**.
+<a id="statistical-rigor"></a>
+> Built-in: Bootstrap CI · Krippendorff α (judge ↔ human) · length-debias · saturation curves · construct-validity isolation. [Why these matter →](docs/statistical-rigor.md)
 
-Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md)
+![omk report](./assets/screenshots/report-overview.png)
 
 ## Quick start
 
@@ -70,46 +37,93 @@ cd my-eval
 omk bench run --dry-run
 
 # run the evaluation (auto-discovers everything under skills/)
-omk bench run
+omk bench run    # → HTML report with verdict in 5 minutes
+                 # (omk doctor + LLM connectivity check both run as mandatory gates;
+                 #  --skip-connectivity available for connectivity, doctor is unconditional)
 
 # CLI output language: zh (default) / en — flag wins over env
 omk bench run --lang en
 OMK_LANG=en omk bench report
 ```
 
-## Use inside Claude Code
+## Use inside AI Coding Agents
 
-After installing omk, talk to it in natural language from Claude Code:
+### Use inside Claude Code
 
-```
+When the `omk` skill is available in Claude Code, you can invoke it directly like this:
+
+```bash
 /omk eval              # evaluate the artifact(s) in the current project
 /omk evolve            # auto-iterate to improve an artifact
 /omk gen-samples       # generate test cases
 ```
 
-You can also just say "compare v1 vs v2 for me" or "improve this artifact" — omk picks the right command.
+You can also just say "compare v1 vs v2 for me" or "improve this artifact" and omk picks the right command.
+
+### Use inside Codex
+
+Codex does not support Claude Code style `/omk ...` slash commands by default. In Codex, the usual pattern is to ask the agent to run the `omk` CLI directly, for example:
+
+```bash
+omk bench run
+omk bench evolve
+omk bench gen-samples skills/my-skill.md
+```
+
+You can also describe the goal in natural language, such as "compare v1 vs v2" or "generate test cases for this skill".
+
+## Why this tool
+
+Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
+
+## Key features
+
+- **Pre-evaluation health check** — `omk doctor` runs as a mandatory gate before `bench run` / `bench gate`; checks skill readability, metadata, dependencies, samples contract — pure static, zero LLM calls (like lint + typecheck for knowledge artifacts). Executor / judge connectivity is a separate phase, controllable via `--skip-connectivity`
+- **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
+- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
+- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
+- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
+- **Pre-merge CI gate** — `omk bench gate` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
+- **One-line ship/no-ship verdict** — `omk bench verdict <reportId>` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
+
+## Why omk over alternatives
+
+| | omk | promptfoo | DeepEval | LangSmith |
+|--|--|--|--|--|
+| Bootstrap CI | ✓ default | ✗ | ✗ | ✗ |
+| Krippendorff α (judge ↔ human) | ✓ default | ✗ | ✗ | ✗ |
+| Length-debias judge prompt | ✓ default | ✗ | ✗ | ✗ |
+| Saturation curve | ✓ | ✗ | ✗ | ✗ |
+| Three-layer scoring isolation | ✓ | ✗ | partial | ✗ |
+| Per-variant skill isolation (construct validity) | ✓ default | ✗ | ✗ | ✗ |
+| Native Claude Code skill | ✓ | ✗ | ✗ | ✗ |
+| Hosted SaaS dashboard | ✗ | ✗ | ✓ | ✓ |
+
+omk's moat is **default-on safety net** — Bootstrap CI, judge ↔ human α, and length-debias aren't advanced flags; they're the default. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
+
+RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md).
 
 ## Features
 
 | Feature | What it does |
 |---|---|
-| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
-| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
-| **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
-| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve |
-| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against: (1) SDK skill auto-discovery, (2) subagent Skill tool, (3) cwd file-system access via the `skills/<name>/` symlink that's normally there for the treatment variant. eval.yaml `allowedSkills` for per-variant whitelists |
-| **Sample design science** | Sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata fields (HF Dataset Cards style). `bench diagnose` shows coverage breakdown + flags `rubric_clarity_low` (short rubric without grading keywords) / `capability_thin` (capability supported by ≤ N×0.2 samples). `bench gen-samples` auto-stamps provenance. See [docs/sample-design-spec.md](docs/sample-design-spec.md) for the 8 industry-gap mapping |
 | **One-line verdict** | `omk bench verdict <id>` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
-| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
+| **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
+| **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / any custom command |
+| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
+| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/statistical-rigor.md) |
 | **Sample diagnostics** | `omk bench diagnose <id>` — 7 issue kinds (low discrimination / duplicates / ambiguous rubric / cost outliers / etc.) + 0-100 healthScore |
 | **Failure clustering** | `omk bench failures <id>` — single LLM call clusters failed samples and emits per-cluster fixes |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
 | **Hard budget caps** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` — abort on total-cost overrun, flag per-sample overruns; partial report persisted |
-| **Multi-executor** | Claude CLI / Claude SDK / OpenAI / Gemini / any custom command |
+| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against: (1) SDK skill auto-discovery, (2) subagent Skill tool, (3) cwd file-system access via the `skills/<name>/` symlink that's normally there for the treatment variant. eval.yaml `allowedSkills` for per-variant whitelists |
+| **Sample design science** | Sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata fields (HF Dataset Cards style). `bench diagnose` shows coverage breakdown + flags `rubric_clarity_low` (short rubric without grading keywords) / `capability_thin` (capability supported by ≤ N×0.2 samples). `bench gen-samples` auto-stamps provenance. See [docs/sample-design-spec.md](docs/sample-design-spec.md) for the 8 industry-gap mapping |
 | **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
 | **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
-| **Parallel execution** | `--concurrency N` runs N tasks at once |
 | **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
+| **Parallel execution** | `--concurrency N` runs N tasks at once |
+| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
 | **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
 | **Traceability** | reports carry CLI version, Node version, artifact version fingerprint, judge prompt hash |
 | **EN / ZH switch** | one-click language toggle in the HTML report |
@@ -135,8 +149,8 @@ flowchart TD
     end
 
     subgraph Exec["④ Executor (fixed model)"]
-        E["claude / claude-sdk / openai / gemini<br/>anthropic-api / openai-api / custom"]
-        T["claude-sdk extracts<br/>turns / toolCalls trace"]
+        E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / custom"]
+        T["claude-sdk / codex extract<br/>turns / toolCalls trace"]
         E -.-> T
     end
 
@@ -373,14 +387,16 @@ options:
   --control <expr>       control-group variant expression (experiment role = control)
   --treatment <v1,v2>    treatment-group variant expressions, comma-separated
                          at least one of --control / --treatment is required
-                         (unless you use --config or --each)
+                         (unless you use --config or --batch)
                          special values: baseline (empty artifact), git:name (git HEAD),
                          git:ref:name (specific commit), path with "/" (read file directly)
   --config <path>        YAML/JSON config file (evaluation-as-code); declares
                          samples + variants + model + executor in one file; CLI
                          flags override config fields when both are provided
-  --model <name>         model under test (default: sonnet)
-  --judge-model <name>   judge model (default: haiku)
+  --model <name>         task execution model (default: sonnet)
+  --judge-models <list>  judge config; 1 entry = single judge (default
+                         claude:haiku), ≥ 2 entries = ensemble. Format:
+                         `executor:model[,executor:model]`
   --output-dir <path>    output dir (default: ~/.oh-my-knowledge/reports/)
   --no-judge             skip the LLM judge
   --no-cache             disable result cache (on by default; identical inputs reuse)
@@ -390,16 +406,17 @@ options:
   --timeout <sec>        per-task executor timeout (default: 120)
   --repeat <n>           repeat N times for variance analysis (default: 1)
   --executor <name>      executor (default: claude); supports custom commands
-  --skip-preflight       skip evaluation model reachability check
+  --skip-connectivity    skip the LLM connectivity check (doctor still runs;
+                         doctor is mandatory and has no skip flag).
+                         Auto-applied on --resume (the original run already
+                         verified connectivity).
   --mcp-config <path>    MCP config for fetching private-doc URLs via MCP Server
                          (default: .mcp.json in cwd)
   --no-serve             don't auto-start the report server after the run
   --verbose              print per-sample details (duration, tokens, output preview)
-  --each                 batch mode: evaluate each artifact independently vs baseline
+  --batch                batch mode: evaluate each artifact independently vs baseline
                          requires {name}.eval-samples.json paired with each artifact
   --judge-repeat <n>     run the LLM judge N times per (sample × dimension) and report stddev
-  --judge-models <list>  multi-judge ensemble: "executor1:model1,executor2:model2"
-                         ≥ 2 judges enables ensemble + inter-judge agreement output
   --bootstrap            enable distribution-free CIs: bootstrap CI per variant +
                          pairwise diff CI (CI containing 0 = not significant)
   --bootstrap-samples N  bootstrap resample count (default 1000)
@@ -418,11 +435,34 @@ options:
 
 **eval.yaml budget**: declare `budget: { totalUSD?, perSampleUSD?, perSampleMs? }` (all optional, must be ≥ 0). CLI flags of the same name override the config values.
 
+**eval.yaml experiment-design fields**: the same flags above can be set in `eval.yaml` for reproducible experiment configuration (CLI > eval.yaml > default):
+
+```yaml
+samples: ./eval-samples.yaml
+model: sonnet
+repeat: 5                    # multi-run variance, ≥ 1
+judgeRepeat: 3               # per (sample × dim) judge self-consistency, ≥ 1
+bootstrap: true              # distribution-free CI per variant
+bootstrapSamples: 2000       # default 1000, ≥ 100
+goldDir: ./gold              # post-run α / κ / Pearson against human anchor
+lengthDebias: true           # default; set false to reproduce pre-v0.21 hash
+strictBaseline: true         # default; set false to disable skill isolation
+noJudge: false               # default; set true to skip LLM judge entirely
+judgeModels:                 # 1 entry = single judge; ≥ 2 = ensemble
+  - { executor: claude, model: opus }
+  - { executor: openai-api, model: gpt-4o }
+variants:
+  - { name: baseline, role: control, artifact: baseline }
+  - { name: my-skill, role: treatment, artifact: ./skills/my-skill.md }
+```
+
+**Field entry points**: `bench run` reads every field above. `bench gate` goes through `parseRunConfig` and picks up the shared subset (`variants` / `executor` / `model` / `judgeModels` — both single-judge and ensemble — / `noJudge` / `noCache` / `blind` / `strictBaseline` / `budget` / `mcpConfig` / `variantAllowedSkills`); the experiment-design fields handled by `handleRun` (`repeat` / `judgeRepeat` / `bootstrap` / `bootstrapSamples` / `goldDir` / `lengthDebias`) are intentionally not read by `gate` and can be extended later. Other subcommands (`evolve` / `verdict` / `diff` / `analyze` / …) do not read `eval.yaml`.
+
 **Difference from `cost_max` / `latency_max` assertions**: assertions are **per-sample scoring rules** (exceeding the cap fails that one assertion, the run continues); budget caps are **workflow-level hard limits** (`totalUSD` overrun aborts the run and persists a partial report; per-sample overruns fail the offending sample but the run continues). Assertions answer "is quality acceptable?"; budgets answer "are cost/time within the envelope?".
 
-### `omk bench run --each` (batch mode)
+### `omk bench run --batch` (batch mode)
 
-When `skills/` contains several **independent** artifacts, use `--each` to evaluate each one against baseline and produce a merged report.
+When `skills/` contains several **independent** artifacts, use `--batch` to evaluate each one against baseline and produce a BatchEvaluationReport with child EvaluationReports.
 
 ```
 skills/
@@ -442,8 +482,8 @@ Pairing rules:
 - artifacts without paired samples are skipped with a warning
 
 ```bash
-omk bench run --each
-omk bench run --each --dry-run
+omk bench run --batch
+omk bench run --batch --dry-run
 ```
 
 ### `omk bench gen-samples` (generate test cases)
@@ -455,7 +495,7 @@ Reads an artifact's content and uses an LLM to auto-generate eval-samples. Revie
 omk bench gen-samples skills/my-skill.md
 
 # batch-generate for every artifact under skills/ that lacks samples
-omk bench gen-samples --each
+omk bench gen-samples --batch
 
 # specify sample count
 omk bench gen-samples skills/my-skill.md --count 10
@@ -464,10 +504,10 @@ omk bench gen-samples skills/my-skill.md --count 10
 Options:
 
 ```
-  --each                 batch-generate for every artifact missing samples
+  --batch                batch-generate for every artifact missing samples
   --count <n>            samples per artifact (default: 5)
   --model <name>         model used for generation (default: sonnet)
-  --skill-dir <path>     artifact dir (default: skills), used with --each
+  --skill-dir <path>     artifact dir (default: skills), used with --batch
 ```
 
 ### `omk bench evolve` (self-iterating improvement)
@@ -505,6 +545,26 @@ omk bench gate [options]
                          independently to fact / behavior / judge
 ```
 
+### `omk doctor` (pre-evaluation health check)
+
+Pure static / zero-LLM checks — analogous to lint + typecheck in the SE toolchain. Runs as a mandatory gate before `bench run` / `bench gate` so a typo'd YAML or missing dependency aborts with an actionable error instead of producing a garbage-in verdict. Also runnable standalone for local iteration or CI.
+
+```bash
+omk doctor                    # batch check every skill in current dir / ./skills
+omk doctor skills/v1.md       # single file
+omk doctor skills/ --json     # JSON output for CI consumption
+omk doctor --gate; echo $?    # silent mode — exit 1 if any fatal check fails
+```
+
+What `doctor` checks:
+
+- **skill readable** — file exists, content non-empty, has minimum length
+- **skill metadata** — front-matter (if present) is valid YAML; directory-skills have `SKILL.md`
+- **dependencies present** — referenced CLI tools, files, env vars all available (reuses `preflightDependencies`)
+- **samples ↔ skill contract** — when samples are provided, validate they're non-empty and have prompt fields (warn-level)
+
+Executor / judge connectivity is verified by a separate evaluation preflight phase, not by doctor — clean boundary: doctor is static, eval is dynamic. `bench run` / `bench gate` abort with `exit 1` and stderr `doctor failed:` prefix when doctor fails. **Doctor is mandatory and not skippable** (static checks have no cost reason to skip); LLM connectivity is separately controllable via `--skip-connectivity` (auto-skipped on `--resume`).
+
 ### `omk bench report`
 
 Start the report server to browse historical reports, submit feedback, and delete reports.
@@ -549,7 +609,7 @@ Re-judges every (sample × variant) of an existing report with the OPPOSITE leng
 ```bash
 omk bench debias-validate length <reportId> [options]
   --variant <name>            check a single variant only
-  --judge-model <id>          override the report's judge model
+  --judge-models <executor:model>  override the report's judge (single-judge only)
   --bootstrap-samples N       bootstrap iterations (default 1000)
   --seed N                    deterministic seed
 ```
@@ -605,8 +665,7 @@ When 14 of 50 samples failed, reading them one by one is slow. This command send
 
 ```bash
 omk bench failures <reportId> [options]
-  --judge-executor <name>     executor (default: claude)
-  --judge-model <id>          clustering model (default: from report.meta.judgeModel)
+  --judge-models <executor:model>  clustering judge (default: from report.meta.judgeModels[0]; single-judge only)
   --max-clusters <n>          maximum clusters (default 5)
   --threshold <num>           failure score threshold (default 3)
   --max-feed <n>              max failures fed to LLM (default 50; takes the worst)
@@ -671,13 +730,16 @@ The command writes `~/.oh-my-knowledge/analyses/<timestamp>-skill-health.json`.
 |---|---|---|
 | `claude` | default | invokes `claude -p` via Claude CLI |
 | `claude-sdk` | structured output | uses Claude Agent SDK — no stdout parsing, avoids buffer truncation |
-| `openai` | cross-vendor comparison | invokes `openai api` CLI |
+| `codex` | OpenAI agent CLI | invokes `codex exec --json` (`@openai/codex` npm); best-effort tool trace; **costUSD not reported** (codex CLI does not emit USD; check usage externally) |
+| `codex-sdk` | OpenAI agent SDK | uses `@openai/codex-sdk` with its bundled `@openai/codex` binary and streamed SDK events; **costUSD not reported** |
 | `gemini` | cross-vendor comparison | invokes `gemini` CLI |
 | `anthropic-api` | no CLI needed | calls Anthropic HTTP API directly (needs `ANTHROPIC_API_KEY`) |
 | `openai-api` | no CLI needed | calls OpenAI HTTP API directly (needs `OPENAI_API_KEY`) |
 
 API-direct executors support custom base URLs via env: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
 
+Codex construct-validity notes: (1) `codex` uses the `codex` binary on `PATH`; `codex-sdk` uses the bundled `@openai/codex` binary resolved by `@openai/codex-sdk`. Reports persist per-variant `meta.executorRuntimes`, `meta.executorRuntime`, and per-judge `meta.judgeModels[].runtime` fingerprints (binary or SDK version + capability snapshot), and `bench diff` / `bench verdict` warn when strict comparability cannot be audited. If runtime fingerprints differ, treat results as an executor-runtime comparison, not only prompt/template behavior. (2) Both executors isolate user-level config: `codex` passes `--ephemeral` + `--ignore-user-config`; `codex-sdk` redirects `$CODEX_HOME` to a per-process tmp dir (auth.json symlinked through). User-level `~/.codex/config.toml` does not leak into eval runs in either case.
+
 ### Custom executor
 
 Any shell command can serve as an executor, communicating via stdin/stdout JSON:
@@ -721,7 +783,7 @@ skills/
 | `./path/to/file.md` | path with `/`: read the file directly as an artifact |
 | `variant@/path/to/project` | attach a run dir to any variant; supports `name@cwd`, `git:name@cwd`, `/file.md@cwd` |
 
-When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--each`. With `--each`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
+When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--batch`. With `--batch`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
 
 ```bash
 # explicit: one control, one or more treatments
@@ -862,25 +924,25 @@ omk bench run \
 export OPENAI_API_KEY="your Zhipu API key"
 export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
 omk bench run --executor openai-api --model glm-4-plus \
-  --judge-model glm-4-plus --no-cache
+  --judge-models openai-api:glm-4-plus --no-cache
 
 # Qwen (Alibaba)
 export OPENAI_API_KEY="your Qwen API key"
 export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
 omk bench run --executor openai-api --model qwen-plus \
-  --judge-model qwen-plus
+  --judge-models openai-api:qwen-plus
 
 # DeepSeek
 export OPENAI_API_KEY="your DeepSeek API key"
 export OPENAI_BASE_URL="https://api.deepseek.com"
 omk bench run --executor openai-api --model deepseek-chat \
-  --judge-model deepseek-chat
+  --judge-models openai-api:deepseek-chat
 
 # Moonshot (Kimi)
 export OPENAI_API_KEY="your Moonshot API key"
 export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
 omk bench run --executor openai-api --model moonshot-v1-8k \
-  --judge-model moonshot-v1-8k
+  --judge-models openai-api:moonshot-v1-8k
 ```
 
 **Ollama local model:**
@@ -890,11 +952,11 @@ omk bench run --executor "python examples/custom-executor/ollama-executor.py" \
   --model llama3 --no-judge
 ```
 
-**About the judge model:**
+**About the judge:**
 
-- `--judge-model` picks the model used by the LLM judge (default `haiku`)
-- `--judge-executor` picks the executor the judge uses (defaults to `--executor`)
-- If you don't have Claude, point `--judge-executor` and `--judge-model` at whatever model you have
+- `--judge-models <list>` picks the LLM judge(s). Format: `executor:model[,executor:model]`. Default: `${executor}:haiku` (or claude:haiku when no `--executor` set)
+- 1 entry = single judge; ≥ 2 entries = multi-judge ensemble + inter-judge agreement
+- If you don't have Claude, point `--judge-models` at whatever you have, e.g. `--judge-models openai-api:glm-4-plus`
 - Add `--no-judge` to skip the LLM judge and rely on assertions alone
 
 ## Environment variables
@@ -927,4 +989,4 @@ This tool is designed for **local trusted environments** (dev machines, CI pipel
 
 ---
 
-See [CHANGELOG](./CHANGELOG.md) for release notes. Contributions welcome — see [CONTRIBUTING](./CONTRIBUTING.md).
+See [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases) for release notes. Contributions welcome — see [CONTRIBUTING](./CONTRIBUTING.md).