oh-my-knowledge 0.24.0 → 0.25.1

package/README.md

@@ -1,6 +1,7 @@
 # oh-my-knowledge
 
 [![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
+[![npm weekly downloads](https://img.shields.io/npm/dw/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
 [![CI](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml/badge.svg)](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
 [![Node.js Version](https://img.shields.io/node/v/oh-my-knowledge.svg)](https://nodejs.org)
@@ -10,7 +11,7 @@
 **omk** — The knowledge you give your LLM — what's it actually worth?
 omk answers with objective data, not gut feeling.
 
-**Fix the model, vary the knowledge artifact.**
+**Evaluation framework for LLM knowledge inputs** — prompts, RAG corpora, skills, agent workflows. Fix the model, vary the artifact.
 
 <a id="statistical-rigor"></a>
 > Built-in: Bootstrap CI · Krippendorff α (judge ↔ human) · length-debias · saturation curves · construct-validity isolation. [Why these matter →](docs/statistical-rigor.md)
@@ -37,23 +38,39 @@ omk bench run --dry-run
 
 # run the evaluation (auto-discovers everything under skills/)
 omk bench run    # → HTML report with verdict in 5 minutes
+                 # (omk doctor + LLM connectivity check both run as mandatory gates;
+                 #  --skip-connectivity available for connectivity, doctor is unconditional)
 
 # CLI output language: zh (default) / en — flag wins over env
 omk bench run --lang en
 OMK_LANG=en omk bench report
 ```
 
-## Use inside Claude Code
+## Use inside AI Coding Agents
 
-After installing omk, talk to it in natural language from Claude Code:
+### Use inside Claude Code
 
-```
+When the `omk` skill is available in Claude Code, you can invoke it directly like this:
+
+```bash
 /omk eval              # evaluate the artifact(s) in the current project
 /omk evolve            # auto-iterate to improve an artifact
 /omk gen-samples       # generate test cases
 ```
 
-You can also just say "compare v1 vs v2 for me" or "improve this artifact" — omk picks the right command.
+You can also just say "compare v1 vs v2 for me" or "improve this artifact" and omk picks the right command.
+
+### Use inside Codex
+
+Codex does not support Claude Code style `/omk ...` slash commands by default. In Codex, the usual pattern is to ask the agent to run the `omk` CLI directly, for example:
+
+```bash
+omk bench run
+omk bench evolve
+omk bench gen-samples skills/my-skill.md
+```
+
+You can also describe the goal in natural language, such as "compare v1 vs v2" or "generate test cases for this skill".
 
 ## Why this tool
 
@@ -61,6 +78,7 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 
 ## Key features
 
+- **Pre-evaluation health check** — `omk doctor` runs as a mandatory gate before `bench run` / `bench gate`; checks skill readability, metadata, dependencies, samples contract — pure static, zero LLM calls (like lint + typecheck for knowledge artifacts). Executor / judge connectivity is a separate phase, controllable via `--skip-connectivity`
 - **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
 - **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
 - **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
@@ -376,7 +394,9 @@ options:
                          samples + variants + model + executor in one file; CLI
                          flags override config fields when both are provided
   --model <name>         task execution model (default: sonnet)
-  --judge-model <name>   judge model (default: haiku)
+  --judge-models <list>  judge config; 1 entry = single judge (default
+                         claude:haiku), ≥ 2 entries = ensemble. Format:
+                         `executor:model[,executor:model]`
   --output-dir <path>    output dir (default: ~/.oh-my-knowledge/reports/)
   --no-judge             skip the LLM judge
   --no-cache             disable result cache (on by default; identical inputs reuse)
@@ -386,7 +406,10 @@ options:
   --timeout <sec>        per-task executor timeout (default: 120)
   --repeat <n>           repeat N times for variance analysis (default: 1)
   --executor <name>      executor (default: claude); supports custom commands
-  --skip-preflight       skip evaluation model reachability check
+  --skip-connectivity    skip the LLM connectivity check (doctor still runs;
+                         doctor is mandatory and has no skip flag).
+                         Auto-applied on --resume (the original run already
+                         verified connectivity).
   --mcp-config <path>    MCP config for fetching private-doc URLs via MCP Server
                          (default: .mcp.json in cwd)
   --no-serve             don't auto-start the report server after the run
@@ -394,8 +417,6 @@ options:
   --batch                batch mode: evaluate each artifact independently vs baseline
                          requires {name}.eval-samples.json paired with each artifact
   --judge-repeat <n>     run the LLM judge N times per (sample × dimension) and report stddev
-  --judge-models <list>  multi-judge ensemble: "executor1:model1,executor2:model2"
-                         ≥ 2 judges enables ensemble + inter-judge agreement output
   --bootstrap            enable distribution-free CIs: bootstrap CI per variant +
                          pairwise diff CI (CI containing 0 = not significant)
   --bootstrap-samples N  bootstrap resample count (default 1000)
@@ -414,6 +435,29 @@ options:
 
 **eval.yaml budget**: declare `budget: { totalUSD?, perSampleUSD?, perSampleMs? }` (all optional, must be ≥ 0). CLI flags of the same name override the config values.
 
+**eval.yaml experiment-design fields**: the same flags above can be set in `eval.yaml` for reproducible experiment configuration (CLI > eval.yaml > default):
+
+```yaml
+samples: ./eval-samples.yaml
+model: sonnet
+repeat: 5                    # multi-run variance, ≥ 1
+judgeRepeat: 3               # per (sample × dim) judge self-consistency, ≥ 1
+bootstrap: true              # distribution-free CI per variant
+bootstrapSamples: 2000       # default 1000, ≥ 100
+goldDir: ./gold              # post-run α / κ / Pearson against human anchor
+lengthDebias: true           # default; set false to reproduce pre-v0.21 hash
+strictBaseline: true         # default; set false to disable skill isolation
+noJudge: false               # default; set true to skip LLM judge entirely
+judgeModels:                 # 1 entry = single judge; ≥ 2 = ensemble
+  - { executor: claude, model: opus }
+  - { executor: openai-api, model: gpt-4o }
+variants:
+  - { name: baseline, role: control, artifact: baseline }
+  - { name: my-skill, role: treatment, artifact: ./skills/my-skill.md }
+```
+
+**Field entry points**: `bench run` reads every field above. `bench gate` goes through `parseRunConfig` and picks up the shared subset (`variants` / `executor` / `model` / `judgeModels` — both single-judge and ensemble — / `noJudge` / `noCache` / `blind` / `strictBaseline` / `budget` / `mcpConfig` / `variantAllowedSkills`); the experiment-design fields handled by `handleRun` (`repeat` / `judgeRepeat` / `bootstrap` / `bootstrapSamples` / `goldDir` / `lengthDebias`) are intentionally not read by `gate` and can be extended later. Other subcommands (`evolve` / `verdict` / `diff` / `analyze` / …) do not read `eval.yaml`.
+
 **Difference from `cost_max` / `latency_max` assertions**: assertions are **per-sample scoring rules** (exceeding the cap fails that one assertion, the run continues); budget caps are **workflow-level hard limits** (`totalUSD` overrun aborts the run and persists a partial report; per-sample overruns fail the offending sample but the run continues). Assertions answer "is quality acceptable?"; budgets answer "are cost/time within the envelope?".
 
 ### `omk bench run --batch` (batch mode)
@@ -501,6 +545,26 @@ omk bench gate [options]
                          independently to fact / behavior / judge
 ```
 
+### `omk doctor` (pre-evaluation health check)
+
+Pure static / zero-LLM checks — analogous to lint + typecheck in the SE toolchain. Runs as a mandatory gate before `bench run` / `bench gate` so a typo'd YAML or missing dependency aborts with an actionable error instead of producing a garbage-in verdict. Also runnable standalone for local iteration or CI.
+
+```bash
+omk doctor                    # batch check every skill in current dir / ./skills
+omk doctor skills/v1.md       # single file
+omk doctor skills/ --json     # JSON output for CI consumption
+omk doctor --gate; echo $?    # silent mode — exit 1 if any fatal check fails
+```
+
+What `doctor` checks:
+
+- **skill readable** — file exists, content non-empty, has minimum length
+- **skill metadata** — front-matter (if present) is valid YAML; directory-skills have `SKILL.md`
+- **dependencies present** — referenced CLI tools, files, env vars all available (reuses `preflightDependencies`)
+- **samples ↔ skill contract** — when samples are provided, validate they're non-empty and have prompt fields (warn-level)
+
+Executor / judge connectivity is verified by a separate evaluation preflight phase, not by doctor — clean boundary: doctor is static, eval is dynamic. `bench run` / `bench gate` abort with `exit 1` and stderr `doctor failed:` prefix when doctor fails. **Doctor is mandatory and not skippable** (static checks have no cost reason to skip); LLM connectivity is separately controllable via `--skip-connectivity` (auto-skipped on `--resume`).
+
 ### `omk bench report`
 
 Start the report server to browse historical reports, submit feedback, and delete reports.
@@ -545,7 +609,7 @@ Re-judges every (sample × variant) of an existing report with the OPPOSITE leng
 ```bash
 omk bench debias-validate length <reportId> [options]
   --variant <name>            check a single variant only
-  --judge-model <id>          override the report's judge model
+  --judge-models <executor:model>  override the report's judge (single-judge only)
   --bootstrap-samples N       bootstrap iterations (default 1000)
   --seed N                    deterministic seed
 ```
@@ -601,8 +665,7 @@ When 14 of 50 samples failed, reading them one by one is slow. This command send
 
 ```bash
 omk bench failures <reportId> [options]
-  --judge-executor <name>     executor (default: claude)
-  --judge-model <id>          clustering model (default: from report.meta.judgeModel)
+  --judge-models <executor:model>  clustering judge (default: from report.meta.judgeModels[0]; single-judge only)
   --max-clusters <n>          maximum clusters (default 5)
   --threshold <num>           failure score threshold (default 3)
   --max-feed <n>              max failures fed to LLM (default 50; takes the worst)
@@ -675,7 +738,7 @@ The command writes `~/.oh-my-knowledge/analyses/<timestamp>-skill-health.json`.
 
 API-direct executors support custom base URLs via env: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
 
-Codex construct-validity notes: (1) `codex` uses the `codex` binary on `PATH`; `codex-sdk` uses the bundled `@openai/codex` binary resolved by `@openai/codex-sdk`. Reports persist per-variant `meta.executorRuntimes` plus `meta.executorRuntime` / `meta.judgeRuntime` fingerprints (binary or SDK version + capability snapshot), and `bench diff` / `bench verdict` warn when strict comparability cannot be audited. If runtime fingerprints differ, treat results as an executor-runtime comparison, not only prompt/template behavior. (2) Both executors isolate user-level config: `codex` passes `--ephemeral` + `--ignore-user-config`; `codex-sdk` redirects `$CODEX_HOME` to a per-process tmp dir (auth.json symlinked through). User-level `~/.codex/config.toml` does not leak into eval runs in either case.
+Codex construct-validity notes: (1) `codex` uses the `codex` binary on `PATH`; `codex-sdk` uses the bundled `@openai/codex` binary resolved by `@openai/codex-sdk`. Reports persist per-variant `meta.executorRuntimes`, `meta.executorRuntime`, and per-judge `meta.judgeModels[].runtime` fingerprints (binary or SDK version + capability snapshot), and `bench diff` / `bench verdict` warn when strict comparability cannot be audited. If runtime fingerprints differ, treat results as an executor-runtime comparison, not only prompt/template behavior. (2) Both executors isolate user-level config: `codex` passes `--ephemeral` + `--ignore-user-config`; `codex-sdk` redirects `$CODEX_HOME` to a per-process tmp dir (auth.json symlinked through). User-level `~/.codex/config.toml` does not leak into eval runs in either case.
 
 ### Custom executor
 
@@ -861,25 +924,25 @@ omk bench run \
 export OPENAI_API_KEY="your Zhipu API key"
 export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
 omk bench run --executor openai-api --model glm-4-plus \
-  --judge-model glm-4-plus --no-cache
+  --judge-models openai-api:glm-4-plus --no-cache
 
 # Qwen (Alibaba)
 export OPENAI_API_KEY="your Qwen API key"
 export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
 omk bench run --executor openai-api --model qwen-plus \
-  --judge-model qwen-plus
+  --judge-models openai-api:qwen-plus
 
 # DeepSeek
 export OPENAI_API_KEY="your DeepSeek API key"
 export OPENAI_BASE_URL="https://api.deepseek.com"
 omk bench run --executor openai-api --model deepseek-chat \
-  --judge-model deepseek-chat
+  --judge-models openai-api:deepseek-chat
 
 # Moonshot (Kimi)
 export OPENAI_API_KEY="your Moonshot API key"
 export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
 omk bench run --executor openai-api --model moonshot-v1-8k \
-  --judge-model moonshot-v1-8k
+  --judge-models openai-api:moonshot-v1-8k
 ```
 
 **Ollama local model:**
@@ -889,11 +952,11 @@ omk bench run --executor "python examples/custom-executor/ollama-executor.py" \
   --model llama3 --no-judge
 ```
 
-**About the judge model:**
+**About the judge:**
 
-- `--judge-model` picks the model used by the LLM judge (default `haiku`)
-- `--judge-executor` picks the executor the judge uses (defaults to `--executor`)
-- If you don't have Claude, point `--judge-executor` and `--judge-model` at whatever model you have
+- `--judge-models <list>` picks the LLM judge(s). Format: `executor:model[,executor:model]`. Default: `${executor}:haiku` (or claude:haiku when no `--executor` set)
+- 1 entry = single judge; ≥ 2 entries = multi-judge ensemble + inter-judge agreement
+- If you don't have Claude, point `--judge-models` at whatever you have, e.g. `--judge-models openai-api:glm-4-plus`
 - Add `--no-judge` to skip the LLM judge and rely on assertions alone
 
 ## Environment variables
@@ -926,4 +989,4 @@ This tool is designed for **local trusted environments** (dev machines, CI pipel
 
 ---
 
-See [CHANGELOG](./CHANGELOG.md) for release notes. Contributions welcome — see [CONTRIBUTING](./CONTRIBUTING.md).
+See [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases) for release notes. Contributions welcome — see [CONTRIBUTING](./CONTRIBUTING.md).

package/README.zh.md

@@ -1,6 +1,7 @@
 # oh-my-knowledge
 
 [![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
+[![npm weekly downloads](https://img.shields.io/npm/dw/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
 [![CI](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml/badge.svg)](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
 [![Node.js Version](https://img.shields.io/node/v/oh-my-knowledge.svg)](https://nodejs.org)
@@ -10,7 +11,7 @@
 **omk** — 你给 LLM 的知识,价值在哪里?
 omk 帮你用客观数据回答,而不是凭感觉。
 
-**固定模型,只变知识载体。**
+**面向 LLM 知识输入(prompt / RAG / skill / agent)的评测框架** —— 固定模型,只变知识载体。
 
 <a id="statistical-rigor"></a>
 > 默认带:Bootstrap 置信区间 · Krippendorff α(评委 ↔ 人工)· 长度去偏 · 饱和曲线 · 用例隔离(construct validity)。[这些为什么重要 →](docs/zh/statistical-rigor.md)
@@ -37,17 +38,21 @@ omk bench run --dry-run
 
 # 运行评测（自动发现 skills/ 目录下的所有 artifact）
 omk bench run    # → 5 分钟出 HTML 报告 + verdict
+                 # （omk doctor 和 LLM 连通性检测都是强制前置门禁；
+                 #  --skip-connectivity 可跳连通性，doctor 无 skip flag）
 
 # CLI 输出语言: zh (默认) / en — flag 优先级高于环境变量
 omk bench run --lang en
 OMK_LANG=en omk bench report
 ```
 
-## 在 Claude Code 中使用
+## 在 AI Coding Agent 中使用
 
-安装 omk 后，在 Claude Code 中直接用自然语言交互：
+### 在 Claude Code 中使用
 
-```
+当 `omk` skill 已在 Claude Code 中可用时，可以直接这样调用：
+
+```bash
 /omk eval              # 评测当前项目的 artifact
 /omk evolve            # 自动迭代改进 artifact
 /omk gen-samples       # 生成测试用例
@@ -55,12 +60,25 @@ OMK_LANG=en omk bench report
 
 或直接说"帮我评测 v1 和 v2 的差异"、"改进一下这个 artifact"，omk 会自动理解意图并调用对应命令。
 
+### 在 Codex 中使用
+
+Codex 默认不支持 `/omk ...` 这种 Claude Code 风格的 slash command。通常直接让 agent 执行 `omk` CLI，例如：
+
+```bash
+omk bench run
+omk bench evolve
+omk bench gen-samples skills/my-skill.md
+```
+
+也可以直接用自然语言描述目标，例如"比较 v1 和 v2 的评测差异"、"为这个 skill 生成测试用例"。
+
 ## 为什么需要这个工具
 
 做知识工程的团队会产出大量知识载体（当前常见是 skill，也包括 prompt、agent、workflow 等）。当被问到"v2 比 v1 好在哪"时，需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题：相同模型、相同测试用例，只改变知识载体。
 
 ## 核心能力
 
+- **评测前置健康检查** — `omk doctor` 在 `bench run` / `bench gate` 之前**强制**运行,检查 skill 可读性、元数据合法性、依赖完整性、samples 契约——纯静态零 LLM 调用,类比 SE 工具栈的 lint + typecheck。executor / judge 连通性是独立阶段,可用 `--skip-connectivity` 单独跳过
 - **控制变量离线评测** — 固定模型和用例，只变知识载体；兼容 Claude Code skill、CLAUDE.md prompt、RAG 知识库等任何 markdown 形式的指令
 - **六维独立打分** — Fact / Behavior / LLM-judge / Cost / Efficiency / Stability 分别出信号，单一维度的回退不会被其他维度的收益掩盖
 - **线上 session 观测** — 解析 Claude Code session JSONL，在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
@@ -374,7 +392,8 @@ omk bench run [选项]
   --config <路径>        YAML/JSON 配置文件（evaluation-as-code）;在一个文件里声明
                          samples + variants + model + executor;CLI 参数会覆盖 config
   --model <名称>         被测模型（默认：sonnet）
-  --judge-model <名称>   评委模型（默认：haiku）
+  --judge-models <list>  评委配置;1 条 = 单评委 (默认 claude:haiku),
+                         ≥ 2 条 = ensemble。格式 `executor:model[,executor:model]`
   --output-dir <路径>    输出目录（默认：~/.oh-my-knowledge/reports/）
   --no-judge             跳过 LLM 评分
   --no-cache             禁用结果缓存（默认开启，相同输入自动复用）
@@ -384,7 +403,8 @@ omk bench run [选项]
   --timeout <秒>         单个任务的执行器超时时间（默认：120）
   --repeat <n>           重复 N 次做方差分析（默认：1）
   --executor <名称>      执行器（默认：claude），支持自定义命令
-  --skip-preflight       跳过评测前的模型连通性检查
+  --skip-connectivity    跳过评测前 LLM 连通性检测(doctor 仍然强制执行,无 skip flag)。
+                         --resume 时自动跳过(原 run 已验过连通性)。
   --mcp-config <路径>    MCP 配置文件，用于通过 MCP Server 获取私有文档 URL 内容
                          （默认：当前目录的 .mcp.json）
   --no-serve             评测完成后不自动启动报告服务
@@ -392,8 +412,6 @@ omk bench run [选项]
   --batch                批量评测：每个 artifact 独立和 baseline 对比
                          需要每个 artifact 配对 {name}.eval-samples.json
   --judge-repeat <n>     每条 sample × dimension 跑 LLM 评委 N 次,输出 stddev (评委自一致性)
-  --judge-models <list>  多评委 ensemble: "executor1:model1,executor2:model2"
-                         ≥ 2 个 judge 触发 ensemble + inter-judge agreement 输出
   --bootstrap            启用 distribution-free CI:每个 variant 加 bootstrap CI,
                          pairwise diff CI 含 0 = 不显著
   --bootstrap-samples N  bootstrap 重采样次数 (默认 1000)
@@ -409,6 +427,29 @@ omk bench run [选项]
 
 **eval.yaml 预算字段**: `budget: { totalUSD?, perSampleUSD?, perSampleMs? }`,所有字段可选且必须 ≥ 0。CLI 同名 flag 覆盖配置值。
 
+**eval.yaml 实验设计字段**: 上面 CLI flag 同样可以写到 `eval.yaml` 让实验配置可复现 (CLI > eval.yaml > 默认):
+
+```yaml
+samples: ./eval-samples.yaml
+model: sonnet
+repeat: 5                    # 多轮方差分析, ≥ 1
+judgeRepeat: 3               # 每条 (sample × dim) 评委自一致性次数, ≥ 1
+bootstrap: true              # 每 variant distribution-free CI
+bootstrapSamples: 2000       # 默认 1000, ≥ 100
+goldDir: ./gold              # 跑完自动对比 human anchor 算 α / κ / Pearson
+lengthDebias: true           # 默认; 设 false 复现 v0.21 之前的 hash
+strictBaseline: true         # 默认; 设 false 关掉 skill 隔离
+noJudge: false               # 默认; 设 true 完全跳过 LLM 评委
+judgeModels:                 # 1 条 = 单评委; ≥ 2 条 = ensemble
+  - { executor: claude, model: opus }
+  - { executor: openai-api, model: gpt-4o }
+variants:
+  - { name: baseline, role: control, artifact: baseline }
+  - { name: my-skill, role: treatment, artifact: ./skills/my-skill.md }
+```
+
+**字段入口**: `bench run` 完整支持上述全部字段; `bench gate` 通过 `parseRunConfig` 共享 variants / executor / model / `judgeModels`(单评委 + ensemble 都生效)/ noJudge / noCache / blind / strictBaseline / budget / mcpConfig / variantAllowedSkills,但 `handleRun` 自己处理的实验设计字段(`repeat` / `judgeRepeat` / `bootstrap` / `bootstrapSamples` / `goldDir` / `lengthDebias`)gate 不读,后续按需扩展到 gate。其他子命令(`evolve` / `verdict` / `diff` / `analyze` 等)完全不读 eval.yaml。
+
 **和 `cost_max` / `latency_max` 断言的区别**: 断言是**单样本评分维度**(超出直接打 0 分,run 继续);budget 是**工作流级硬阈值**(`totalUSD` 超出整个 run abort 保留 partial report,per-sample 超出该样本失败但 run 继续)。一个回答"质量是否达标",一个回答"花钱/时间是否在预算内"。
 
 ### `omk bench run --batch`（批量评测）
@@ -496,6 +537,26 @@ omk bench gate [选项]
                          fact / behavior / judge 三层
 ```
 
+### `omk doctor`(评测前置健康检查)
+
+纯静态 / 零 LLM 调用,类比 SE 工具栈的 lint + typecheck。`bench run` / `bench gate` 之前强制运行,YAML 写错、依赖缺失这类问题会 abort 评测并给可操作错误,而不是让你拿到 garbage-in 的 verdict 数字。也可独立调用,适合本地迭代或 CI 单跑。
+
+```bash
+omk doctor                    # 批量检查当前目录或 ./skills 下所有 skill
+omk doctor skills/v1.md       # 单个文件
+omk doctor skills/ --json     # JSON 输出供 CI 消费
+omk doctor --gate; echo $?    # 静默模式 — 任意 fatal 失败 exit 1
+```
+
+doctor 检查项:
+
+- **skill 文件可读** — 文件存在、内容非空、有最低长度
+- **skill 元数据合法** — front-matter(若有)YAML 合法;directory-skill 有 `SKILL.md`
+- **前置依赖完整** — 引用的 CLI 工具、文件、环境变量都可用(复用 `preflightDependencies`)
+- **用例 ↔ skill 输入约定** — 传 samples 时校验非空且含 prompt 字段(warn 级)
+
+executor / judge 连通性由独立的 evaluation preflight 阶段负责,不在 doctor 范围内 — 边界清晰:doctor 静态,eval 动态。`bench run` / `bench gate` 在 doctor 失败时 abort(exit 1,stderr 前缀 `doctor failed:`)。**doctor 是评测必经环节,无 skip flag**(静态检查零成本无理由跳过);LLM 连通性可用 `--skip-connectivity` 单独控制(`--resume` 时自动跳过)。
+
 ### `omk bench report`
 
 启动报告服务，浏览历史报告、提交反馈、删除报告。
@@ -540,7 +601,7 @@ gold-dir/
 ```bash
 omk bench debias-validate length <reportId> [选项]
   --variant <name>            只测一个 variant
-  --judge-model <id>          override report 的 judge model
+  --judge-models <executor:model>  override report 的评委(仅支持单评委)
   --bootstrap-samples N       bootstrap 迭代数 (默认 1000)
   --seed N                    确定性种子
 ```
@@ -596,8 +657,7 @@ omk bench diagnose <reportId> [选项]
 
 ```bash
 omk bench failures <reportId> [选项]
-  --judge-executor <name>     执行器 (默认 claude)
-  --judge-model <id>          聚类用 model (默认沿用 report.meta.judgeModel)
+  --judge-models <executor:model>  聚类评委 (默认沿用 report.meta.judgeModels[0],仅支持单评委)
   --max-clusters <n>          最多多少 cluster (默认 5)
   --threshold <num>           算失败的分数阈值 (默认 3)
   --max-feed <n>              最多喂给 LLM 多少条 (默认 50,超出取最差)
@@ -670,7 +730,7 @@ omk analyze ~/.claude/projects/my-project --kb /path/to/project
 
 API 直调执行器支持通过环境变量自定义 Base URL：`ANTHROPIC_BASE_URL`、`OPENAI_BASE_URL`。
 
-Codex construct-validity 说明:(1) `codex` 使用 `PATH` 上找到的 `codex` binary;`codex-sdk` 使用 `@openai/codex-sdk` 解析到的自带 `@openai/codex` binary。报告会持久化 per-variant `meta.executorRuntimes` 以及 `meta.executorRuntime` / `meta.judgeRuntime` 指纹(binary 或 SDK 版本 + 能力快照),`bench diff` / `bench verdict` 会在 strict comparability 无法审计时提示。runtime 指纹不一致时,结果应解释为 executor runtime 对比,而不只是 prompt/template 行为对比。(2) 两个 executor 都隔离用户级 config:`codex` 传 `--ephemeral` + `--ignore-user-config`,`codex-sdk` 把 `$CODEX_HOME` 重定向到 per-process tmp 目录(auth.json 通过 symlink 透传)。用户的 `~/.codex/config.toml` 不会渗入任意一个 executor 的 eval。
+Codex construct-validity 说明:(1) `codex` 使用 `PATH` 上找到的 `codex` binary;`codex-sdk` 使用 `@openai/codex-sdk` 解析到的自带 `@openai/codex` binary。报告会持久化 per-variant `meta.executorRuntimes`、`meta.executorRuntime`,以及每个评委的 `meta.judgeModels[].runtime` 指纹(binary 或 SDK 版本 + 能力快照),`bench diff` / `bench verdict` 会在 strict comparability 无法审计时提示。runtime 指纹不一致时,结果应解释为 executor runtime 对比,而不只是 prompt/template 行为对比。(2) 两个 executor 都隔离用户级 config:`codex` 传 `--ephemeral` + `--ignore-user-config`,`codex-sdk` 把 `$CODEX_HOME` 重定向到 per-process tmp 目录(auth.json 通过 symlink 透传)。用户的 `~/.codex/config.toml` 不会渗入任意一个 executor 的 eval。
 
 ### 自定义执行器
 
@@ -856,25 +916,25 @@ omk bench run \
 export OPENAI_API_KEY="你的智谱 API Key"
 export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
 omk bench run --executor openai-api --model glm-4-plus \
-  --judge-model glm-4-plus --no-cache
+  --judge-models openai-api:glm-4-plus --no-cache
 
 # 通义千问
 export OPENAI_API_KEY="你的通义 API Key"
 export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
 omk bench run --executor openai-api --model qwen-plus \
-  --judge-model qwen-plus
+  --judge-models openai-api:qwen-plus
 
 # DeepSeek
 export OPENAI_API_KEY="你的 DeepSeek API Key"
 export OPENAI_BASE_URL="https://api.deepseek.com"
 omk bench run --executor openai-api --model deepseek-chat \
-  --judge-model deepseek-chat
+  --judge-models openai-api:deepseek-chat
 
 # Moonshot（Kimi）
 export OPENAI_API_KEY="你的 Moonshot API Key"
 export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
 omk bench run --executor openai-api --model moonshot-v1-8k \
-  --judge-model moonshot-v1-8k
+  --judge-models openai-api:moonshot-v1-8k
 ```
 
 **Ollama 本地模型：**
@@ -884,12 +944,12 @@ omk bench run --executor "python examples/custom-executor/ollama-executor.py" \
   --model llama3 --no-judge
 ```
 
-**关于评委模型：**
+**关于评委:**
 
-- `--judge-model` 指定 LLM 评委使用的模型，默认 `haiku`
-- `--judge-executor` 指定评委使用的执行器（默认与 `--executor` 相同）
-- 如果你没有 Claude，用 `--judge-executor` 和 `--judge-model` 指向你可用的模型
-- 加 `--no-judge` 可跳过 LLM 评委，仅使用断言评分
+- `--judge-models <list>` 指定评委,格式 `executor:model[,executor:model]`。默认 `${executor}:haiku`(没设 `--executor` 时为 claude:haiku)
+- 1 条 = 单评委;≥ 2 条 = 多评委 ensemble + inter-judge agreement
+- 没有 Claude 时把 `--judge-models` 指向你可用的模型,例如 `--judge-models openai-api:glm-4-plus`
+- 加 `--no-judge` 可跳过 LLM 评委,仅使用断言评分
 
 ## 环境变量
 
@@ -921,4 +981,4 @@ omk bench run --executor "python examples/custom-executor/ollama-executor.py" \
 
 ---
 
-版本变更记录见 [CHANGELOG](./CHANGELOG.md)。欢迎贡献 — 详见 [CONTRIBUTING](./CONTRIBUTING.md)。
+版本变更记录见 [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases)。欢迎贡献 — 详见 [CONTRIBUTING](./CONTRIBUTING.md)。

package/dist/src/analysis/saturation.d.ts

@@ -7,7 +7,7 @@
  * is "is N=30 enough, or should I keep running?". Without a principled
  * answer, users either over-pay (running 200 when 50 was enough) or
  * under-pay (calling skill effects null at N=30 when N=80 would have
- * shown clearly significant — Bootstrap CI from Phase 1 just hasn't
+ * shown clearly significant — Bootstrap CI just hasn't
  * converged yet).
  *
  * Saturation analysis fits the right tool to the question: track a
@@ -19,7 +19,7 @@
  *    checkpoints. Easy to explain. Fragile to outlier samples.
  *  - **bootstrap-ci-width** (default): CI shrinks as O(1/√N); when its
  *    decay rate flattens, more samples buy little. Statistically
- *    grounded; pairs naturally with Phase 1.
+ *    grounded; pairs naturally with the Bootstrap CI module.
  *  - **plateau-height**: range of mean across the last K checkpoints.
  *    Conservative — slow to declare saturation, hard to fool.
  *

package/dist/src/analysis/saturation.js

@@ -7,7 +7,7 @@
  * is "is N=30 enough, or should I keep running?". Without a principled
  * answer, users either over-pay (running 200 when 50 was enough) or
  * under-pay (calling skill effects null at N=30 when N=80 would have
- * shown clearly significant — Bootstrap CI from Phase 1 just hasn't
+ * shown clearly significant — Bootstrap CI just hasn't
  * converged yet).
  *
  * Saturation analysis fits the right tool to the question: track a
@@ -19,7 +19,7 @@
  *    checkpoints. Easy to explain. Fragile to outlier samples.
  *  - **bootstrap-ci-width** (default): CI shrinks as O(1/√N); when its
  *    decay rate flattens, more samples buy little. Statistically
- *    grounded; pairs naturally with Phase 1.
+ *    grounded; pairs naturally with the Bootstrap CI module.
  *  - **plateau-height**: range of mean across the last K checkpoints.
  *    Conservative — slow to declare saturation, hard to fool.
  *

package/dist/src/authoring/evolver.d.ts

@@ -1,4 +1,4 @@
-import type { ProgressCallback, Report, Sample } from '../types/index.js';
+import type { JudgeConfig, ProgressCallback, Report, Sample } from '../types/index.js';
 interface WeakSample {
     sample_id: string;
     compositeScore: number;
@@ -29,12 +29,15 @@ interface EvolveOptions {
     rounds?: number;
     target?: number | null;
     model?: string;
-    judgeModel?: string;
+    /** Single-judge config. evolve 不支持 ensemble — CLI 在 length>=2 时 exit 2,
+     *  programmatic API 在 evolveSkill 入口同样 throw,二者一致。Default
+     *  `[{ executor: <executorName>, model: 'haiku' }]`. */
+    judgeModels?: JudgeConfig[];
     improveModel?: string;
     executorName?: string;
     concurrency?: number;
     timeoutMs?: number;
-    skipPreflight?: boolean;
+    skipConnectivity?: boolean;
     onProgress?: ProgressCallback | null;
     onRoundProgress?: ((progress: EvolveRoundProgressInfo) => void) | null;
 }
@@ -64,6 +67,6 @@ export interface RoundReport {
     report: Report;
 }
 export declare function mergeEvolveReports(roundReports: RoundReport[], skillName: string, totalCostUSD: number, samples?: Sample[]): Report;
-export declare function evolveSkill({ skillPath, samplesPath, rounds, target, model, judgeModel, improveModel, executorName, concurrency, timeoutMs, skipPreflight, onProgress, onRoundProgress, }: EvolveOptions): Promise<EvolveResult>;
+export declare function evolveSkill({ skillPath, samplesPath, rounds, target, model, judgeModels, improveModel, executorName, concurrency, timeoutMs, skipConnectivity, onProgress, onRoundProgress, }: EvolveOptions): Promise<EvolveResult>;
 export {};
 //# sourceMappingURL=evolver.d.ts.map

package/dist/src/authoring/evolver.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"evolver.d.ts","sourceRoot":"","sources":["../../../src/authoring/evolver.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,EAAe,MAAM,EAAiB,MAAM,mBAAmB,CAAC;AAYtG,UAAU,UAAU;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,gBAAgB,EAAE,MAAM,EAAE,CAAC;IAC3B,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAAC;CAC3C;AAmBD,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,KAAK,GAAE,MAAU,GAAG,UAAU,EAAE,CAkBtG;AAED,wBAAgB,sBAAsB,CAAC,YAAY,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,GAAG,MAAM,CAqB7G;AAgBD,+DAA+D;AAC/D,MAAM,MAAM,kBAAkB,GAAG,UAAU,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC;AAEjE,MAAM,WAAW,uBAAuB;IACtC,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;qCACiC;IACjC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,UAAU,aAAa;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,UAAU,CAAC,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACrC,eAAe,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,uBAAuB,KAAK,IAAI,CAAC,GAAG,IAAI,CAAC;CACxE;AAED,UAAU,eAAe;IACvB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,YAAY,EAAE,MAAM,CAAC;IACrB,2EAA2E;IAC3E,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,UAAU,EAAE,eAAe,EAAE,CAAC;IAC9B,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,WAAW;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,YAAY,EAAE,WAAW,EAAE,EAC3B,SAAS,EAAE,MAAM,EACjB,YAAY,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,MAAM,EAAE,GACjB,MAAM,CAoER;AAED,wBAAsB,WAAW,CAAC,EAChC,SAAS,EACT,WAAW,EACX,MAAU,EACV,MAAa,EACb,KAAqB,EACrB,UAAwB,EACxB,YAA4B,EAC5B,YAAuB,EACvB,WAAe,EACf,SAAS,EACT,aAAqB,EACrB,UAAiB,EACjB,eAAsB,GACvB,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC,CA+IvC"}
+{"version":3,"file":"evolver.d.ts","sourceRoot":"","sources":["../../../src/authoring/evolver.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAE,WAAW,EAAE,gBAAgB,EAAE,MAAM,EAAe,MAAM,EAAiB,MAAM,mBAAmB,CAAC;AAYnH,UAAU,UAAU;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,cAAc,EAAE,MAAM,CAAC;IACvB,SAAS,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,gBAAgB,EAAE,MAAM,EAAE,CAAC;IAC3B,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,IAAI,CAAC;CAC3C;AAmBD,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,KAAK,GAAE,MAAU,GAAG,UAAU,EAAE,CAkBtG;AAED,wBAAgB,sBAAsB,CAAC,YAAY,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,GAAG,MAAM,CAqB7G;AAgBD,+DAA+D;AAC/D,MAAM,MAAM,kBAAkB,GAAG,UAAU,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC;AAEjE,MAAM,WAAW,uBAAuB;IACtC,KAAK,EAAE,MAAM,CAAC;IACd,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;qCACiC;IACjC,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,UAAU,aAAa;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;4DAEwD;IACxD,WAAW,CAAC,EAAE,WAAW,EAAE,CAAC;IAC5B,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,UAAU,CAAC,EAAE,gBAAgB,GAAG,IAAI,CAAC;IACrC,eAAe,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,uBAAuB,KAAK,IAAI,CAAC,GAAG,IAAI,CAAC;CACxE;AAED,UAAU,eAAe;IACvB,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,YAAY;IAC3B,UAAU,EAAE,MAAM,CAAC;IACnB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,YAAY,EAAE,MAAM,CAAC;IACrB,2EAA2E;IAC3E,YAAY,CAAC,EAAE,OAAO,CAAC;IACvB,UAAU,EAAE,eAAe,EAAE,CAAC;IAC9B,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,EAAE,CAAC;IACtB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,WAAW;IAC1B,KAAK,EAAE,MAAM,CAAC;IACd,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,YAAY,EAAE,WAAW,EAAE,EAC3B,SAAS,EAAE,MAAM,EACjB,YAAY,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,MAAM,EAAE,GACjB,MAAM,CAoER;AAED,wBAAsB,WAAW,CAAC,EAChC,SAAS,EACT,WAAW,EACX,MAAU,EACV,MAAa,EACb,KAAqB,EACrB,WAAW,EACX,YAA4B,EAC5B,YAAuB,EACvB,WAAe,EACf,SAAS,EACT,gBAAwB,EACxB,UAAiB,EACjB,eAAsB,GACvB,EAAE,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC,CAyJvC"}

package/dist/src/authoring/evolver.js

@@ -149,7 +149,15 @@ export function mergeEvolveReports(roundReports, skillName, totalCostUSD, sample
     report.analysis = analyzeResults(report, { samples });
     return report;
 }
-export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target = null, model = DEFAULT_MODEL, judgeModel = JUDGE_MODEL, improveModel = DEFAULT_MODEL, executorName = 'claude', concurrency = 1, timeoutMs, skipPreflight = false, onProgress = null, onRoundProgress = null, }) {
+export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target = null, model = DEFAULT_MODEL, judgeModels, improveModel = DEFAULT_MODEL, executorName = 'claude', concurrency = 1, timeoutMs, skipConnectivity = false, onProgress = null, onRoundProgress = null, }) {
+    if (judgeModels && judgeModels.length > 1) {
+        throw new Error('evolveSkill does not support multi-judge ensemble (received '
+            + `${judgeModels.length} judges). Pass a single-judge array, e.g. `
+            + `[{ executor: 'claude', model: 'haiku' }]`);
+    }
+    const effectiveJudgeModels = judgeModels && judgeModels.length > 0
+        ? judgeModels
+        : [{ executor: executorName, model: JUDGE_MODEL }];
     const absSkillPath = resolve(skillPath);
     const absSamplesPath = resolve(samplesPath);
     const skillDir = dirname(absSkillPath);
@@ -177,7 +185,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
     const reportHasUnreportedCost = (rep) => Object.values(rep.summary).some((v) => v.execCostReported === false || v.judgeCostReported === false);
     // Round 0: baseline evaluation
     const baselineReport = await evaluate(r0Path, {
-        samplesPath: absSamplesPath, skillDir, model, judgeModel, executorName, concurrency, timeoutMs, skipPreflight, onProgress,
+        samplesPath: absSamplesPath, skillDir, model, judgeModels: effectiveJudgeModels, executorName, concurrency, timeoutMs, skipConnectivity, onProgress,
     });
     const baselineVariantKey = Object.keys(baselineReport.summary)[0];
     bestScore = baselineReport.summary[baselineVariantKey]?.avgCompositeScore ?? 0;
@@ -203,7 +211,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         }
         else {
             lastReport = await evaluate(allVersions[bestRound], {
-                samplesPath: absSamplesPath, skillDir, model, judgeModel, executorName, concurrency, timeoutMs, skipPreflight, onProgress,
+                samplesPath: absSamplesPath, skillDir, model, judgeModels: effectiveJudgeModels, executorName, concurrency, timeoutMs, skipConnectivity, onProgress,
             });
             totalCostUSD += lastReport.meta.totalCostUSD;
             if (reportHasUnreportedCost(lastReport))
@@ -235,7 +243,7 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         allVersions.push(candidatePath);
         // Evaluate candidate
         const candidateReport = await evaluate(candidatePath, {
-            samplesPath: absSamplesPath, skillDir, model, judgeModel, executorName, concurrency, timeoutMs, skipPreflight, onProgress,
+            samplesPath: absSamplesPath, skillDir, model, judgeModels: effectiveJudgeModels, executorName, concurrency, timeoutMs, skipConnectivity, onProgress,
         });
         const candidateVariantKey = Object.keys(candidateReport.summary)[0];
         const candidateScore = candidateReport.summary[candidateVariantKey]?.avgCompositeScore ?? 0;
@@ -289,19 +297,19 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
         reportId,
     };
 }
-async function evaluate(skillFilePath, { samplesPath, skillDir, model, judgeModel, executorName, concurrency, timeoutMs, skipPreflight, onProgress }) {
+async function evaluate(skillFilePath, { samplesPath, skillDir, model, judgeModels, executorName, concurrency, timeoutMs, skipConnectivity, onProgress }) {
     const { report } = await runEvaluation({
         samplesPath,
         skillDir,
         // evolve 评测每一轮只跑当前迭代的 skill，没有对照组；标为 treatment。
         variantSpecs: [{ name: skillFilePath, role: 'treatment', expr: skillFilePath }],
         model,
-        judgeModel,
+        judgeModels,
         outputDir: null, // don't persist intermediate reports
         concurrency,
         timeoutMs,
         executorName,
-        skipPreflight,
+        skipConnectivity,
         onProgress,
     });
     return report;