oh-my-knowledge 0.40.0 → 0.42.0

package/README.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
+Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
+
+Prerequisite: the default executor and judge use the `claude` CLI — install and log in first (see [Requirements](#requirements)); to use another model or run offline (no API key) see [executors](docs/reference/executors.md).
+
+> The first run has only 3 cases, so the verdict will usually be `UNDERPOWERED` (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.
 
 > The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
 
-Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
+Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users). More runnable examples (A/B, offline executor, batch, evolve, agent, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
 
 Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
@@ -105,8 +109,7 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 | **Git & remote sources** | install / eval from a local git ref or a remote git URL (`--git-url`); directory-skills run in a content-addressed **isolated copy** so `references/` assets are real measured input, not just `SKILL.md` |
 | **Evidence-gated management** | `omk install` registers a managed record; `omk eval` auto-writes evidence bound by content fingerprint, moving a skill `installed → measurable`; `omk list` surfaces each managed skill's status (installed / measurable / promoted / stale); `omk promote` accepts a version once its evidence passes the gate (default PROGRESS only); `omk rollback` revokes that acceptance, returning the skill to `measurable`. [spec →](docs/specs/evidence-gated-management.md) |
 | **Sample design science** | sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata (HF Dataset Cards style); studio surfaces coverage breakdown plus `rubric_clarity_low` / `capability_thin` flags. [docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
-| **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
-| **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
+| **Multi-judge ensemble** | `--judge-models claude:opus,openai-api:gpt-4o` cross-vendor scoring + agreement metrics |
 | **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
 | **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
@@ -123,6 +126,7 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
 - **[Executors](docs/reference/executors.md)** & **[artifact layout](docs/reference/artifact-layout.md)** — built-in / custom executors; how `variant` resolves to an artifact + runtime context
 - **[How-to guides](docs/guides/agent-eval.md)** — [evaluate an agent](docs/guides/agent-eval.md) (project runtime context) and [use non-Claude models](docs/guides/non-claude-models.md) (GLM / Qwen / DeepSeek / Moonshot / Ollama)
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
+- **[Example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** — a set of runnable examples in the repo, arranged simplest-to-richest
 - **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
 - **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
 - **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai

package/README.zh.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例；`omk eval` 跑控制变量 A/B，5 分钟内出 HTML 报告 + 一行 verdict。
+开箱即跑：`omk init` 脚手架好两版 skill 和三条评测用例，不用先改任何文件，`omk eval` 跑控制变量 A/B，约 5 分钟出 HTML 报告 + 一行 verdict；跑通后再把 skill 和用例换成你自己的。
+
+前置：默认执行器与评委用 `claude` CLI，需先安装并登录（见[系统要求](#系统要求)）；想用别的模型或离线跑（无需 API key）见[执行器](docs/zh/reference/executors.md)。
+
+> 首跑只有 3 条用例，verdict 多半是「数据不足（UNDERPOWERED）」——这是正常起点而非出错；把用例加到约 20 条以上，再看「可发布」结论。
 
 > 命令行有新版本时会自动提示（每 20 小时最多一次）；想永久关闭该提醒，设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
 
-手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。
+手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。更多可跑示例（A/B、离线执行器、batch、evolve、agent、RAG）见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
 
 深入：[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
 
@@ -105,8 +109,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 | **Git / 远端源** | install / eval 支持本地 git ref 或远端 git URL（`--git-url`）；目录-skill 在内容寻址**隔离副本**里执行，`references/` 资产是真实测量输入，不只是 `SKILL.md` |
 | **证据门控管理** | `omk install` 登记受管记录；`omk eval` 按内容指纹自动写入证据，把 skill 从 `installed` 推到 `measurable`；`omk list` 查看各受管 skill 的状态（installed / measurable / promoted / stale）；`omk promote` 在证据过门禁（默认仅 PROGRESS）后把该版本接受为当前版本；`omk rollback` 撤销这次接受，让 skill 回到 `measurable`。[规范 →](docs/zh/specs/evidence-gated-management.md) |
 | **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段（HF Dataset Cards 风），studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/zh/specs/sample-design-spec.md](docs/zh/specs/sample-design-spec.md) |
-| **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
-| **盲测 A/B** | `--blind` 隐藏变体名称，HTML 报告有揭晓按钮 |
+| **多评委 ensemble** | `--judge-models claude:opus,openai-api:gpt-4o` 跨厂商评分 + agreement 度量 |
 | **多轮方差分析** | `--repeat N` 重复 N 次，计算均值/标准差/置信区间/t 检验 |
 | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容（SSO 保护的知识库等） |
 | **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
@@ -123,6 +126,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 - **[执行器](docs/zh/reference/executors.md)** & **[artifact 布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器；variant 如何解析为 artifact + runtime context
 - **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)（项目 runtime context）与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)（GLM / 通义 / DeepSeek / Moonshot / Ollama）
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
+- **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例，按由简到全排成上手路径
 - **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射
 - **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
 - **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai 等 25+ 维度横评

package/dist/analysis/report-diagnostics.d.ts

@@ -1,7 +1,7 @@
 /**
  * Auto-analysis: detect patterns and generate insights from evaluation results.
  */
-import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Lang } from '../types/index.js';
+import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Representativeness, Lang } from '../types/index.js';
 /** opts for `analyzeResults`. Optional because most older callers don't have
  *  samples in scope; new callers (evaluation-pipeline / evolver) pass them in to
  *  populate `analysis.sampleQuality`. */
@@ -30,4 +30,11 @@ export declare function analyzeResults(report: Report, opts?: AnalyzeResultsOpti
  * participate in grading / judge / verdict. See docs/specs/sample-design-spec.md.
  */
 export declare function buildSampleQualityAggregate(samples: Sample[]): SampleQualityAggregate;
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export declare function buildRepresentativeness(aggregate: SampleQualityAggregate): Representativeness;
 export declare function generateAnalysisSummary(report: Report, lang?: Lang): string | undefined;

package/dist/analysis/report-diagnostics.js

@@ -2,6 +2,7 @@
  * Auto-analysis: detect patterns and generate insights from evaluation results.
  */
 import { normalizeCapability } from './sample-diagnostics.js';
+import { analyzeJudgeIndependence } from '../eval-core/judge-independence.js';
 /**
  * Analyze an evaluation report and produce structured insights.
  */
@@ -15,6 +16,10 @@ export function analyzeResults(report, opts = {}) {
     const sampleQuality = opts.samples
         ? buildSampleQualityAggregate(opts.samples)
         : undefined;
+    // 评委独立性对**单变体**报告同样适用(SOLO 只有绝对分、没有 A/B 差值去抵消自我偏好,
+    // 反而更该报),故放在 variants<2 早退之前;仅 results 为空(dry-run)时不评估。
+    if (results.length > 0)
+        detectJudgeIndependence(report, insights);
     if (results.length === 0 || variants.length < 2) {
         return { insights, ...(sampleQuality && { sampleQuality }) };
     }
@@ -38,11 +43,33 @@ export function analyzeResults(report, opts = {}) {
     detectAgentAssertionDiscrimination(results, variants, insights);
     // 10. Suggest --repeat when score variance is high and no repeat data
     detectNeedRepeat(report, results, variants, insights);
+    // 11. Judge independence — 已在 variants<2 早退前调过(单变体也适用),此处不重复。
     return {
         insights,
         ...(sampleQuality && { sampleQuality }),
     };
 }
+/**
+ * 评委独立性诊断:同厂商评委(自我偏好敞口)/ 单厂商 ensemble(一致性不反驳共有偏置)。
+ * 判定走单一来源 `analyzeJudgeIndependence`;挂了 gold 校准则软化为 info。
+ */
+function detectJudgeIndependence(report, insights) {
+    const ind = analyzeJudgeIndependence(report);
+    if (ind.sameVendorJudge) {
+        insights.push({
+            type: 'judge_self_preference',
+            severity: ind.goldCalibrated ? 'info' : 'warning',
+            details: { judgeVendors: ind.judgeVendors, outputVendors: ind.outputVendors, goldCalibrated: ind.goldCalibrated },
+        });
+    }
+    if (ind.singleVendorEnsemble) {
+        insights.push({
+            type: 'single_vendor_ensemble',
+            severity: ind.goldCalibrated ? 'info' : 'warning',
+            details: { judgeVendors: ind.judgeVendors, goldCalibrated: ind.goldCalibrated },
+        });
+    }
+}
 /**
  * Build sample design science aggregate from sample metadata.
  *
@@ -117,7 +144,7 @@ export function buildSampleQualityAggregate(samples) {
             rubricCount++;
         }
     }
-    return {
+    const aggregate = {
         capabilityCoverage,
         difficultyDistribution,
         constructDistribution,
@@ -128,6 +155,56 @@ export function buildSampleQualityAggregate(samples) {
         sampleCountWithConstruct: withConstruct,
         sampleCountWithProvenance: withProvenance,
     };
+    aggregate.representativeness = buildRepresentativeness(aggregate);
+    return aggregate;
+}
+/** Largest entry of a `label → count` map, as `[label, share]` over a given total.
+ *  Returns `[undefined, 0]` when the map is empty or total ≤ 0. */
+function dominantShare(counts, total) {
+    let label;
+    let max = 0;
+    for (const [k, v] of Object.entries(counts)) {
+        if (v > max) {
+            max = v;
+            label = k;
+        }
+    }
+    return total > 0 ? [label, max / total] : [undefined, 0];
+}
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export function buildRepresentativeness(aggregate) {
+    const capTotal = Object.values(aggregate.capabilityCoverage).reduce((a, b) => a + b, 0);
+    const [dominantCapability, capabilityConcentration] = dominantShare(aggregate.capabilityCoverage, capTotal);
+    // difficulty / construct concentration over *declared* samples only — `unspecified`
+    // is "didn't say", not a bucket the set is skewed toward.
+    const declaredDifficulty = {
+        easy: aggregate.difficultyDistribution.easy,
+        medium: aggregate.difficultyDistribution.medium,
+        hard: aggregate.difficultyDistribution.hard,
+    };
+    const diffTotal = declaredDifficulty.easy + declaredDifficulty.medium + declaredDifficulty.hard;
+    const [dominantDifficulty, difficultyConcentration] = dominantShare(declaredDifficulty, diffTotal);
+    const declaredConstruct = {};
+    for (const [k, v] of Object.entries(aggregate.constructDistribution)) {
+        if (k !== 'unspecified')
+            declaredConstruct[k] = v;
+    }
+    const consTotal = Object.values(declaredConstruct).reduce((a, b) => a + b, 0);
+    const [dominantConstruct, constructConcentration] = dominantShare(declaredConstruct, consTotal);
+    return {
+        capabilityCount: Object.keys(aggregate.capabilityCoverage).length,
+        capabilityConcentration,
+        ...(dominantCapability ? { dominantCapability } : {}),
+        difficultyConcentration,
+        ...(dominantDifficulty ? { dominantDifficulty: dominantDifficulty } : {}),
+        constructConcentration,
+        ...(dominantConstruct ? { dominantConstruct } : {}),
+    };
 }
 export function generateAnalysisSummary(report, lang = 'zh') {
     const variants = report.meta?.variants || [];
@@ -356,12 +433,43 @@ export function generateAnalysisSummary(report, lang = 'zh') {
             ? `【综合洞察】${synthesis.join('；')}。`
             : `【Synthesis】${synthesis.join('; ')}.`);
     }
+    // ── Sample-composition skew ── flags over-representation ("70% are easy") so the
+    // reader knows the set may not be representative of the real task distribution.
+    // Diagnostic only — never gates the verdict. N≥10 guard mirrors capability_thin.
+    const rep = report.analysis?.sampleQuality?.representativeness;
+    const sampleCount = report.meta?.sampleCount ?? 0;
+    if (rep && sampleCount >= 10) {
+        const skews = [];
+        if (rep.difficultyConcentration > SKEW_CONCENTRATION_BAND && rep.dominantDifficulty) {
+            skews.push(lang === 'zh'
+                ? `难度 ${(rep.difficultyConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantDifficulty}`
+                : `${(rep.difficultyConcentration * 100).toFixed(0)}% of declared difficulty is ${rep.dominantDifficulty}`);
+        }
+        if (rep.capabilityConcentration > SKEW_CONCENTRATION_BAND && rep.dominantCapability) {
+            skews.push(lang === 'zh'
+                ? `能力标签 ${(rep.capabilityConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantCapability}`
+                : `${(rep.capabilityConcentration * 100).toFixed(0)}% of capability tags are ${rep.dominantCapability}`);
+        }
+        if (rep.constructConcentration > SKEW_CONCENTRATION_BAND && rep.dominantConstruct) {
+            skews.push(lang === 'zh'
+                ? `construct ${(rep.constructConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantConstruct}`
+                : `${(rep.constructConcentration * 100).toFixed(0)}% of declared construct is ${rep.dominantConstruct}`);
+        }
+        if (skews.length > 0) {
+            lines.push(lang === 'zh'
+                ? `【用例构成】偏斜：${skews.join('；')}——补充其它维度可提升代表性（构成提示，不影响 verdict）。`
+                : `【Sample composition】Skewed: ${skews.join('; ')} — adding other dimensions improves representativeness (informational; does not affect the verdict).`);
+        }
+    }
     // Caveats and recommendations are handled by the issues table below,
     // so the summary focuses only on verdict + differentiators + synthesis.
     if (lines.length === 0)
         return undefined;
     return lines.join('\n');
 }
+/** Dominant-bucket share above which the sample set is flagged as skewed. Pragmatic
+ *  default — 60% of declared samples in one bucket is a clear over-representation. */
+const SKEW_CONCENTRATION_BAND = 0.6;
 const AGENT_ASSERTION_TYPES = new Set([
     'tools_called',
     'tools_not_called',

package/dist/assets/agent-skills/omk/references/commands.md

@@ -67,7 +67,6 @@ omk eval [flags]
 **Flags:**
 
 - `--batch` `boolean`:batch 模式:baseline vs 每个 skill
-- `--blind` `boolean`:judge blind 模式
 - `--bootstrap` `boolean`:加 bootstrap CI
 - `--bootstrap-samples` `option`:bootstrap 重采样次数，默认 1000
 - `--budget-per-sample-ms` `option`:单 sample 时长上限 ms（必须 > 0，不传则无上限）
@@ -82,7 +81,8 @@ omk eval [flags]
 - `--executor` `option`:执行器:claude / claude-sdk / codex / codex-sdk / openai-api / gemini / 自定义命令（默认 claude）。
 - `--global` `boolean`:报告写全局 ~/.oh-my-knowledge/reports，而非项目 .omk/
 - `--gold-dir` `option`:gold dataset 目录
-- `--judge-models` `option`:评委配置，格式 executor:model[,...]，例 claude:haiku 或 claude:opus,openai:gpt-4o(≥ 2 个 = ensemble）。默认 <executor>:haiku。
+- `--holdout-ratio` `option`:留出比例 0-1（如 0.3）；切出 holdout 子集，对比 train/holdout 综合分检测过拟合
+- `--judge-models` `option`:评委配置，格式 executor:model[,...]，例 claude:haiku 或 claude:opus,openai-api:gpt-4o(≥ 2 个 = ensemble）。默认 <executor>:haiku。
 - `--judge-repeat` `option`:每个 dim 评 N 次
 - `--lang` `option` (默认 `zh`):输出语言 zh|en，优先级 CLI > OMK_LANG env > zh。
 - `--layered-stats` `boolean`:输出分层统计

package/dist/authoring/evolver.d.ts

@@ -16,14 +16,11 @@ interface WeakSample {
     };
 }
 export declare function extractWeakSamples(report: Report, variantKey: string, count?: number, sampleIdFilter?: Set<string>): WeakSample[];
-/** A train / holdout partition of a sample set. */
-interface HoldoutSplit {
-    trainIds: Set<string>;
-    holdoutIds: Set<string>;
-}
 /** A train / val / test partition. `val` drives the accept decision; `test` is
  *  locked — never seen during the loop, read once at the end for an unbiased
- *  generalization score. */
+ *  generalization score. Two-way holdout (`splitHoldout`), the stride picker
+ *  (`pickByStride`), `MIN_HOLDOUT_SUBSET`, and `subsetCompositeScore` live in
+ *  `src/eval-core/holdout.ts` so `omk eval --holdout-ratio` reuses them. */
 interface TrainValTestSplit {
     trainIds: Set<string>;
     valIds: Set<string>;
@@ -34,14 +31,6 @@ interface TrainValTestSplit {
  *  every candidate. Under that floor evolve degrades to the point-estimate accept
  *  and flags `gate.underpowered`. */
 export declare const MIN_GATE_SAMPLES = 8;
-/**
- * Deterministically split sample ids into train / holdout by `ratio` (fraction
- * held out). Holdout members are picked at an even stride so the partition is
- * representative of the ordering, and the split is stable across rounds and runs
- * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
- * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
- */
-export declare function splitHoldout(sampleIds: string[], ratio: number): HoldoutSplit | null;
 /**
  * Deterministically split sample ids into train / val / test. `val` is carved
  * first at an even stride; `test` is carved at an even stride over what remains,

package/dist/authoring/evolver.js

@@ -8,7 +8,7 @@ import { projectReportsDir, globalReportsDir } from '../eval-core/measurement-di
 import { analyzeResults } from '../analysis/report-diagnostics.js';
 import { loadSamples } from '../inputs/load-samples.js';
 import { hashArtifactSource } from '../inputs/content-hash.js';
-import { buildVariantSummary } from '../eval-core/schema.js';
+import { MIN_HOLDOUT_SUBSET, pickByStride, splitHoldout, subsetCompositeScore } from '../eval-core/holdout.js';
 import { bootstrapDiffCI, DEFAULT_BOOTSTRAP_ALPHA, DEFAULT_BOOTSTRAP_SAMPLES } from '../eval-core/bootstrap.js';
 import { fixSamples } from './sample-fixer.js';
 const IMPROVE_SYSTEM_PROMPT = `你是一个 AI 提示词改进专家。你的任务是分析评测结果中的薄弱环节，针对性地改进 skill（系统提示词），使其在评测中获得更高的分数。
@@ -165,43 +165,11 @@ export function extractWeakSamples(report, variantKey, count = 5, sampleIdFilter
         .sort((a, b) => a.compositeScore - b.compositeScore)
         .slice(0, count);
 }
-/** Below this many samples on any side, a split is too small to be meaningful —
- *  evolve falls back to full-set scoring and warns. */
-const MIN_HOLDOUT_SUBSET = 3;
 /** Below this many decision (val) samples the bootstrap diff CI almost never
  *  excludes 0 for realistic effect sizes, so the significance gate would reject
  *  every candidate. Under that floor evolve degrades to the point-estimate accept
  *  and flags `gate.underpowered`. */
 export const MIN_GATE_SAMPLES = 8;
-/** Pick `count` ids at an even stride across `ids` (deterministic, no RNG) so the
- *  picked subset is representative of the ordering and stable across rounds/runs. */
-function pickByStride(ids, count) {
-    const picked = new Set();
-    if (count <= 0)
-        return picked;
-    const stride = ids.length / count;
-    for (let k = 0; k < count; k++)
-        picked.add(ids[Math.floor(k * stride)]);
-    return picked;
-}
-/**
- * Deterministically split sample ids into train / holdout by `ratio` (fraction
- * held out). Holdout members are picked at an even stride so the partition is
- * representative of the ordering, and the split is stable across rounds and runs
- * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
- * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
- */
-export function splitHoldout(sampleIds, ratio) {
-    if (!(ratio > 0) || sampleIds.length === 0)
-        return null;
-    const holdoutCount = Math.round(sampleIds.length * ratio);
-    const trainCount = sampleIds.length - holdoutCount;
-    if (holdoutCount < MIN_HOLDOUT_SUBSET || trainCount < MIN_HOLDOUT_SUBSET)
-        return null;
-    const holdoutIds = pickByStride(sampleIds, holdoutCount);
-    const trainIds = new Set(sampleIds.filter((id) => !holdoutIds.has(id)));
-    return { trainIds, holdoutIds };
-}
 /**
  * Deterministically split sample ids into train / val / test. `val` is carved
  * first at an even stride; `test` is carved at an even stride over what remains,
@@ -223,25 +191,6 @@ export function splitTrainValTest(sampleIds, valRatio, testRatio) {
     const trainIds = new Set(sampleIds.filter((id) => !valIds.has(id) && !testIds.has(id)));
     return { trainIds, valIds, testIds };
 }
-/**
- * Mean composite over the subset of a report's results whose sample_id is in
- * `ids`, using the same aggregation as the full-run summary
- * (`buildVariantSummary`) so train / holdout scores stay comparable to the
- * headline composite. Returns 0 when the subset has no scorable entries.
- */
-function subsetCompositeScore(report, variantKey, ids) {
-    const entries = [];
-    for (const r of report.results) {
-        if (!ids.has(r.sample_id))
-            continue;
-        const v = r.variants[variantKey];
-        if (v)
-            entries.push(v);
-    }
-    if (entries.length === 0)
-        return 0;
-    return buildVariantSummary(entries).avgCompositeScore ?? 0;
-}
 /**
  * Per-sample composite scores over the subset of a report's results whose
  * sample_id is in `ids`, in result order. Feeds `bootstrapDiffCI` for the

package/dist/authoring/generator.d.ts

@@ -31,6 +31,30 @@ export declare function generateSamples({ skillContent, count, model, executorNa
     costUSD: number;
 }>;
 type TraceSignalItem = Pick<ObservationInboxItem, 'skillName' | 'signalType' | 'signalSubtype' | 'severity' | 'evidence' | 'messageWindow' | 'occurrences'>;
+export interface StratifiedTraceSignal extends TraceSignalItem {
+    /** Share of total occurrences across all signals (0-1) — drives proportional
+     *  sample allocation in the prompt. */
+    weight: number;
+}
+/**
+ * Rank trace signals by frequency and annotate each with its share of the total
+ * occurrences. Lets `omk sample --from-traces` allocate samples *proportional to
+ * how often a failure actually happened* instead of a flat "1-2 per signal" — a
+ * failure seen 100× deserves more regression coverage than one seen twice.
+ *
+ * Deliberately does NOT re-merge signals here. The observation inbox is the only
+ * source of these items and already aggregates `occurrences` by the FULL identity
+ * — `skillName + cwd + sourceKind + signalType + signalSubtype + evidence`
+ * (`inbox.ts` `keyFor`). Re-merging on a narrower key (e.g. type+subtype+evidence)
+ * would fold *different skills / cwd* with the same failure shape into one entry,
+ * mis-attributing the summed occurrences to the first skill and skewing the
+ * regenerated distribution. So we trust the upstream dedup and only sort + weight.
+ *
+ * Does NOT fix the underlying selection bias (traces only capture *failures*),
+ * which is why the `omk sample --from-traces` draft warning still stands — this
+ * only makes the within-failure distribution representative of frequency.
+ */
+export declare function stratifyTraceSignals(items: TraceSignalItem[]): StratifiedTraceSignal[];
 /**
  * Build the generation prompt for `omk sample --from-traces`. Renders each
  * observation-inbox signal (evidence + message window) into a section and asks

package/dist/authoring/generator.js

@@ -1,4 +1,5 @@
 import { createExecutor } from '../executors/index.js';
+import { DEFAULT_GATE_THRESHOLD } from '../eval-core/verdict.js';
 /**
  * Generator 默认模型 'opus' (跟 eval 默认对齐)。
  * lean=true 路径会自动追加 `--effort low`,关掉 opus 默认的扩展思考,
@@ -76,7 +77,7 @@ const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据
   来涨数量)。omk 的评分体系是 layered scoring: **fact 层**(deterministic 字面/工具断言)
   + **behavior 层**(代价指标如 turn 数 / 工具失败率) + **judge 层**(主观语义评分,从
   sample.rubric 派生维度,judge LLM 看 trace 评 1-5)三层独立计分,verdict 是三层
-  独立过 threshold(默认 3.5)。**fact 层的本职是测 deterministic 端点,不是测轨迹**。
+  独立过 threshold(默认 ${DEFAULT_GATE_THRESHOLD})。**fact 层的本职是测 deterministic 端点,不是测轨迹**。
 
   **断言哲学(关键):fact 测结果+里程碑,过程质量交 judge**
   ─────────────────────────────────────────────────────────────
@@ -109,7 +110,7 @@ const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据
     - tools_not_called 反模式断言 0-1 条(禁止接触某禁忌工具,如 tripwire sample)
     - rubric 3-5 个判分维度(细致写明 judge 该看什么),由 sample.rubric 字段承载
 
-  *测量学背景:* 当前 omk verdict 三层独立 threshold(默认 3.5),fact 条目少之后单条
+  *测量学背景:* 当前 omk verdict 三层独立 threshold(默认 ${DEFAULT_GATE_THRESHOLD}),fact 条目少之后单条
   权重大、单次评测方差大,**强烈建议** 评测时带 \`--repeat 2\` 或更大测稳定性(coefficient
   of variation),并参考 bootstrap CI 而非点估计。这是 fact 层稀疏化的代价,换来的是
   fact 信号干净(不被 trajectory 字面噪音污染)。
@@ -464,10 +465,34 @@ async function finalizeSamples(samples, costUSD, skillContent) {
 const TRACE_GEN_INSTRUCTIONS = `下面给出的不是 skill，而是从生产会话 trace 中观测到的失败 / 异常信号。请为这些信号生成评测用例（eval samples），使评测能复现并守住这些失败模式——把线上真实发生过的问题沉淀成回归用例。
 
 要求：
-- 每个信号生成 1-2 条聚焦回归用例；信号若是噪声 / 证据不足 / 无法复现，跳过它，不要硬凑。
+- 按各信号标注的「占比」分配用例数：高频信号多生成、低频少生成，让用例集覆盖线上失败的真实频次分布（高占比信号 2-3 条，低占比 1 条即可）；信号若是噪声 / 证据不足 / 无法复现，跳过它，不要硬凑。
 - prompt 要还原触发该信号的场景（自然语言任务），不要直接复述证据文本。
 - 断言优先用 mock_hit / tools_called / tools_not_called / tool_input_contains 精确锚定失败步骤，再用 contains 兜底；按「原子型」配比处理（无需工作流编号步骤），除非证据明显是多步流程。
 - 不要在输出里说明判断过程，直接输出 JSON 数组。`;
+/**
+ * Rank trace signals by frequency and annotate each with its share of the total
+ * occurrences. Lets `omk sample --from-traces` allocate samples *proportional to
+ * how often a failure actually happened* instead of a flat "1-2 per signal" — a
+ * failure seen 100× deserves more regression coverage than one seen twice.
+ *
+ * Deliberately does NOT re-merge signals here. The observation inbox is the only
+ * source of these items and already aggregates `occurrences` by the FULL identity
+ * — `skillName + cwd + sourceKind + signalType + signalSubtype + evidence`
+ * (`inbox.ts` `keyFor`). Re-merging on a narrower key (e.g. type+subtype+evidence)
+ * would fold *different skills / cwd* with the same failure shape into one entry,
+ * mis-attributing the summed occurrences to the first skill and skewing the
+ * regenerated distribution. So we trust the upstream dedup and only sort + weight.
+ *
+ * Does NOT fix the underlying selection bias (traces only capture *failures*),
+ * which is why the `omk sample --from-traces` draft warning still stands — this
+ * only makes the within-failure distribution representative of frequency.
+ */
+export function stratifyTraceSignals(items) {
+    const total = items.reduce((sum, it) => sum + (it.occurrences ?? 0), 0) || 1;
+    return items
+        .map((it) => ({ ...it, occurrences: it.occurrences ?? 0, weight: (it.occurrences ?? 0) / total }))
+        .sort((a, b) => b.occurrences - a.occurrences);
+}
 /**
  * Build the generation prompt for `omk sample --from-traces`. Renders each
  * observation-inbox signal (evidence + message window) into a section and asks
@@ -476,7 +501,9 @@ const TRACE_GEN_INSTRUCTIONS = `下面给出的不是 skill，而是从生产会
  * judge prompt — so judge-prompt isolation is unaffected.
  */
 export function buildSamplesFromTracesPrompt(items, count) {
-    const sections = items.map((it, i) => {
+    // 先按频次分层（合并重复 + 算占比 + 降序），让模型按「占比」分配配额，而非每信号一刀切。
+    const stratified = stratifyTraceSignals(items);
+    const sections = stratified.map((it, i) => {
         const ev = it.evidence ?? {};
         const evLines = [
             ev.tool && `工具: ${ev.tool}`,
@@ -490,14 +517,15 @@ export function buildSamplesFromTracesPrompt(items, count) {
             ? '\n上下文消息:\n' + [...it.messageWindow.before, ...it.messageWindow.event, ...it.messageWindow.after]
                 .map((m) => `  [${m.role}] ${m.snippet}`).join('\n')
             : '';
-        return `### 信号 ${i + 1}：${it.signalType} / ${it.signalSubtype}（严重度 ${it.severity}，出现 ${it.occurrences} 次，skill: ${it.skillName}）\n${evLines}${win}`;
+        const pct = (it.weight * 100).toFixed(0);
+        return `### 信号 ${i + 1}：${it.signalType} / ${it.signalSubtype}（严重度 ${it.severity}，出现 ${it.occurrences} 次 · 占比 ${pct}%，skill: ${it.skillName}）\n${evLines}${win}`;
     }).join('\n\n---\n\n');
     const countLine = typeof count === 'number'
-        ? `共生成约 ${count} 条评测用例。`
-        : '按上面的「每个信号 1-2 条」生成。';
+        ? `共生成约 ${count} 条评测用例，按各信号的「占比」分配配额（高频多、低频少），覆盖整体失败分布。`
+        : '按各信号「占比」分配：高频信号多生成、低频少生成，覆盖整体失败分布。';
     return `${TRACE_GEN_INSTRUCTIONS}
 
-## 观测到的失败信号（共 ${items.length} 个）
+## 观测到的失败信号（共 ${stratified.length} 个，已按出现频次降序）
 
 ${sections}
 

package/dist/cli/commands/eval/index.d.ts

@@ -37,8 +37,8 @@ export default class Eval extends BaseCommand {
         'no-strict-baseline': import("@oclif/core/interfaces").BooleanFlag<boolean>;
         effort: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         'no-diagnostic': import("@oclif/core/interfaces").BooleanFlag<boolean>;
-        blind: import("@oclif/core/interfaces").BooleanFlag<boolean>;
         repeat: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'holdout-ratio': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         'judge-repeat': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         bootstrap: import("@oclif/core/interfaces").BooleanFlag<boolean>;
         'bootstrap-samples': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;