oh-my-knowledge 0.41.0 → 0.42.0

package/README.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
+Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
+
+Prerequisite: the default executor and judge use the `claude` CLI — install and log in first (see [Requirements](#requirements)); to use another model or run offline (no API key) see [executors](docs/reference/executors.md).
+
+> The first run has only 3 cases, so the verdict will usually be `UNDERPOWERED` (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.
 
 > The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
 
-Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
+Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users). More runnable examples (A/B, offline executor, batch, evolve, agent, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
 
 Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
@@ -122,6 +126,7 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
 - **[Executors](docs/reference/executors.md)** & **[artifact layout](docs/reference/artifact-layout.md)** — built-in / custom executors; how `variant` resolves to an artifact + runtime context
 - **[How-to guides](docs/guides/agent-eval.md)** — [evaluate an agent](docs/guides/agent-eval.md) (project runtime context) and [use non-Claude models](docs/guides/non-claude-models.md) (GLM / Qwen / DeepSeek / Moonshot / Ollama)
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
+- **[Example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** — a set of runnable examples in the repo, arranged simplest-to-richest
 - **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
 - **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
 - **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai

package/README.zh.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例；`omk eval` 跑控制变量 A/B，5 分钟内出 HTML 报告 + 一行 verdict。
+开箱即跑：`omk init` 脚手架好两版 skill 和三条评测用例，不用先改任何文件，`omk eval` 跑控制变量 A/B，约 5 分钟出 HTML 报告 + 一行 verdict；跑通后再把 skill 和用例换成你自己的。
+
+前置：默认执行器与评委用 `claude` CLI，需先安装并登录（见[系统要求](#系统要求)）；想用别的模型或离线跑（无需 API key）见[执行器](docs/zh/reference/executors.md)。
+
+> 首跑只有 3 条用例，verdict 多半是「数据不足（UNDERPOWERED）」——这是正常起点而非出错；把用例加到约 20 条以上，再看「可发布」结论。
 
 > 命令行有新版本时会自动提示（每 20 小时最多一次）；想永久关闭该提醒，设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
 
-手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。
+手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。更多可跑示例（A/B、离线执行器、batch、evolve、agent、RAG）见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
 
 深入：[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
 
@@ -122,6 +126,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 - **[执行器](docs/zh/reference/executors.md)** & **[artifact 布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器；variant 如何解析为 artifact + runtime context
 - **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)（项目 runtime context）与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)（GLM / 通义 / DeepSeek / Moonshot / Ollama）
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
+- **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例，按由简到全排成上手路径
 - **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射
 - **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
 - **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai 等 25+ 维度横评

package/dist/analysis/report-diagnostics.d.ts

@@ -1,7 +1,7 @@
 /**
  * Auto-analysis: detect patterns and generate insights from evaluation results.
  */
-import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Lang } from '../types/index.js';
+import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Representativeness, Lang } from '../types/index.js';
 /** opts for `analyzeResults`. Optional because most older callers don't have
  *  samples in scope; new callers (evaluation-pipeline / evolver) pass them in to
  *  populate `analysis.sampleQuality`. */
@@ -30,4 +30,11 @@ export declare function analyzeResults(report: Report, opts?: AnalyzeResultsOpti
  * participate in grading / judge / verdict. See docs/specs/sample-design-spec.md.
  */
 export declare function buildSampleQualityAggregate(samples: Sample[]): SampleQualityAggregate;
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export declare function buildRepresentativeness(aggregate: SampleQualityAggregate): Representativeness;
 export declare function generateAnalysisSummary(report: Report, lang?: Lang): string | undefined;

package/dist/analysis/report-diagnostics.js

@@ -144,7 +144,7 @@ export function buildSampleQualityAggregate(samples) {
             rubricCount++;
         }
     }
-    return {
+    const aggregate = {
         capabilityCoverage,
         difficultyDistribution,
         constructDistribution,
@@ -155,6 +155,56 @@ export function buildSampleQualityAggregate(samples) {
         sampleCountWithConstruct: withConstruct,
         sampleCountWithProvenance: withProvenance,
     };
+    aggregate.representativeness = buildRepresentativeness(aggregate);
+    return aggregate;
+}
+/** Largest entry of a `label → count` map, as `[label, share]` over a given total.
+ *  Returns `[undefined, 0]` when the map is empty or total ≤ 0. */
+function dominantShare(counts, total) {
+    let label;
+    let max = 0;
+    for (const [k, v] of Object.entries(counts)) {
+        if (v > max) {
+            max = v;
+            label = k;
+        }
+    }
+    return total > 0 ? [label, max / total] : [undefined, 0];
+}
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export function buildRepresentativeness(aggregate) {
+    const capTotal = Object.values(aggregate.capabilityCoverage).reduce((a, b) => a + b, 0);
+    const [dominantCapability, capabilityConcentration] = dominantShare(aggregate.capabilityCoverage, capTotal);
+    // difficulty / construct concentration over *declared* samples only — `unspecified`
+    // is "didn't say", not a bucket the set is skewed toward.
+    const declaredDifficulty = {
+        easy: aggregate.difficultyDistribution.easy,
+        medium: aggregate.difficultyDistribution.medium,
+        hard: aggregate.difficultyDistribution.hard,
+    };
+    const diffTotal = declaredDifficulty.easy + declaredDifficulty.medium + declaredDifficulty.hard;
+    const [dominantDifficulty, difficultyConcentration] = dominantShare(declaredDifficulty, diffTotal);
+    const declaredConstruct = {};
+    for (const [k, v] of Object.entries(aggregate.constructDistribution)) {
+        if (k !== 'unspecified')
+            declaredConstruct[k] = v;
+    }
+    const consTotal = Object.values(declaredConstruct).reduce((a, b) => a + b, 0);
+    const [dominantConstruct, constructConcentration] = dominantShare(declaredConstruct, consTotal);
+    return {
+        capabilityCount: Object.keys(aggregate.capabilityCoverage).length,
+        capabilityConcentration,
+        ...(dominantCapability ? { dominantCapability } : {}),
+        difficultyConcentration,
+        ...(dominantDifficulty ? { dominantDifficulty: dominantDifficulty } : {}),
+        constructConcentration,
+        ...(dominantConstruct ? { dominantConstruct } : {}),
+    };
 }
 export function generateAnalysisSummary(report, lang = 'zh') {
     const variants = report.meta?.variants || [];
@@ -383,12 +433,43 @@ export function generateAnalysisSummary(report, lang = 'zh') {
             ? `【综合洞察】${synthesis.join('；')}。`
             : `【Synthesis】${synthesis.join('; ')}.`);
     }
+    // ── Sample-composition skew ── flags over-representation ("70% are easy") so the
+    // reader knows the set may not be representative of the real task distribution.
+    // Diagnostic only — never gates the verdict. N≥10 guard mirrors capability_thin.
+    const rep = report.analysis?.sampleQuality?.representativeness;
+    const sampleCount = report.meta?.sampleCount ?? 0;
+    if (rep && sampleCount >= 10) {
+        const skews = [];
+        if (rep.difficultyConcentration > SKEW_CONCENTRATION_BAND && rep.dominantDifficulty) {
+            skews.push(lang === 'zh'
+                ? `难度 ${(rep.difficultyConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantDifficulty}`
+                : `${(rep.difficultyConcentration * 100).toFixed(0)}% of declared difficulty is ${rep.dominantDifficulty}`);
+        }
+        if (rep.capabilityConcentration > SKEW_CONCENTRATION_BAND && rep.dominantCapability) {
+            skews.push(lang === 'zh'
+                ? `能力标签 ${(rep.capabilityConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantCapability}`
+                : `${(rep.capabilityConcentration * 100).toFixed(0)}% of capability tags are ${rep.dominantCapability}`);
+        }
+        if (rep.constructConcentration > SKEW_CONCENTRATION_BAND && rep.dominantConstruct) {
+            skews.push(lang === 'zh'
+                ? `construct ${(rep.constructConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantConstruct}`
+                : `${(rep.constructConcentration * 100).toFixed(0)}% of declared construct is ${rep.dominantConstruct}`);
+        }
+        if (skews.length > 0) {
+            lines.push(lang === 'zh'
+                ? `【用例构成】偏斜：${skews.join('；')}——补充其它维度可提升代表性（构成提示，不影响 verdict）。`
+                : `【Sample composition】Skewed: ${skews.join('; ')} — adding other dimensions improves representativeness (informational; does not affect the verdict).`);
+        }
+    }
     // Caveats and recommendations are handled by the issues table below,
     // so the summary focuses only on verdict + differentiators + synthesis.
     if (lines.length === 0)
         return undefined;
     return lines.join('\n');
 }
+/** Dominant-bucket share above which the sample set is flagged as skewed. Pragmatic
+ *  default — 60% of declared samples in one bucket is a clear over-representation. */
+const SKEW_CONCENTRATION_BAND = 0.6;
 const AGENT_ASSERTION_TYPES = new Set([
     'tools_called',
     'tools_not_called',

package/dist/assets/agent-skills/omk/references/commands.md

@@ -81,6 +81,7 @@ omk eval [flags]
 - `--executor` `option`:执行器:claude / claude-sdk / codex / codex-sdk / openai-api / gemini / 自定义命令（默认 claude）。
 - `--global` `boolean`:报告写全局 ~/.oh-my-knowledge/reports，而非项目 .omk/
 - `--gold-dir` `option`:gold dataset 目录
+- `--holdout-ratio` `option`:留出比例 0-1（如 0.3）；切出 holdout 子集，对比 train/holdout 综合分检测过拟合
 - `--judge-models` `option`:评委配置，格式 executor:model[,...]，例 claude:haiku 或 claude:opus,openai-api:gpt-4o(≥ 2 个 = ensemble）。默认 <executor>:haiku。
 - `--judge-repeat` `option`:每个 dim 评 N 次
 - `--lang` `option` (默认 `zh`):输出语言 zh|en，优先级 CLI > OMK_LANG env > zh。

package/dist/authoring/evolver.d.ts

@@ -16,14 +16,11 @@ interface WeakSample {
     };
 }
 export declare function extractWeakSamples(report: Report, variantKey: string, count?: number, sampleIdFilter?: Set<string>): WeakSample[];
-/** A train / holdout partition of a sample set. */
-interface HoldoutSplit {
-    trainIds: Set<string>;
-    holdoutIds: Set<string>;
-}
 /** A train / val / test partition. `val` drives the accept decision; `test` is
  *  locked — never seen during the loop, read once at the end for an unbiased
- *  generalization score. */
+ *  generalization score. Two-way holdout (`splitHoldout`), the stride picker
+ *  (`pickByStride`), `MIN_HOLDOUT_SUBSET`, and `subsetCompositeScore` live in
+ *  `src/eval-core/holdout.ts` so `omk eval --holdout-ratio` reuses them. */
 interface TrainValTestSplit {
     trainIds: Set<string>;
     valIds: Set<string>;
@@ -34,14 +31,6 @@ interface TrainValTestSplit {
  *  every candidate. Under that floor evolve degrades to the point-estimate accept
  *  and flags `gate.underpowered`. */
 export declare const MIN_GATE_SAMPLES = 8;
-/**
- * Deterministically split sample ids into train / holdout by `ratio` (fraction
- * held out). Holdout members are picked at an even stride so the partition is
- * representative of the ordering, and the split is stable across rounds and runs
- * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
- * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
- */
-export declare function splitHoldout(sampleIds: string[], ratio: number): HoldoutSplit | null;
 /**
  * Deterministically split sample ids into train / val / test. `val` is carved
  * first at an even stride; `test` is carved at an even stride over what remains,

package/dist/authoring/evolver.js

@@ -8,7 +8,7 @@ import { projectReportsDir, globalReportsDir } from '../eval-core/measurement-di
 import { analyzeResults } from '../analysis/report-diagnostics.js';
 import { loadSamples } from '../inputs/load-samples.js';
 import { hashArtifactSource } from '../inputs/content-hash.js';
-import { buildVariantSummary } from '../eval-core/schema.js';
+import { MIN_HOLDOUT_SUBSET, pickByStride, splitHoldout, subsetCompositeScore } from '../eval-core/holdout.js';
 import { bootstrapDiffCI, DEFAULT_BOOTSTRAP_ALPHA, DEFAULT_BOOTSTRAP_SAMPLES } from '../eval-core/bootstrap.js';
 import { fixSamples } from './sample-fixer.js';
 const IMPROVE_SYSTEM_PROMPT = `你是一个 AI 提示词改进专家。你的任务是分析评测结果中的薄弱环节，针对性地改进 skill（系统提示词），使其在评测中获得更高的分数。
@@ -165,43 +165,11 @@ export function extractWeakSamples(report, variantKey, count = 5, sampleIdFilter
         .sort((a, b) => a.compositeScore - b.compositeScore)
         .slice(0, count);
 }
-/** Below this many samples on any side, a split is too small to be meaningful —
- *  evolve falls back to full-set scoring and warns. */
-const MIN_HOLDOUT_SUBSET = 3;
 /** Below this many decision (val) samples the bootstrap diff CI almost never
  *  excludes 0 for realistic effect sizes, so the significance gate would reject
  *  every candidate. Under that floor evolve degrades to the point-estimate accept
  *  and flags `gate.underpowered`. */
 export const MIN_GATE_SAMPLES = 8;
-/** Pick `count` ids at an even stride across `ids` (deterministic, no RNG) so the
- *  picked subset is representative of the ordering and stable across rounds/runs. */
-function pickByStride(ids, count) {
-    const picked = new Set();
-    if (count <= 0)
-        return picked;
-    const stride = ids.length / count;
-    for (let k = 0; k < count; k++)
-        picked.add(ids[Math.floor(k * stride)]);
-    return picked;
-}
-/**
- * Deterministically split sample ids into train / holdout by `ratio` (fraction
- * held out). Holdout members are picked at an even stride so the partition is
- * representative of the ordering, and the split is stable across rounds and runs
- * (no RNG). Returns null when ratio ≤ 0 or either side would drop below
- * MIN_HOLDOUT_SUBSET — the caller then scores on the full set.
- */
-export function splitHoldout(sampleIds, ratio) {
-    if (!(ratio > 0) || sampleIds.length === 0)
-        return null;
-    const holdoutCount = Math.round(sampleIds.length * ratio);
-    const trainCount = sampleIds.length - holdoutCount;
-    if (holdoutCount < MIN_HOLDOUT_SUBSET || trainCount < MIN_HOLDOUT_SUBSET)
-        return null;
-    const holdoutIds = pickByStride(sampleIds, holdoutCount);
-    const trainIds = new Set(sampleIds.filter((id) => !holdoutIds.has(id)));
-    return { trainIds, holdoutIds };
-}
 /**
  * Deterministically split sample ids into train / val / test. `val` is carved
  * first at an even stride; `test` is carved at an even stride over what remains,
@@ -223,25 +191,6 @@ export function splitTrainValTest(sampleIds, valRatio, testRatio) {
     const trainIds = new Set(sampleIds.filter((id) => !valIds.has(id) && !testIds.has(id)));
     return { trainIds, valIds, testIds };
 }
-/**
- * Mean composite over the subset of a report's results whose sample_id is in
- * `ids`, using the same aggregation as the full-run summary
- * (`buildVariantSummary`) so train / holdout scores stay comparable to the
- * headline composite. Returns 0 when the subset has no scorable entries.
- */
-function subsetCompositeScore(report, variantKey, ids) {
-    const entries = [];
-    for (const r of report.results) {
-        if (!ids.has(r.sample_id))
-            continue;
-        const v = r.variants[variantKey];
-        if (v)
-            entries.push(v);
-    }
-    if (entries.length === 0)
-        return 0;
-    return buildVariantSummary(entries).avgCompositeScore ?? 0;
-}
 /**
  * Per-sample composite scores over the subset of a report's results whose
  * sample_id is in `ids`, in result order. Feeds `bootstrapDiffCI` for the

package/dist/authoring/generator.d.ts

@@ -31,6 +31,30 @@ export declare function generateSamples({ skillContent, count, model, executorNa
     costUSD: number;
 }>;
 type TraceSignalItem = Pick<ObservationInboxItem, 'skillName' | 'signalType' | 'signalSubtype' | 'severity' | 'evidence' | 'messageWindow' | 'occurrences'>;
+export interface StratifiedTraceSignal extends TraceSignalItem {
+    /** Share of total occurrences across all signals (0-1) — drives proportional
+     *  sample allocation in the prompt. */
+    weight: number;
+}
+/**
+ * Rank trace signals by frequency and annotate each with its share of the total
+ * occurrences. Lets `omk sample --from-traces` allocate samples *proportional to
+ * how often a failure actually happened* instead of a flat "1-2 per signal" — a
+ * failure seen 100× deserves more regression coverage than one seen twice.
+ *
+ * Deliberately does NOT re-merge signals here. The observation inbox is the only
+ * source of these items and already aggregates `occurrences` by the FULL identity
+ * — `skillName + cwd + sourceKind + signalType + signalSubtype + evidence`
+ * (`inbox.ts` `keyFor`). Re-merging on a narrower key (e.g. type+subtype+evidence)
+ * would fold *different skills / cwd* with the same failure shape into one entry,
+ * mis-attributing the summed occurrences to the first skill and skewing the
+ * regenerated distribution. So we trust the upstream dedup and only sort + weight.
+ *
+ * Does NOT fix the underlying selection bias (traces only capture *failures*),
+ * which is why the `omk sample --from-traces` draft warning still stands — this
+ * only makes the within-failure distribution representative of frequency.
+ */
+export declare function stratifyTraceSignals(items: TraceSignalItem[]): StratifiedTraceSignal[];
 /**
  * Build the generation prompt for `omk sample --from-traces`. Renders each
  * observation-inbox signal (evidence + message window) into a section and asks

package/dist/authoring/generator.js

@@ -465,10 +465,34 @@ async function finalizeSamples(samples, costUSD, skillContent) {
 const TRACE_GEN_INSTRUCTIONS = `下面给出的不是 skill，而是从生产会话 trace 中观测到的失败 / 异常信号。请为这些信号生成评测用例（eval samples），使评测能复现并守住这些失败模式——把线上真实发生过的问题沉淀成回归用例。
 
 要求：
-- 每个信号生成 1-2 条聚焦回归用例；信号若是噪声 / 证据不足 / 无法复现，跳过它，不要硬凑。
+- 按各信号标注的「占比」分配用例数：高频信号多生成、低频少生成，让用例集覆盖线上失败的真实频次分布（高占比信号 2-3 条，低占比 1 条即可）；信号若是噪声 / 证据不足 / 无法复现，跳过它，不要硬凑。
 - prompt 要还原触发该信号的场景（自然语言任务），不要直接复述证据文本。
 - 断言优先用 mock_hit / tools_called / tools_not_called / tool_input_contains 精确锚定失败步骤，再用 contains 兜底；按「原子型」配比处理（无需工作流编号步骤），除非证据明显是多步流程。
 - 不要在输出里说明判断过程，直接输出 JSON 数组。`;
+/**
+ * Rank trace signals by frequency and annotate each with its share of the total
+ * occurrences. Lets `omk sample --from-traces` allocate samples *proportional to
+ * how often a failure actually happened* instead of a flat "1-2 per signal" — a
+ * failure seen 100× deserves more regression coverage than one seen twice.
+ *
+ * Deliberately does NOT re-merge signals here. The observation inbox is the only
+ * source of these items and already aggregates `occurrences` by the FULL identity
+ * — `skillName + cwd + sourceKind + signalType + signalSubtype + evidence`
+ * (`inbox.ts` `keyFor`). Re-merging on a narrower key (e.g. type+subtype+evidence)
+ * would fold *different skills / cwd* with the same failure shape into one entry,
+ * mis-attributing the summed occurrences to the first skill and skewing the
+ * regenerated distribution. So we trust the upstream dedup and only sort + weight.
+ *
+ * Does NOT fix the underlying selection bias (traces only capture *failures*),
+ * which is why the `omk sample --from-traces` draft warning still stands — this
+ * only makes the within-failure distribution representative of frequency.
+ */
+export function stratifyTraceSignals(items) {
+    const total = items.reduce((sum, it) => sum + (it.occurrences ?? 0), 0) || 1;
+    return items
+        .map((it) => ({ ...it, occurrences: it.occurrences ?? 0, weight: (it.occurrences ?? 0) / total }))
+        .sort((a, b) => b.occurrences - a.occurrences);
+}
 /**
  * Build the generation prompt for `omk sample --from-traces`. Renders each
  * observation-inbox signal (evidence + message window) into a section and asks
@@ -477,7 +501,9 @@ const TRACE_GEN_INSTRUCTIONS = `下面给出的不是 skill，而是从生产会
  * judge prompt — so judge-prompt isolation is unaffected.
  */
 export function buildSamplesFromTracesPrompt(items, count) {
-    const sections = items.map((it, i) => {
+    // 先按频次分层（合并重复 + 算占比 + 降序），让模型按「占比」分配配额，而非每信号一刀切。
+    const stratified = stratifyTraceSignals(items);
+    const sections = stratified.map((it, i) => {
         const ev = it.evidence ?? {};
         const evLines = [
             ev.tool && `工具: ${ev.tool}`,
@@ -491,14 +517,15 @@ export function buildSamplesFromTracesPrompt(items, count) {
             ? '\n上下文消息:\n' + [...it.messageWindow.before, ...it.messageWindow.event, ...it.messageWindow.after]
                 .map((m) => `  [${m.role}] ${m.snippet}`).join('\n')
             : '';
-        return `### 信号 ${i + 1}：${it.signalType} / ${it.signalSubtype}（严重度 ${it.severity}，出现 ${it.occurrences} 次，skill: ${it.skillName}）\n${evLines}${win}`;
+        const pct = (it.weight * 100).toFixed(0);
+        return `### 信号 ${i + 1}：${it.signalType} / ${it.signalSubtype}（严重度 ${it.severity}，出现 ${it.occurrences} 次 · 占比 ${pct}%，skill: ${it.skillName}）\n${evLines}${win}`;
     }).join('\n\n---\n\n');
     const countLine = typeof count === 'number'
-        ? `共生成约 ${count} 条评测用例。`
-        : '按上面的「每个信号 1-2 条」生成。';
+        ? `共生成约 ${count} 条评测用例，按各信号的「占比」分配配额（高频多、低频少），覆盖整体失败分布。`
+        : '按各信号「占比」分配：高频信号多生成、低频少生成，覆盖整体失败分布。';
     return `${TRACE_GEN_INSTRUCTIONS}
 
-## 观测到的失败信号（共 ${items.length} 个）
+## 观测到的失败信号（共 ${stratified.length} 个，已按出现频次降序）
 
 ${sections}
 

package/dist/cli/commands/eval/index.d.ts

@@ -38,6 +38,7 @@ export default class Eval extends BaseCommand {
         effort: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         'no-diagnostic': import("@oclif/core/interfaces").BooleanFlag<boolean>;
         repeat: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
+        'holdout-ratio': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         'judge-repeat': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
         bootstrap: import("@oclif/core/interfaces").BooleanFlag<boolean>;
         'bootstrap-samples': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;

package/dist/cli/commands/eval/index.js

@@ -41,10 +41,30 @@ function applyGateExitCode(code, values, lang) {
     process.stderr.write(tCli('cli.run.report_only_gate_skipped', lang));
     return 0;
 }
+/**
+ * 完整 report JSON 是**机器输出**:重定向 / 管道(`omk eval > r.json`、`| jq`)时吐到 stdout 供下游消费。
+ * 交互式 TTY 下报告已存盘、(默认)还起了 report server,再刷上千行 JSON 只会把 verdict 淹没在屏幕外 ——
+ * 故只在非 TTY(stdout 被重定向 / 管道)时 dump。dry-run 的 JSON 是用户显式索取的产物,不走此门控。
+ */
+function emitReportJson(report) {
+    if (!process.stdout.isTTY) {
+        console.log(JSON.stringify(report, null, 2));
+    }
+}
+/**
+ * 给人读的 verdict 文案:stdout 是 TTY 时进 stdout(交互终端没有 JSON,verdict 就是答案),
+ * 否则进 stderr —— 与 emitReportJson 配对,保证非 TTY 的 stdout 是**纯 report JSON**,
+ * `omk eval | jq` / `> report.json` 不会被末尾拼上的人类文案噎住(否则 JSON.parse 直接失败)。
+ */
+function emitVerdictText(text) {
+    // 与 console.log 等价(对单个字符串 = write(text + '\n')),只切换目标流,逐字节保留既有文案。
+    const stream = process.stdout.isTTY ? process.stdout : process.stderr;
+    stream.write(text + '\n');
+}
 async function emitEvaluationVerdict(report, values, lang) {
     const { computeVerdict, formatVerdictText } = await import('../../../eval-core/verdict.js');
     const result = computeVerdict(report, verdictOptions(values));
-    console.log(formatVerdictText(result, { verbose: true }));
+    emitVerdictText(formatVerdictText(result, { verbose: true, lang }));
     await recordEvidenceSafely(report, result.level, values, lang);
     return verdictPasses(result.level, result.headline) ? 0 : 1;
 }
@@ -123,13 +143,13 @@ async function emitBatchVerdict(report, reportsDir, values, lang) {
     const status = lang === 'zh'
         ? (failed === 0 ? '通过' : '未通过')
         : (failed === 0 ? 'PASS' : 'FAIL');
-    console.log(tCli('cli.run.batch_verdict_header', lang, {
+    emitVerdictText(tCli('cli.run.batch_verdict_header', lang, {
         status,
         passed,
         total: results.length,
     }));
     for (const result of results) {
-        console.log(`  ${result.verdict.level}: ${result.treatment} — ${result.verdict.headline}`);
+        emitVerdictText(`  ${result.verdict.level}: ${result.treatment} — ${result.verdict.headline}`);
     }
     return failed === 0 ? 0 : 1;
 }
@@ -168,6 +188,13 @@ async function runEval(_args, flags, lang) {
         process.stderr.write(tCli('cli.run.invalid_repeat', lang, { value: repeatRaw }));
     }
     const repeatCount = Math.max(1, Math.floor(parsedRepeat) || 1);
+    const holdoutRatioRaw = values['holdout-ratio'];
+    const parsedHoldoutRatio = holdoutRatioRaw !== undefined ? Number(holdoutRatioRaw) : (evalConfig?.holdoutRatio ?? 0);
+    if (holdoutRatioRaw !== undefined && (!Number.isFinite(parsedHoldoutRatio) || parsedHoldoutRatio <= 0 || parsedHoldoutRatio >= 1)) {
+        process.stderr.write(tCli('cli.run.invalid_holdout_ratio', lang, { value: holdoutRatioRaw }));
+    }
+    if (parsedHoldoutRatio > 0 && parsedHoldoutRatio < 1)
+        config.holdoutRatio = parsedHoldoutRatio;
     const judgeRepeatRaw = values['judge-repeat'];
     const parsedJudgeRepeat = judgeRepeatRaw !== undefined ? Number(judgeRepeatRaw) : (evalConfig?.judgeRepeat ?? 1);
     if (judgeRepeatRaw !== undefined && (!Number.isFinite(parsedJudgeRepeat) || parsedJudgeRepeat < 1)) {
@@ -224,11 +251,12 @@ async function runEval(_args, flags, lang) {
                     }
                 },
             });
-            console.log(JSON.stringify(report, null, 2));
             if (isDryRunBatchReport(report)) {
+                console.log(JSON.stringify(report, null, 2));
                 console.log(tCli('cli.run.dry_run_no_scores', lang));
                 throw new CliExit(0);
             }
+            emitReportJson(report);
             if (filePath) {
                 await announceSavedReport({ report, filePath, reportsDir: config.outputDir, values, lang });
             }
@@ -285,7 +313,7 @@ async function runEval(_args, flags, lang) {
                 }
             }
         }
-        console.log(JSON.stringify(report, null, 2));
+        emitReportJson(report);
         if (filePath) {
             await announceSavedReport({ report, filePath, reportsDir: config.outputDir, values, lang });
         }
@@ -457,6 +485,13 @@ export default class Eval extends BaseCommand {
             description: bilingual({ zh: '每个 sample 重复跑 N 次', en: 'Repeat each sample N times' }),
             parse: integerStringParser('--repeat', { min: 1 }),
         }),
+        'holdout-ratio': Flags.string({
+            description: bilingual({
+                zh: '留出比例 0-1（如 0.3）；切出 holdout 子集，对比 train/holdout 综合分检测过拟合',
+                en: 'Holdout fraction 0-1 (e.g. 0.3); splits a holdout subset, compares train/holdout composite to flag overfitting',
+            }),
+            parse: numberStringParser('--holdout-ratio', { min: 0, max: 1 }),
+        }),
         'judge-repeat': Flags.string({
             description: bilingual({ zh: '每个 dim 评 N 次', en: 'Judge each dim N times' }),
             parse: integerStringParser('--judge-repeat', { min: 1 }),

package/dist/cli/commands/init.js

@@ -12,17 +12,21 @@ const INIT_OMK_GITIGNORE = `# omk 测量 bulk + doctor --fix 备份(项目本地
 /reports/
 /backups/
 `;
+// 脚手架用例必须过 omk 自身的断言合规校验(load-samples.ts Rule A),否则新用户照
+// 快速开始跑的第一条 omk eval 会直接硬报错。约束:contains / not_contains 的 value
+// 只能是单个 ASCII token(长度 [2,40]、无内部空白、无 CJK);多词 / 中文语义匹配一律
+// 走 rubric 交评委判;regex pattern 不能含 CJK。改这里前先跑 `omk eval --dry-run`
+// (非 lenient 合规 oracle)与 test/cli/init-scaffold-conformance 回归测试。
 const INIT_SAMPLES = `[
   {
     "sample_id": "s001",
     "prompt": "审查以下代码",
     "context": "function authenticate(username, password) {\\n  const query = \`SELECT * FROM users WHERE name='\${username}' AND pass='\${password}'\`;\\n  return db.execute(query);\\n}",
-    "rubric": "应识别 SQL 注入风险，建议使用参数化查询",
+    "rubric": "应识别 SQL 注入风险，建议使用参数化查询；不应把这段代码判为安全无问题。",
     "assertions": [
       { "type": "contains", "value": "SQL", "weight": 1 },
       { "type": "contains", "value": "injection", "weight": 1 },
-      { "type": "regex", "pattern": "parameterized|prepared|placeholder|bind", "flags": "i", "weight": 0.5 },
-      { "type": "not_contains", "value": "looks good", "weight": 0.5 }
+      { "type": "regex", "pattern": "parameterized|prepared|placeholder|bind", "flags": "i", "weight": 0.5 }
     ],
     "dimensions": {
       "security": "是否准确识别出 SQL 注入漏洞并说明其危害",
@@ -35,8 +39,7 @@ const INIT_SAMPLES = `[
     "context": "async function fetchData(url) {\\n  const res = await fetch(url);\\n  const data = await res.json();\\n  return data;\\n}",
     "rubric": "应指出缺少错误处理（网络异常、非 JSON 响应、HTTP 错误状态码）",
     "assertions": [
-      { "type": "contains", "value": "error handling", "weight": 1 },
-      { "type": "regex", "pattern": "try[\\\\s\\\\S]*catch|exception|error", "flags": "i", "weight": 1 },
+      { "type": "regex", "pattern": "try[\\\\s\\\\S]*catch|catch|exception|error", "flags": "i", "weight": 1 },
       { "type": "contains", "value": "status", "weight": 0.5 }
     ],
     "dimensions": {
@@ -154,9 +157,9 @@ export default class Init extends BaseCommand {
             console.log(tCli('cli.init.scaffolded', lang, { dir: targetDir }));
             console.log('');
             console.log(tCli('cli.init.next_steps_title', lang));
-            console.log(tCli('cli.init.next_step_edit_samples', lang));
-            console.log(tCli('cli.init.next_step_edit_skills', lang));
             console.log(tCli('cli.init.next_step_run', lang));
+            console.log(tCli('cli.init.next_step_executor', lang));
+            console.log(tCli('cli.init.next_step_customize', lang));
             console.log(tCli('cli.init.note_codex_executor', lang));
         });
     }

package/dist/cli/lib/i18n-dict/init.d.ts

@@ -1,3 +1,3 @@
 import type { CliMessage } from './types.js';
-export type InitMessageKey = 'cli.init.scaffolded' | 'cli.init.next_steps_title' | 'cli.init.next_step_edit_samples' | 'cli.init.next_step_edit_skills' | 'cli.init.next_step_run' | 'cli.init.note_codex_executor';
+export type InitMessageKey = 'cli.init.scaffolded' | 'cli.init.next_steps_title' | 'cli.init.next_step_run' | 'cli.init.next_step_executor' | 'cli.init.next_step_customize' | 'cli.init.note_codex_executor';
 export declare const initDict: Record<InitMessageKey, CliMessage>;

package/dist/cli/lib/i18n-dict/init.js

@@ -1,23 +1,26 @@
 export const initDict = {
     'cli.init.scaffolded': {
-        zh: '已初始化 omk 项目: {dir}',
+        zh: '已初始化 omk 项目：{dir}',
         en: 'omk project initialized at: {dir}',
     },
     'cli.init.next_steps_title': {
-        zh: '下一步:',
+        zh: '下一步：',
         en: 'Next steps:',
     },
-    'cli.init.next_step_edit_samples': {
-        zh: '  1. 编辑 eval-samples.json，加入你要测的评测用例',
-        en: '  1. Edit eval-samples.json to add your test cases',
+    // 先让用户「无需改任何文件直接跑通」——脚手架的用例与 skill 本身可跑(已过合规校验),
+    // 跑出第一份报告是冷启动最该先发生的事;「换成你自己的」放到跑通之后。这也消除了
+    // 主 README「不用改任何文件」与旧 init「先编辑」的矛盾。
+    'cli.init.next_step_run': {
+        zh: '  1. 直接跑通（无需先改任何文件）：omk eval --control code-review-v1 --treatment code-review-v2',
+        en: '  1. Run it as-is (no edits needed): omk eval --control code-review-v1 --treatment code-review-v2',
     },
-    'cli.init.next_step_edit_skills': {
-        zh: '  2. 编辑 skills/code-review-v1/SKILL.md 和 skills/code-review-v2/SKILL.md, 为两个 skill 版本填入实际内容',
-        en: '  2. Edit skills/code-review-v1/SKILL.md and skills/code-review-v2/SKILL.md with your skill versions',
+    'cli.init.next_step_executor': {
+        zh: '     默认执行器与评委用 claude CLI，需先装好并登录；想换别的模型或离线跑（无需 API key）见文档「执行器」。',
+        en: '     The default executor and judge use the claude CLI (install and log in first); to use another model or run offline (no API key) see the Executors docs.',
     },
-    'cli.init.next_step_run': {
-        zh: '  3. 运行: omk eval --control code-review-v1 --treatment code-review-v2',
-        en: '  3. Run: omk eval --control code-review-v1 --treatment code-review-v2',
+    'cli.init.next_step_customize': {
+        zh: '  2. 跑通后，把 skills/code-review-v1/SKILL.md 和 skills/code-review-v2/SKILL.md 与 eval-samples.json 换成你自己的 skill 和用例',
+        en: '  2. Once it runs, replace skills/code-review-v1/SKILL.md and skills/code-review-v2/SKILL.md and eval-samples.json with your own skills and cases',
     },
     'cli.init.note_codex_executor': {
         zh: '\n注: omk 评测时把 SKILL.md 整文(含 frontmatter)作为 system prompt 注入——跨 executor 一致(claude / codex / openai-api / gemini 都走同一条路径,不依赖任何 executor 的 native skill auto-discovery 或 Skill 工具机制)。frontmatter 在 prompt 头部对 model 行为无显著影响。\n模板带 Claude Code 兼容的 frontmatter(name + description)是为了让同一份 directory-skill 也能 deploy 到 Claude Code:把整个目录复制到 ~/.claude/skills/code-review-v1/(整目录,不是单个 SKILL.md),Claude SDK 才能识别。这是 omk 评测之外的 bonus,一份文件双向 dogfood。',

package/dist/cli/lib/i18n-dict/run.d.ts

@@ -1,3 +1,3 @@
 import type { CliMessage } from './types.js';
-export type RunMessageKey = 'cli.progress.preflight_starting' | 'cli.progress.sample_retry' | 'cli.progress.sample_error' | 'cli.progress.sample_executing' | 'cli.progress.sample_exec_done' | 'cli.progress.output_preview' | 'cli.progress.judging' | 'cli.progress.judged' | 'cli.progress.skipped' | 'cli.progress.sample_done' | 'cli.progress.sample_failed_done' | 'cli.run.invalid_repeat' | 'cli.run.invalid_judge_repeat' | 'cli.run.no_debias_length_active' | 'cli.run.invalid_bootstrap_samples' | 'cli.run.bootstrap_samples_too_large' | 'cli.run.dry_run_no_scores' | 'cli.run.skill_section' | 'cli.run.run_section' | 'cli.run.batch_complete' | 'cli.run.batch_verdict_header' | 'cli.run.batch_child_report_missing' | 'cli.run.eval_complete' | 'cli.run.tally' | 'cli.run.report_saved' | 'cli.run.evidence_recorded' | 'cli.run.evidence_recorded_unbound' | 'cli.run.report_only_gate_skipped' | 'cli.run.report_server_running' | 'cli.run.report_server_view' | 'cli.run.report_server_stop' | 'cli.run.no_serve_in_non_tty' | 'cli.run.no_serve_view_hint' | 'cli.run.gold_load_failed' | 'cli.run.gold_load_issue' | 'cli.run.contamination_warning' | 'cli.run.skip_connectivity_warning';
+export type RunMessageKey = 'cli.progress.preflight_starting' | 'cli.progress.sample_retry' | 'cli.progress.sample_error' | 'cli.progress.sample_executing' | 'cli.progress.sample_exec_done' | 'cli.progress.output_preview' | 'cli.progress.judging' | 'cli.progress.judged' | 'cli.progress.skipped' | 'cli.progress.sample_done' | 'cli.progress.sample_failed_done' | 'cli.run.invalid_repeat' | 'cli.run.invalid_holdout_ratio' | 'cli.run.invalid_judge_repeat' | 'cli.run.no_debias_length_active' | 'cli.run.invalid_bootstrap_samples' | 'cli.run.bootstrap_samples_too_large' | 'cli.run.dry_run_no_scores' | 'cli.run.skill_section' | 'cli.run.run_section' | 'cli.run.batch_complete' | 'cli.run.batch_verdict_header' | 'cli.run.batch_child_report_missing' | 'cli.run.eval_complete' | 'cli.run.tally' | 'cli.run.report_saved' | 'cli.run.evidence_recorded' | 'cli.run.evidence_recorded_unbound' | 'cli.run.report_only_gate_skipped' | 'cli.run.report_server_running' | 'cli.run.report_server_view' | 'cli.run.report_server_stop' | 'cli.run.no_serve_in_non_tty' | 'cli.run.no_serve_view_hint' | 'cli.run.gold_load_failed' | 'cli.run.gold_load_issue' | 'cli.run.contamination_warning' | 'cli.run.skip_connectivity_warning';
 export declare const runDict: Record<RunMessageKey, CliMessage>;