oh-my-knowledge 0.41.0 → 0.43.0

package/README.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
+Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
+
+Prerequisite: the default executor and judge use the `claude` CLI — install and log in first (see [Requirements](#requirements)); to use another model or run offline (no API key) see [executors](docs/reference/executors.md).
+
+> The first run has only 3 cases, so the verdict will usually be `UNDERPOWERED` (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.
 
 > The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
 
-Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
+Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
 
 Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
@@ -122,6 +126,7 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
 - **[Executors](docs/reference/executors.md)** & **[artifact layout](docs/reference/artifact-layout.md)** — built-in / custom executors; how `variant` resolves to an artifact + runtime context
 - **[How-to guides](docs/guides/agent-eval.md)** — [evaluate an agent](docs/guides/agent-eval.md) (project runtime context) and [use non-Claude models](docs/guides/non-claude-models.md) (GLM / Qwen / DeepSeek / Moonshot / Ollama)
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
+- **[Example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** — a set of runnable examples in the repo, arranged simplest-to-richest
 - **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
 - **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
 - **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai

package/README.zh.md

@@ -23,11 +23,15 @@ omk init demo && cd demo
 omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例；`omk eval` 跑控制变量 A/B，5 分钟内出 HTML 报告 + 一行 verdict。
+开箱即跑：`omk init` 脚手架好两版 skill 和三条评测用例，不用先改任何文件，`omk eval` 跑控制变量 A/B，约 5 分钟出 HTML 报告 + 一行 verdict；跑通后再把 skill 和用例换成你自己的。
+
+前置：默认执行器与评委用 `claude` CLI，需先安装并登录（见[系统要求](#系统要求)）；想用别的模型或离线跑（无需 API key）见[执行器](docs/zh/reference/executors.md)。
+
+> 首跑只有 3 条用例，verdict 多半是「数据不足（UNDERPOWERED）」——这是正常起点而非出错；把用例加到约 20 条以上，再看「可发布」结论。
 
 > 命令行有新版本时会自动提示（每 20 小时最多一次）；想永久关闭该提醒，设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
 
-手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。
+手把手教程：[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)（推荐第一次跑评测的用户）。更多可跑示例（Skill Map、A/B、离线执行器、agent runtime、RAG）见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
 
 深入：[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
 
@@ -122,6 +126,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 - **[执行器](docs/zh/reference/executors.md)** & **[artifact 布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器；variant 如何解析为 artifact + runtime context
 - **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)（项目 runtime context）与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)（GLM / 通义 / DeepSeek / Moonshot / Ollama）
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
+- **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例，按由简到全排成上手路径
 - **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射
 - **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
 - **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai 等 25+ 维度横评

package/dist/analysis/report-diagnostics.d.ts

@@ -1,7 +1,7 @@
 /**
  * Auto-analysis: detect patterns and generate insights from evaluation results.
  */
-import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Lang } from '../types/index.js';
+import type { Report, AnalysisResult, Sample, SampleQualityAggregate, Representativeness, Lang } from '../types/index.js';
 /** opts for `analyzeResults`. Optional because most older callers don't have
  *  samples in scope; new callers (evaluation-pipeline / evolver) pass them in to
  *  populate `analysis.sampleQuality`. */
@@ -30,4 +30,11 @@ export declare function analyzeResults(report: Report, opts?: AnalyzeResultsOpti
  * participate in grading / judge / verdict. See docs/specs/sample-design-spec.md.
  */
 export declare function buildSampleQualityAggregate(samples: Sample[]): SampleQualityAggregate;
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export declare function buildRepresentativeness(aggregate: SampleQualityAggregate): Representativeness;
 export declare function generateAnalysisSummary(report: Report, lang?: Lang): string | undefined;

package/dist/analysis/report-diagnostics.js

@@ -144,7 +144,7 @@ export function buildSampleQualityAggregate(samples) {
             rubricCount++;
         }
     }
-    return {
+    const aggregate = {
         capabilityCoverage,
         difficultyDistribution,
         constructDistribution,
@@ -155,6 +155,56 @@ export function buildSampleQualityAggregate(samples) {
         sampleCountWithConstruct: withConstruct,
         sampleCountWithProvenance: withProvenance,
     };
+    aggregate.representativeness = buildRepresentativeness(aggregate);
+    return aggregate;
+}
+/** Largest entry of a `label → count` map, as `[label, share]` over a given total.
+ *  Returns `[undefined, 0]` when the map is empty or total ≤ 0. */
+function dominantShare(counts, total) {
+    let label;
+    let max = 0;
+    for (const [k, v] of Object.entries(counts)) {
+        if (v > max) {
+            max = v;
+            label = k;
+        }
+    }
+    return total > 0 ? [label, max / total] : [undefined, 0];
+}
+/**
+ * Relative-balance / skew of the sample set. Pure function of the aggregate's
+ * distributions — no external "expected" denominator exists (capabilities are
+ * free-form), so it reports concentration (dominant bucket share) + the dominant
+ * label per dimension, not absolute coverage. Diagnostic only.
+ */
+export function buildRepresentativeness(aggregate) {
+    const capTotal = Object.values(aggregate.capabilityCoverage).reduce((a, b) => a + b, 0);
+    const [dominantCapability, capabilityConcentration] = dominantShare(aggregate.capabilityCoverage, capTotal);
+    // difficulty / construct concentration over *declared* samples only — `unspecified`
+    // is "didn't say", not a bucket the set is skewed toward.
+    const declaredDifficulty = {
+        easy: aggregate.difficultyDistribution.easy,
+        medium: aggregate.difficultyDistribution.medium,
+        hard: aggregate.difficultyDistribution.hard,
+    };
+    const diffTotal = declaredDifficulty.easy + declaredDifficulty.medium + declaredDifficulty.hard;
+    const [dominantDifficulty, difficultyConcentration] = dominantShare(declaredDifficulty, diffTotal);
+    const declaredConstruct = {};
+    for (const [k, v] of Object.entries(aggregate.constructDistribution)) {
+        if (k !== 'unspecified')
+            declaredConstruct[k] = v;
+    }
+    const consTotal = Object.values(declaredConstruct).reduce((a, b) => a + b, 0);
+    const [dominantConstruct, constructConcentration] = dominantShare(declaredConstruct, consTotal);
+    return {
+        capabilityCount: Object.keys(aggregate.capabilityCoverage).length,
+        capabilityConcentration,
+        ...(dominantCapability ? { dominantCapability } : {}),
+        difficultyConcentration,
+        ...(dominantDifficulty ? { dominantDifficulty: dominantDifficulty } : {}),
+        constructConcentration,
+        ...(dominantConstruct ? { dominantConstruct } : {}),
+    };
 }
 export function generateAnalysisSummary(report, lang = 'zh') {
     const variants = report.meta?.variants || [];
@@ -383,12 +433,43 @@ export function generateAnalysisSummary(report, lang = 'zh') {
             ? `【综合洞察】${synthesis.join('；')}。`
             : `【Synthesis】${synthesis.join('; ')}.`);
     }
+    // ── Sample-composition skew ── flags over-representation ("70% are easy") so the
+    // reader knows the set may not be representative of the real task distribution.
+    // Diagnostic only — never gates the verdict. N≥10 guard mirrors capability_thin.
+    const rep = report.analysis?.sampleQuality?.representativeness;
+    const sampleCount = report.meta?.sampleCount ?? 0;
+    if (rep && sampleCount >= 10) {
+        const skews = [];
+        if (rep.difficultyConcentration > SKEW_CONCENTRATION_BAND && rep.dominantDifficulty) {
+            skews.push(lang === 'zh'
+                ? `难度 ${(rep.difficultyConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantDifficulty}`
+                : `${(rep.difficultyConcentration * 100).toFixed(0)}% of declared difficulty is ${rep.dominantDifficulty}`);
+        }
+        if (rep.capabilityConcentration > SKEW_CONCENTRATION_BAND && rep.dominantCapability) {
+            skews.push(lang === 'zh'
+                ? `能力标签 ${(rep.capabilityConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantCapability}`
+                : `${(rep.capabilityConcentration * 100).toFixed(0)}% of capability tags are ${rep.dominantCapability}`);
+        }
+        if (rep.constructConcentration > SKEW_CONCENTRATION_BAND && rep.dominantConstruct) {
+            skews.push(lang === 'zh'
+                ? `construct ${(rep.constructConcentration * 100).toFixed(0)}% 集中在 ${rep.dominantConstruct}`
+                : `${(rep.constructConcentration * 100).toFixed(0)}% of declared construct is ${rep.dominantConstruct}`);
+        }
+        if (skews.length > 0) {
+            lines.push(lang === 'zh'
+                ? `【用例构成】偏斜：${skews.join('；')}——补充其它维度可提升代表性（构成提示，不影响 verdict）。`
+                : `【Sample composition】Skewed: ${skews.join('; ')} — adding other dimensions improves representativeness (informational; does not affect the verdict).`);
+        }
+    }
     // Caveats and recommendations are handled by the issues table below,
     // so the summary focuses only on verdict + differentiators + synthesis.
     if (lines.length === 0)
         return undefined;
     return lines.join('\n');
 }
+/** Dominant-bucket share above which the sample set is flagged as skewed. Pragmatic
+ *  default — 60% of declared samples in one bucket is a clear over-representation. */
+const SKEW_CONCENTRATION_BAND = 0.6;
 const AGENT_ASSERTION_TYPES = new Set([
     'tools_called',
     'tools_not_called',

package/dist/artifact-graph/doctor.d.ts

@@ -0,0 +1,21 @@
+import type { ArtifactGraphDocument, DoctorReport, DoctorSkillReport, Lang } from '../types/index.js';
+export interface BuildDoctorGraphOptions {
+    report: DoctorReport;
+    skill: DoctorSkillReport;
+    sourcePath: string;
+    generatedAt?: string;
+}
+export interface PersistDoctorGraphOptions extends BuildDoctorGraphOptions {
+    outputDir: string;
+    fileStem: string;
+    lang: Lang;
+}
+export interface PersistDoctorGraphResult {
+    graphPath: string;
+    evidenceCardPath: string;
+}
+export declare function doctorGraphDirForDoctorOutput(doctorOutputDir: string): string;
+export declare function buildDoctorArtifactGraph(options: BuildDoctorGraphOptions): ArtifactGraphDocument;
+export declare function renderDoctorEvidenceCard(graph: ArtifactGraphDocument, skill: DoctorSkillReport, lang: Lang): string;
+export declare function persistDoctorGraphSidecars(options: PersistDoctorGraphOptions): PersistDoctorGraphResult;
+export declare function removeDoctorGraphSidecars(doctorOutputDir: string, fileStem: string): void;