oh-my-knowledge 0.46.0 → 0.47.0

package/README.md

@@ -8,8 +8,8 @@
 
 **English** | [简体中文](./README.zh.md)
 
-**Did your prompt actually get better?**
-A/B test your prompts and skills with statistical rigor — bootstrap CI and length-debias on by default, Krippendorff α the moment you add a gold set.
+**Is this knowledge input (prompt / skill / RAG / agent) any good, and can you ship it with evidence?**
+`doctor` checks whether this knowledge input is coherent enough to measure; `eval` fixes the model and samples, changes only the knowledge input, and tells you whether the new version is genuinely better. Bootstrap CI and length-debias are on by default; Krippendorff α appears the moment you add a gold set.
 
 📖 **Full documentation: [oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** (searchable, English / 简体中文)
 
@@ -35,6 +35,20 @@ Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recomme
 
 Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
+## The first workflow
+
+omk is primarily for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The first workflow is deliberately small:
+
+```text
+change a skill / prompt / agent artifact
+→ run omk doctor to catch structure, dependency, and measurability problems
+→ run omk eval to compare against a baseline on the same samples
+→ read the report / Studio view for the next concrete fix
+→ decide ship / don't ship
+```
+
+`observe` is the later production-feedback loop: useful once real usage traces exist, but not required for omk's first value. The trunk is the pre-ship doctor → eval decision.
+
 ## Use inside AI Coding Agents
 
 Install the official omk Agent Skill to let your coding agent run omk workflows from natural language:
@@ -73,7 +87,7 @@ You can also describe the goal in natural language, such as "compare v1 vs v2" o
 
 ## Why this tool
 
-Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
+Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "can we ship v2, and why?", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
 
 ## Why omk over alternatives
 

package/README.zh.md

@@ -8,8 +8,8 @@
 
 [English](./README.md) | **简体中文**
 
-**你改完 prompt，真的变好了吗？**
-用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、长度去偏默认开，配一份人工 gold 就自动算 Krippendorff α。
+**一段知识输入（prompt / skill / RAG / agent）好不好，能不能有证据地发布？**
+`doctor` 先检查这类知识输入是否清楚到值得测；`eval` 固定模型和用例，只改变知识输入，判断新版本是否真的更好。Bootstrap 置信区间、长度去偏默认开，配一份人工 gold 就自动算 Krippendorff α。
 
 📖 **完整文档：[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)**（可搜索，可切换英文）
 
@@ -35,6 +35,20 @@ omk eval --control code-review-v1 --treatment code-review-v2
 
 深入：[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
 
+## 第一条工作流
+
+omk 主要给 LLM 知识载体的作者 / 维护者用，帮他们做发布判断；它不是给被动安装 skill 的普通使用者用的。第一条工作流应该很短：
+
+```text
+改了一个 skill / prompt / agent artifact
+→ 跑 omk doctor，先抓结构、依赖和可测性问题
+→ 跑 omk eval，在同一批用例上和 baseline 对比
+→ 看报告 / Studio，知道下一步具体该改哪里
+→ 决定发布 / 不发布
+```
+
+`observe` 是后续的生产反馈闭环：等真实使用 trace 存在后很有价值，但不是 omk 第一价值的前提。主干先是发布前的 doctor → eval 判断。
+
 ## 在 AI Coding Agent 中使用
 
 安装 omk 官方 Agent Skill 后，可以直接用自然语言让 coding agent 跑 omk 工作流：
@@ -73,7 +87,7 @@ omk sample skills/my-skill.md
 
 ## 为什么需要这个工具
 
-做知识工程的团队会产出大量知识载体（当前常见是 skill，也包括 prompt、agent、workflow 等）。当被问到「v2 比 v1 好在哪」时，需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题：**相同模型、相同评测用例，只改变知识载体。**
+做知识工程的团队会产出大量知识载体（当前常见是 skill，也包括 prompt、agent、workflow 等）。当被问到「v2 能不能发、为什么」时，需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题：**相同模型、相同评测用例，只改变知识载体。**
 
 ## 为什么选 omk
 

package/dist/cli/commands/doctor.js

@@ -178,7 +178,7 @@ export default class Doctor extends BaseCommand {
                 }
             }
             const { runDoctor } = await import('../../doctor/index.js');
-            const { renderDoctorReportText, renderDoctorReportJson } = await import('../../doctor/renderer.js');
+            const { renderDoctorReportText, renderDoctorReportJson, renderDoctorActionPlanText } = await import('../../doctor/renderer.js');
             const { getRegisteredRules } = await import('../../doctor/rules.js');
             const { isComposerRule } = await import('../../types/doctor.js');
             // 默认:静态规则 + 在线检查(LLM health composer + endpoint 自定义维度 external=true)。
@@ -236,6 +236,7 @@ export default class Doctor extends BaseCommand {
                         ? `doctor failed: ${report.totals.fail} 个 skill 未通过 (${report.totals.warn} warn / ${report.totals.pass} pass)`
                         : `doctor failed: ${report.totals.fail} skills did not pass (${report.totals.warn} warn / ${report.totals.pass} pass)`;
                     console.error(summary);
+                    process.stderr.write(renderDoctorActionPlanText(report, lang));
                 }
             }
             else {

package/dist/cli/commands/eval/index.js

@@ -152,6 +152,14 @@ async function emitBatchVerdict(report, reportsDir, values, lang) {
     for (const result of results) {
         emitVerdictText(`  ${result.verdict.level}: ${result.treatment} — ${result.verdict.headline}`);
     }
+    const next = lang === 'zh'
+        ? (failed === 0
+            ? '全部通过，可以进入发布流程：请留存这些报告作为发布证据；如果是受管 skill，再运行 `omk promote`。'
+            : '先处理未通过的 skill；逐个打开对应报告，看最差层和失败用例后再重跑。')
+        : (failed === 0
+            ? 'all skills passed and are ready for release: keep these reports as release evidence; for managed skills, run `omk promote`.'
+            : 'fix the failing skills first; open each report, inspect the weakest layer and failing samples, then re-run.');
+    emitVerdictText(tCli('cli.run.batch_verdict_next_step', lang, { next }));
     return failed === 0 ? 0 : 1;
 }
 async function announceSavedReport({ report, filePath, reportsDir, values, lang, }) {

package/dist/cli/lib/i18n-dict/run.d.ts

@@ -1,3 +1,3 @@
 import type { CliMessage } from './types.js';
-export type RunMessageKey = 'cli.progress.preflight_starting' | 'cli.progress.sample_retry' | 'cli.progress.sample_error' | 'cli.progress.sample_executing' | 'cli.progress.sample_exec_done' | 'cli.progress.output_preview' | 'cli.progress.judging' | 'cli.progress.judged' | 'cli.progress.skipped' | 'cli.progress.sample_done' | 'cli.progress.sample_failed_done' | 'cli.run.invalid_repeat' | 'cli.run.invalid_holdout_ratio' | 'cli.run.invalid_judge_repeat' | 'cli.run.no_debias_length_active' | 'cli.run.invalid_bootstrap_samples' | 'cli.run.bootstrap_samples_too_large' | 'cli.run.dry_run_no_scores' | 'cli.run.skill_section' | 'cli.run.run_section' | 'cli.run.batch_complete' | 'cli.run.batch_verdict_header' | 'cli.run.batch_child_report_missing' | 'cli.run.eval_complete' | 'cli.run.tally' | 'cli.run.report_saved' | 'cli.run.evidence_recorded' | 'cli.run.evidence_recorded_unbound' | 'cli.run.report_only_gate_skipped' | 'cli.run.report_server_running' | 'cli.run.report_server_view' | 'cli.run.report_server_stop' | 'cli.run.no_serve_in_non_tty' | 'cli.run.no_serve_view_hint' | 'cli.run.gold_load_failed' | 'cli.run.gold_load_issue' | 'cli.run.contamination_warning' | 'cli.run.skip_connectivity_warning';
+export type RunMessageKey = 'cli.progress.preflight_starting' | 'cli.progress.sample_retry' | 'cli.progress.sample_error' | 'cli.progress.sample_executing' | 'cli.progress.sample_exec_done' | 'cli.progress.output_preview' | 'cli.progress.judging' | 'cli.progress.judged' | 'cli.progress.skipped' | 'cli.progress.sample_done' | 'cli.progress.sample_failed_done' | 'cli.run.invalid_repeat' | 'cli.run.invalid_holdout_ratio' | 'cli.run.invalid_judge_repeat' | 'cli.run.no_debias_length_active' | 'cli.run.invalid_bootstrap_samples' | 'cli.run.bootstrap_samples_too_large' | 'cli.run.dry_run_no_scores' | 'cli.run.skill_section' | 'cli.run.run_section' | 'cli.run.batch_complete' | 'cli.run.batch_verdict_header' | 'cli.run.batch_verdict_next_step' | 'cli.run.batch_child_report_missing' | 'cli.run.eval_complete' | 'cli.run.tally' | 'cli.run.report_saved' | 'cli.run.evidence_recorded' | 'cli.run.evidence_recorded_unbound' | 'cli.run.report_only_gate_skipped' | 'cli.run.report_server_running' | 'cli.run.report_server_view' | 'cli.run.report_server_stop' | 'cli.run.no_serve_in_non_tty' | 'cli.run.no_serve_view_hint' | 'cli.run.gold_load_failed' | 'cli.run.gold_load_issue' | 'cli.run.contamination_warning' | 'cli.run.skip_connectivity_warning';
 export declare const runDict: Record<RunMessageKey, CliMessage>;

package/dist/cli/lib/i18n-dict/run.js

@@ -87,6 +87,10 @@ export const runDict = {
         zh: '批量评测结论：{status}（{passed}/{total} 通过）',
         en: 'Batch verdict: {status} ({passed}/{total} passed)',
     },
+    'cli.run.batch_verdict_next_step': {
+        zh: '  下一步：{next}',
+        en: '  Next: {next}',
+    },
     'cli.run.batch_child_report_missing': {
         zh: '⚠ 子报告缺失：{id}，将按不可 ship 处理。\n',
         en: '⚠ Child report missing: {id}; treating it as not shippable.\n',

package/dist/doctor/renderer.d.ts

@@ -12,5 +12,6 @@
  */
 import type { Lang } from '../types/shared.js';
 import type { DoctorReport } from '../types/index.js';
+export declare function renderDoctorActionPlanText(report: DoctorReport, lang: Lang): string;
 export declare function renderDoctorReportText(report: DoctorReport, lang: Lang, write?: (s: string) => void): void;
 export declare function renderDoctorReportJson(report: DoctorReport): string;

package/dist/doctor/renderer.js

@@ -43,6 +43,68 @@ function renderRuleLine(result, ruleLabel, lang, indent = '  ') {
     }
     return line;
 }
+function compactActionText(text) {
+    return (text ?? '').replace(/\s+/g, ' ').trim();
+}
+function collectDoctorRepairItems(report, lang) {
+    const allItems = [];
+    for (const skill of report.skills) {
+        for (const result of skill.results) {
+            if (result.status !== 'fail' && result.status !== 'warn')
+                continue;
+            const action = compactActionText(result.hint || result.message);
+            allItems.push({
+                skillName: skill.skillName,
+                status: result.status,
+                label: renderRuleLabel(result, lang),
+                action,
+                isSummary: result.ruleId.endsWith(':_summary'),
+            });
+        }
+    }
+    const focusedItems = allItems.filter((item) => !item.isSummary);
+    const source = focusedItems.length > 0 ? focusedItems : allItems;
+    const sorted = [...source].sort((a, b) => STATUS_RANK[a.status] - STATUS_RANK[b.status]);
+    return { items: sorted.slice(0, 6), total: source.length };
+}
+export function renderDoctorActionPlanText(report, lang) {
+    const { items, total } = collectDoctorRepairItems(report, lang);
+    if (items.length === 0) {
+        return lang === 'zh'
+            ? '\n下一步：doctor 已通过，可以继续运行 `omk eval`。\n'
+            : '\nNext: doctor passed; continue with `omk eval`.\n';
+    }
+    const hasFail = items.some((item) => item.status === 'fail');
+    const lines = [
+        '',
+        lang === 'zh'
+            ? (hasFail ? '修复清单（先处理阻塞项）：' : '修复清单（建议处理，不阻断 eval）：')
+            : (hasFail ? 'Repair checklist (fix blocking items first):' : 'Repair checklist (recommended; does not block eval):'),
+    ];
+    items.forEach((item, index) => {
+        const status = statusLabel(item.status, lang);
+        if (lang === 'zh') {
+            lines.push(`  ${index + 1}. [${item.skillName}] ${status}：${item.label}。${item.action}`);
+        }
+        else {
+            lines.push(`  ${index + 1}. [${item.skillName}] ${status}: ${item.label}. ${item.action}`);
+        }
+    });
+    const remaining = total - items.length;
+    if (remaining > 0) {
+        lines.push(lang === 'zh'
+            ? `  ……还有 ${remaining} 项，运行 \`omk doctor\` 查看完整明细。`
+            : `  ...and ${remaining} more; run \`omk doctor\` for full detail.`);
+    }
+    lines.push(hasFail
+        ? (lang === 'zh'
+            ? '下一步：先修阻塞项，重跑 `omk doctor --gate`；通过后再跑 `omk eval`。'
+            : 'Next: fix the blocking items, re-run `omk doctor --gate`, then run `omk eval`.')
+        : (lang === 'zh'
+            ? '下一步：可以先跑 `omk eval`，但发布前建议把这些 warning 处理掉。'
+            : 'Next: you can run `omk eval`, but clear these warnings before shipping.'));
+    return `${lines.join('\n')}\n`;
+}
 function segmentResults(results) {
     const segs = [];
     const groupSegMap = new Map();
@@ -115,6 +177,7 @@ export function renderDoctorReportText(report, lang, write = (s) => process.stde
         ? `\n总览: ${report.totals.pass} 通过 / ${report.totals.warn} 警告 / ${report.totals.fail} 失败\n`
         : `\nSummary: ${report.totals.pass} pass / ${report.totals.warn} warn / ${report.totals.fail} fail\n`;
     write(summary);
+    write(renderDoctorActionPlanText(report, lang));
 }
 export function renderDoctorReportJson(report) {
     return JSON.stringify(report, null, 2);

package/dist/eval-core/verdict.d.ts

@@ -166,8 +166,8 @@ export declare function medianStabilityCV(report: Report): {
     cv: number;
 } | null;
 /**
- * Plain-text formatter for the `omk eval` verdict. Stays under 6 lines per the
- *  spec — one verdict + four rationale bullets + one ship recommendation.
+ * Plain-text formatter for the `omk eval` verdict. Stays terse for the
+ *  spec — one verdict, rationale bullets, one ship recommendation, and one next step.
  */
 export declare function formatVerdictText(result: VerdictResult, options?: {
     verbose?: boolean;

package/dist/eval-core/verdict.js

@@ -108,6 +108,7 @@ export function computeVerdict(report, options = {}) {
                 ...(judgeInd.rationale ? { judgeAgreement: judgeInd.rationale } : {}),
                 ...(overfit.rationale ? { overfitting: overfit.rationale } : {}),
                 ...(gap.rationale ? { gapSignal: gap.rationale } : {}),
+                shipRecommendation: recommendation('SOLO', []),
             },
             ...((overfit.data || gap.data) ? {
                 caveats: {
@@ -583,9 +584,41 @@ function recommendation(level, _perPair, lang = 'en') {
             return 'ADD A CONTROL — single-variant report. Re-run with --control baseline --treatment <name>.';
     }
 }
+function releaseNextStep(level, lang) {
+    if (lang === 'zh') {
+        switch (level) {
+            case 'PROGRESS':
+                return '可以进入发布流程：请留存本次报告作为发布证据；如果这是受管 skill，再运行 `omk promote`。';
+            case 'CAUTIOUS':
+                return '先看触发的告警（分层门控、评委分歧、稳定性或 holdout），修完再重跑。';
+            case 'REGRESS':
+                return '不要发布；定位最差层和失败用例，修复后重跑。';
+            case 'NOISE':
+                return '先别发布；增加样本数或提高用例区分度，再重跑。';
+            case 'UNDERPOWERED':
+                return '把样本数加到至少 20，或先按当前规模 2× 扩充后重跑。';
+            case 'SOLO':
+                return '补一个 baseline 对照，再跑 `omk eval --control baseline --treatment <名字>`。';
+        }
+    }
+    switch (level) {
+        case 'PROGRESS':
+            return 'ready for release: keep this report as release evidence; for a managed skill, run `omk promote`.';
+        case 'CAUTIOUS':
+            return 'inspect the warnings (layer gates, judge dissent, stability, or holdout), fix them, then re-run.';
+        case 'REGRESS':
+            return 'do not ship; inspect the weakest layer and failing samples, fix them, then re-run.';
+        case 'NOISE':
+            return 'do not ship yet; add samples or sharpen the test set, then re-run.';
+        case 'UNDERPOWERED':
+            return 'increase the sample set to at least 20, or roughly 2x the current size, then re-run.';
+        case 'SOLO':
+            return 'add a baseline control and re-run `omk eval --control baseline --treatment <name>`.';
+    }
+}
 /**
- * Plain-text formatter for the `omk eval` verdict. Stays under 6 lines per the
- *  spec — one verdict + four rationale bullets + one ship recommendation.
+ * Plain-text formatter for the `omk eval` verdict. Stays terse for the
+ *  spec — one verdict, rationale bullets, one ship recommendation, and one next step.
  */
 export function formatVerdictText(result, options = {}) {
     // lang 默认 'en':保留既有英文输出逐字节不变(verdict.test 与历史 CLI 行为)。zh 只本地化
@@ -609,6 +642,7 @@ export function formatVerdictText(result, options = {}) {
         lines.push(zh ? `  知识缺口：${result.rationale.gapSignal}` : `  Gap signal:    ${result.rationale.gapSignal}`);
     if (result.rationale.shipRecommendation) {
         lines.push(`  ${zh ? recommendation(result.level, [], 'zh') : result.rationale.shipRecommendation}`);
+        lines.push(zh ? `  下一步：${releaseNextStep(result.level, 'zh')}` : `  Next: ${releaseNextStep(result.level, 'en')}`);
     }
     if (options.verbose && result.perPair && result.perPair.length > 1) {
         lines.push('');

package/dist/eval-workflows/messages.js

@@ -12,8 +12,8 @@ const MESSAGES = {
         en: '⚠ --repeat=1: single-run cannot measure stability (CV will be marked "not measured"). Use --repeat 3+ to detect within-variant variance.',
     },
     doctor_gate_blocked: {
-        zh: 'skill 健康检查未通过，评测已中止。doctor 是评测必经环节，无 skip 选项——请修复上述问题后重跑。',
-        en: 'skill health check failed; evaluation aborted. doctor is mandatory and not skippable — fix the issues above and re-run.',
+        zh: '发布前 doctor 门禁未通过，评测已中止。\n下一步：先修复上面的阻塞项，再重跑 `omk eval`。\n原因：这段输入还不值得测，继续比较分数会是 garbage-in。\n如果依赖确实由 mock / stub 提供、doctor 误报，可用 `--skip-doctor` 绕过，但这次结果由你承担不可比风险。',
+        en: 'pre-ship doctor gate failed; evaluation aborted.\nNext: fix the blocking findings above, then re-run `omk eval`.\nWhy: this input is not measurable enough yet, and comparing scores now would be garbage-in.\nIf deps are truly supplied by mocks/stubs and doctor is a false positive, use `--skip-doctor`, but you own the comparability risk.',
     },
 };
 const DEFAULT_LANG = 'zh';

package/dist/eval-workflows/run-evaluation.js

@@ -62,7 +62,7 @@ export async function runEvaluation({ samplesPath, skillDir, variantSpecs = [],
             });
             if (doctorReport.outcome === 'failed') {
                 renderDoctorReportText(doctorReport, lang);
-                throw new Error(`doctor failed: ${tEvalWorkflowMessage('doctor_gate_blocked', lang)}`);
+                throw new Error(`doctor failed:\n${tEvalWorkflowMessage('doctor_gate_blocked', lang)}`);
             }
         }
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "oh-my-knowledge",
-  "version": "0.46.0",
+  "version": "0.47.0",
   "packageManager": "yarn@4.16.0",
   "description": "Evaluation framework for LLM knowledge inputs — prompts, RAG corpora, skills, agent workflows. Fix the model, vary the artifact. Built-in statistical rigor: bootstrap CI, Krippendorff α, length-debias, saturation curves.",
   "type": "module",
@@ -93,11 +93,11 @@
   "license": "MIT",
   "dependencies": {
     "@anthropic-ai/claude-agent-sdk": "^0.3.143",
-    "@anthropic-ai/sdk": "^0.105.0",
+    "@anthropic-ai/sdk": "^0.107.0",
     "@inquirer/prompts": "^8.4.3",
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@oclif/core": "^4",
-    "@openai/codex-sdk": "0.141.0",
+    "@openai/codex-sdk": "0.142.4",
     "ajv": "^8.18.0",
     "chart.js": "^4.5.1",
     "js-yaml": "^4.1.1",
@@ -112,11 +112,11 @@
     "@types/node": "^25.5.0",
     "eslint": "^10.1.0",
     "husky": "^9.1.7",
-    "lint-staged": "17.0.7",
+    "lint-staged": "17.0.8",
     "npm-run-all2": "^9.0.1",
     "typescript": "^6.0.2",
     "typescript-eslint": "^8.58.0",
     "vitepress": "^1.6.4",
-    "vitest": "4.1.8"
+    "vitest": "4.1.9"
   }
 }