oh-my-knowledge 0.48.0 → 0.50.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (243) hide show
  1. package/README.md +50 -18
  2. package/README.zh.md +55 -23
  3. package/dist/analysis/coverage-analyzer.d.ts +1 -0
  4. package/dist/analysis/coverage-analyzer.js +125 -62
  5. package/dist/analysis/failure-clusterer.js +2 -1
  6. package/dist/analysis/gap-analyzer.d.ts +2 -2
  7. package/dist/analysis/gap-analyzer.js +13 -3
  8. package/dist/analysis/hedging-classifier.d.ts +2 -2
  9. package/dist/analysis/hedging-classifier.js +3 -4
  10. package/dist/analysis/report-diagnostics.js +9 -7
  11. package/dist/analysis/sample-diagnostics.js +6 -6
  12. package/dist/artifact-graph/doctor.js +15 -7
  13. package/dist/assets/agent-skills/omk/SKILL.md +27 -7
  14. package/dist/assets/agent-skills/omk/references/commands.md +20 -19
  15. package/dist/authoring/evolver.d.ts +10 -6
  16. package/dist/authoring/evolver.js +496 -83
  17. package/dist/authoring/generator.d.ts +3 -3
  18. package/dist/authoring/generator.js +5 -10
  19. package/dist/authoring/sample-fixer.d.ts +8 -6
  20. package/dist/authoring/sample-fixer.js +76 -5
  21. package/dist/cli/commands/doctor.js +31 -14
  22. package/dist/cli/commands/eval/index.d.ts +3 -0
  23. package/dist/cli/commands/eval/index.js +163 -17
  24. package/dist/cli/commands/evolve.d.ts +4 -4
  25. package/dist/cli/commands/evolve.js +27 -13
  26. package/dist/cli/commands/init.js +16 -3
  27. package/dist/cli/commands/observe/inbox.js +30 -23
  28. package/dist/cli/commands/observe/index.js +20 -11
  29. package/dist/cli/commands/observe/ingest.d.ts +3 -0
  30. package/dist/cli/commands/observe/ingest.js +31 -3
  31. package/dist/cli/commands/sample.d.ts +6 -3
  32. package/dist/cli/commands/sample.js +75 -71
  33. package/dist/cli/lib/codex-model-hint.d.ts +9 -0
  34. package/dist/cli/lib/codex-model-hint.js +45 -0
  35. package/dist/cli/lib/generation-failure-hint.d.ts +2 -0
  36. package/dist/cli/lib/generation-failure-hint.js +61 -0
  37. package/dist/cli/lib/i18n-dict/common.d.ts +1 -1
  38. package/dist/cli/lib/i18n-dict/common.js +4 -0
  39. package/dist/cli/lib/i18n-dict/gen.d.ts +1 -1
  40. package/dist/cli/lib/i18n-dict/gen.js +38 -6
  41. package/dist/cli/lib/i18n-dict/help.js +6 -6
  42. package/dist/cli/lib/i18n-dict/init.d.ts +1 -1
  43. package/dist/cli/lib/i18n-dict/init.js +13 -9
  44. package/dist/cli/lib/i18n-dict/run.d.ts +1 -1
  45. package/dist/cli/lib/i18n-dict/run.js +34 -2
  46. package/dist/cli/lib/llm-failure-classifier.d.ts +2 -0
  47. package/dist/cli/lib/llm-failure-classifier.js +8 -0
  48. package/dist/cli/lib/parse-run-config.d.ts +6 -5
  49. package/dist/cli/lib/parse-run-config.js +16 -9
  50. package/dist/cli/lib/runtime-defaults.d.ts +21 -0
  51. package/dist/cli/lib/runtime-defaults.js +79 -0
  52. package/dist/diagnosis/observe-mapper.js +14 -15
  53. package/dist/diagnosis/observe-producer.js +3 -1
  54. package/dist/diagnosis/studio-projection.js +14 -7
  55. package/dist/diagnosis/types.d.ts +2 -0
  56. package/dist/diagnosis/types.js +12 -0
  57. package/dist/doctor/endpoint-rule.js +2 -1
  58. package/dist/eval-core/artifact-file-names.js +18 -1
  59. package/dist/eval-core/artifact-index.d.ts +7 -11
  60. package/dist/eval-core/artifact-index.js +139 -80
  61. package/dist/eval-core/cache.d.ts +12 -3
  62. package/dist/eval-core/cache.js +89 -29
  63. package/dist/eval-core/comparability.js +10 -6
  64. package/dist/eval-core/evaluation-execution.d.ts +2 -1
  65. package/dist/eval-core/evaluation-execution.js +122 -37
  66. package/dist/eval-core/evaluation-job.d.ts +4 -1
  67. package/dist/eval-core/evaluation-job.js +4 -1
  68. package/dist/eval-core/evaluation-reporting.d.ts +15 -13
  69. package/dist/eval-core/evaluation-reporting.js +54 -52
  70. package/dist/eval-core/execution-strategy.d.ts +2 -0
  71. package/dist/eval-core/execution-strategy.js +11 -9
  72. package/dist/eval-core/fact-checker.js +15 -7
  73. package/dist/eval-core/holdout.js +3 -2
  74. package/dist/eval-core/judge-independence.d.ts +2 -2
  75. package/dist/eval-core/mock-hook.cjs +23 -6
  76. package/dist/eval-core/mocks-runtime.js +30 -8
  77. package/dist/eval-core/report-document.d.ts +12 -0
  78. package/dist/eval-core/report-document.js +1151 -0
  79. package/dist/eval-core/report-extensions.d.ts +4 -0
  80. package/dist/eval-core/report-extensions.js +500 -0
  81. package/dist/eval-core/report-file-migration.js +7 -2
  82. package/dist/eval-core/resume-compatibility.d.ts +31 -0
  83. package/dist/eval-core/resume-compatibility.js +141 -0
  84. package/dist/eval-core/sample-fingerprint.d.ts +12 -0
  85. package/dist/eval-core/sample-fingerprint.js +193 -0
  86. package/dist/eval-core/schema.js +86 -31
  87. package/dist/eval-core/verdict.d.ts +8 -4
  88. package/dist/eval-core/verdict.js +24 -10
  89. package/dist/eval-workflows/batch-evaluation-workflow.d.ts +2 -1
  90. package/dist/eval-workflows/batch-evaluation-workflow.js +25 -12
  91. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts +10 -5
  92. package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js +58 -21
  93. package/dist/eval-workflows/evaluation-pipeline/report-finalize.js +3 -1
  94. package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts +4 -1
  95. package/dist/eval-workflows/evaluation-pipeline/run-state.js +4 -1
  96. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts +6 -5
  97. package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js +17 -10
  98. package/dist/eval-workflows/evaluation-pipeline.js +12 -7
  99. package/dist/eval-workflows/run-evaluation.d.ts +9 -7
  100. package/dist/eval-workflows/run-evaluation.js +79 -51
  101. package/dist/executors/anthropic-api.js +65 -9
  102. package/dist/executors/claude-cli.js +16 -79
  103. package/dist/executors/claude-protocol.d.ts +28 -0
  104. package/dist/executors/claude-protocol.js +180 -0
  105. package/dist/executors/claude-sdk-trace.js +56 -28
  106. package/dist/executors/claude-sdk.d.ts +1 -0
  107. package/dist/executors/claude-sdk.js +39 -93
  108. package/dist/executors/codex-cli-trace.js +166 -31
  109. package/dist/executors/codex-cli.d.ts +6 -8
  110. package/dist/executors/codex-cli.js +49 -151
  111. package/dist/executors/codex-protocol.d.ts +24 -0
  112. package/dist/executors/codex-protocol.js +234 -0
  113. package/dist/executors/codex-sdk.js +68 -120
  114. package/dist/executors/gemini.js +88 -13
  115. package/dist/executors/index.d.ts +2 -3
  116. package/dist/executors/index.js +5 -3
  117. package/dist/executors/openai-api.js +70 -9
  118. package/dist/executors/runtime-fingerprint.js +88 -11
  119. package/dist/executors/script-command.d.ts +8 -0
  120. package/dist/executors/script-command.js +87 -0
  121. package/dist/executors/script.js +202 -29
  122. package/dist/executors/shared.d.ts +35 -3
  123. package/dist/executors/shared.js +113 -15
  124. package/dist/grading/assertions.d.ts +1 -1
  125. package/dist/grading/assertions.js +19 -9
  126. package/dist/grading/diagnostic.d.ts +9 -2
  127. package/dist/grading/diagnostic.js +25 -2
  128. package/dist/grading/index.js +10 -4
  129. package/dist/grading/judge.js +19 -6
  130. package/dist/grading/layered-scores.d.ts +2 -3
  131. package/dist/grading/layered-scores.js +2 -3
  132. package/dist/inputs/load-samples.d.ts +1 -2
  133. package/dist/inputs/load-samples.js +23 -1
  134. package/dist/inputs/mcp-resolver.js +6 -3
  135. package/dist/inputs/sample-document.d.ts +11 -0
  136. package/dist/inputs/sample-document.js +96 -0
  137. package/dist/managed/evidence.d.ts +1 -0
  138. package/dist/managed/evidence.js +1 -1
  139. package/dist/managed/store.js +200 -91
  140. package/dist/observability/codex-trace-adapter.d.ts +5 -0
  141. package/dist/observability/codex-trace-adapter.js +850 -0
  142. package/dist/observability/experience.d.ts +32 -6
  143. package/dist/observability/experience.js +2779 -472
  144. package/dist/observability/feedback-matchers.js +16 -1
  145. package/dist/observability/inbox-view-model.d.ts +1 -1
  146. package/dist/observability/inbox-view-model.js +19 -14
  147. package/dist/observability/inbox.d.ts +7 -1
  148. package/dist/observability/inbox.js +632 -124
  149. package/dist/observability/problem-patterns.js +2 -0
  150. package/dist/observability/review-state.d.ts +6 -0
  151. package/dist/observability/review-state.js +235 -63
  152. package/dist/observability/skill-chain-advisories.js +1 -1
  153. package/dist/observability/skill-chain.js +17 -4
  154. package/dist/observability/skill-health-analyzer.d.ts +32 -7
  155. package/dist/observability/skill-health-analyzer.js +194 -121
  156. package/dist/observability/skill-health-report.d.ts +10 -0
  157. package/dist/observability/skill-health-report.js +620 -0
  158. package/dist/observability/soft-standards/constants.d.ts +0 -1
  159. package/dist/observability/soft-standards/constants.js +0 -1
  160. package/dist/observability/soft-standards/index.d.ts +1 -1
  161. package/dist/observability/soft-standards/index.js +1 -1
  162. package/dist/observability/soft-standards/llm-extractor.js +8 -10
  163. package/dist/observability/soft-standards/skill-standards-store.d.ts +2 -1
  164. package/dist/observability/soft-standards/skill-standards-store.js +59 -18
  165. package/dist/observability/soft-standards/types.d.ts +2 -2
  166. package/dist/observability/trace-adapter.d.ts +12 -7
  167. package/dist/observability/trace-adapter.js +11 -9
  168. package/dist/observability/trace-attribution.d.ts +13 -5
  169. package/dist/observability/trace-attribution.js +315 -21
  170. package/dist/observability/trace-ingestion.d.ts +9 -0
  171. package/dist/observability/trace-ingestion.js +80 -0
  172. package/dist/observability/trace-ir.d.ts +113 -0
  173. package/dist/observability/trace-ir.js +87 -0
  174. package/dist/observability/trace-segmenter.d.ts +19 -6
  175. package/dist/observability/trace-segmenter.js +377 -196
  176. package/dist/observability/trace-session-index.d.ts +19 -0
  177. package/dist/observability/trace-session-index.js +68 -0
  178. package/dist/observability/trace-source.d.ts +12 -4
  179. package/dist/observability/trace-source.js +939 -215
  180. package/dist/renderer/html-renderer.js +37 -6
  181. package/dist/renderer/icons.js +3 -0
  182. package/dist/renderer/observation-inbox-renderer.js +208 -90
  183. package/dist/renderer/skill-detail-renderer.js +452 -109
  184. package/dist/renderer/skill-health-renderer.js +69 -12
  185. package/dist/renderer/summary.js +28 -7
  186. package/dist/renderer/table.js +21 -4
  187. package/dist/renderer/test-view.d.ts +1 -0
  188. package/dist/renderer/test-view.js +44 -9
  189. package/dist/server/indexed-report-store.js +14 -18
  190. package/dist/server/job-store.js +64 -26
  191. package/dist/server/report-server.js +190 -78
  192. package/dist/server/report-store.js +57 -80
  193. package/dist/server/skill-index.js +143 -49
  194. package/dist/server/skill-insights.js +44 -5
  195. package/dist/shared/artifact-graph.d.ts +3 -0
  196. package/dist/shared/artifact-graph.js +224 -0
  197. package/dist/shared/assertion-types.d.ts +8 -0
  198. package/dist/shared/assertion-types.js +46 -0
  199. package/dist/shared/atomic-json.d.ts +8 -0
  200. package/dist/shared/atomic-json.js +33 -0
  201. package/dist/shared/diagnosis-schema.d.ts +9 -0
  202. package/dist/shared/diagnosis-schema.js +181 -0
  203. package/dist/shared/doctor-report.d.ts +3 -0
  204. package/dist/shared/doctor-report.js +103 -0
  205. package/dist/shared/evaluation-job.d.ts +6 -0
  206. package/dist/shared/evaluation-job.js +217 -0
  207. package/dist/shared/executor-result.d.ts +17 -0
  208. package/dist/shared/executor-result.js +221 -0
  209. package/dist/shared/file-lock.d.ts +12 -0
  210. package/dist/shared/file-lock.js +129 -0
  211. package/dist/shared/json-value.d.ts +5 -0
  212. package/dist/shared/json-value.js +36 -0
  213. package/dist/shared/keyed-mutex.d.ts +7 -0
  214. package/dist/shared/keyed-mutex.js +24 -0
  215. package/dist/shared/record-count.d.ts +8 -0
  216. package/dist/shared/record-count.js +43 -0
  217. package/dist/shared/sample-contract.d.ts +3 -0
  218. package/dist/shared/sample-contract.js +332 -0
  219. package/dist/shared/timestamp.d.ts +6 -0
  220. package/dist/shared/timestamp.js +64 -0
  221. package/dist/shared/token-usage.d.ts +19 -0
  222. package/dist/shared/token-usage.js +50 -0
  223. package/dist/shared/tool-call-status.d.ts +8 -0
  224. package/dist/shared/tool-call-status.js +28 -0
  225. package/dist/shared/tool-identity.d.ts +21 -0
  226. package/dist/shared/tool-identity.js +84 -0
  227. package/dist/shared/tool-search.js +73 -16
  228. package/dist/shared/trace-projection.d.ts +5 -0
  229. package/dist/shared/trace-projection.js +20 -0
  230. package/dist/shared/trace-source-kind.d.ts +3 -0
  231. package/dist/shared/trace-source-kind.js +12 -0
  232. package/dist/types/diagnosis.d.ts +2 -0
  233. package/dist/types/eval.d.ts +4 -0
  234. package/dist/types/executor.d.ts +32 -5
  235. package/dist/types/index.d.ts +1 -0
  236. package/dist/types/index.js +1 -0
  237. package/dist/types/judge.d.ts +2 -0
  238. package/dist/types/observability.d.ts +116 -9
  239. package/dist/types/report.d.ts +58 -6
  240. package/dist/types/skill-index.d.ts +7 -0
  241. package/dist/types/trace.d.ts +2 -0
  242. package/dist/types/trace.js +1 -0
  243. package/package.json +9 -5
package/README.md CHANGED
@@ -8,11 +8,23 @@
8
8
 
9
9
  **English** | [简体中文](./README.zh.md)
10
10
 
11
- **Is this knowledge input (prompt / skill / RAG / agent) any good, and can you ship it with evidence?**
12
- `doctor` checks whether this knowledge input is coherent enough to measure; `eval` fixes the model and samples, changes only the knowledge input, and tells you whether the new version is genuinely better. Bootstrap CI and length-debias are on by default; Krippendorff α appears the moment you add a gold set.
11
+ **Stop editing LLM knowledge inputs by gut feel.**
12
+ `oh-my-knowledge` (omk) is a measurement workflow for prompts, RAG context, skills, agents, and workflows. It fixes the executor model and the evaluation samples, changes only the knowledge artifact, then answers the release question that matters: **can v2 ship, and where is it better?**
13
+
14
+ ![omk knowledge artifact evaluation flow: doctor / eval / observe / sample / evolve loop](./docs/public/omk-knowledge-flow-en-animated.gif)
13
15
 
14
16
  📖 **Full documentation: [oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** (searchable, English / 简体中文)
15
17
 
18
+ ## What omk makes measurable
19
+
20
+ | Decision | Command | Evidence you get |
21
+ |---|---|---|
22
+ | Is this artifact coherent enough to evaluate? | `omk doctor` | structure, dependencies, safety, and measurability checks |
23
+ | Is v2 actually better than v1? | `omk eval` | one-line verdict, confidence interval, failed samples, cost |
24
+ | Why did it pass or fail? | `omk studio` | report view with scores, diagnostics, and examples |
25
+ | Should this version become the accepted one? | `omk promote` / `omk evolve` | evidence-gated accept or generate a better candidate |
26
+ | What did real usage expose? | `omk observe` / `omk sample --from-traces` | production gaps drafted for review; reviewed drafts can become eval samples |
27
+
16
28
  ![omk report — verdict pill "v2 is clearly better than v1 — ready to ship"](./assets/screenshots/report-overview.png)
17
29
 
18
30
  ## Quick start
@@ -20,34 +32,45 @@
20
32
  ```bash
21
33
  npm i -g oh-my-knowledge
22
34
  omk init demo && cd demo
35
+ omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
23
36
  omk eval --control code-review-v1 --treatment code-review-v2
24
37
  ```
25
38
 
26
- Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
39
+ Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `--dry-run` previews calls and cost; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
40
+
41
+ Prerequisite: configure one authenticated model runtime (Codex CLI, Claude Code, or an API executor; see [Requirements](#requirements)). Inside a Codex task in the ChatGPT desktop app, omk automatically selects `codex`, reads the model from `~/.codex/config.toml`, and uses the same Codex model as the default judge. Claude is not required.
27
42
 
28
- Prerequisite: the default executor and judge use the `claude` CLI — install and log in first (see [Requirements](#requirements)); to use another model or run offline (no API key) see [executors](docs/reference/executors.md).
43
+ To make Codex the default in regular terminals, add the preference to your shell profile (for example `~/.zshrc`):
44
+
45
+ ```bash
46
+ export OMK_EXECUTOR=codex
47
+ # Optional: export OMK_MODEL="your-codex-model"
48
+ ```
49
+
50
+ Without `OMK_MODEL`, omk reads the model from `~/.codex/config.toml`. You can still pass `--executor codex --model <codex-model>` per command. Pass `--judge-models` or set `OMK_JUDGE_MODELS` only when you want a different judge.
29
51
 
30
52
  > The first run has only 3 cases, so the verdict will usually be `UNDERPOWERED` (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.
31
53
 
32
54
  > The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
33
55
 
34
- Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
56
+ Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users; includes demo → own skill → verdict actions). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
35
57
 
36
58
  Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
37
59
 
38
- ## The first workflow
60
+ ## The omk loop
39
61
 
40
- omk is primarily for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The first workflow is deliberately small:
62
+ omk is for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The main loop is deliberately controlled:
41
63
 
42
64
  ```text
43
- change a skill / prompt / agent artifact
44
- → run omk doctor to catch structure, dependency, and measurability problems
45
- → run omk eval to compare against a baseline on the same samples
46
- → read the report / Studio view for the next concrete fix
47
- decide ship / don't ship
65
+ change a prompt / RAG / skill / agent artifact
66
+ → run omk doctor before evaluation
67
+ → run omk eval with the same model and the same samples
68
+ → read the report / Studio evidence
69
+ promote a proven version or evolve a candidate
70
+ → observe real usage and draft gap-derived samples for review
48
71
  ```
49
72
 
50
- `observe` is the later production-feedback loop: useful once real usage traces exist, but not required for omk's first value. The trunk is the pre-ship doctor eval decision.
73
+ The first value is the pre-ship `doctor → eval` decision. The long-term value is the closed loop: `observe` surfaces production gaps, `sample --from-traces` drafts regression samples for human review, and reviewed drafts can become fixed eval samples that make the next `eval` harder to game.
51
74
 
52
75
  ## Use inside AI Coding Agents
53
76
 
@@ -73,7 +96,7 @@ These slash commands are natural-language entry points — the agent reads the c
73
96
 
74
97
  ### Use inside Codex
75
98
 
76
- Codex does not support Claude Code style `/omk ...` slash commands. Ask the agent to run the `omk` CLI directly:
99
+ Codex does not support Claude Code style `/omk ...` slash commands. Ask the agent to run the `omk` CLI directly. Inside a Codex task, omk automatically selects the Codex runtime and locally configured model:
77
100
 
78
101
  ```bash
79
102
  omk eval
@@ -83,11 +106,15 @@ omk sample skills/my-skill.md
83
106
 
84
107
  You can also describe the goal in natural language, such as "compare v1 vs v2" or "generate test cases for this skill".
85
108
 
109
+ `eval`, `doctor`, `sample`, `evolve`, and the LLM-enhanced observe review share the same runtime resolution. Once Codex is selected, the default judge reuses the evaluated Codex model instead of falling back to `claude:haiku`.
110
+
86
111
  > `omk evolve` is a one-shot loop: it runs the doctor gate first, auto-generates eval samples when the target skill has none, then self-iterates. For a brand-new skill, just run `omk evolve skills/foo.md`.
87
112
 
88
113
  ## Why this tool
89
114
 
90
- Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "can we ship v2, and why?", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
115
+ Knowledge engineering creates a versioning problem: every prompt, RAG recipe, skill, agent, or workflow can change behavior without changing application code. When someone asks "can we ship v2, and why?", a prettier answer or a higher anecdotal success rate is not enough.
116
+
117
+ omk treats the knowledge artifact as the variable under test: **same model, same evaluation samples, only the artifact changes.** That makes the comparison explainable, repeatable, and suitable for CI or release review.
91
118
 
92
119
  ## Why omk over alternatives
93
120
 
@@ -117,7 +144,7 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
117
144
  | **Statistical rigor** | Bootstrap CI / length-debias / saturation curve on by default; Krippendorff α auto-computed with a gold set. [Details →](docs/explanation/statistical-rigor.md) |
118
145
  | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage |
119
146
  | **LLM health audit** | `omk doctor` grades 7 builtin dimensions; repeats the audit (`--repeat`) and merges findings by k/n consensus |
120
- | **Production observability** | parse Claude Code session JSONL traces; measure per-skill failure rate / latency / cost / knowledge-gap signals |
147
+ | **Production observability** | normalize Codex, Claude Code, OpenClaw, and markdown logs into source-neutral Trace IR; measure per-skill outcomes / latency / token use / knowledge-gap signals |
121
148
  | **Knowledge-gap detection** | severity-weighted signals quantify risk exposure instead of claiming completeness |
122
149
  | **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against |
123
150
  | **Git & remote sources** | install / eval from a local git ref or a remote git URL (`--git-url`); directory-skills run in a content-addressed **isolated copy** so `references/` assets are real measured input, not just `SKILL.md` |
@@ -150,14 +177,19 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
150
177
 
151
178
  | Variable | Description |
152
179
  |---|---|
180
+ | `OMK_EXECUTOR` | default executor preference, e.g. `codex` / `codex-sdk` / `claude` |
181
+ | `OMK_MODEL` | default evaluated model; Codex reads local `config.toml` when unset |
182
+ | `OMK_JUDGE_MODELS` | default judge list in `executor:model[,...]` format |
153
183
  | `CCV_PROXY_URL` | proxy requests through cc-viewer for live eval-traffic visualization |
154
184
  | `OMK_REPORT_PORT` | report server port (default: 7799) |
155
185
 
156
186
  ## Requirements
157
187
 
158
188
  - Node.js >= 22
159
- - `claude` CLI (for the default executor and LLM judge; see [Claude Code](https://claude.ai/code))
160
- - not needed if you use other executors (openai-api / anthropic-api / gemini) with `--no-judge`
189
+ - At least one authenticated model runtime:
190
+ - Codex: install and authenticate the Codex CLI (`npm i -g @openai/codex`); Codex tasks in the ChatGPT desktop app select it automatically
191
+ - Claude: install and authenticate [Claude Code](https://claude.ai/code)
192
+ - API / other executors: configure them as described in [Executors](docs/reference/executors.md)
161
193
 
162
194
  ## Security notice
163
195
 
package/README.zh.md CHANGED
@@ -8,11 +8,23 @@
8
8
 
9
9
  [English](./README.md) | **简体中文**
10
10
 
11
- **一段知识输入(prompt / skill / RAG / agent)好不好,能不能有证据地发布?**
12
- `doctor` 先检查这类知识输入是否清楚到值得测;`eval` 固定模型和用例,只改变知识输入,判断新版本是否真的更好。Bootstrap 置信区间、长度去偏默认开,配一份人工 gold 就自动算 Krippendorff α。
11
+ **别再凭感觉改 LLM 知识输入。**
12
+ `oh-my-knowledge`(omk)是一套面向 prompt、RAG、skill、agent、workflow 的测量工作流。它固定执行模型和评测用例,只改变知识载体,然后回答发布时真正要追问的问题:**v2 能不能发,好在哪里?**
13
+
14
+ ![omk 知识载体评测流程:doctor / eval / observe / sample / evolve 闭环](./docs/public/omk-knowledge-flow-animated.gif)
13
15
 
14
16
  📖 **完整文档:[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)**(可搜索,可切换英文)
15
17
 
18
+ ## omk 让什么变得可测量
19
+
20
+ | 决策问题 | 命令 | 你会得到的证据 |
21
+ |------|------|------|
22
+ | 这份知识载体是否清楚到值得评测? | `omk doctor` | 结构、依赖、安全性、可测性检查 |
23
+ | v2 是否真的优于 v1? | `omk eval` | 一行 verdict、置信区间、失败样本、成本 |
24
+ | 它为什么通过或失败? | `omk studio` | 分数、诊断、样本证据的报告视图 |
25
+ | 这个版本是否应成为接受版本? | `omk promote` / `omk evolve` | 基于证据接受,或生成更好的候选版 |
26
+ | 真实使用暴露了哪些知识缺口? | `omk observe` / `omk sample --from-traces` | 将线上缺口生成待复核草稿,复核后再沉淀为评测样本 |
27
+
16
28
  ![omk 报告 — verdict pill「v2 明显优于 v1,可以发布」](./assets/screenshots/report-overview-zh.png)
17
29
 
18
30
  ## 快速开始
@@ -20,34 +32,45 @@
20
32
  ```bash
21
33
  npm i -g oh-my-knowledge
22
34
  omk init demo && cd demo
35
+ omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
23
36
  omk eval --control code-review-v1 --treatment code-review-v2
24
37
  ```
25
38
 
26
- 开箱即跑:`omk init` 脚手架好两版 skill 和三条评测用例,不用先改任何文件,`omk eval` 跑控制变量 A/B,约 5 分钟出 HTML 报告 + 一行 verdict;跑通后再把 skill 和用例换成你自己的。
39
+ 开箱即跑:`omk init` 脚手架好两版 skill 和三条评测用例,不用先改任何文件;`--dry-run` 预览调用次数和成本;`omk eval` 跑控制变量 A/B,约 5 分钟出 HTML 报告 + 一行 verdict。跑通后再把 skill 和用例换成你自己的。
40
+
41
+ 前置:准备一个已认证的模型 runtime(Codex CLI、Claude Code 或 API 执行器,见[系统要求](#系统要求))。在 ChatGPT desktop 的 Codex 任务里,omk 会自动使用 `codex`,从 `~/.codex/config.toml` 读取模型,并默认用同一个 Codex 模型担任评委,不依赖 Claude。
42
+
43
+ 普通终端想固定使用 Codex,可以把偏好加入 shell 配置,例如 `~/.zshrc`:
27
44
 
28
- 前置:默认执行器与评委用 `claude` CLI,需先安装并登录(见[系统要求](#系统要求));想用别的模型或离线跑(无需 API key)见[执行器](docs/zh/reference/executors.md)。
45
+ ```bash
46
+ export OMK_EXECUTOR=codex
47
+ # 可选:export OMK_MODEL="你的 Codex 模型"
48
+ ```
49
+
50
+ 不设置 `OMK_MODEL` 时,omk 会读取 `~/.codex/config.toml` 的模型。也可以继续逐次显式传 `--executor codex --model <codex-model>`。自定义评委时再传 `--judge-models` 或设置 `OMK_JUDGE_MODELS`。
29
51
 
30
52
  > 首跑只有 3 条用例,verdict 多半是「数据不足(UNDERPOWERED)」——这是正常起点而非出错;把用例加到约 20 条以上,再看「可发布」结论。
31
53
 
32
54
  > 命令行有新版本时会自动提示(每 20 小时最多一次);想永久关闭该提醒,设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
33
55
 
34
- 手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)(推荐第一次跑评测的用户)。更多可跑示例(Skill Map、A/B、离线执行器、agent runtime、RAG)见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
56
+ 手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)(推荐第一次跑评测的用户,覆盖 demo → 自己的 skill → verdict 动作)。更多可跑示例(Skill Map、A/B、离线执行器、agent runtime、RAG)见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
35
57
 
36
- 深入:[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [artifact 布局](docs/zh/reference/artifact-layout.md)
58
+ 深入:[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [知识载体布局](docs/zh/reference/artifact-layout.md)
37
59
 
38
- ## 第一条工作流
60
+ ## omk 的闭环
39
61
 
40
- omk 主要给 LLM 知识载体的作者 / 维护者用,帮他们做发布判断;它不是给被动安装 skill 的普通使用者用的。第一条工作流应该很短:
62
+ omk 主要给 LLM 知识载体的作者 / 维护者用,帮他们做发布判断;它不是给被动安装 skill 的普通使用者用的。主流程刻意保持受控:
41
63
 
42
64
  ```text
43
- 改了一个 skill / prompt / agent artifact
44
- omk doctor,先抓结构、依赖和可测性问题
45
- omk eval,在同一批用例上和 baseline 对比
46
- 看报告 / Studio,知道下一步具体该改哪里
47
- 决定发布 / 不发布
65
+ 改了一份 prompt / RAG / skill / agent 知识载体
66
+ 先跑 omk doctor
67
+ 用相同模型、相同评测用例跑 omk eval
68
+ report / Studio 里的证据
69
+ 证据足够则 promote,证据不足则 evolve 候选版
70
+ → observe 真实使用,把缺口生成待复核评测草稿
48
71
  ```
49
72
 
50
- `observe` 是后续的生产反馈闭环:等真实使用 trace 存在后很有价值,但不是 omk 第一价值的前提。主干先是发布前的 doctor eval 判断。
73
+ 第一价值是发布前的 `doctor eval` 判断。长期价值是闭环:`observe` 暴露真实使用里的知识缺口,`sample --from-traces` 先生成待人工复核的评测用例草稿,复核后的草稿再沉淀为固定评测样本,下一次 `eval` 就更难被偶然样本骗过。
51
74
 
52
75
  ## 在 AI Coding Agent 中使用
53
76
 
@@ -64,16 +87,16 @@ omk install omk-agent-skill
64
87
  当 `omk` skill 已在 Claude Code 中可用时,可以直接这样调用:
65
88
 
66
89
  ```bash
67
- /omk eval # 评测当前项目的 artifact
90
+ /omk eval # 评测当前项目的知识载体
68
91
  /omk evolve # 多轮自动迭代改进 skill
69
92
  /omk sample # 生成或补齐评测用例
70
93
  ```
71
94
 
72
- 这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个 artifact」,omk 会自动理解意图并调用对应命令。
95
+ 这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个知识载体」,omk 会自动理解意图并调用对应命令。
73
96
 
74
97
  ### 在 Codex 中使用
75
98
 
76
- Codex 默认不支持 `/omk ...` 这种 Claude Code 风格的 slash command。通常直接让 agent 执行 `omk` CLI,例如:
99
+ Codex 默认不支持 `/omk ...` 这种 Claude Code 风格的 slash command。直接让 agent 执行 `omk` CLI 即可;在 Codex 任务里,omk 会自动选择 Codex runtime 和本机配置的模型:
77
100
 
78
101
  ```bash
79
102
  omk eval
@@ -83,11 +106,15 @@ omk sample skills/my-skill.md
83
106
 
84
107
  也可以直接用自然语言描述目标,例如「比较 v1 和 v2 的评测差异」、「为这个 skill 生成评测用例」。
85
108
 
109
+ `eval`、`doctor`、`sample`、`evolve` 和 observe 的 LLM 增强复盘共用同一套 runtime 解析。Codex 被选中后,默认评委沿用被测 Codex 模型,不会回落到 `claude:haiku`。
110
+
86
111
  > `omk evolve` 是一键闭环:默认先跑 doctor 体检,目标 skill 没有评测用例时会自动生成一批,再进入多轮自迭代。全新 skill 直接 `omk evolve skills/foo.md` 即可。
87
112
 
88
113
  ## 为什么需要这个工具
89
114
 
90
- 做知识工程的团队会产出大量知识载体(当前常见是 skill,也包括 prompt、agent、workflow 等)。当被问到「v2 能不能发、为什么」时,需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题:**相同模型、相同评测用例,只改变知识载体。**
115
+ 知识工程带来的是一个版本治理问题:prompt、RAG 配方、skill、agent、workflow 都会改变模型行为,但这些改动未必体现在应用代码里。当有人追问「v2 能不能发、为什么」时,回答更顺眼、体感更好,远远不够。
116
+
117
+ omk 把知识载体当作被测变量:**相同模型、相同评测用例,只改变知识载体。** 这样得到的对比才可解释、可复跑,也适合进入 CI 或发布评审。
91
118
 
92
119
  ## 为什么选 omk
93
120
 
@@ -117,7 +144,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
117
144
  | **统计严谨性** | Bootstrap CI / 长度去偏 / 饱和曲线默认开,Krippendorff α 提供 gold 集即自动计算。[详情 →](docs/zh/explanation/statistical-rigor.md) |
118
145
  | **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
119
146
  | **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分;重复采样(`--repeat`)+ k/n 共识归并 |
120
- | **线上 session 观测** | 解析 Claude Code session JSONL,测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
147
+ | **线上 session 观测** | Codex、Claude Code、OpenClaw markdown 日志统一为 source-neutral Trace IR,测量各 skill 的执行结果、耗时、token 使用和知识缺口信号 |
121
148
  | **知识缺口识别** | 严重度加权的信号量化风险敞口,不宣称完备性 |
122
149
  | **用例隔离 (construct validity)** | `--strict-baseline`(默认开)三堵 baseline 拿到被测 skill 的污染路径 |
123
150
  | **Git / 远端源** | install / eval 支持本地 git ref 或远端 git URL(`--git-url`);目录-skill 在内容寻址**隔离副本**里执行,`references/` 资产是真实测量输入,不只是 `SKILL.md` |
@@ -127,7 +154,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
127
154
  | **多轮方差分析** | `--repeat N` 重复 N 次,计算均值/标准差/置信区间/t 检验 |
128
155
  | **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
129
156
  | **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
130
- | **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹、judge prompt hash |
157
+ | **可追溯性** | 报告含 CLI 版本、Node 版本、知识载体版本指纹、judge prompt hash |
131
158
  | **中英切换** | HTML 报告右上角一键切换语言 |
132
159
 
133
160
  ## 文档
@@ -137,7 +164,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
137
164
  - **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
138
165
  - **[评测用例格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、30+ 断言类型、自定义 JS 断言
139
166
  - **[CLI 参考](docs/zh/reference/cli.md)** —— 顶层命令的 bash 示例和 flag 表
140
- - **[执行器](docs/zh/reference/executors.md)** & **[artifact 布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器;variant 如何解析为 artifact + runtime context
167
+ - **[执行器](docs/zh/reference/executors.md)** & **[知识载体布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器;variant 如何解析为 artifact + runtime context
141
168
  - **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)(项目 runtime context)与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)(GLM / 通义 / DeepSeek / Moonshot / Ollama)
142
169
  - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
143
170
  - **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例,按由简到全排成上手路径
@@ -150,14 +177,19 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
150
177
 
151
178
  | 变量 | 说明 |
152
179
  |------|------|
180
+ | `OMK_EXECUTOR` | 默认执行器偏好,例如 `codex` / `codex-sdk` / `claude` |
181
+ | `OMK_MODEL` | 默认被测模型;Codex 未设置时读取本机 `config.toml` |
182
+ | `OMK_JUDGE_MODELS` | 默认评委列表,格式 `executor:model[,...]` |
153
183
  | `CCV_PROXY_URL` | 通过 cc-viewer 代理请求,实时可视化评测流量 |
154
184
  | `OMK_REPORT_PORT` | 报告服务端口(默认 7799) |
155
185
 
156
186
  ## 系统要求
157
187
 
158
188
  - Node.js >= 22
159
- - `claude` CLI(默认执行器和 LLM 评委需要,参考 [Claude Code](https://claude.ai/code))
160
- - 如果使用其它执行器(openai-api / anthropic-api / gemini)+ `--no-judge` 则可不需要
189
+ - 至少一个已认证的模型 runtime:
190
+ - Codex:安装并登录 Codex CLI(`npm i -g @openai/codex`);ChatGPT desktop Codex 任务会自动选择它
191
+ - Claude:安装并登录 [Claude Code](https://claude.ai/code)
192
+ - API / 其它执行器:按[执行器文档](docs/zh/reference/executors.md)配置
161
193
 
162
194
  ## 安全说明
163
195
 
@@ -5,6 +5,7 @@
5
5
  import type { ToolCallInfo, ResultEntry, Report } from '../types/index.js';
6
6
  export interface KnowledgeEntry {
7
7
  path: string;
8
+ aliases?: string[];
8
9
  type: 'principle' | 'semantic' | 'design' | 'script' | 'code' | 'other';
9
10
  lineCount?: number;
10
11
  }