oh-my-knowledge 0.48.0 → 0.50.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -18
- package/README.zh.md +55 -23
- package/dist/analysis/coverage-analyzer.d.ts +1 -0
- package/dist/analysis/coverage-analyzer.js +125 -62
- package/dist/analysis/failure-clusterer.js +2 -1
- package/dist/analysis/gap-analyzer.d.ts +2 -2
- package/dist/analysis/gap-analyzer.js +13 -3
- package/dist/analysis/hedging-classifier.d.ts +2 -2
- package/dist/analysis/hedging-classifier.js +3 -4
- package/dist/analysis/report-diagnostics.js +9 -7
- package/dist/analysis/sample-diagnostics.js +6 -6
- package/dist/artifact-graph/doctor.js +15 -7
- package/dist/assets/agent-skills/omk/SKILL.md +27 -7
- package/dist/assets/agent-skills/omk/references/commands.md +20 -19
- package/dist/authoring/evolver.d.ts +10 -6
- package/dist/authoring/evolver.js +496 -83
- package/dist/authoring/generator.d.ts +3 -3
- package/dist/authoring/generator.js +5 -10
- package/dist/authoring/sample-fixer.d.ts +8 -6
- package/dist/authoring/sample-fixer.js +76 -5
- package/dist/cli/commands/doctor.js +31 -14
- package/dist/cli/commands/eval/index.d.ts +3 -0
- package/dist/cli/commands/eval/index.js +163 -17
- package/dist/cli/commands/evolve.d.ts +4 -4
- package/dist/cli/commands/evolve.js +27 -13
- package/dist/cli/commands/init.js +16 -3
- package/dist/cli/commands/observe/inbox.js +30 -23
- package/dist/cli/commands/observe/index.js +20 -11
- package/dist/cli/commands/observe/ingest.d.ts +3 -0
- package/dist/cli/commands/observe/ingest.js +31 -3
- package/dist/cli/commands/sample.d.ts +6 -3
- package/dist/cli/commands/sample.js +75 -71
- package/dist/cli/lib/codex-model-hint.d.ts +9 -0
- package/dist/cli/lib/codex-model-hint.js +45 -0
- package/dist/cli/lib/generation-failure-hint.d.ts +2 -0
- package/dist/cli/lib/generation-failure-hint.js +61 -0
- package/dist/cli/lib/i18n-dict/common.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/common.js +4 -0
- package/dist/cli/lib/i18n-dict/gen.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/gen.js +38 -6
- package/dist/cli/lib/i18n-dict/help.js +6 -6
- package/dist/cli/lib/i18n-dict/init.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/init.js +13 -9
- package/dist/cli/lib/i18n-dict/run.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/run.js +34 -2
- package/dist/cli/lib/llm-failure-classifier.d.ts +2 -0
- package/dist/cli/lib/llm-failure-classifier.js +8 -0
- package/dist/cli/lib/parse-run-config.d.ts +6 -5
- package/dist/cli/lib/parse-run-config.js +16 -9
- package/dist/cli/lib/runtime-defaults.d.ts +21 -0
- package/dist/cli/lib/runtime-defaults.js +79 -0
- package/dist/diagnosis/observe-mapper.js +14 -15
- package/dist/diagnosis/observe-producer.js +3 -1
- package/dist/diagnosis/studio-projection.js +14 -7
- package/dist/diagnosis/types.d.ts +2 -0
- package/dist/diagnosis/types.js +12 -0
- package/dist/doctor/endpoint-rule.js +2 -1
- package/dist/eval-core/artifact-file-names.js +18 -1
- package/dist/eval-core/artifact-index.d.ts +7 -11
- package/dist/eval-core/artifact-index.js +139 -80
- package/dist/eval-core/cache.d.ts +12 -3
- package/dist/eval-core/cache.js +89 -29
- package/dist/eval-core/comparability.js +10 -6
- package/dist/eval-core/evaluation-execution.d.ts +2 -1
- package/dist/eval-core/evaluation-execution.js +122 -37
- package/dist/eval-core/evaluation-job.d.ts +4 -1
- package/dist/eval-core/evaluation-job.js +4 -1
- package/dist/eval-core/evaluation-reporting.d.ts +15 -13
- package/dist/eval-core/evaluation-reporting.js +54 -52
- package/dist/eval-core/execution-strategy.d.ts +2 -0
- package/dist/eval-core/execution-strategy.js +11 -9
- package/dist/eval-core/fact-checker.js +15 -7
- package/dist/eval-core/holdout.js +3 -2
- package/dist/eval-core/judge-independence.d.ts +2 -2
- package/dist/eval-core/mock-hook.cjs +23 -6
- package/dist/eval-core/mocks-runtime.js +30 -8
- package/dist/eval-core/report-document.d.ts +12 -0
- package/dist/eval-core/report-document.js +1151 -0
- package/dist/eval-core/report-extensions.d.ts +4 -0
- package/dist/eval-core/report-extensions.js +500 -0
- package/dist/eval-core/report-file-migration.js +7 -2
- package/dist/eval-core/resume-compatibility.d.ts +31 -0
- package/dist/eval-core/resume-compatibility.js +141 -0
- package/dist/eval-core/sample-fingerprint.d.ts +12 -0
- package/dist/eval-core/sample-fingerprint.js +193 -0
- package/dist/eval-core/schema.js +86 -31
- package/dist/eval-core/verdict.d.ts +8 -4
- package/dist/eval-core/verdict.js +24 -10
- package/dist/eval-workflows/batch-evaluation-workflow.d.ts +2 -1
- package/dist/eval-workflows/batch-evaluation-workflow.js +25 -12
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts +10 -5
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js +58 -21
- package/dist/eval-workflows/evaluation-pipeline/report-finalize.js +3 -1
- package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts +4 -1
- package/dist/eval-workflows/evaluation-pipeline/run-state.js +4 -1
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts +6 -5
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js +17 -10
- package/dist/eval-workflows/evaluation-pipeline.js +12 -7
- package/dist/eval-workflows/run-evaluation.d.ts +9 -7
- package/dist/eval-workflows/run-evaluation.js +79 -51
- package/dist/executors/anthropic-api.js +65 -9
- package/dist/executors/claude-cli.js +16 -79
- package/dist/executors/claude-protocol.d.ts +28 -0
- package/dist/executors/claude-protocol.js +180 -0
- package/dist/executors/claude-sdk-trace.js +56 -28
- package/dist/executors/claude-sdk.d.ts +1 -0
- package/dist/executors/claude-sdk.js +39 -93
- package/dist/executors/codex-cli-trace.js +166 -31
- package/dist/executors/codex-cli.d.ts +6 -8
- package/dist/executors/codex-cli.js +49 -151
- package/dist/executors/codex-protocol.d.ts +24 -0
- package/dist/executors/codex-protocol.js +234 -0
- package/dist/executors/codex-sdk.js +68 -120
- package/dist/executors/gemini.js +88 -13
- package/dist/executors/index.d.ts +2 -3
- package/dist/executors/index.js +5 -3
- package/dist/executors/openai-api.js +70 -9
- package/dist/executors/runtime-fingerprint.js +88 -11
- package/dist/executors/script-command.d.ts +8 -0
- package/dist/executors/script-command.js +87 -0
- package/dist/executors/script.js +202 -29
- package/dist/executors/shared.d.ts +35 -3
- package/dist/executors/shared.js +113 -15
- package/dist/grading/assertions.d.ts +1 -1
- package/dist/grading/assertions.js +19 -9
- package/dist/grading/diagnostic.d.ts +9 -2
- package/dist/grading/diagnostic.js +25 -2
- package/dist/grading/index.js +10 -4
- package/dist/grading/judge.js +19 -6
- package/dist/grading/layered-scores.d.ts +2 -3
- package/dist/grading/layered-scores.js +2 -3
- package/dist/inputs/load-samples.d.ts +1 -2
- package/dist/inputs/load-samples.js +23 -1
- package/dist/inputs/mcp-resolver.js +6 -3
- package/dist/inputs/sample-document.d.ts +11 -0
- package/dist/inputs/sample-document.js +96 -0
- package/dist/managed/evidence.d.ts +1 -0
- package/dist/managed/evidence.js +1 -1
- package/dist/managed/store.js +200 -91
- package/dist/observability/codex-trace-adapter.d.ts +5 -0
- package/dist/observability/codex-trace-adapter.js +850 -0
- package/dist/observability/experience.d.ts +32 -6
- package/dist/observability/experience.js +2779 -472
- package/dist/observability/feedback-matchers.js +16 -1
- package/dist/observability/inbox-view-model.d.ts +1 -1
- package/dist/observability/inbox-view-model.js +19 -14
- package/dist/observability/inbox.d.ts +7 -1
- package/dist/observability/inbox.js +632 -124
- package/dist/observability/problem-patterns.js +2 -0
- package/dist/observability/review-state.d.ts +6 -0
- package/dist/observability/review-state.js +235 -63
- package/dist/observability/skill-chain-advisories.js +1 -1
- package/dist/observability/skill-chain.js +17 -4
- package/dist/observability/skill-health-analyzer.d.ts +32 -7
- package/dist/observability/skill-health-analyzer.js +194 -121
- package/dist/observability/skill-health-report.d.ts +10 -0
- package/dist/observability/skill-health-report.js +620 -0
- package/dist/observability/soft-standards/constants.d.ts +0 -1
- package/dist/observability/soft-standards/constants.js +0 -1
- package/dist/observability/soft-standards/index.d.ts +1 -1
- package/dist/observability/soft-standards/index.js +1 -1
- package/dist/observability/soft-standards/llm-extractor.js +8 -10
- package/dist/observability/soft-standards/skill-standards-store.d.ts +2 -1
- package/dist/observability/soft-standards/skill-standards-store.js +59 -18
- package/dist/observability/soft-standards/types.d.ts +2 -2
- package/dist/observability/trace-adapter.d.ts +12 -7
- package/dist/observability/trace-adapter.js +11 -9
- package/dist/observability/trace-attribution.d.ts +13 -5
- package/dist/observability/trace-attribution.js +315 -21
- package/dist/observability/trace-ingestion.d.ts +9 -0
- package/dist/observability/trace-ingestion.js +80 -0
- package/dist/observability/trace-ir.d.ts +113 -0
- package/dist/observability/trace-ir.js +87 -0
- package/dist/observability/trace-segmenter.d.ts +19 -6
- package/dist/observability/trace-segmenter.js +377 -196
- package/dist/observability/trace-session-index.d.ts +19 -0
- package/dist/observability/trace-session-index.js +68 -0
- package/dist/observability/trace-source.d.ts +12 -4
- package/dist/observability/trace-source.js +939 -215
- package/dist/renderer/html-renderer.js +37 -6
- package/dist/renderer/icons.js +3 -0
- package/dist/renderer/observation-inbox-renderer.js +208 -90
- package/dist/renderer/skill-detail-renderer.js +452 -109
- package/dist/renderer/skill-health-renderer.js +69 -12
- package/dist/renderer/summary.js +28 -7
- package/dist/renderer/table.js +21 -4
- package/dist/renderer/test-view.d.ts +1 -0
- package/dist/renderer/test-view.js +44 -9
- package/dist/server/indexed-report-store.js +14 -18
- package/dist/server/job-store.js +64 -26
- package/dist/server/report-server.js +190 -78
- package/dist/server/report-store.js +57 -80
- package/dist/server/skill-index.js +143 -49
- package/dist/server/skill-insights.js +44 -5
- package/dist/shared/artifact-graph.d.ts +3 -0
- package/dist/shared/artifact-graph.js +224 -0
- package/dist/shared/assertion-types.d.ts +8 -0
- package/dist/shared/assertion-types.js +46 -0
- package/dist/shared/atomic-json.d.ts +8 -0
- package/dist/shared/atomic-json.js +33 -0
- package/dist/shared/diagnosis-schema.d.ts +9 -0
- package/dist/shared/diagnosis-schema.js +181 -0
- package/dist/shared/doctor-report.d.ts +3 -0
- package/dist/shared/doctor-report.js +103 -0
- package/dist/shared/evaluation-job.d.ts +6 -0
- package/dist/shared/evaluation-job.js +217 -0
- package/dist/shared/executor-result.d.ts +17 -0
- package/dist/shared/executor-result.js +221 -0
- package/dist/shared/file-lock.d.ts +12 -0
- package/dist/shared/file-lock.js +129 -0
- package/dist/shared/json-value.d.ts +5 -0
- package/dist/shared/json-value.js +36 -0
- package/dist/shared/keyed-mutex.d.ts +7 -0
- package/dist/shared/keyed-mutex.js +24 -0
- package/dist/shared/record-count.d.ts +8 -0
- package/dist/shared/record-count.js +43 -0
- package/dist/shared/sample-contract.d.ts +3 -0
- package/dist/shared/sample-contract.js +332 -0
- package/dist/shared/timestamp.d.ts +6 -0
- package/dist/shared/timestamp.js +64 -0
- package/dist/shared/token-usage.d.ts +19 -0
- package/dist/shared/token-usage.js +50 -0
- package/dist/shared/tool-call-status.d.ts +8 -0
- package/dist/shared/tool-call-status.js +28 -0
- package/dist/shared/tool-identity.d.ts +21 -0
- package/dist/shared/tool-identity.js +84 -0
- package/dist/shared/tool-search.js +73 -16
- package/dist/shared/trace-projection.d.ts +5 -0
- package/dist/shared/trace-projection.js +20 -0
- package/dist/shared/trace-source-kind.d.ts +3 -0
- package/dist/shared/trace-source-kind.js +12 -0
- package/dist/types/diagnosis.d.ts +2 -0
- package/dist/types/eval.d.ts +4 -0
- package/dist/types/executor.d.ts +32 -5
- package/dist/types/index.d.ts +1 -0
- package/dist/types/index.js +1 -0
- package/dist/types/judge.d.ts +2 -0
- package/dist/types/observability.d.ts +116 -9
- package/dist/types/report.d.ts +58 -6
- package/dist/types/skill-index.d.ts +7 -0
- package/dist/types/trace.d.ts +2 -0
- package/dist/types/trace.js +1 -0
- package/package.json +9 -5
package/README.md
CHANGED
|
@@ -8,11 +8,23 @@
|
|
|
8
8
|
|
|
9
9
|
**English** | [简体中文](./README.zh.md)
|
|
10
10
|
|
|
11
|
-
**
|
|
12
|
-
`
|
|
11
|
+
**Stop editing LLM knowledge inputs by gut feel.**
|
|
12
|
+
`oh-my-knowledge` (omk) is a measurement workflow for prompts, RAG context, skills, agents, and workflows. It fixes the executor model and the evaluation samples, changes only the knowledge artifact, then answers the release question that matters: **can v2 ship, and where is it better?**
|
|
13
|
+
|
|
14
|
+

|
|
13
15
|
|
|
14
16
|
📖 **Full documentation: [oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** (searchable, English / 简体中文)
|
|
15
17
|
|
|
18
|
+
## What omk makes measurable
|
|
19
|
+
|
|
20
|
+
| Decision | Command | Evidence you get |
|
|
21
|
+
|---|---|---|
|
|
22
|
+
| Is this artifact coherent enough to evaluate? | `omk doctor` | structure, dependencies, safety, and measurability checks |
|
|
23
|
+
| Is v2 actually better than v1? | `omk eval` | one-line verdict, confidence interval, failed samples, cost |
|
|
24
|
+
| Why did it pass or fail? | `omk studio` | report view with scores, diagnostics, and examples |
|
|
25
|
+
| Should this version become the accepted one? | `omk promote` / `omk evolve` | evidence-gated accept or generate a better candidate |
|
|
26
|
+
| What did real usage expose? | `omk observe` / `omk sample --from-traces` | production gaps drafted for review; reviewed drafts can become eval samples |
|
|
27
|
+
|
|
16
28
|

|
|
17
29
|
|
|
18
30
|
## Quick start
|
|
@@ -20,34 +32,45 @@
|
|
|
20
32
|
```bash
|
|
21
33
|
npm i -g oh-my-knowledge
|
|
22
34
|
omk init demo && cd demo
|
|
35
|
+
omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
|
|
23
36
|
omk eval --control code-review-v1 --treatment code-review-v2
|
|
24
37
|
```
|
|
25
38
|
|
|
26
|
-
Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
|
|
39
|
+
Runs out of the box — no edits needed first. `omk init` scaffolds two skill variants and three sample cases; `--dry-run` previews calls and cost; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.
|
|
40
|
+
|
|
41
|
+
Prerequisite: configure one authenticated model runtime (Codex CLI, Claude Code, or an API executor; see [Requirements](#requirements)). Inside a Codex task in the ChatGPT desktop app, omk automatically selects `codex`, reads the model from `~/.codex/config.toml`, and uses the same Codex model as the default judge. Claude is not required.
|
|
27
42
|
|
|
28
|
-
|
|
43
|
+
To make Codex the default in regular terminals, add the preference to your shell profile (for example `~/.zshrc`):
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
export OMK_EXECUTOR=codex
|
|
47
|
+
# Optional: export OMK_MODEL="your-codex-model"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Without `OMK_MODEL`, omk reads the model from `~/.codex/config.toml`. You can still pass `--executor codex --model <codex-model>` per command. Pass `--judge-models` or set `OMK_JUDGE_MODELS` only when you want a different judge.
|
|
29
51
|
|
|
30
52
|
> The first run has only 3 cases, so the verdict will usually be `UNDERPOWERED` (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.
|
|
31
53
|
|
|
32
54
|
> The CLI notifies you when a newer version is available (at most once per 20h); set `OMK_SKIP_UPDATE_CHECK=1` to silence it permanently.
|
|
33
55
|
|
|
34
|
-
Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
|
|
56
|
+
Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users; includes demo → own skill → verdict actions). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's [example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples).
|
|
35
57
|
|
|
36
58
|
Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
|
|
37
59
|
|
|
38
|
-
## The
|
|
60
|
+
## The omk loop
|
|
39
61
|
|
|
40
|
-
omk is
|
|
62
|
+
omk is for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The main loop is deliberately controlled:
|
|
41
63
|
|
|
42
64
|
```text
|
|
43
|
-
change a
|
|
44
|
-
→ run omk doctor
|
|
45
|
-
→ run omk eval
|
|
46
|
-
→ read the report / Studio
|
|
47
|
-
→
|
|
65
|
+
change a prompt / RAG / skill / agent artifact
|
|
66
|
+
→ run omk doctor before evaluation
|
|
67
|
+
→ run omk eval with the same model and the same samples
|
|
68
|
+
→ read the report / Studio evidence
|
|
69
|
+
→ promote a proven version or evolve a candidate
|
|
70
|
+
→ observe real usage and draft gap-derived samples for review
|
|
48
71
|
```
|
|
49
72
|
|
|
50
|
-
`
|
|
73
|
+
The first value is the pre-ship `doctor → eval` decision. The long-term value is the closed loop: `observe` surfaces production gaps, `sample --from-traces` drafts regression samples for human review, and reviewed drafts can become fixed eval samples that make the next `eval` harder to game.
|
|
51
74
|
|
|
52
75
|
## Use inside AI Coding Agents
|
|
53
76
|
|
|
@@ -73,7 +96,7 @@ These slash commands are natural-language entry points — the agent reads the c
|
|
|
73
96
|
|
|
74
97
|
### Use inside Codex
|
|
75
98
|
|
|
76
|
-
Codex does not support Claude Code style `/omk ...` slash commands. Ask the agent to run the `omk` CLI directly:
|
|
99
|
+
Codex does not support Claude Code style `/omk ...` slash commands. Ask the agent to run the `omk` CLI directly. Inside a Codex task, omk automatically selects the Codex runtime and locally configured model:
|
|
77
100
|
|
|
78
101
|
```bash
|
|
79
102
|
omk eval
|
|
@@ -83,11 +106,15 @@ omk sample skills/my-skill.md
|
|
|
83
106
|
|
|
84
107
|
You can also describe the goal in natural language, such as "compare v1 vs v2" or "generate test cases for this skill".
|
|
85
108
|
|
|
109
|
+
`eval`, `doctor`, `sample`, `evolve`, and the LLM-enhanced observe review share the same runtime resolution. Once Codex is selected, the default judge reuses the evaluated Codex model instead of falling back to `claude:haiku`.
|
|
110
|
+
|
|
86
111
|
> `omk evolve` is a one-shot loop: it runs the doctor gate first, auto-generates eval samples when the target skill has none, then self-iterates. For a brand-new skill, just run `omk evolve skills/foo.md`.
|
|
87
112
|
|
|
88
113
|
## Why this tool
|
|
89
114
|
|
|
90
|
-
|
|
115
|
+
Knowledge engineering creates a versioning problem: every prompt, RAG recipe, skill, agent, or workflow can change behavior without changing application code. When someone asks "can we ship v2, and why?", a prettier answer or a higher anecdotal success rate is not enough.
|
|
116
|
+
|
|
117
|
+
omk treats the knowledge artifact as the variable under test: **same model, same evaluation samples, only the artifact changes.** That makes the comparison explainable, repeatable, and suitable for CI or release review.
|
|
91
118
|
|
|
92
119
|
## Why omk over alternatives
|
|
93
120
|
|
|
@@ -117,7 +144,7 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
|
|
|
117
144
|
| **Statistical rigor** | Bootstrap CI / length-debias / saturation curve on by default; Krippendorff α auto-computed with a gold set. [Details →](docs/explanation/statistical-rigor.md) |
|
|
118
145
|
| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage |
|
|
119
146
|
| **LLM health audit** | `omk doctor` grades 7 builtin dimensions; repeats the audit (`--repeat`) and merges findings by k/n consensus |
|
|
120
|
-
| **Production observability** |
|
|
147
|
+
| **Production observability** | normalize Codex, Claude Code, OpenClaw, and markdown logs into source-neutral Trace IR; measure per-skill outcomes / latency / token use / knowledge-gap signals |
|
|
121
148
|
| **Knowledge-gap detection** | severity-weighted signals quantify risk exposure instead of claiming completeness |
|
|
122
149
|
| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against |
|
|
123
150
|
| **Git & remote sources** | install / eval from a local git ref or a remote git URL (`--git-url`); directory-skills run in a content-addressed **isolated copy** so `references/` assets are real measured input, not just `SKILL.md` |
|
|
@@ -150,14 +177,19 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
|
|
|
150
177
|
|
|
151
178
|
| Variable | Description |
|
|
152
179
|
|---|---|
|
|
180
|
+
| `OMK_EXECUTOR` | default executor preference, e.g. `codex` / `codex-sdk` / `claude` |
|
|
181
|
+
| `OMK_MODEL` | default evaluated model; Codex reads local `config.toml` when unset |
|
|
182
|
+
| `OMK_JUDGE_MODELS` | default judge list in `executor:model[,...]` format |
|
|
153
183
|
| `CCV_PROXY_URL` | proxy requests through cc-viewer for live eval-traffic visualization |
|
|
154
184
|
| `OMK_REPORT_PORT` | report server port (default: 7799) |
|
|
155
185
|
|
|
156
186
|
## Requirements
|
|
157
187
|
|
|
158
188
|
- Node.js >= 22
|
|
159
|
-
-
|
|
160
|
-
-
|
|
189
|
+
- At least one authenticated model runtime:
|
|
190
|
+
- Codex: install and authenticate the Codex CLI (`npm i -g @openai/codex`); Codex tasks in the ChatGPT desktop app select it automatically
|
|
191
|
+
- Claude: install and authenticate [Claude Code](https://claude.ai/code)
|
|
192
|
+
- API / other executors: configure them as described in [Executors](docs/reference/executors.md)
|
|
161
193
|
|
|
162
194
|
## Security notice
|
|
163
195
|
|
package/README.zh.md
CHANGED
|
@@ -8,11 +8,23 @@
|
|
|
8
8
|
|
|
9
9
|
[English](./README.md) | **简体中文**
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
`
|
|
11
|
+
**别再凭感觉改 LLM 知识输入。**
|
|
12
|
+
`oh-my-knowledge`(omk)是一套面向 prompt、RAG、skill、agent、workflow 的测量工作流。它固定执行模型和评测用例,只改变知识载体,然后回答发布时真正要追问的问题:**v2 能不能发,好在哪里?**
|
|
13
|
+
|
|
14
|
+

|
|
13
15
|
|
|
14
16
|
📖 **完整文档:[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)**(可搜索,可切换英文)
|
|
15
17
|
|
|
18
|
+
## omk 让什么变得可测量
|
|
19
|
+
|
|
20
|
+
| 决策问题 | 命令 | 你会得到的证据 |
|
|
21
|
+
|------|------|------|
|
|
22
|
+
| 这份知识载体是否清楚到值得评测? | `omk doctor` | 结构、依赖、安全性、可测性检查 |
|
|
23
|
+
| v2 是否真的优于 v1? | `omk eval` | 一行 verdict、置信区间、失败样本、成本 |
|
|
24
|
+
| 它为什么通过或失败? | `omk studio` | 分数、诊断、样本证据的报告视图 |
|
|
25
|
+
| 这个版本是否应成为接受版本? | `omk promote` / `omk evolve` | 基于证据接受,或生成更好的候选版 |
|
|
26
|
+
| 真实使用暴露了哪些知识缺口? | `omk observe` / `omk sample --from-traces` | 将线上缺口生成待复核草稿,复核后再沉淀为评测样本 |
|
|
27
|
+
|
|
16
28
|

|
|
17
29
|
|
|
18
30
|
## 快速开始
|
|
@@ -20,34 +32,45 @@
|
|
|
20
32
|
```bash
|
|
21
33
|
npm i -g oh-my-knowledge
|
|
22
34
|
omk init demo && cd demo
|
|
35
|
+
omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
|
|
23
36
|
omk eval --control code-review-v1 --treatment code-review-v2
|
|
24
37
|
```
|
|
25
38
|
|
|
26
|
-
开箱即跑:`omk init` 脚手架好两版 skill
|
|
39
|
+
开箱即跑:`omk init` 脚手架好两版 skill 和三条评测用例,不用先改任何文件;`--dry-run` 预览调用次数和成本;`omk eval` 跑控制变量 A/B,约 5 分钟出 HTML 报告 + 一行 verdict。跑通后再把 skill 和用例换成你自己的。
|
|
40
|
+
|
|
41
|
+
前置:准备一个已认证的模型 runtime(Codex CLI、Claude Code 或 API 执行器,见[系统要求](#系统要求))。在 ChatGPT desktop 的 Codex 任务里,omk 会自动使用 `codex`,从 `~/.codex/config.toml` 读取模型,并默认用同一个 Codex 模型担任评委,不依赖 Claude。
|
|
42
|
+
|
|
43
|
+
普通终端想固定使用 Codex,可以把偏好加入 shell 配置,例如 `~/.zshrc`:
|
|
27
44
|
|
|
28
|
-
|
|
45
|
+
```bash
|
|
46
|
+
export OMK_EXECUTOR=codex
|
|
47
|
+
# 可选:export OMK_MODEL="你的 Codex 模型"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
不设置 `OMK_MODEL` 时,omk 会读取 `~/.codex/config.toml` 的模型。也可以继续逐次显式传 `--executor codex --model <codex-model>`。自定义评委时再传 `--judge-models` 或设置 `OMK_JUDGE_MODELS`。
|
|
29
51
|
|
|
30
52
|
> 首跑只有 3 条用例,verdict 多半是「数据不足(UNDERPOWERED)」——这是正常起点而非出错;把用例加到约 20 条以上,再看「可发布」结论。
|
|
31
53
|
|
|
32
54
|
> 命令行有新版本时会自动提示(每 20 小时最多一次);想永久关闭该提醒,设环境变量 `OMK_SKIP_UPDATE_CHECK=1` 即可。
|
|
33
55
|
|
|
34
|
-
手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)
|
|
56
|
+
手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)(推荐第一次跑评测的用户,覆盖 demo → 自己的 skill → verdict 动作)。更多可跑示例(Skill Map、A/B、离线执行器、agent runtime、RAG)见仓库的[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)。
|
|
35
57
|
|
|
36
|
-
深入:[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [
|
|
58
|
+
深入:[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [知识载体布局](docs/zh/reference/artifact-layout.md)
|
|
37
59
|
|
|
38
|
-
##
|
|
60
|
+
## omk 的闭环
|
|
39
61
|
|
|
40
|
-
omk 主要给 LLM 知识载体的作者 / 维护者用,帮他们做发布判断;它不是给被动安装 skill
|
|
62
|
+
omk 主要给 LLM 知识载体的作者 / 维护者用,帮他们做发布判断;它不是给被动安装 skill 的普通使用者用的。主流程刻意保持受控:
|
|
41
63
|
|
|
42
64
|
```text
|
|
43
|
-
|
|
44
|
-
→
|
|
45
|
-
→
|
|
46
|
-
→
|
|
47
|
-
→
|
|
65
|
+
改了一份 prompt / RAG / skill / agent 知识载体
|
|
66
|
+
→ 先跑 omk doctor
|
|
67
|
+
→ 用相同模型、相同评测用例跑 omk eval
|
|
68
|
+
→ 看 report / Studio 里的证据
|
|
69
|
+
→ 证据足够则 promote,证据不足则 evolve 候选版
|
|
70
|
+
→ observe 真实使用,把缺口生成待复核评测草稿
|
|
48
71
|
```
|
|
49
72
|
|
|
50
|
-
`
|
|
73
|
+
第一价值是发布前的 `doctor → eval` 判断。长期价值是闭环:`observe` 暴露真实使用里的知识缺口,`sample --from-traces` 先生成待人工复核的评测用例草稿,复核后的草稿再沉淀为固定评测样本,下一次 `eval` 就更难被偶然样本骗过。
|
|
51
74
|
|
|
52
75
|
## 在 AI Coding Agent 中使用
|
|
53
76
|
|
|
@@ -64,16 +87,16 @@ omk install omk-agent-skill
|
|
|
64
87
|
当 `omk` skill 已在 Claude Code 中可用时,可以直接这样调用:
|
|
65
88
|
|
|
66
89
|
```bash
|
|
67
|
-
/omk eval #
|
|
90
|
+
/omk eval # 评测当前项目的知识载体
|
|
68
91
|
/omk evolve # 多轮自动迭代改进 skill
|
|
69
92
|
/omk sample # 生成或补齐评测用例
|
|
70
93
|
```
|
|
71
94
|
|
|
72
|
-
这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2
|
|
95
|
+
这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个知识载体」,omk 会自动理解意图并调用对应命令。
|
|
73
96
|
|
|
74
97
|
### 在 Codex 中使用
|
|
75
98
|
|
|
76
|
-
Codex 默认不支持 `/omk ...` 这种 Claude Code 风格的 slash command
|
|
99
|
+
Codex 默认不支持 `/omk ...` 这种 Claude Code 风格的 slash command。直接让 agent 执行 `omk` CLI 即可;在 Codex 任务里,omk 会自动选择 Codex runtime 和本机配置的模型:
|
|
77
100
|
|
|
78
101
|
```bash
|
|
79
102
|
omk eval
|
|
@@ -83,11 +106,15 @@ omk sample skills/my-skill.md
|
|
|
83
106
|
|
|
84
107
|
也可以直接用自然语言描述目标,例如「比较 v1 和 v2 的评测差异」、「为这个 skill 生成评测用例」。
|
|
85
108
|
|
|
109
|
+
`eval`、`doctor`、`sample`、`evolve` 和 observe 的 LLM 增强复盘共用同一套 runtime 解析。Codex 被选中后,默认评委沿用被测 Codex 模型,不会回落到 `claude:haiku`。
|
|
110
|
+
|
|
86
111
|
> `omk evolve` 是一键闭环:默认先跑 doctor 体检,目标 skill 没有评测用例时会自动生成一批,再进入多轮自迭代。全新 skill 直接 `omk evolve skills/foo.md` 即可。
|
|
87
112
|
|
|
88
113
|
## 为什么需要这个工具
|
|
89
114
|
|
|
90
|
-
|
|
115
|
+
知识工程带来的是一个版本治理问题:prompt、RAG 配方、skill、agent、workflow 都会改变模型行为,但这些改动未必体现在应用代码里。当有人追问「v2 能不能发、为什么」时,回答更顺眼、体感更好,远远不够。
|
|
116
|
+
|
|
117
|
+
omk 把知识载体当作被测变量:**相同模型、相同评测用例,只改变知识载体。** 这样得到的对比才可解释、可复跑,也适合进入 CI 或发布评审。
|
|
91
118
|
|
|
92
119
|
## 为什么选 omk
|
|
93
120
|
|
|
@@ -117,7 +144,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
117
144
|
| **统计严谨性** | Bootstrap CI / 长度去偏 / 饱和曲线默认开,Krippendorff α 提供 gold 集即自动计算。[详情 →](docs/zh/explanation/statistical-rigor.md) |
|
|
118
145
|
| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
|
|
119
146
|
| **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分;重复采样(`--repeat`)+ k/n 共识归并 |
|
|
120
|
-
| **线上 session 观测** |
|
|
147
|
+
| **线上 session 观测** | 将 Codex、Claude Code、OpenClaw 与 markdown 日志统一为 source-neutral Trace IR,测量各 skill 的执行结果、耗时、token 使用和知识缺口信号 |
|
|
121
148
|
| **知识缺口识别** | 严重度加权的信号量化风险敞口,不宣称完备性 |
|
|
122
149
|
| **用例隔离 (construct validity)** | `--strict-baseline`(默认开)三堵 baseline 拿到被测 skill 的污染路径 |
|
|
123
150
|
| **Git / 远端源** | install / eval 支持本地 git ref 或远端 git URL(`--git-url`);目录-skill 在内容寻址**隔离副本**里执行,`references/` 资产是真实测量输入,不只是 `SKILL.md` |
|
|
@@ -127,7 +154,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
127
154
|
| **多轮方差分析** | `--repeat N` 重复 N 次,计算均值/标准差/置信区间/t 检验 |
|
|
128
155
|
| **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
|
|
129
156
|
| **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
|
|
130
|
-
| **可追溯性** | 报告含 CLI 版本、Node
|
|
157
|
+
| **可追溯性** | 报告含 CLI 版本、Node 版本、知识载体版本指纹、judge prompt hash |
|
|
131
158
|
| **中英切换** | HTML 报告右上角一键切换语言 |
|
|
132
159
|
|
|
133
160
|
## 文档
|
|
@@ -137,7 +164,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
137
164
|
- **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
|
|
138
165
|
- **[评测用例格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、30+ 断言类型、自定义 JS 断言
|
|
139
166
|
- **[CLI 参考](docs/zh/reference/cli.md)** —— 顶层命令的 bash 示例和 flag 表
|
|
140
|
-
- **[执行器](docs/zh/reference/executors.md)** & **[
|
|
167
|
+
- **[执行器](docs/zh/reference/executors.md)** & **[知识载体布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器;variant 如何解析为 artifact + runtime context
|
|
141
168
|
- **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)(项目 runtime context)与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)(GLM / 通义 / DeepSeek / Moonshot / Ollama)
|
|
142
169
|
- **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
|
|
143
170
|
- **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例,按由简到全排成上手路径
|
|
@@ -150,14 +177,19 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
150
177
|
|
|
151
178
|
| 变量 | 说明 |
|
|
152
179
|
|------|------|
|
|
180
|
+
| `OMK_EXECUTOR` | 默认执行器偏好,例如 `codex` / `codex-sdk` / `claude` |
|
|
181
|
+
| `OMK_MODEL` | 默认被测模型;Codex 未设置时读取本机 `config.toml` |
|
|
182
|
+
| `OMK_JUDGE_MODELS` | 默认评委列表,格式 `executor:model[,...]` |
|
|
153
183
|
| `CCV_PROXY_URL` | 通过 cc-viewer 代理请求,实时可视化评测流量 |
|
|
154
184
|
| `OMK_REPORT_PORT` | 报告服务端口(默认 7799) |
|
|
155
185
|
|
|
156
186
|
## 系统要求
|
|
157
187
|
|
|
158
188
|
- Node.js >= 22
|
|
159
|
-
-
|
|
160
|
-
-
|
|
189
|
+
- 至少一个已认证的模型 runtime:
|
|
190
|
+
- Codex:安装并登录 Codex CLI(`npm i -g @openai/codex`);ChatGPT desktop 的 Codex 任务会自动选择它
|
|
191
|
+
- Claude:安装并登录 [Claude Code](https://claude.ai/code)
|
|
192
|
+
- API / 其它执行器:按[执行器文档](docs/zh/reference/executors.md)配置
|
|
161
193
|
|
|
162
194
|
## 安全说明
|
|
163
195
|
|