oh-my-knowledge 0.30.0 → 0.32.0

package/README.md

@@ -8,32 +8,30 @@
 
 **English** | [简体中文](./README.zh.md)
 
-**omk** — The knowledge you give your LLM — what's it actually worth?
-omk answers with objective data, not gut feeling.
+**Did your prompt actually get better?**
+A/B test your prompts and skills with statistical rigor — bootstrap CI, Krippendorff α, length-debias — all on by default.
 
-**Evaluation framework for LLM knowledge inputs** — prompts, RAG corpora, skills, agent workflows. Fix the model, vary the artifact.
-
-<a id="statistical-rigor"></a>
-> Built-in: Bootstrap CI · Krippendorff α (judge ↔ human) · length-debias · saturation curves · construct-validity isolation. [Why these matter →](docs/statistical-rigor.md)
-
-![omk report](./assets/screenshots/report-overview.png)
+![omk report — verdict pill "v2 is clearly better than v1 — ready to ship"](./assets/screenshots/report-overview.png)
 
 ## Quick start
 
 ```bash
-npm i oh-my-knowledge -g
-omk init my-eval && cd my-eval
-# edit skills/code-review-v1/SKILL.md and skills/code-review-v2/SKILL.md with your two versions
-omk eval --control code-review-v1 --treatment code-review-v2    # → HTML report with verdict in 5 minutes
+npm i -g oh-my-knowledge
+omk init demo && cd demo
+omk eval --control code-review-v1 --treatment code-review-v2
 ```
 
-Deeper: [use inside Claude Code / Codex](#use-inside-ai-coding-agents) · [`omk eval` flags](#omk-eval) · [artifact directory layout](#artifact-directory-layout) · [`--lang` / `OMK_LANG`](#environment-variables)
+That's it — no editing required. `omk init` scaffolds two skill variants and three sample cases; `omk eval` runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes.
+
+Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recommended for first-time users).
+
+Deeper: [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors & artifact layout](docs/reference/executors.md)
 
 ## Use inside AI Coding Agents
 
 ### Use inside Claude Code
 
-When the `omk` skill is available in Claude Code, you can invoke it directly like this:
+When the `omk` skill is available in Claude Code, you can invoke it directly:
 
 ```bash
 /omk eval              # evaluate the artifact(s) in the current project
@@ -41,11 +39,11 @@ When the `omk` skill is available in Claude Code, you can invoke it directly lik
 /omk sample            # generate or fill test cases
 ```
 
-These slash commands are natural-language entry points — the agent reads the conversation context to figure out which skill to operate on, so you usually don't pass the path explicitly. (See the Codex section below for the literal `omk evolve <skill>` CLI form.) You can also just say "compare v1 vs v2 for me" or "improve this artifact" and omk picks the right command.
+These slash commands are natural-language entry points — the agent reads the conversation context to figure out which skill to operate on. You can also just say "compare v1 vs v2 for me" or "improve this artifact" and omk picks the right command.
 
 ### Use inside Codex
 
-Codex does not support Claude Code style `/omk ...` slash commands by default. In Codex, the usual pattern is to ask the agent to run the `omk` CLI directly, for example:
+Codex does not support Claude Code style `/omk ...` slash commands. Ask the agent to run the `omk` CLI directly:
 
 ```bash
 omk eval
@@ -59,16 +57,6 @@ You can also describe the goal in natural language, such as "compare v1 vs v2" o
 
 Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
 
-## Key features
-
-- **LLM health audit** — `omk doctor` runs a single LLM session that emits a multi-dimension report; 7 builtin dimensions (trigger & boundary / doc clarity / instruction precision / dependency / tool conventions / security & compliance / example completeness) each get a *healthy / sub-healthy / unhealthy / N-A* grade plus findings and suggestions; dimensions are extensible, and `--html` produces a visual report. Pass `--static-only` for an offline mode (CI nodes without an LLM, debugging without network) that runs the static checks only (readability / metadata / dependencies / samples contract). `omk eval` still runs static readability / metadata / dependency gates internally to protect eval quality (separation of roles: doctor = audit, eval = evaluate)
-- **Controlled-variable offline eval** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
-- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
-- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
-- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
-- **Pre-merge CI gate** — `omk eval` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
-- **One-line ship/no-ship verdict** — `omk eval` aggregates bootstrap CI / three-layer ci-gate / saturation / human α into a six-tier verdict (PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO) plus an action recommendation; the exit code reflects whether to ship
-
 ## Why omk over alternatives
 
 | | omk | promptfoo | DeepEval | LangSmith |
@@ -84,7 +72,7 @@ Teams doing knowledge engineering produce lots of knowledge artifacts (skills to
 
 omk's moat is **default-on safety net** — Bootstrap CI, judge ↔ human α, and length-debias aren't advanced flags; they're the default. Other tools let you opt into confidence intervals; omk makes them unavoidable. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. **Shipping to production and someone will ask "why should I trust this number?" Choose omk.**
 
-RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/comparison.md](docs/comparison.md).
+RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: [docs/reference/comparison.md](docs/reference/comparison.md).
 
 ## Features
 
@@ -93,659 +81,32 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
 | **One-line verdict** | `omk eval` six-tier verdict + ship recommendation + exit-code routing; HTML pill shares the same rules |
 | **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
 | **Multi-executor** | Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini / any custom command |
-| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS, and more |
-| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/statistical-rigor.md) |
-| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage; auto-inherits length-debias |
-| **Hard budget caps** | `--budget-usd / --budget-per-sample-usd / --budget-per-sample-ms` — abort on total-cost overrun, flag per-sample overruns; partial report persisted |
-| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against: (1) SDK skill auto-discovery, (2) subagent Skill tool, (3) cwd file-system access via the `skills/<name>/` symlink that's normally there for the treatment variant. eval.yaml `allowedSkills` for per-variant whitelists |
-| **Sample design science** | Sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata fields (HF Dataset Cards style); studio surfaces coverage breakdown plus `rubric_clarity_low` / `capability_thin` flags. `omk sample` auto-stamps provenance on generated cases. See [docs/sample-design-spec.md](docs/sample-design-spec.md) for the 8 industry-gap mapping |
+| **21+ assertion types** | substring, regex, JSON Schema, ROUGE/BLEU/Levenshtein similarity, agent tool-call assertions, semantic similarity, custom JS |
+| **Statistical rigor** | Bootstrap CI / Krippendorff α / length-debias / saturation curve — all on by default. [Details →](docs/explanation/statistical-rigor.md) |
+| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` — anti-hallucination + answer relevance + context coverage |
+| **LLM health audit** | `omk doctor` grades 7 builtin dimensions; `--static-only` runs offline without an LLM |
+| **Production observability** | parse Claude Code session JSONL traces; measure per-skill failure rate / latency / cost / knowledge-gap signals |
+| **Knowledge-gap detection** | severity-weighted signals quantify risk exposure instead of claiming completeness |
+| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against |
+| **Sample design science** | sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata (HF Dataset Cards style); studio surfaces coverage breakdown plus `rubric_clarity_low` / `capability_thin` flags. [docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
 | **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
-| **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
 | **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
-| **Parallel execution** | `--concurrency N` runs N tasks at once |
-| **Assertion negation + composition** | universal `not: true` field + `assert-set` (any/all) with arbitrary nesting |
+| **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
 | **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
 | **Traceability** | reports carry CLI version, Node version, artifact version fingerprint, judge prompt hash |
 | **EN / ZH switch** | one-click language toggle in the HTML report |
 
-## How it works
-
-Core idea: **fix the model and the samples, vary only the artifact and runtime context**, use interleaved scheduling to cancel time drift, score via assertions + LLM judge (dual channel), then layer on knowledge-gap signals to quantify risk exposure.
-
-```mermaid
-flowchart TD
-    subgraph Input["① Input"]
-        S["eval-samples<br/>(JSON / YAML)"]
-        A["artifacts<br/>skills/*.md · SKILL.md<br/>baseline · git:name · @cwd"]
-    end
-
-    subgraph Prep["② Preprocess (resolve & fetch)"]
-        V["variant resolution<br/>variant → artifact + runtime context<br/>(cwd / project CLAUDE.md / local skills)"]
-        U["URL fetching<br/>URLs in prompt / context<br/>MCP Server(private docs) → HTTP"]
-    end
-
-    subgraph Schedule["③ Interleaved + concurrent scheduling"]
-        Q["s1-v1 → s1-v2 → s2-v1 → s2-v2 …<br/>--concurrency N · --repeat N"]
-    end
-
-    subgraph Exec["④ Executor (fixed model)"]
-        E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / custom"]
-        T["claude-sdk / codex extract<br/>turns / toolCalls trace"]
-        E -.-> T
-    end
-
-    subgraph Score["⑤ Dual-channel scoring"]
-        AS["assertions (18 types)<br/>content / structure / cost / latency<br/>agent: tools_called · turns_min …"]
-        LS["LLM judge<br/>rubric · dimensions (independent per-dim scores)"]
-        CS["composite score<br/>mean of assertion & LLM when both present"]
-        AS --> CS
-        LS --> CS
-    end
-
-    subgraph Analyze["⑥ Auto analysis + knowledge gaps"]
-        D["low-discrimination / flat scores / all-pass or all-fail<br/>expensive samples · variance · t-test"]
-        G["knowledge-gap signals<br/>(quantify risk exposure, not completeness proof)"]
-    end
-
-    subgraph Report["⑦ Report"]
-        R["Six dims: Fact / Behavior / LLM-judge / Cost / Efficiency / Stability<br/>JSON + HTML · top verdict pill · blind reveal<br/>CLI/Node/version fingerprint traceable"]
-    end
-
-    S --> U
-    A --> V
-    V --> Q
-    U --> Q
-    Q --> E
-    T --> AS
-    E --> AS
-    E --> LS
-    CS --> D
-    CS --> G
-    D --> R
-    G --> R
-```
-
-**Key design choices:**
-
-- **Interleaved scheduling** removes time drift: different variants of the same sample are dispatched alternately rather than "all of v1 then all of v2", so model load / network jitter can't be mis-attributed to the artifact.
-- **variant = artifact + runtime context**: `name@cwd` lets control groups explicitly declare the "project directory" input, separating "project-level accumulated knowledge" from "explicit artifact injection".
-- **Dual-channel scoring is complementary**: assertions catch deterministic defects (must call tool X, must contain field Y); the LLM judge catches subjective quality (readability, completeness). Mean is taken when both are present.
-- **Knowledge-gap signals** are not part of the score — they are an independent tracking channel that tells you "how much risk exposure this evaluation covered", for convergence tracking, not as a completeness proof.
-
-## Eval sample format
-
-Supports JSON and YAML (`eval-samples.json`, `eval-samples.yaml`, `eval-samples.yml`).
-
-```json
-[
-  {
-    "sample_id": "s001",
-    "prompt": "Review this code for security issues",
-    "context": "function auth(u, p) { db.query('SELECT * FROM users WHERE name=' + u); }",
-    "rubric": "Should identify SQL injection risk and recommend parameterized queries",
-    "assertions": [
-      { "type": "contains", "value": "SQL injection", "weight": 1 },
-      { "type": "contains", "value": "parameterized", "weight": 1 },
-      { "type": "not_contains", "value": "looks fine", "weight": 0.5 }
-    ],
-    "dimensions": {
-      "security": "did it identify the injection vulnerability?",
-      "actionability": "did it give directly usable fix code?"
-    }
-  }
-]
-```
-
-### Fields
-
-| Field | Type | Required | Description |
-|---|---|---|---|
-| `sample_id` | `string` | **yes** | Unique sample ID |
-| `prompt` | `string` | **yes** | User prompt sent to the model |
-| `context` | `string` | no | Extra context (e.g. code). Wrapped in a code block and appended to the prompt. URLs are auto-fetched at runtime. |
-| `rubric` | `string` | no | Scoring guideline for the LLM judge (1-5 scale) |
-| `assertions` | `array` | no | Assertion checks; see [assertion types](#assertion-types) |
-| `assertions[].type` | `string` | **yes** | Assertion type |
-| `assertions[].value` | `string\|number` | depends | Check value (required for `contains`, `min_length`, `cost_max`, etc.) |
-| `assertions[].values` | `array` | depends | String array (required for `contains_all`, `contains_any`) |
-| `assertions[].pattern` | `string` | depends | Regex pattern (required for `regex`) |
-| `assertions[].flags` | `string` | no | Regex flags (default `"i"`) |
-| `assertions[].schema` | `object` | depends | JSON Schema object (required for `json_schema`, via [ajv](https://ajv.js.org/)) |
-| `assertions[].reference` | `string` | depends | Reference text (required for `semantic_similarity`) |
-| `assertions[].threshold` | `number` | no | Pass threshold for semantic similarity (default 3) |
-| `assertions[].fn` | `string` | depends | Path to a custom assertion JS file (required for `custom`) |
-| `assertions[].weight` | `number` | no | Weight (default 1) |
-| `dimensions` | `object` | no | Multi-dimension scoring; key = dimension name, value = scoring guideline |
-
-### URL auto-fetching
-
-URLs in `prompt` and `context` are auto-fetched before evaluation and inlined into the text. Useful when referencing online docs, API references, etc.:
-
-```json
-{
-  "sample_id": "s001",
-  "prompt": "Generate test cases from this PRD: https://wiki.example.com/prd/feature-x"
-}
-```
-
-At runtime, URLs are replaced with the actual content. Fetch order: MCP Server first for matching URLs (e.g. SSO-protected private docs), then plain HTTP for the rest. URLs already resolved by MCP are not re-fetched via HTTP.
-
-**Private-doc URLs**: drop a `.mcp.json` config file into the project dir, or pass `--mcp-config <path>`:
-
-```json
-{
-  "mcpServers": {
-    "docs": {
-      "command": "npx",
-      "args": ["@example/docs-mcp-server"],
-      "env": { "DOCS_API_TOKEN": "xxx" },
-      "urlPatterns": ["docs.example.com"],
-      "fetchTool": {
-        "name": "fetch_doc",
-        "urlTransform": {
-          "regex": "docs\\.example\\.com/([^/]+/[^/]+)/([^/?#]+)",
-          "params": { "namespace": "$1", "slug": "$2" }
-        },
-        "contentExtract": "data.body"
-      }
-    }
-  }
-}
-```
-
-**Public URLs**: fetched via plain HTTP. If they require auth, make sure the shell already has network access configured (VPN, proxy, etc.).
-
-### Scoring strategy
-
-#### 1. Assertion score
-
-Rule-based local checks; each assertion yields pass/fail.
-
-**Formula:**
-
-- Pass rate = sum of passed assertion weights / total weight (0–1)
-- Score = 1 + pass_rate × 4 (mapped to 1–5)
-- Example: 3 assertions (weight 1 each), 2 pass → pass rate 2/3 → score = 1 + 0.67 × 4 = **3.67**
-
-#### 2. Rubric / Dimensions score
-
-The judge model (default `haiku`) scores 1–5 against the rubric. In `dimensions` mode, each dimension is scored independently and then averaged.
-
-#### 3. Composite score
-
-| Condition | Formula |
-|---|---|
-| Only assertions | `assertionScore` |
-| Only LLM judge | `llmScore` |
-| Both present | `(assertionScore + llmScore) / 2` |
-| Neither | `0` |
-
-### Assertion types
-
-**Deterministic assertions (21+ total):**
-
-| Type | Description |
-|---|---|
-| `contains` / `not_contains` | substring must / must-not appear |
-| `regex` | regex match |
-| `min_length` / `max_length` | length bounds |
-| `json_valid` / `json_schema` | JSON validation |
-| `starts_with` / `ends_with` | prefix / suffix |
-| `equals` / `not_equals` | exact match |
-| `word_count_min` / `word_count_max` | word-count bounds |
-| `contains_all` / `contains_any` | multi-value match |
-| `cost_max` / `latency_max` | cost / latency caps |
-| `tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max` | agent tool-call assertions |
-| `tool_output_contains` / `tool_input_contains` | match content of a tool's input or output |
-| `turns_min` / `turns_max` | conversation-turn bounds |
-| `rouge_n_min` | ROUGE-N recall ≥ threshold (`reference` field holds the gold text; `n` defaults to 1; `threshold` defaults to 0.5) |
-| `levenshtein_max` | edit distance ≤ value (for "output should be near-identical to reference") |
-| `bleu_min` | BLEU-4 ≥ threshold (unsmoothed; degenerates to 0 on short text) |
-| `faithfulness` | output stays grounded in `sample.context` (anti-hallucination); LLM judge 1-5; threshold defaults to 3 |
-| `answer_relevancy` | output directly answers `sample.prompt`; catches dodging, topic drift, verbosity; threshold defaults to 3 |
-| `context_recall` | gold facts in `sample.context` are actually used in the output; `reference` may explicitly enumerate gold facts; threshold defaults to 3 |
-| `semantic_similarity` | LLM-based holistic semantic similarity (complementary to the three RAG metrics above) |
-| `custom` | custom JS function (30 s timeout) |
-
-**Universal modifier:**
-
-Any assertion takes `not: true` to invert (replaces paired `not_contains` / `not_equals` etc; legacy types remain as aliases):
-
-```yaml
-- type: regex
-  pattern: "TODO|FIXME"
-  not: true              # output must NOT contain TODO/FIXME
-```
-
-**Composition (assert-set):**
-
-`assert-set` combines child assertions with `any` (OR) or `all` (AND) and supports nesting:
-
-```yaml
-- type: assert-set
-  mode: any              # at least one child must pass (mode: 'all' = all must pass)
-  children:
-    - { type: contains, value: "parameterized" }
-    - { type: contains, value: "prepared statement" }
-    - { type: regex, pattern: "bind\\(.*\\?" }
-```
-
-Children can independently use `not: true`; nested `assert-set`s can express any boolean shape.
-
-### Custom assertion
-
-```js
-// my-assertion.mjs
-export default function(output, { sample, assertion }) {
-  return { pass: output.includes('SQL'), message: 'checked for SQL keyword' };
-}
-```
-
-## Six-dim evaluation
-
-Reports display results across six independent dimensions. The three scoring layers — Fact / Behavior / LLM-judge — are shown separately so you see **which layer regressed** instead of a single composite number:
-
-| Dimension | Metric | Description |
-|---|---|---|
-| 📋 **Fact** | fact-assertion pass rate | rule-verifiable assertions like `contains` / `json_schema` / `fact_check`, mapped to 1-5 |
-| 🛠️ **Behavior** | behavior-assertion pass rate | execution-compliance assertions like `tools_called` / `tool_output_contains` / `turns_max` |
-| 💬 **LLM-judge** | rubric score | 1-5 scored by the judge model against a predefined rubric; subjective, catches what rules miss |
-| 💰 **Cost** | total cost, input/output tokens | API cost based on token usage and model pricing |
-| ⚡ **Efficiency** | average latency (ms) | end-to-end latency from request to full response |
-| 🛡️ **Stability** | CV (coefficient of variation) | score consistency across repeated runs (`--repeat ≥ 2`); single-run shows `—`, **honestly acknowledging what can't be measured** |
-
-## CLI reference
-
-omk exposes a workflow CLI for knowledge artifacts. Seven top-level commands cover the full loop: `init` (scaffold) · `doctor` (static check) · `eval` (offline A/B) · `observe` (online trace) · `evolve` (auto-iterate a skill) · `sample` (generate or fill test cases) · `studio` (local web UI for reports & analysis).
-
-### `omk init`
-
-```bash
-omk init [dir]
-```
-
-Scaffolds an evaluation project with two starter skill variants and an `eval-samples.json` file.
-
-### `omk doctor`
-
-```bash
-omk doctor                              # audit current dir / ./skills
-omk doctor skills/v1.md                 # audit one skill file
-omk doctor skills/ --html report.html   # produce a visual HTML report
-omk doctor skills/ --json > r.json      # JSON for CI / external tools
-omk doctor --gate; echo $?              # silent gate; exit 1 on fatal failures, warnings do not block
-omk doctor --static-only                # offline mode: static checks only, no LLM call
-```
-
-LLM health audit: a single LLM session emits per-dimension grades, findings, and suggestions for the 7 builtin dimensions; the HTML report sorts dimensions fail→warn→pass→skipped with errors first within each dim. Dimensions are extensible — call `registerHealthDimension` in your own code and the new section is folded into the same LLM call's prompt and report (order = registration order).
-
-Static-only mode (`--static-only`): for CI nodes without claude / codex installed, or local debugging without network — runs the four static rules (readability / metadata / dependencies / samples contract) with zero LLM calls and zero cost. Output goes through the same `DoctorReport` shape and combines with `--json` / `--gate` / `--html`.
-
-`omk eval` still runs its own static readability / metadata / dependency / samples-contract gates internally to protect eval quality; that path is separate from this user-facing `omk doctor` command and the two roles do not overlap.
-
-### `omk eval`
-
-```bash
-omk eval --control baseline --treatment my-skill                # single-skill necessity test (baseline = reserved "no skill" variant)
-omk eval --control code-review-v1 --treatment code-review-v2    # multi-variant A/B
-omk eval --config eval.yaml
-omk eval --batch
-omk eval gold compare <report-id> --gold-dir gold-dataset
-```
-
-Runs the offline evaluation, applies the verdict gate, persists the report, and returns a ship/no-ship exit code. Bootstrap CI is enabled by default on this workflow.
-
-Common options:
-
-```text
-  --samples <path>       sample file (default: eval-samples.json, also detects .yaml/.yml; auto-discovers <skill>/.omk/samples.json under --skill-dir)
-  --skill-dir <path>     artifact dir (default: skills)
-  --control <expr>       control variant expression
-  --treatment <v1,v2>    treatment variants, comma-separated
-  --config <path>        YAML/JSON evaluation config
-  --model <name>         task execution model (default: opus alias)
-  --effort <level>       reasoning effort for executor LLM: low/medium/high/xhigh/max (default: low; reports across efforts not strictly comparable)
-  --judge-models <list>  judge config, e.g. claude:haiku or claude:opus,openai:gpt-4o
-  --executor <name>      claude / claude-sdk / codex / codex-sdk / openai-api / gemini / custom
-  --no-judge             skip LLM judge subjective scoring
-  --no-diagnostic        skip the diagnostic LLM call (default: on; emits "what went wrong + how to fix" advice for failed samples)
-  --skip-doctor          escape hatch: bypass the doctor health-check gate (default: on). Use when sandbox mocks supply deps and doctor's path/env checks misfire; caller owns garbage-in risk
-  --dry-run              preview only
-  --blind                blind A/B mode
-  --concurrency <n>      parallel tasks
-  --timeout <sec>        per-task timeout
-  --repeat <n>           repeat N times for variance analysis
-  --batch                evaluate each artifact independently vs baseline
-  --bootstrap-samples N  bootstrap resample count (default 1000)
-  --threshold <number>   verdict layer-gate threshold (default 3.5)
-  --trivial-diff <num>   practically tiny diff cutoff (default 0.1)
-  --report-only          produce the report and print verdict, but always exit 0
-  --no-gate              alias for --report-only
-  --skip-connectivity    skip model connectivity check (internal static gates still run)
-  --no-serve             do not auto-start the report server after evaluation
-```
-
-The HTML report has two tabs:
-- **📊 Score view** — the verdict-driven A/B comparison (fact / behavior / judge layers, bootstrap CI, length-debias).
-- **✅ Functional view** — each sample as a unit test: design (prompt / rubric / mocks / environment) + execution trace + assertion results + actionable diagnostic. Diagnostic emits root cause (skill_doc_unclear / llm_misread / sample_design / tripwire_intentional / ...), workflow checks (rubric step ✓/✗ with evidence), and failure-mode tags (工作流跳步 / 硬编码值 / 幻觉输出 / 工具误用 / 环境拦截 / 误读约束 / 其他). For the sandbox-mock semantics behind `mocks` / `environment` / `tripwire` / `mocksStrict`, see [docs/sample-design-spec.md §三](./docs/sample-design-spec.md).
-
-### `omk observe`
-
-`omk observe` ships two workflows: the default skill-health report, and the new observe inbox for human review.
-
-#### A. Skill-health report (default)
-
-```bash
-omk observe ~/.claude/projects/-Users-you-Documents-my-project
-omk observe ~/.claude/projects/my-project --last 7d
-omk observe ~/.claude/projects/my-project --from 2026-04-01T00:00:00Z --to 2026-04-15T23:59:59Z
-omk observe ~/.claude/projects/my-project --skills audit,polish
-omk observe ~/.claude/projects/my-project --kb /path/to/project
-```
-
-Turns real Claude Code session traces into skill-health reports: knowledge usage, gap signals, execution stability, tokens, and latency. This is production observation, not production scoring.
-
-#### B. observe inbox: reviewer loop
-
-Parses, aggregates, and de-noises real session traces into a per-observation list a human can review. The whole pipeline is local-only and LLM-free.
-
-```bash
-# 1. Parse traces, aggregate signals, write to .omk/observations/
-omk observe ingest ~/.claude/projects/my-project
-omk observe ingest ~/.claude/projects/my-project --output-dir ./custom-dir
-
-# 2. Read the inbox (default: top 20, sorted by severity / confidence / lastSeen)
-omk observe inbox
-omk observe inbox --limit 50
-omk observe inbox --skill audit                    # filter by skill
-omk observe inbox --by-skill                       # rollup view (one row per skill)
-omk observe inbox --explore 10                     # sample 10 long-tail items from medium/low
-omk observe inbox --explore 10 --include-noise     # explicitly include the noise bucket
-omk observe inbox --json                           # JSON output for automation
-
-# 3. Inspect a single observation with its event triplet (surrounding messages)
-omk observe show <inbox_id>
-```
-
-Every observation carries:
-
-- `confidence` and `attributionConfidence` — signal credibility plus skill-attribution credibility, displayed side by side
-- `severityReasonCode` — stable structured reason code for the assigned severity; human-readable reasons are generated by CLI / studio rendering
-- `messageWindow` — 3 messages before / trigger / 3 messages after, plus `resolutionAfter` (whether the agent recovered)
-- `evidence.{messageIndex,messageUuid,toolUseId}` — anchors for round-tripping back to the original jsonl
-
-Supported trace formats: Claude Code session JSONL (`.jsonl`), OpenClaw session JSONL (`.jsonl`), and markdown conversation logs (`.log`).
-
-### `omk evolve`
-
-```bash
-omk evolve <skill>                  # multi-round auto-iteration on a skill
-omk evolve skills/foo.md --rounds 10 --target 4.5
-```
-
-Auto-iterates a skill through repeated eval → judge → rewrite loops until it hits `--target` or exhausts `--rounds`. Cost scales with `rounds × samples × variants`; a typical run takes minutes to tens of minutes. Original skill files are versioned under `skills/evolve/*.r0.md`.
-
-### `omk sample`
-
-```bash
-omk sample <skill>                  # generate or fill eval-samples test cases for one skill
-omk sample --batch                  # generate for skills missing eval-samples
-```
-
-One-shot generation. Auto-stamps `provenance` on generated cases. Generated assertions use English, numbers, or code tokens so they compare cleanly across bilingual outputs.
-
-### `omk studio`
-
-```bash
-omk studio
-omk studio --port 7799
-omk studio --host 0.0.0.0                          # LAN access (default: 127.0.0.1)
-omk studio --reports-dir ~/.oh-my-knowledge/reports
-omk studio --observations-dir .omk/observations    # observe inbox data directory
-omk studio --no-open
-```
-
-Starts the local knowledge workbench for browsing reports and observation analyses. Verdict, sample diffs, regressions, saturation curves, and per-sample drill-downs all live in the studio UI — there is no CLI export / analysis subcommand. For CI gates, use `omk eval`'s exit code (0 on `PROGRESS`, non-zero otherwise) or `jq` over the report JSON.
-
-Studio is skill-centric — the list page (`/`) shows skill cards with health band / 0-100 reference score / open-issue count / trend; the detail page (`/skills/<name>`) puts a prioritized issue checklist on the left (skill issues / sample issues / tool advisories), and a chart.js health trend plus three compact stage cards (doctor / eval / observe) on the right, with modals for deeper drill-down. The legacy run list moved to `/runs`. Visit `/observations/inbox` for the observe inbox dashboard: per-skill rollup view, reviewer action list, observability funnel, and a per-observation detail panel with the event triplet (surrounding messages).
-
-## Executors
-
-### Built-in executors
-
-| Executor | When to use | Description |
-|---|---|---|
-| `claude` | default | invokes `claude -p` via Claude CLI |
-| `claude-sdk` | structured output | uses Claude Agent SDK — no stdout parsing, avoids buffer truncation |
-| `codex` | OpenAI agent CLI | invokes `codex exec --json` (`@openai/codex` npm); best-effort tool trace; **costUSD not reported** (codex CLI does not emit USD; check usage externally) |
-| `codex-sdk` | OpenAI agent SDK | uses `@openai/codex-sdk` with its bundled `@openai/codex` binary and streamed SDK events; **costUSD not reported** |
-| `gemini` | cross-vendor comparison | invokes `gemini` CLI |
-| `anthropic-api` | no CLI needed | calls Anthropic HTTP API directly (needs `ANTHROPIC_API_KEY`) |
-| `openai-api` | no CLI needed | calls OpenAI HTTP API directly (needs `OPENAI_API_KEY`) |
-
-API-direct executors support custom base URLs via env: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
-
-Codex construct-validity notes: (1) `codex` uses the `codex` binary on `PATH`; `codex-sdk` uses the bundled `@openai/codex` binary resolved by `@openai/codex-sdk`. Reports persist per-variant `meta.executorRuntimes`, `meta.executorRuntime`, and per-judge `meta.judgeModels[].runtime` fingerprints (binary or SDK version + capability snapshot), and strict comparability checks warn when runtime fingerprints cannot be audited. If runtime fingerprints differ, treat results as an executor-runtime comparison, not only prompt/template behavior. (2) Both executors isolate user-level config: `codex` passes `--ephemeral` + `--ignore-user-config`; `codex-sdk` redirects `$CODEX_HOME` to a per-process tmp dir (auth.json symlinked through). User-level `~/.codex/config.toml` does not leak into eval runs in either case.
-
-### Custom executor
-
-Any shell command can serve as an executor, communicating via stdin/stdout JSON:
-
-```bash
-omk eval --executor "python my_provider.py"
-omk eval --executor "./my-executor.sh"
-```
-
-**Protocol:**
-
-- **input** (stdin): JSON `{"model":"...","system":"...","prompt":"..."}`
-- **output** (stdout): JSON `{"output":"model reply","inputTokens":0,"outputTokens":0,"costUSD":0}`
-- stdout only needs to return the fields you care about; others default to 0. Plain-text output (no tokens/cost parsing) is also fine.
-- non-zero exit code counts as failure
-
-### Artifact directory layout
-
-The built-in executors (claude / openai / gemini) support two artifact layouts, mixable in the same run:
-
-```
-skills/
-├── v1.md                    # option 1: plain .md file
-└── my-skill/                # option 2: full artifact dir
-    ├── SKILL.md             #   this file is auto-loaded as system prompt
-    ├── config.json          #   other files don't participate in eval, kept for completeness
-    └── scripts/
-```
-
-**Variant resolution rules:**
-
-`variant` is the experiment-group expression. After resolution, OMK produces an `artifact` plus an optional `runtime context` (currently mainly `cwd`).
-
-| Format | Meaning |
-|---|---|
-| `name` | looks up `name.md` or `name/SKILL.md` in the artifact dir, resolves to one artifact |
-| `baseline` | empty artifact, no system prompt — think "nothing at all" |
-| `project-env@/path/to/project` | empty artifact, but run in the specified project dir — observe project-level runtime context alone |
-| `git:name` | reads the last-committed version of an artifact from git HEAD |
-| `git:ref:name` | reads an artifact from a specific commit |
-| `./path/to/file.md` | path with `/`: read the file directly as an artifact |
-| `variant@/path/to/project` | attach a run dir to any variant; supports `name@cwd`, `git:name@cwd`, `/file.md@cwd` |
-
-When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--batch`. With `--batch`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
-
-```bash
-# explicit: one control, one or more treatments
-omk eval --control v1 --treatment v2
-omk eval --control baseline --treatment v1,v2,v3
-
-# compare empty artifact vs explicit artifact
-omk eval --control baseline --treatment my-skill
-
-# observe project-level runtime context in isolation (use a self-describing label)
-omk eval --control baseline --treatment project-env@/path/to/target-project
-
-# compare "project-level runtime context" vs "explicit artifact injection"
-omk eval \
-  --control project-env@/path/to/target-project \
-  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
-
-# before vs after (old version read from git history)
-omk eval --control git:my-skill --treatment my-skill
-
-# direct file paths
-omk eval --control ./old-skill.md --treatment ./new-skill.md
-
-# config-file driven (evaluation-as-code)
-omk eval --config eval.yaml
-```
-
-**Prerequisites:**
-
-- **claude**: install [Claude Code](https://claude.ai/code) and authenticate
-- **claude-sdk**: install [Claude Code](https://claude.ai/code) and authenticate (uses Agent SDK, no CLI stdout parsing)
-- **anthropic-api**: set the `ANTHROPIC_API_KEY` env var
-- **openai**: `pip install openai` and set `OPENAI_API_KEY`
-- **openai-api**: set the `OPENAI_API_KEY` env var
-- **gemini**: `npm i -g @google/gemini-cli` and authenticate
-
-### Agent evaluation and project-level runtime context
-
-When the executor is `claude-sdk`, OMK supports a first pass of agent-aware evaluation.
-
-A few concepts worth keeping separate:
-
-- `artifact`: the thing being evaluated — baseline, skill, prompt, agent
-- `variant`: the CLI expression for an experiment group
-- `runtime context`: the runtime environment; currently mainly `cwd`. In project-type agent scenarios it includes the project dir, its `CLAUDE.md`, local skills, and any other environmental factors that affect behavior
-
-In OMK, `agent` is not a catch-all term and neither is `skill`. A cleaner phrasing: **you are comparing how different artifacts behave under different runtime contexts.**
-
-- auto-extracts turns / toolCalls traces
-- supports assertions on tool-call behavior
-- supports running under a specified `cwd`, so Claude Code auto-loads the project's `CLAUDE.md`, skills, and local runtime context
-
-#### Recommended executor
-
-```bash
-omk eval --executor claude-sdk
-```
-
-#### Agent-related assertions
-
-| Assertion | Meaning |
-|---|---|
-| `tools_called` | must call the specified tool(s) |
-| `tools_not_called` | must not call the specified tool(s) |
-| `tools_count_min` / `tools_count_max` | tool-call-count bounds |
-| `tool_output_contains` | output of a specific tool must contain given content |
-| `turns_min` / `turns_max` | turn-count bounds |
-
-#### Three common control setups
-
-**1. Bare-model baseline**
-
-No system prompt and no knowledge-carrying project dir. Requires at least one treatment to compare against:
-
-```bash
-omk eval \
-  --executor claude-sdk \
-  --control baseline \
-  --treatment my-skill
-```
-
-**2. Empty artifact + project-level runtime context**
-
-No system prompt, but runs inside a project dir. This is **not** a strict "bare baseline" — it is "empty artifact + project-level runtime context".
-
-```bash
-omk eval \
-  --executor claude-sdk \
-  --control baseline \
-  --treatment project-env@/path/to/target-project
-```
-
-**3. Explicit artifact injection**
-
-Inject an external `SKILL.md` as the artifact while also keeping the project dir. Good for contrasting "project-level runtime context" vs "explicit single-artifact injection".
-
-```bash
-omk eval \
-  --executor claude-sdk \
-  --control project-env@/path/to/target-project \
-  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
-```
-
-#### Recommended first-round design
-
-For PRD / complex business-knowledge scenarios, start with:
-
-```bash
-omk eval \
-  --executor claude-sdk \
-  --samples skills/evaluate-review/eval-samples.yaml \
-  --control baseline \
-  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
-```
-
-If you want to prove whether "the knowledge sitting inside the project directory" is effective on its own, add a second treatment:
-
-```bash
-omk eval \
-  --executor claude-sdk \
-  --samples skills/evaluate-review/eval-samples.yaml \
-  --control baseline \
-  --treatment project-env@/path/to/target-project,/path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
-```
-
-#### Design tips
-
-- **Always start with `--dry-run`** to confirm samples, variants, and `cwd` are parsed correctly
-- **Project-level controls must differ in `cwd`**: the same prompt under different project dirs hits different runtime contexts
-- **Try PRD scenarios first**: compared to pure coding, they make it easier to validate knowledge completeness, impact-area detection, and business correctness
-
-### Common model configurations
-
-**Don't have Claude?** Most Chinese LLMs (GLM, Qwen, Moonshot, DeepSeek, etc.) are OpenAI-API compatible — use the `openai-api` executor directly:
-
-```bash
-# GLM (Zhipu)
-export OPENAI_API_KEY="your Zhipu API key"
-export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
-omk eval --executor openai-api --model glm-4-plus \
-  --judge-models openai-api:glm-4-plus --no-cache
-
-# Qwen (Alibaba)
-export OPENAI_API_KEY="your Qwen API key"
-export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
-omk eval --executor openai-api --model qwen-plus \
-  --judge-models openai-api:qwen-plus
-
-# DeepSeek
-export OPENAI_API_KEY="your DeepSeek API key"
-export OPENAI_BASE_URL="https://api.deepseek.com"
-omk eval --executor openai-api --model deepseek-chat \
-  --judge-models openai-api:deepseek-chat
-
-# Moonshot (Kimi)
-export OPENAI_API_KEY="your Moonshot API key"
-export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
-omk eval --executor openai-api --model moonshot-v1-8k \
-  --judge-models openai-api:moonshot-v1-8k
-```
-
-**Ollama local model:**
-
-```bash
-omk eval --executor "python examples/custom-executor/ollama-executor.py" \
-  --model llama3 --no-judge
-```
-
-**About the judge:**
+## Documentation
 
-- `--judge-models <list>` picks the LLM judge(s). Format: `executor:model[,executor:model]`. Default: `${executor}:haiku` (or claude:haiku when no `--executor` set)
-- 1 entry = single judge; ≥ 2 entries = multi-judge ensemble + inter-judge agreement
-- If you don't have Claude, point `--judge-models` at whatever you have, e.g. `--judge-models openai-api:glm-4-plus`
-- Add `--no-judge` to skip the LLM judge and rely on assertions alone
+- **[How it works](docs/explanation/architecture.md)** — interleaved scheduling, variant resolution, dual-channel scoring, six-dim report
+- **[Eval sample format](docs/reference/eval-sample-format.md)** — sample schema, scoring formulas, 21+ assertion types, custom JS assertions
+- **[CLI reference](docs/reference/cli.md)** — all seven commands with bash examples and flag tables
+- **[Executors & artifact layout](docs/reference/executors.md)** — built-in / custom executors, agent evaluation, common model configs (Claude / OpenAI / GLM / Qwen / DeepSeek / Moonshot / Ollama)
+- **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
+- **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
+- **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
+- **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / LangSmith / Langfuse / Braintrust etc.
 
 ## Environment variables