oh-my-knowledge 0.0.1 → 0.19.0

package/README.md

@@ -1,448 +1,726 @@
 # oh-my-knowledge
 
-Knowledge artifact evaluation toolkit — benchmark your skills with objective data.
+[![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
+[![CI](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml/badge.svg)](https://github.com/lizhiyao/oh-my-knowledge/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
+[![Node.js Version](https://img.shields.io/node/v/oh-my-knowledge.svg)](https://nodejs.org)
 
-English | [中文](./README.zh-CN.md)
+**English** | [简体中文](./README.zh.md)
 
-**Fixed model, variable knowledge artifact, data speaks.**
+Knowledge-artifact evaluation toolkit — measure your artifact's quality with objective data.
 
-## Why
+**Fix the model, vary the knowledge artifact, let the data speak.**
 
-Teams building AI skills (system prompts, knowledge packages, rule sets) need objective data to prove v2 is better than v1. `oh-my-knowledge` runs controlled experiments: same model, same test cases, only the knowledge artifact changes.
+## Why this tool
 
-## Quick Start
+Teams doing knowledge engineering produce lots of knowledge artifacts (skills today, but also prompts, agents, workflows…). When someone asks "why is v2 better than v1", you need objective data instead of gut feeling. `oh-my-knowledge` solves this with controlled experiments: **same model, same test samples, only the knowledge artifact changes.**
+
+## Key features
+
+- **Controlled-variable offline bench** — fix the model and samples, vary only the artifact; works with Claude Code skills, CLAUDE.md prompts, RAG knowledge bases, or any markdown-based instruction
+- **Six-dimension scoring** — separate signals for Fact / Behavior / LLM-judge / Cost / Efficiency / Stability, so a regression in one axis isn't hidden by gains in another
+- **Production session observability** — parse Claude Code session JSONL traces, measure per-skill failure rate, latency, token cost, and knowledge-gap signals on real user sessions
+- **Knowledge-gap detection** — severity-weighted signals (explicit markers / failed searches / hedging language / repeated failures) quantify risk exposure instead of claiming completeness
+- **Pre-merge CI gate** — `omk bench ci` enforces three-layer all-pass (fact + behavior + llm-judge) semantics, catching single-layer regressions a composite score would hide
+
+## Quick start
 
 ```bash
-# Install globally
-npm i -g oh-my-knowledge
+# install
+npm i oh-my-knowledge -g
 
-# Scaffold a new eval project
+# scaffold an eval project
 omk bench init my-eval
 cd my-eval
 
-# Preview the evaluation plan
+# drop the artifacts you want to compare into skills/
+# option 1: plain .md files (skills/v1.md, skills/v2.md)
+# option 2: full artifact dirs (skills/my-skill-v1/SKILL.md, ...)
+# a single artifact also works — baseline is auto-added as control
+
+# preview the plan
 omk bench run --dry-run
 
-# Run the evaluation
-omk bench run --variants v1,v2
-
-# View the report
-omk bench report
-# Open http://127.0.0.1:7799
-```
-
-## How It Works
-
-```
-eval-samples.json     skills/v1.md     skills/v2.md
-       │                    │                │
-       └────────┬───────────┘                │
-                │                            │
-         ┌──────▼──────┐              ┌──────▼──────┐
-         │  sample +   │              │  sample +   │
-         │  skill v1   │              │  skill v2   │
-         └──────┬──────┘              └──────┬──────┘
-                │                            │
-         ┌──────▼──────┐              ┌──────▼──────┐
-         │  Executor   │              │  Executor   │
-         │ claude      │              │ claude      │
-         │ openai      │              │ openai      │
-         │ gemini      │              │ gemini      │
-         └──────┬──────┘              └──────┬──────┘
-                │                            │
-         ┌──────▼──────────────────────▼──────┐
-         │          Grading                    │
-         │  ┌─────────────┐ ┌──────────────┐  │
-         │  │ Assertions  │ │ LLM Judge    │  │
-         │  │ (18 types)  │ │ (rubric or   │  │
-         │  │             │ │  dimensions) │  │
-         │  └─────────────┘ └──────────────┘  │
-         └──────────────────┬─────────────────┘
-                            │
-                  ┌─────────▼─────────┐
-                  │  Report + Analysis │
-                  │  (JSON/HTML)       │
-                  └───────────────────┘
-```
-
-## Eval Sample Format
-
-Supports both JSON and YAML (`eval-samples.json`, `eval-samples.yaml`, `eval-samples.yml`).
-
-The file contains an array of sample objects. Each sample represents one test case for evaluating a skill.
+# run the evaluation (auto-discovers everything under skills/)
+omk bench run
+```
+
+## Use inside Claude Code
+
+After installing omk, talk to it in natural language from Claude Code:
+
+```
+/omk eval              # evaluate the artifact(s) in the current project
+/omk evolve            # auto-iterate to improve an artifact
+/omk gen-samples       # generate test cases
+```
+
+You can also just say "compare v1 vs v2 for me" or "improve this artifact" — omk picks the right command.
+
+## Features
+
+| Feature | What it does |
+|---|---|
+| **18 assertion types** | substring, regex, JSON Schema, semantic similarity, custom JS function, and more |
+| **Six-dim evaluation** | Fact / Behavior / LLM-judge / Cost / Efficiency / Stability shown independently |
+| **Multi-executor** | Claude CLI / Claude SDK / OpenAI / Gemini / any custom command |
+| **MCP URL fetching** | pull content from private-doc URLs via an MCP server (SSO-protected knowledge bases, etc.) |
+| **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
+| **Parallel execution** | `--concurrency N` runs N tasks at once |
+| **Multi-run variance** | `--repeat N` repeats the eval and computes mean / SD / CI / t-test |
+| **Auto analysis** | detects low-discrimination assertions, flat scores, all-pass / all-fail, expensive samples |
+| **Traceability** | reports carry CLI version, Node version, artifact version fingerprint |
+| **EN / ZH switch** | one-click language toggle in the HTML report |
+
+## How it works
+
+Core idea: **fix the model and the samples, vary only the artifact and runtime context**, use interleaved scheduling to cancel time drift, score via assertions + LLM judge (dual channel), then layer on knowledge-gap signals to quantify risk exposure.
+
+```mermaid
+flowchart TD
+    subgraph Input["① Input"]
+        S["eval-samples<br/>(JSON / YAML)"]
+        A["artifacts<br/>skills/*.md · SKILL.md<br/>baseline · git:name · @cwd"]
+    end
+
+    subgraph Prep["② Preprocess (resolve & fetch)"]
+        V["variant resolution<br/>variant → artifact + runtime context<br/>(cwd / project CLAUDE.md / local skills)"]
+        U["URL fetching<br/>URLs in prompt / context<br/>MCP Server(private docs) → HTTP"]
+    end
+
+    subgraph Schedule["③ Interleaved + concurrent scheduling"]
+        Q["s1-v1 → s1-v2 → s2-v1 → s2-v2 …<br/>--concurrency N · --repeat N"]
+    end
+
+    subgraph Exec["④ Executor (fixed model)"]
+        E["claude / claude-sdk / openai / gemini<br/>anthropic-api / openai-api / custom"]
+        T["claude-sdk extracts<br/>turns / toolCalls trace"]
+        E -.-> T
+    end
+
+    subgraph Score["⑤ Dual-channel scoring"]
+        AS["assertions (18 types)<br/>content / structure / cost / latency<br/>agent: tools_called · turns_min …"]
+        LS["LLM judge<br/>rubric · dimensions (independent per-dim scores)"]
+        CS["composite score<br/>mean of assertion & LLM when both present"]
+        AS --> CS
+        LS --> CS
+    end
+
+    subgraph Analyze["⑥ Auto analysis + knowledge gaps"]
+        D["low-discrimination / flat scores / all-pass or all-fail<br/>expensive samples · variance · t-test"]
+        G["knowledge-gap signals<br/>(quantify risk exposure, not completeness proof)"]
+    end
+
+    subgraph Report["⑦ Report"]
+        R["Six dims: Fact / Behavior / LLM-judge / Cost / Efficiency / Stability<br/>JSON + HTML · blind reveal<br/>CLI/Node/version fingerprint traceable"]
+    end
+
+    S --> U
+    A --> V
+    V --> Q
+    U --> Q
+    Q --> E
+    T --> AS
+    E --> AS
+    E --> LS
+    CS --> D
+    CS --> G
+    D --> R
+    G --> R
+```
+
+**Key design choices:**
+
+- **Interleaved scheduling** removes time drift: different variants of the same sample are dispatched alternately rather than "all of v1 then all of v2", so model load / network jitter can't be mis-attributed to the artifact.
+- **variant = artifact + runtime context**: `name@cwd` lets control groups explicitly declare the "project directory" input, separating "project-level accumulated knowledge" from "explicit artifact injection".
+- **Dual-channel scoring is complementary**: assertions catch deterministic defects (must call tool X, must contain field Y); the LLM judge catches subjective quality (readability, completeness). Mean is taken when both are present.
+- **Knowledge-gap signals** are not part of the score — they are an independent tracking channel that tells you "how much risk exposure this evaluation covered", for convergence tracking, not as a completeness proof.
+
+## Eval sample format
+
+Supports JSON and YAML (`eval-samples.json`, `eval-samples.yaml`, `eval-samples.yml`).
 
 ```json
 [
   {
     "sample_id": "s001",
-    "prompt": "Review this code",
+    "prompt": "Review this code for security issues",
     "context": "function auth(u, p) { db.query('SELECT * FROM users WHERE name=' + u); }",
-    "rubric": "Should identify SQL injection and suggest parameterized queries",
+    "rubric": "Should identify SQL injection risk and recommend parameterized queries",
     "assertions": [
       { "type": "contains", "value": "SQL injection", "weight": 1 },
       { "type": "contains", "value": "parameterized", "weight": 1 },
-      { "type": "not_contains", "value": "looks good", "weight": 0.5 },
-      { "type": "json_valid" },
-      { "type": "cost_max", "value": 0.01 },
-      { "type": "custom", "fn": "my-assertion.mjs", "weight": 1 }
+      { "type": "not_contains", "value": "looks fine", "weight": 0.5 }
     ],
     "dimensions": {
-      "security": "Should identify injection vulnerability",
-      "actionability": "Should provide concrete fix with code"
+      "security": "did it identify the injection vulnerability?",
+      "actionability": "did it give directly usable fix code?"
     }
   }
 ]
 ```
 
-### Field Reference
+### Fields
 
 | Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| `sample_id` | `string` | **Yes** | Unique identifier for the sample (e.g., `"s001"`). Used in reports and analysis to reference this test case. |
-| `prompt` | `string` | **Yes** | The user prompt sent to the model. This is the task or question the model should answer. |
-| `context` | `string` | No | Additional context appended to the prompt (e.g., code snippet, document text). If provided, it is wrapped in a code block and concatenated after `prompt`. |
-| `rubric` | `string` | No | Natural language scoring criteria for the LLM judge. The judge model reads this rubric and scores the output 1-5. Use when you need semantic/qualitative evaluation. |
-| `assertions` | `array` | No | List of deterministic and async checks applied to the model output. Each assertion is an object with a `type` field (see [Assertion Types](#assertion-types)). |
-| `assertions[].type` | `string` | **Yes** | The assertion type (e.g., `"contains"`, `"json_valid"`, `"custom"`). See full list below. |
-| `assertions[].value` | `string\|number` | Varies | The value to check against. Required for `contains`, `starts_with`, `equals`, `min_length`, `cost_max`, etc. |
-| `assertions[].values` | `array` | Varies | Array of strings. Required for `contains_all` and `contains_any`. |
-| `assertions[].pattern` | `string` | Varies | Regex pattern. Required for `regex` type. |
-| `assertions[].flags` | `string` | No | Regex flags (default: `"i"`). Only used with `regex` type. |
-| `assertions[].schema` | `object` | Varies | JSON Schema object. Required for `json_schema` type. Validated via [ajv](https://ajv.js.org/) (full JSON Schema spec). |
-| `assertions[].reference` | `string` | Varies | Reference text for semantic comparison. Required for `semantic_similarity` type. |
-| `assertions[].threshold` | `number` | No | Minimum score (1-5) to consider a semantic similarity match passing. Default: `3`. |
-| `assertions[].fn` | `string` | Varies | Path to a `.mjs` file exporting the check function. Required for `custom` type. Resolved relative to the samples file directory. |
-| `assertions[].weight` | `number` | No | Weight of this assertion in the composite score calculation. Default: `1`. Higher weight = more influence on the final assertion score. |
-| `dimensions` | `object` | No | Key-value map for multi-dimensional LLM scoring. Each key is a dimension name (e.g., `"security"`), and the value is the rubric text the LLM judge uses to score that dimension (1-5). Scores are averaged into a single LLM score. |
+|---|---|---|---|
+| `sample_id` | `string` | **yes** | Unique sample ID |
+| `prompt` | `string` | **yes** | User prompt sent to the model |
+| `context` | `string` | no | Extra context (e.g. code). Wrapped in a code block and appended to the prompt. URLs are auto-fetched at runtime. |
+| `rubric` | `string` | no | Scoring guideline for the LLM judge (1-5 scale) |
+| `assertions` | `array` | no | Assertion checks; see [assertion types](#assertion-types) |
+| `assertions[].type` | `string` | **yes** | Assertion type |
+| `assertions[].value` | `string\|number` | depends | Check value (required for `contains`, `min_length`, `cost_max`, etc.) |
+| `assertions[].values` | `array` | depends | String array (required for `contains_all`, `contains_any`) |
+| `assertions[].pattern` | `string` | depends | Regex pattern (required for `regex`) |
+| `assertions[].flags` | `string` | no | Regex flags (default `"i"`) |
+| `assertions[].schema` | `object` | depends | JSON Schema object (required for `json_schema`, via [ajv](https://ajv.js.org/)) |
+| `assertions[].reference` | `string` | depends | Reference text (required for `semantic_similarity`) |
+| `assertions[].threshold` | `number` | no | Pass threshold for semantic similarity (default 3) |
+| `assertions[].fn` | `string` | depends | Path to a custom assertion JS file (required for `custom`) |
+| `assertions[].weight` | `number` | no | Weight (default 1) |
+| `dimensions` | `object` | no | Multi-dimension scoring; key = dimension name, value = scoring guideline |
+
+### URL auto-fetching
+
+URLs in `prompt` and `context` are auto-fetched before evaluation and inlined into the text. Useful when referencing online docs, API references, etc.:
 
-**Scoring priority:** If both `assertions` and `rubric`/`dimensions` are present, the composite score is a 50/50 weighted average. If only one is present, that score is used directly. If none are present, the score is 0.
+```json
+{
+  "sample_id": "s001",
+  "prompt": "Generate test cases from this PRD: https://wiki.example.com/prd/feature-x"
+}
+```
+
+At runtime, URLs are replaced with the actual content. Fetch order: MCP Server first for matching URLs (e.g. SSO-protected private docs), then plain HTTP for the rest. URLs already resolved by MCP are not re-fetched via HTTP.
+
+**Private-doc URLs**: drop a `.mcp.json` config file into the project dir, or pass `--mcp-config <path>`:
+
+```json
+{
+  "mcpServers": {
+    "docs": {
+      "command": "npx",
+      "args": ["@example/docs-mcp-server"],
+      "env": { "DOCS_API_TOKEN": "xxx" },
+      "urlPatterns": ["docs.example.com"],
+      "fetchTool": {
+        "name": "fetch_doc",
+        "urlTransform": {
+          "regex": "docs\\.example\\.com/([^/]+/[^/]+)/([^/?#]+)",
+          "params": { "namespace": "$1", "slug": "$2" }
+        },
+        "contentExtract": "data.body"
+      }
+    }
+  }
+}
+```
 
-**Prompt construction:** The final prompt sent to the model is: `prompt` alone if no `context`, or `prompt + "\n\n```\n" + context + "\n```"` if `context` is provided.
+**Public URLs**: fetched via plain HTTP. If they require auth, make sure the shell already has network access configured (VPN, proxy, etc.).
 
-### Grading Strategy
+### Scoring strategy
 
-Each sample can use up to three grading methods. They can be used alone or combined.
+#### 1. Assertion score
 
-#### 1. Assertions (deterministic scoring)
+Rule-based local checks; each assertion yields pass/fail.
 
-Assertions are rule-based checks that run locally without any LLM calls (except `semantic_similarity` and `custom`). Each assertion produces a **pass/fail** result.
+**Formula:**
 
-**How the assertion score is calculated:**
+- Pass rate = sum of passed assertion weights / total weight (0–1)
+- Score = 1 + pass_rate × 4 (mapped to 1–5)
+- Example: 3 assertions (weight 1 each), 2 pass → pass rate 2/3 → score = 1 + 0.67 × 4 = **3.67**
 
-1. Each assertion has a `weight` (default: 1)
-2. Sum the weights of all passing assertions → `passedWeight`
-3. Sum the weights of all assertions → `totalWeight`
-4. Compute ratio: `passedWeight / totalWeight` (0.0 ~ 1.0)
-5. Normalize to 1-5 scale: **`score = 1 + ratio × 4`**
+#### 2. Rubric / Dimensions score
 
-Example: 3 assertions (weight 1 each), 2 pass → ratio = 2/3 → score = 1 + 2.67 = **3.67**
+The judge model (default `haiku`) scores 1–5 against the rubric. In `dimensions` mode, each dimension is scored independently and then averaged.
 
-#### 2. Rubric (single LLM judge)
+#### 3. Composite score
 
-A judge model (default: `haiku`, configurable via `--judge-model`) reads the model output and scores it against the rubric text. Returns an integer score from **1** (fail) to **5** (excellent) with a brief reason.
+| Condition | Formula |
+|---|---|
+| Only assertions | `assertionScore` |
+| Only LLM judge | `llmScore` |
+| Both present | `(assertionScore + llmScore) / 2` |
+| Neither | `0` |
 
-Only one of `rubric` or `dimensions` should be used per sample. If both are present, `dimensions` takes priority.
+### Assertion types
 
-#### 3. Dimensions (multi-dimensional LLM judge)
+**Deterministic assertions (18 total):**
 
-Each dimension is scored independently by the judge model (1-5). The dimension scores are **averaged** to produce a single LLM score.
+| Type | Description |
+|---|---|
+| `contains` / `not_contains` | substring must / must-not appear |
+| `regex` | regex match |
+| `min_length` / `max_length` | length bounds |
+| `json_valid` / `json_schema` | JSON validation |
+| `starts_with` / `ends_with` | prefix / suffix |
+| `equals` / `not_equals` | exact match |
+| `word_count_min` / `word_count_max` | word-count bounds |
+| `contains_all` / `contains_any` | multi-value match |
+| `cost_max` / `latency_max` | cost / latency caps |
+| `semantic_similarity` | LLM-based semantic similarity |
+| `custom` | custom JS function (30 s timeout) |
 
-Example: `security: 5`, `actionability: 3` → LLM score = **(5 + 3) / 2 = 4.0**
+### Custom assertion
 
-#### Composite Score
+```js
+// my-assertion.mjs
+export default function(output, { sample, assertion }) {
+  return { pass: output.includes('SQL'), message: 'checked for SQL keyword' };
+}
+```
 
-| What's present | Composite score formula |
-|----------------|----------------------|
-| Assertions only | `assertionScore` |
-| LLM only (rubric or dimensions) | `llmScore` |
-| Both | `(assertionScore + llmScore) / 2` |
-| Neither | `0` |
+## Six-dim evaluation
 
-All scores are on a **1-5 scale**. A score of 0 means no grading criteria were defined.
+Reports display results across six independent dimensions. The three scoring layers — Fact / Behavior / LLM-judge — are shown separately so you see **which layer regressed** instead of a single composite number:
 
-### Assertion Types
+| Dimension | Metric | Description |
+|---|---|---|
+| 📋 **Fact** | fact-assertion pass rate | rule-verifiable assertions like `contains` / `json_schema` / `fact_check`, mapped to 1-5 |
+| 🛠️ **Behavior** | behavior-assertion pass rate | execution-compliance assertions like `tools_called` / `tool_output_contains` / `turns_max` |
+| 💬 **LLM-judge** | rubric score | 1-5 scored by the judge model against a predefined rubric; subjective, catches what rules miss |
+| 💰 **Cost** | total cost, input/output tokens | API cost based on token usage and model pricing |
+| ⚡ **Efficiency** | average latency (ms) | end-to-end latency from request to full response |
+| 🛡️ **Stability** | CV (coefficient of variation) | score consistency across repeated runs (`--repeat ≥ 2`); single-run shows `—`, **honestly acknowledging what can't be measured** |
 
-**Deterministic (sync, no LLM):**
+## CLI reference
 
-| Type | Fields | Description |
-|------|--------|-------------|
-| `contains` | `value`, `weight` | Output contains substring (case-insensitive) |
-| `not_contains` | `value`, `weight` | Output does NOT contain substring |
-| `regex` | `pattern`, `flags`, `weight` | Output matches regex |
-| `min_length` | `value`, `weight` | Output length >= value |
-| `max_length` | `value`, `weight` | Output length <= value |
-| `json_valid` | `weight` | Output is valid JSON |
-| `json_schema` | `schema`, `weight` | Output matches JSON Schema (full spec via ajv) |
-| `starts_with` | `value`, `weight` | Output starts with string (case-insensitive) |
-| `ends_with` | `value`, `weight` | Output ends with string (case-insensitive) |
-| `equals` | `value`, `weight` | Output exactly equals value (after trim) |
-| `not_equals` | `value`, `weight` | Output does not equal value (after trim) |
-| `word_count_min` | `value`, `weight` | Word count >= value |
-| `word_count_max` | `value`, `weight` | Word count <= value |
-| `contains_all` | `values`, `weight` | Output contains ALL substrings |
-| `contains_any` | `values`, `weight` | Output contains at least one substring |
-| `cost_max` | `value`, `weight` | Execution cost (USD) <= value |
-| `latency_max` | `value`, `weight` | Execution latency (ms) <= value |
+### `omk bench run`
 
-**Async (LLM-based):**
+```bash
+omk bench run [options]
 
-| Type | Fields | Description |
-|------|--------|-------------|
-| `semantic_similarity` | `reference`, `threshold`, `weight` | LLM judges similarity to reference text (threshold default: 3) |
-| `custom` | `fn`, `weight` | Load external JS function (see below) |
+options:
+  --samples <path>       sample file (default: eval-samples.json, also detects .yaml/.yml)
+  --skill-dir <path>     artifact dir (default: skills)
+  --control <expr>       control-group variant expression (experiment role = control)
+  --treatment <v1,v2>    treatment-group variant expressions, comma-separated
+                         at least one of --control / --treatment is required
+                         (unless you use --config or --each)
+                         special values: baseline (empty artifact), git:name (git HEAD),
+                         git:ref:name (specific commit), path with "/" (read file directly)
+  --config <path>        YAML/JSON config file (evaluation-as-code); declares
+                         samples + variants + model + executor in one file; CLI
+                         flags override config fields when both are provided
+  --model <name>         model under test (default: sonnet)
+  --judge-model <name>   judge model (default: haiku)
+  --output-dir <path>    output dir (default: ~/.oh-my-knowledge/reports/)
+  --no-judge             skip the LLM judge
+  --no-cache             disable result cache (on by default; identical inputs reuse)
+  --dry-run              preview only
+  --blind                blind mode
+  --concurrency <n>      parallel tasks (default: 1)
+  --timeout <sec>        per-task executor timeout (default: 120)
+  --repeat <n>           repeat N times for variance analysis (default: 1)
+  --executor <name>      executor (default: claude); supports custom commands
+  --skip-preflight       skip pre-evaluation model reachability check
+  --mcp-config <path>    MCP config for fetching private-doc URLs via MCP Server
+                         (default: .mcp.json in cwd)
+  --no-serve             don't auto-start the report server after the run
+  --verbose              print per-sample details (duration, tokens, output preview)
+  --each                 batch mode: evaluate each artifact independently vs baseline
+                         requires {name}.eval-samples.json paired with each artifact
+```
 
-### Custom Assertions
+### `omk bench run --each` (batch mode)
 
-Create a `.mjs` file that exports a function:
+When `skills/` contains several **independent** artifacts, use `--each` to evaluate each one against baseline and produce a merged report.
 
-```js
-// my-assertion.mjs
-export default function(output, { sample, assertion }) {
-  const hasKeyword = output.includes('SQL');
-  return { pass: hasKeyword, message: 'Checked for SQL keyword' };
-}
+```
+skills/
+├── asset.md                       ← artifact file
+├── asset.eval-samples.json        ← paired samples
+├── home.md
+├── home.eval-samples.json
+└── product/                       ← directory format also supported
+    ├── SKILL.md
+    └── eval-samples.json
 ```
 
-Reference it in your sample: `{ "type": "custom", "fn": "my-assertion.mjs" }`. The `fn` path is resolved relative to the samples file directory.
+Pairing rules:
 
-## Report Example
+- `{name}.md` → looks for `{name}.eval-samples.json` in the same dir
+- `{name}/SKILL.md` → looks for `{name}/eval-samples.json`
+- artifacts without paired samples are skipped with a warning
 
-After running `omk bench run --variants v1,v2`, the tool outputs a JSON report (also saved to `~/.oh-my-knowledge/reports/`):
+```bash
+omk bench run --each
+omk bench run --each --dry-run
+```
 
-```json
-{
-  "id": "2026-03-24T15-30-45-v1-v2",
-  "meta": {
-    "variants": ["v1", "v2"],
-    "model": "sonnet",
-    "judgeModel": "haiku",
-    "executor": "claude",
-    "sampleCount": 3,
-    "taskCount": 6,
-    "totalCostUSD": 0.0234,
-    "timestamp": "2026-03-24T15:30:45.000Z",
-    "cliVersion": "0.3.0",
-    "nodeVersion": "v22.0.0",
-    "skillHashes": { "v1": "a1b2c3d4e5f6", "v2": "f6e5d4c3b2a1" }
-  },
-  "summary": {
-    "v1": {
-      "totalSamples": 3,
-      "successCount": 3,
-      "errorCount": 0,
-      "avgCompositeScore": 3.67,
-      "avgAssertionScore": 3.0,
-      "avgLlmScore": 4.33,
-      "avgDurationMs": 2500,
-      "avgTotalTokens": 1850,
-      "totalCostUSD": 0.0112
-    },
-    "v2": {
-      "totalSamples": 3,
-      "successCount": 3,
-      "errorCount": 0,
-      "avgCompositeScore": 4.5,
-      "avgAssertionScore": 5.0,
-      "avgLlmScore": 4.0,
-      "avgDurationMs": 2800,
-      "avgTotalTokens": 2100,
-      "totalCostUSD": 0.0122
-    }
-  },
-  "results": [
-    {
-      "sample_id": "s001",
-      "variants": {
-        "v1": {
-          "ok": true,
-          "compositeScore": 3.5,
-          "assertions": {
-            "passed": 1,
-            "total": 2,
-            "score": 3.0,
-            "details": [
-              { "type": "contains", "value": "SQL injection", "weight": 1, "passed": true },
-              { "type": "contains", "value": "parameterized", "weight": 1, "passed": false }
-            ]
-          },
-          "llmScore": 4,
-          "llmReason": "Identified the vulnerability but did not provide a complete fix",
-          "durationMs": 2300,
-          "inputTokens": 850,
-          "outputTokens": 1200,
-          "totalTokens": 2050,
-          "costUSD": 0.0038,
-          "outputPreview": "This code has a SQL injection vulnerability..."
-        },
-        "v2": {
-          "ok": true,
-          "compositeScore": 4.5,
-          "assertions": {
-            "passed": 2,
-            "total": 2,
-            "score": 5.0,
-            "details": [
-              { "type": "contains", "value": "SQL injection", "weight": 1, "passed": true },
-              { "type": "contains", "value": "parameterized", "weight": 1, "passed": true }
-            ]
-          },
-          "llmScore": 4,
-          "llmReason": "Thorough analysis with actionable fix code",
-          "durationMs": 2600,
-          "inputTokens": 900,
-          "outputTokens": 1400,
-          "totalTokens": 2300,
-          "costUSD": 0.0042,
-          "outputPreview": "## Security Issue: SQL Injection\n\nThe code is vulnerable..."
-        }
-      }
-    }
-  ],
-  "analysis": {
-    "insights": [
-      {
-        "type": "uniform_scores",
-        "severity": "info",
-        "message": "1/3 samples show score difference < 0.5 between variants"
-      }
-    ],
-    "suggestions": []
-  }
-}
+### `omk bench gen-samples` (generate test cases)
+
+Reads an artifact's content and uses an LLM to auto-generate eval-samples. Review and edit them before running eval.
+
+```bash
+# generate for a specific artifact (writes eval-samples.json)
+omk bench gen-samples skills/my-skill.md
+
+# batch-generate for every artifact under skills/ that lacks samples
+omk bench gen-samples --each
+
+# specify sample count
+omk bench gen-samples skills/my-skill.md --count 10
 ```
 
-**Reading the report:**
+Options:
 
-- **`summary`** gives a quick comparison — in this example, v2 scores higher (4.5 vs 3.67) because it passes more assertions
-- **`results`** shows per-sample detail — you can see exactly which assertions passed/failed and why
-- **`analysis`** flags patterns — here it notes one sample has similar scores across variants
-- View the HTML version at `http://127.0.0.1:7799/run/{id}` after running `omk bench report`
+```
+  --each                 batch-generate for every artifact missing samples
+  --count <n>            samples per artifact (default: 5)
+  --model <name>         model used for generation (default: sonnet)
+  --skill-dir <path>     artifact dir (default: skills), used with --each
+```
 
-## CLI Reference
+### `omk bench evolve` (self-iterating improvement)
 
-### `omk bench run`
+Lets the AI iterate an artifact automatically: evaluate → analyze weak spots → LLM rewrites → evaluate again → keep if the score went up, drop otherwise → repeat.
 
 ```bash
-omk bench run [options]
+# basic: iterate 5 rounds
+omk bench evolve skills/my-skill.md
+
+# set rounds and target score
+omk bench evolve skills/my-skill.md --rounds 10 --target 4.5
+```
 
 Options:
-  --samples <path>       Sample file (default: eval-samples.json, auto-detects .yaml/.yml)
-  --skill-dir <path>     Skill directory (default: skills)
-  --variants <v1,v2>     Variant names (default: v1,v2)
-  --model <name>         Model under test (default: sonnet)
-  --judge-model <name>   Judge model (default: haiku)
-  --output-dir <path>    Output directory (default: ~/.oh-my-knowledge/reports/)
-  --no-judge             Skip LLM judging
-  --dry-run              Preview only
-  --blind                Blind A/B mode: hide variant names in report
-  --concurrency <n>      Number of parallel tasks (default: 1)
-  --repeat <n>           Run evaluation N times for variance analysis (default: 1)
-  --executor <name>      Executor (default: claude)
+
 ```
+  --rounds <n>           max iteration rounds (default: 5)
+  --target <score>       stop early when the score reaches this threshold
+  --samples <path>       sample file (default: eval-samples.json)
+  --improve-model <name> model used for rewrites (default: sonnet)
+```
+
+Each round's output is saved under `skills/evolve/` (`my-skill.r0.md`, `my-skill.r1.md`…), so you can `diff` to see what the AI changed. The best round is written back to the original file.
 
 ### `omk bench ci`
 
-Run evaluation in CI and exit with pass/fail code.
+Run the evaluation inside CI. Exit code 0 on pass, 1 on fail — can be wired into gates directly.
+
+The gate is **three-layer all-pass**: `avgFactScore >= threshold AND avgBehaviorScore >= threshold AND avgJudgeScore >= threshold`. Any layer below threshold is FAIL, and the output shows which layer broke. This stops cases like `fact 4.5→2.5 but judge 3→5` from passing via composite averaging — if one layer regresses, the gate catches it.
 
 ```bash
 omk bench ci [options]
-
-Options:
-  (same as "bench run", plus:)
-  --threshold <number>   Minimum composite score to pass (default: 3.5)
+  --threshold <number>   per-layer minimum score (default: 3.5); applied
+                         independently to fact / behavior / judge
 ```
 
-Exit code 0 = all variants pass, 1 = at least one variant below threshold.
-
 ### `omk bench report`
 
+Start the report server to browse historical reports, submit feedback, and delete reports.
+
 ```bash
 omk bench report [options]
-
-Options:
-  --port <number>        Server port (default: 7799)
-  --reports-dir <path>   Reports directory (default: ~/.oh-my-knowledge/reports/)
+  --port <number>        server port (default: 7799)
 ```
 
 ### `omk bench init`
 
 ```bash
-omk bench init [dir]    # Scaffold a new eval project
+omk bench init [dir]     # scaffold an eval project
 ```
 
-## Features
+## `omk analyze` — production observability
 
-### Blind A/B Testing
+`omk bench run` is **offline evaluation** (fixed controls, repeatable, scored). Production is different — no control group, no ground truth, no repetition, so scoring isn't valid there. `omk analyze` turns existing Claude Code session traces into **skill-health reports** (coverage, gap signals, execution stability, tokens/latency per skill). It gives you clues about **which skill is worth re-evaluating offline**, not a production score.
 
-Use `--blind` to hide variant names in reports. Variants are randomly labeled as "Variant A", "Variant B", etc. A reveal button in the HTML report shows the mapping.
+```bash
+# analyze all cc sessions of the current project (auto-infers kb from the trace)
+omk analyze ~/.claude/projects/-Users-you-Documents-my-project
 
-### Parallel Execution
+# restrict to the last 7 days / 24 hours / 30 minutes
+omk analyze ~/.claude/projects/my-project --last 7d
 
-Use `--concurrency N` to run N tasks in parallel. Tasks maintain interleaved scheduling order to reduce time bias.
+# absolute time window
+omk analyze ~/.claude/projects/my-project --from 2026-04-01T00:00:00Z --to 2026-04-15T23:59:59Z
 
-### Multi-run Variance Analysis
+# whitelist specific skills
+omk analyze ~/.claude/projects/my-project --skills audit,polish
 
-Use `--repeat N` to run the evaluation N times. The report includes:
-- Per-variant mean, standard deviation, 95% confidence interval
-- Pairwise Welch's t-test between variants (significance at p < 0.05)
+# override the inferred knowledge-base root
+omk analyze ~/.claude/projects/my-project --kb /path/to/project
+```
 
-### Auto-analysis
+The command writes `~/.oh-my-knowledge/analyses/<timestamp>-skill-health.json`. Browse results alongside bench reports with `omk bench report` — the homepage has a "📊 Skill Health Reports" link, and each skill card also has a "trend →" link to its time-series view. For two reports side-by-side, use the compare selector on `/analyses`.
 
-After each evaluation, the toolkit automatically detects:
-- **Low-discrimination assertions**: assertions with identical results across all variants
-- **Uniform scores**: samples where variants score within 0.5 of each other
-- **All-pass / all-fail**: assertions that may be too loose or too strict
-- **High-cost samples**: samples with disproportionately high cost
+**What you get per skill:**
 
-Insights and suggestions are shown in the HTML report.
+- **Knowledge usage** — which KB files this skill actually read (coverage %)
+- **Knowledge gaps** — four weighted signals (failed search / model-flagged gap / hedging / repeated miss); hedging goes through an LLM-assisted classifier to filter out business-possibility hedging vs genuine knowledge uncertainty
+- **Execution stability** — tool-failure rate; a skill with > 20% failures gets a warning that its gap signals may be environmental noise rather than real knowledge gaps
+- **Usage cost** — billable tokens (input+output) separate from cached tokens, total duration
 
-### Human Feedback
+**What this is NOT:**
 
-The HTML report includes star rating (1-5) and comment forms for each sample-variant pair. Feedback is persisted to the report JSON via `POST /api/run/:id/feedback`.
+- Not a general APM (request/response/latency tracing is Langfuse / Datadog territory)
+- Not streaming / alerting (batch only — run on a cron if you want periodic snapshots)
+- Not a production score (no control group, no ground truth — use `omk bench run` for scoring)
 
-### Traceability
+## Executors
 
-Reports include `cliVersion`, `nodeVersion`, and `skillHashes` (SHA-256 of each skill file) in metadata for reproducibility.
+### Built-in executors
 
-## Executors
+| Executor | When to use | Description |
+|---|---|---|
+| `claude` | default | invokes `claude -p` via Claude CLI |
+| `claude-sdk` | structured output | uses Claude Agent SDK — no stdout parsing, avoids buffer truncation |
+| `openai` | cross-vendor comparison | invokes `openai api` CLI |
+| `gemini` | cross-vendor comparison | invokes `gemini` CLI |
+| `anthropic-api` | no CLI needed | calls Anthropic HTTP API directly (needs `ANTHROPIC_API_KEY`) |
+| `openai-api` | no CLI needed | calls OpenAI HTTP API directly (needs `OPENAI_API_KEY`) |
+
+API-direct executors support custom base URLs via env: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
 
-Use `--executor` to select which model provider to use.
+### Custom executor
 
-| Executor | CLI Tool | Default Model | Auth |
-|----------|----------|---------------|------|
-| `claude` | `claude -p` | `sonnet` | Claude Max plan or API key |
-| `openai` | `openai api chat.completions.create` | `gpt-4o` | `OPENAI_API_KEY` env var |
-| `gemini` | `gemini` (stdin pipe) | Default Gemini model | Google account or `GOOGLE_API_KEY` |
+Any shell command can serve as an executor, communicating via stdin/stdout JSON:
 
 ```bash
-# Use OpenAI
-omk bench run --executor openai --model gpt-4o --variants v1,v2
+omk bench run --executor "python my_provider.py"
+omk bench run --executor "./my-executor.sh"
+```
+
+**Protocol:**
+
+- **input** (stdin): JSON `{"model":"...","system":"...","prompt":"..."}`
+- **output** (stdout): JSON `{"output":"model reply","inputTokens":0,"outputTokens":0,"costUSD":0}`
+- stdout only needs to return the fields you care about; others default to 0. Plain-text output (no tokens/cost parsing) is also fine.
+- non-zero exit code counts as failure
 
-# Use Gemini
-omk bench run --executor gemini --model gemini-2.5-pro --variants v1,v2
+### Artifact directory layout
 
-# Compare the same skill across providers (run separately, compare reports)
-omk bench run --executor claude --model sonnet --variants v1,v2
-omk bench run --executor openai --model gpt-4o --variants v1,v2
+The built-in executors (claude / openai / gemini) support two artifact layouts, mixable in the same run:
+
+```
+skills/
+├── v1.md                    # option 1: plain .md file
+└── my-skill/                # option 2: full artifact dir
+    ├── SKILL.md             #   this file is auto-loaded as system prompt
+    ├── config.json          #   other files don't participate in eval, kept for completeness
+    └── scripts/
+```
+
+**Variant resolution rules:**
+
+`variant` is the experiment-group expression. After resolution, OMK produces an `artifact` plus an optional `runtime context` (currently mainly `cwd`).
+
+| Format | Meaning |
+|---|---|
+| `name` | looks up `name.md` or `name/SKILL.md` in the artifact dir, resolves to one artifact |
+| `baseline` | empty artifact, no system prompt — think "nothing at all" |
+| `project-env@/path/to/project` | empty artifact, but run in the specified project dir — observe project-level runtime context alone |
+| `git:name` | reads the last-committed version of an artifact from git HEAD |
+| `git:ref:name` | reads an artifact from a specific commit |
+| `./path/to/file.md` | path with `/`: read the file directly as an artifact |
+| `variant@/path/to/project` | attach a run dir to any variant; supports `name@cwd`, `git:name@cwd`, `/file.md@cwd` |
+
+When both `--control` and `--treatment` are omitted, use `--config eval.yaml` or `--each`. With `--each`, `baseline` is auto-added as control and every discovered artifact becomes a treatment.
+
+```bash
+# explicit: one control, one or more treatments
+omk bench run --control v1 --treatment v2
+omk bench run --control baseline --treatment v1,v2,v3
+
+# compare empty artifact vs explicit artifact
+omk bench run --control baseline --treatment my-skill
+
+# observe project-level runtime context in isolation (use a self-describing label)
+omk bench run --control baseline --treatment project-env@/path/to/target-project
+
+# compare "project-level runtime context" vs "explicit artifact injection"
+omk bench run \
+  --control project-env@/path/to/target-project \
+  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
+
+# before vs after (old version read from git history)
+omk bench run --control git:my-skill --treatment my-skill
+
+# direct file paths
+omk bench run --control ./old-skill.md --treatment ./new-skill.md
+
+# config-file driven (evaluation-as-code)
+omk bench run --config eval.yaml
 ```
 
 **Prerequisites:**
-- **claude**: Install [Claude Code](https://claude.ai/code) and authenticate
+
+- **claude**: install [Claude Code](https://claude.ai/code) and authenticate
+- **claude-sdk**: install [Claude Code](https://claude.ai/code) and authenticate (uses Agent SDK, no CLI stdout parsing)
+- **anthropic-api**: set the `ANTHROPIC_API_KEY` env var
 - **openai**: `pip install openai` and set `OPENAI_API_KEY`
-- **gemini**: `npm i -g @google/gemini-cli` and authenticate with Google
+- **openai-api**: set the `OPENAI_API_KEY` env var
+- **gemini**: `npm i -g @google/gemini-cli` and authenticate
+
+### Agent evaluation and project-level runtime context
+
+When the executor is `claude-sdk`, OMK supports a first pass of agent-aware evaluation.
+
+A few concepts worth keeping separate:
+
+- `artifact`: the thing being evaluated — baseline, skill, prompt, agent
+- `variant`: the CLI expression for an experiment group
+- `runtime context`: the runtime environment; currently mainly `cwd`. In project-type agent scenarios it includes the project dir, its `CLAUDE.md`, local skills, and any other environmental factors that affect behavior
+
+In OMK, `agent` is not a catch-all term and neither is `skill`. A cleaner phrasing: **you are comparing how different artifacts behave under different runtime contexts.**
+
+- auto-extracts turns / toolCalls traces
+- supports assertions on tool-call behavior
+- supports running under a specified `cwd`, so Claude Code auto-loads the project's `CLAUDE.md`, skills, and local runtime context
+
+#### Recommended executor
+
+```bash
+omk bench run --executor claude-sdk
+```
+
+#### Agent-related assertions
+
+| Assertion | Meaning |
+|---|---|
+| `tools_called` | must call the specified tool(s) |
+| `tools_not_called` | must not call the specified tool(s) |
+| `tools_count_min` / `tools_count_max` | tool-call-count bounds |
+| `tool_output_contains` | output of a specific tool must contain given content |
+| `turns_min` / `turns_max` | turn-count bounds |
+
+#### Three common control setups
+
+**1. Bare-model baseline**
 
-## Environment Variables
+No system prompt and no knowledge-carrying project dir. Requires at least one treatment to compare against:
+
+```bash
+omk bench run \
+  --executor claude-sdk \
+  --control baseline \
+  --treatment my-skill
+```
+
+**2. Empty artifact + project-level runtime context**
+
+No system prompt, but runs inside a project dir. This is **not** a strict "bare baseline" — it is "empty artifact + project-level runtime context".
+
+```bash
+omk bench run \
+  --executor claude-sdk \
+  --control baseline \
+  --treatment project-env@/path/to/target-project
+```
+
+**3. Explicit artifact injection**
+
+Inject an external `SKILL.md` as the artifact while also keeping the project dir. Good for contrasting "project-level runtime context" vs "explicit single-artifact injection".
+
+```bash
+omk bench run \
+  --executor claude-sdk \
+  --control project-env@/path/to/target-project \
+  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
+```
+
+#### Recommended first-round design
+
+For PRD / complex business-knowledge scenarios, start with:
+
+```bash
+omk bench run \
+  --executor claude-sdk \
+  --samples skills/evaluate-review/eval-samples.yaml \
+  --control baseline \
+  --treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
+```
+
+If you want to prove whether "the knowledge sitting inside the project directory" is effective on its own, add a second treatment:
+
+```bash
+omk bench run \
+  --executor claude-sdk \
+  --samples skills/evaluate-review/eval-samples.yaml \
+  --control baseline \
+  --treatment project-env@/path/to/target-project,/path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
+```
+
+#### Design tips
+
+- **Always start with `--dry-run`** to confirm samples, variants, and `cwd` are parsed correctly
+- **Project-level controls must differ in `cwd`**: the same prompt under different project dirs hits different runtime contexts
+- **Try PRD scenarios first**: compared to pure coding, they make it easier to validate knowledge completeness, impact-area detection, and business correctness
+
+### Common model configurations
+
+**Don't have Claude?** Most Chinese LLMs (GLM, Qwen, Moonshot, DeepSeek, etc.) are OpenAI-API compatible — use the `openai-api` executor directly:
+
+```bash
+# GLM (Zhipu)
+export OPENAI_API_KEY="your Zhipu API key"
+export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
+omk bench run --executor openai-api --model glm-4-plus \
+  --judge-model glm-4-plus --no-cache
+
+# Qwen (Alibaba)
+export OPENAI_API_KEY="your Qwen API key"
+export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
+omk bench run --executor openai-api --model qwen-plus \
+  --judge-model qwen-plus
+
+# DeepSeek
+export OPENAI_API_KEY="your DeepSeek API key"
+export OPENAI_BASE_URL="https://api.deepseek.com"
+omk bench run --executor openai-api --model deepseek-chat \
+  --judge-model deepseek-chat
+
+# Moonshot (Kimi)
+export OPENAI_API_KEY="your Moonshot API key"
+export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
+omk bench run --executor openai-api --model moonshot-v1-8k \
+  --judge-model moonshot-v1-8k
+```
+
+**Ollama local model:**
+
+```bash
+omk bench run --executor "python examples/custom-executor/ollama-executor.py" \
+  --model llama3 --no-judge
+```
+
+**About the judge model:**
+
+- `--judge-model` picks the model used by the LLM judge (default `haiku`)
+- `--judge-executor` picks the executor the judge uses (defaults to `--executor`)
+- If you don't have Claude, point `--judge-executor` and `--judge-model` at whatever model you have
+- Add `--no-judge` to skip the LLM judge and rely on assertions alone
+
+## Environment variables
 
 | Variable | Description |
-|----------|-------------|
-| `CCV_PROXY_URL` | Route requests through cc-viewer proxy for real-time visualization |
-| `OMK_BENCH_PORT` | Report server port (default: 7799) |
+|---|---|
+| `CCV_PROXY_URL` | proxy requests through cc-viewer for live eval-traffic visualization |
+| `OMK_BENCH_PORT` | report server port (default: 7799) |
 
 ## Requirements
 
 - Node.js >= 20
-- `claude` CLI installed and authenticated (Max plan works, no API key needed)
+- `claude` CLI (for the default executor and LLM judge; see [Claude Code](https://claude.ai/code))
+  - not needed if you use other executors (openai / gemini) with `--no-judge`
+
+## Security notice
+
+This tool is designed for **local trusted environments** (dev machines, CI pipelines). The following features execute local code — make sure inputs come from a trusted source:
+
+| Feature | Risk | Scope |
+|---|---|---|
+| **Custom assertions** (`custom`) | dynamically loads and executes user-specified `.mjs` files | only use assertion files you authored or reviewed |
+| **eval-samples.json** | assertion configs can reference external file paths | don't use sample files from untrusted sources |
+
+**Recommendations:**
+
+- Do not expose `omk bench report` on the public internet (no auth)
+- Don't use third-party eval-samples you haven't vetted
+- Custom assertions have a 30-second timeout but no sandbox isolation
 
-## License
+---
 
-MIT
+See [CHANGELOG](./CHANGELOG.md) for release notes. Contributions welcome — see [CONTRIBUTING](./CONTRIBUTING.md).