oh-my-knowledge 0.51.1 → 0.52.0

package/README.md

@@ -1,4 +1,4 @@
-# oh-my-knowledge
+# OMK
 
 [![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
 [![npm weekly downloads](https://img.shields.io/npm/dw/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
@@ -8,14 +8,19 @@
 
 **English** | [简体中文](./README.zh.md)
 
-**Stop editing LLM knowledge inputs by gut feel.**
-`oh-my-knowledge` (omk) is a measurement workflow for prompts, RAG context, skills, agents, and workflows. It fixes the executor model and the evaluation samples, changes only the knowledge artifact, then answers the release question that matters: **can v2 ship, and where is it better?**
+**Observe. Measure. Know.**
+
+**OMK makes every knowledge change in your AI application evidence-backed.**
+
+Observe real-world performance, measure version differences, and determine whether the change is effective and the version is ready to ship.
+
+**Same model. Same evaluation samples. Only the knowledge artifact changes.**
 
 ![omk knowledge artifact evaluation flow: doctor / eval / observe / sample / evolve loop](./docs/public/omk-knowledge-flow-en-animated.gif)
 
 📖 **Full documentation: [oh-my-knowledge.pages.dev](https://oh-my-knowledge.pages.dev)** (searchable, English / 简体中文)
 
-## What omk makes measurable
+## What OMK helps you know
 
 | Decision | Command | Evidence you get |
 |---|---|---|
@@ -23,6 +28,7 @@
 | Is v2 actually better than v1? | `omk eval` | one-line verdict, confidence interval, failed samples, cost |
 | Why did it pass or fail? | `omk studio` | report view with scores, diagnostics, and examples |
 | Should this version become the accepted one? | `omk promote` / `omk evolve` | evidence-gated accept or generate a better candidate |
+| What happened during one real AI task? | `omk observe` / Studio Task Trajectory | a trace-backed view of the request, visible Knowledge, tool calls, results, response, and user correction |
 | What did real usage expose? | `omk observe` / `omk sample --from-traces` | production gaps drafted for review; reviewed drafts can become eval samples |
 
 ![omk report — verdict pill "v2 is clearly better than v1 — ready to ship"](./assets/screenshots/report-overview.png)
@@ -57,9 +63,23 @@ Walkthrough: [5-minute quickstart guide](docs/quickstart-skill-eval.md) (recomme
 
 Deeper: [who omk is for](docs/explanation/who-omk-is-for.md) · [CLI reference](docs/reference/cli.md) · [how it works](docs/explanation/architecture.md) · [eval sample format](docs/reference/eval-sample-format.md) · [executors](docs/reference/executors.md) · [artifact layout](docs/reference/artifact-layout.md)
 
-## The omk loop
+## Inspect one Codex task
+
+If you only want to see what happened behind one Codex conversation, you do not need to run `observe ingest` first:
+
+```bash
+omk studio
+```
+
+Studio opens the local Codex conversation overview at `http://127.0.0.1:7799` by default. Select a conversation, then a task, to open **Task Trajectory**. Its four lanes — **Conversation, Actions, Results, and Knowledge** — show the request, AI responses, tool calls, tool returns, and observable context, with drill-downs into normalized events and raw logs.
+
+Running tasks are prioritized and update live. While **Following**, the trajectory advances smoothly as events arrive; after you inspect an earlier point, Studio keeps your position and offers **View updates**. Old logs without a terminal event are marked **End status not recorded** instead of remaining live forever.
+
+Task Trajectory only reconstructs facts observable in the log. It does not reveal or infer hidden reasoning. See [Observe production traces](docs/guides/observe-production.md#inspect-one-task) for the full model.
+
+## The OMK loop
 
-omk is for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The main loop is deliberately controlled:
+OMK is for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The main loop is deliberately controlled:
 
 ```text
 change a prompt / RAG / skill / agent artifact
@@ -166,6 +186,7 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
 - **[CLI reference](docs/reference/cli.md)** — all top-level commands with bash examples and flag tables
 - **[Executors](docs/reference/executors.md)** & **[artifact layout](docs/reference/artifact-layout.md)** — built-in / custom executors; how `variant` resolves to an artifact + runtime context
 - **[How-to guides](docs/guides/agent-eval.md)** — [evaluate an agent](docs/guides/agent-eval.md) (project runtime context) and [use non-Claude models](docs/guides/non-claude-models.md) (GLM / Qwen / DeepSeek / Moonshot / Ollama)
+- **[Observe & inspect task trajectories](docs/guides/observe-production.md)** — browse local Codex conversations, drill into one task, and follow its observable execution live
 - **[Quickstart](docs/quickstart-skill-eval.md)** — first-time five-minute walkthrough
 - **[Example gallery](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** — a set of runnable examples in the repo, arranged simplest-to-richest
 - **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping

package/README.zh.md

@@ -1,4 +1,4 @@
-# oh-my-knowledge
+# OMK
 
 [![npm version](https://img.shields.io/npm/v/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
 [![npm weekly downloads](https://img.shields.io/npm/dw/oh-my-knowledge.svg)](https://www.npmjs.com/package/oh-my-knowledge)
@@ -8,14 +8,19 @@
 
 [English](./README.md) | **简体中文**
 
-**别再凭感觉改 LLM 知识输入。**
-`oh-my-knowledge`（omk）是一套面向 prompt、RAG、skill、agent、workflow 的测量工作流。它固定执行模型和评测用例，只改变知识载体，然后回答发布时真正要追问的问题：**v2 能不能发，好在哪里？**
+**Observe. Measure. Know.**
+
+**OMK，让 AI 应用的知识改动有据可依。**
+
+观测真实表现，量出版本差异，判断改动是否有效、版本能否发布。
+
+**相同模型，相同评测用例，只改变知识载体。**
 
 ![omk 知识载体评测流程：doctor / eval / observe / sample / evolve 闭环](./docs/public/omk-knowledge-flow-animated.gif)
 
 📖 **完整文档：[oh-my-knowledge.pages.dev/zh](https://oh-my-knowledge.pages.dev/zh/)**（可搜索，可切换英文）
 
-## omk 让什么变得可测量
+## OMK 让你知道什么
 
 | 决策问题 | 命令 | 你会得到的证据 |
 |------|------|------|
@@ -23,6 +28,7 @@
 | v2 是否真的优于 v1？ | `omk eval` | 一行 verdict、置信区间、失败样本、成本 |
 | 它为什么通过或失败？ | `omk studio` | 分数、诊断、样本证据的报告视图 |
 | 这个版本是否应成为接受版本？ | `omk promote` / `omk evolve` | 基于证据接受，或生成更好的候选版 |
+| 一次真实 AI 任务中发生了什么？ | `omk observe` / Studio 任务轨迹 | 请求、可见知识、工具调用与结果、回答和用户纠正的可核验轨迹 |
 | 真实使用暴露了哪些知识缺口？ | `omk observe` / `omk sample --from-traces` | 将线上缺口生成待复核草稿，复核后再沉淀为评测样本 |
 
 ![omk 报告 — verdict pill「v2 明显优于 v1，可以发布」](./assets/screenshots/report-overview-zh.png)
@@ -57,9 +63,23 @@ export OMK_EXECUTOR=codex
 
 深入：[为谁、解决什么](docs/zh/explanation/who-omk-is-for.md) · [CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测用例格式](docs/zh/reference/eval-sample-format.md) · [执行器](docs/zh/reference/executors.md) · [知识载体布局](docs/zh/reference/artifact-layout.md)
 
-## omk 的闭环
+## 先看清一次 Codex 任务
+
+只想知道一次 Codex 对话背后发生了什么，不需要先运行 `observe ingest`：
+
+```bash
+omk studio
+```
+
+Studio 默认在 `http://127.0.0.1:7799` 打开本机 Codex 对话总览。先选择一段对话，再选择其中一次任务，即可进入「任务轨迹」：按**对话、执行、结果、知识**四条泳道查看请求、AI 回答、工具调用、工具返回和可见上下文，并可下钻到规范化事件与原始日志。
+
+进行中的任务会优先显示并实时更新。保持「跟随中」时，轨迹会随新事件平滑前进；手动查看历史位置后，页面保留当前位置并提示「查看更新」。旧日志如果没有记录结束事件，会标记为「未记录结束状态」，不会一直冒充进行中。
+
+任务轨迹只还原日志中可观测的事实，不展示或推断隐藏思维。完整说明见[观测与任务轨迹](docs/zh/guides/observe-production.md#查看一次任务)。
+
+## OMK 的闭环
 
-omk 主要给 LLM 知识载体的作者 / 维护者用，帮他们做发布判断；它不是给被动安装 skill 的普通使用者用的。主流程刻意保持受控：
+OMK 主要给 LLM 知识载体的作者 / 维护者用，帮他们做发布判断；它不是给被动安装 skill 的普通使用者用的。主流程刻意保持受控：
 
 ```text
 改了一份 prompt / RAG / skill / agent 知识载体
@@ -166,6 +186,7 @@ RAG 专项评测请看 RAGAS（独立 niche，跟 omk 互补）。完整对比
 - **[CLI 参考](docs/zh/reference/cli.md)** —— 顶层命令的 bash 示例和 flag 表
 - **[执行器](docs/zh/reference/executors.md)** & **[知识载体布局](docs/zh/reference/artifact-layout.md)** —— 内置 / 自定义执行器；variant 如何解析为 artifact + runtime context
 - **[操作指南](docs/zh/guides/agent-eval.md)** —— [评测 agent](docs/zh/guides/agent-eval.md)（项目 runtime context）与[使用非 Claude 模型](docs/zh/guides/non-claude-models.md)（GLM / 通义 / DeepSeek / Moonshot / Ollama）
+- **[观测与任务轨迹](docs/zh/guides/observe-production.md)** —— 浏览本机 Codex 对话，下钻一次任务，并实时跟随可观测执行过程
 - **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
 - **[示例画廊](https://github.com/lizhiyao/oh-my-knowledge/tree/main/examples)** —— 仓库里一组可直接跑的示例，按由简到全排成上手路径
 - **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据；行业 gap 映射

package/dist/assets/agent-skills/omk/SKILL.md

@@ -1,15 +1,15 @@
 ---
 name: omk
 description: |
-  oh-my-knowledge 知识载体评测工具的智能代理。评测 skill（系统提示词）质量，对比不同版本效果，自动迭代改进。
+  OMK（Observe. Measure. Know.）让 AI 应用的知识改动有据可依。观测真实表现，受控测量 prompt / RAG / skill / agent / workflow 的版本差异，判断改动是否有效、版本能否发布，并支持自动迭代改进。
   Use when: 用户提到"评测"、"测评"、"eval"、"benchmark"、"对比 skill"、"改进 skill"、"evolve"、"生成测试用例"、"gen-samples"、"omk"。
 user-invocable: true
 argument-hint: "<doctor|eval|evolve|init|install|list|observe|promote|rollback|sample|studio> [options]"
 ---
 
-# OMK — 知识载体评测
+# OMK — Observe. Measure. Know.
 
-你是 oh-my-knowledge（omk）的智能代理。帮助用户评测、对比、改进 AI skill（系统提示词），用数据说话。
+你是 OMK 的智能代理。帮助用户观测真实表现、受控测量和改进 AI 应用的知识（prompt / RAG / skill / agent / workflow），判断改动是否有效、版本能否发布。
 
 ## 第一步：检查环境
 
@@ -37,7 +37,7 @@ Codex 是 omk 的一等 runtime。运行在 Codex 任务中时，`omk eval` / `d
 | 改进 / 优化 skill | → `omk evolve`（自动多轮迭代） |
 | 生成测试用例 | → `omk sample` |
 | 体检 skill 写法 | → `omk doctor` |
-| 查看 / 浏览报告 | → `omk studio`（启动本地报告浏览器） |
+| 浏览对话、任务轨迹与报告 | → `omk studio`（启动本地知识工作台） |
 | 看真实使用 trace | → `omk observe` |
 | 查看受管 skill 状态 | → `omk list` |
 | 按证据接受 / 回退某版本 | → `omk promote` / `omk rollback` |
@@ -123,6 +123,8 @@ omk sample skills/my-skill/SKILL.md --focus "重点覆盖搜索失败 / 权限
 omk sample --batch
 ```
 
+目标执行器不支持工具拦截时，`omk sample` 会自动生成无 mock 用例。当前 `codex` / `codex-sdk` 属于这种情况；不要手工补 `mocks` 或 `mock_hit`。已有 mocks 用例会被 `omk eval` 在模型调用前拒绝，避免把执行器能力缺口误判成模型失败。`environment.files_available` 只提供题设上下文，不会在 `cwd` 物化文件。
+
 输出位置：目录 skill（`<skill>/SKILL.md`）→ `<skill>/.omk/samples.json`（标准）；扁平 `.md` 单次生成 → 当前目录 `eval-samples.json`（项目级兜底）；扁平 `.md` 的 `--batch` 兼容生成 `<skill-dir>/<name>.eval-samples.json`。
 
 ### 观测真实使用
@@ -136,7 +138,7 @@ omk observe ingest ~/.codex/sessions
 omk observe ~/.claude/projects/<project> --last 7d
 ```
 
-Codex rollout 会保留 `sourceKind=codex`、模型、父子任务、tool call 和 token 证据，并从实际读取的 `skills/<name>/SKILL.md` 归因 skill。`observe ingest` 生成待复核 observation；确认真实知识缺口后，再用 `omk sample --from-traces` 草拟评测用例。
+Codex rollout 会保留 `sourceKind=codex`、模型、父子任务、tool call 和 token 证据，并从实际读取的 `skills/<name>/SKILL.md` 归因 skill。`omk studio` 无需先 ingest，即可从本机 Codex 对话总览进入某次任务的实时轨迹；`observe ingest` 只在需要生成待复核 observation 时运行。任务轨迹按对话、执行、结果和知识呈现结构化事实，并联动检查配对后的工具调用与结果、AI 回答和用户纠正。该页面只呈现 trace 中可观测的执行过程，不推断隐藏思维或失败根因。确认真实知识缺口后，再用 `omk sample --from-traces` 草拟评测用例。
 
 ### 体检 skill 写法
 
@@ -153,16 +155,16 @@ omk doctor skills/my-skill.md
 
 `omk eval` 默认会先跑一次 doctor 当 preflight 门禁，所以一般不用单独跑；想在 eval 之前先把结构问题先扫一遍再跑评测，就单独跑 `omk doctor`。
 
-### 查看 / 浏览报告
+### 浏览对话、任务轨迹与报告
 
 ```bash
-omk studio                                # 启动本地 web 报告浏览器
+omk studio                                # 启动本地知识工作台（默认端口 7799）
 omk studio --port 8080                    # 改端口
 omk studio --host 0.0.0.0                 # 局域网访问（默认 127.0.0.1）
 omk studio --no-open                      # 不自动开浏览器
 ```
 
-Studio 是 skill-centric：列表页（`/`）按 skill 卡片展示健康等级 / 0-100 参考分 / 待优化数 / 趋势；详情页（`/skills/<name>`）左栏列关键问题清单，右栏画健康趋势 + 三档阶段卡（doctor / eval / observe）。访问 `/observe-inbox` 查看 observe inbox 看板。
+Studio 首页直接索引本机 Codex 对话。先选择对话，再选择任务查看四泳道任务轨迹；进行中的任务支持实时跟随。顶部「知识载体」入口用于浏览 doctor / eval / observe 报告，`/observe-inbox` 用于复核 observation。无需为了浏览本机 Codex 对话而先运行 `omk observe ingest`。
 
 ## 第五步：解读结果
 

package/dist/assets/agent-skills/omk/references/commands.md

@@ -602,7 +602,7 @@ omk sample [skillPath] [flags]
 - `--from-traces` `boolean`:from-traces 模式：从 observe inbox 的失败信号回流生成评测用例草稿（provenance: production-trace），落草稿待人工 review。
 - `--lang` `option` (默认 `zh`):输出语言 zh|en，优先级 CLI > OMK_LANG env > zh。
 - `--model` `option`:生成 LLM model 名。Codex 自动读取本机配置；也可用 OMK_MODEL 设置环境偏好。
-- `--no-mock` `boolean`:不生成 mocks，eval 时所有工具调用真实执行。
+- `--no-mock` `boolean`:不生成 mocks。执行器不支持工具拦截时会自动启用，避免产生必然失败的 mock_hit。
 - `--observations-dir` `option`:observe inbox 目录（from-traces 模式用），默认项目 .omk/observe-inbox。
 - `--reports-dir` `option`:报告目录（fix 模式用），默认 ~/.oh-my-knowledge/reports。
 - `--skill` `option`:仅从指定 skill 的 observe inbox 信号生成草稿（仅 from-traces 模式用）。
@@ -689,7 +689,7 @@ omk studio --port 8080 --no-open
 | `construct` | 否 | 测的是什么构念 |
 | `provenance` | 否 | 用例来源（`omk sample` 自动打） |
 | `mocks` | 否 | 工具调用 mock 返回（sandbox 评测） |
-| `environment` | 否 | 评测环境前置「已就绪」声明 |
+| `environment` | 否 | 题设环境声明（仅注入 prompt，不物化） |
 | `tripwire` | 否 | 标记为「故意诱错」样本，failed 时 diagnostic 不建议改 skill |
 
 完整 schema 见 [docs/specs/sample-design-spec.md](https://github.com/lizhiyao/oh-my-knowledge/blob/main/docs/specs/sample-design-spec.md)。

package/dist/authoring/generator.d.ts

@@ -1,6 +1,7 @@
-import type { Sample, ExecutorFn } from '../types/index.js';
+import type { ExecutorFn, Sample } from '../types/index.js';
 import type { ObservationInboxItem } from '../types/observability.js';
-interface GenerateSamplesOptions {
+export declare function sampleGenerationUsesMocks(executorName: string | undefined, noMock?: boolean): boolean;
+export interface GenerateSamplesOptions {
     skillContent: string;
     count?: number;
     model: string;
@@ -12,6 +13,8 @@ interface GenerateSamplesOptions {
     focus?: string;
     /** 不生成 mocks/mocksStrict，eval 时真实执行所有工具调用。 */
     noMock?: boolean;
+    /** Injectable executor for tests. Defaults to createExecutor(executorName). */
+    executor?: ExecutorFn;
 }
 /**
  * 拼出送给 LLM 的 user prompt。抽出来便于单测验证 focus 是否真的注入了。
@@ -26,7 +29,7 @@ export declare function buildSamplesPrompt({ skillContent, count, focus, noMock
     focus?: string;
     noMock?: boolean;
 }): string;
-export declare function generateSamples({ skillContent, count, model, executorName, focus, noMock }: GenerateSamplesOptions): Promise<{
+export declare function generateSamples({ skillContent, count, model, executorName, focus, noMock, executor: injectedExecutor, }: GenerateSamplesOptions): Promise<{
     samples: Sample[];
     costUSD: number;
 }>;
@@ -62,12 +65,16 @@ export declare function stratifyTraceSignals(items: TraceSignalItem[]): Stratifi
  * production failures. The trace text feeds the *generator* only — never the
  * judge prompt — so judge-prompt isolation is unaffected.
  */
-export declare function buildSamplesFromTracesPrompt(items: TraceSignalItem[], count?: number): string;
+export declare function buildSamplesFromTracesPrompt(items: TraceSignalItem[], count?: number, options?: {
+    noMock?: boolean;
+}): string;
 export interface GenerateSamplesFromTracesOptions {
     items: TraceSignalItem[];
     count?: number;
     model: string;
     executorName?: string;
+    /** 不生成 mocks。目标 executor 不支持时会自动启用。 */
+    noMock?: boolean;
     /** Injectable executor (tests). Defaults to createExecutor(executorName). */
     executor?: ExecutorFn;
 }
@@ -77,12 +84,15 @@ export interface GenerateSamplesFromTracesOptions {
  * stamps `provenance: 'production-trace'`. Output is meant to land in a review draft,
  * not the live dataset (the CLI enforces that).
  */
-export declare function generateSamplesFromTraces({ items, count, model, executorName, executor: injectedExecutor, }: GenerateSamplesFromTracesOptions): Promise<{
+export declare function generateSamplesFromTraces({ items, count, model, executorName, noMock, executor: injectedExecutor, }: GenerateSamplesFromTracesOptions): Promise<{
     samples: Sample[];
     costUSD: number;
 }>;
 export declare function sanitizeGeneratedSamples(samples: Sample[], opts?: {
     skillContent?: string;
+    mockless?: boolean;
+    migrateMocklessEnvironment?: boolean;
+    preserveMocklessEnvironment?: boolean;
 }): {
     stripped: string[];
 };

package/dist/authoring/generator.js

@@ -1,5 +1,7 @@
 import { createExecutor } from '../executors/index.js';
+import { executorSupportsSampleMocks } from '../executors/capabilities.js';
 import { DEFAULT_GATE_THRESHOLD } from '../eval-core/verdict.js';
+import { sampleMockReferenceKeys } from '../shared/sample-contract.js';
 const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据用户提供的 skill（系统提示词）内容，生成高质量的评测用例。
 
 样本结构决策（必须先做）：先扫一遍 skill 内容判断它属于哪一类，按对应配比和数量生成。
@@ -195,14 +197,14 @@ const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据
         tool_input_not_contains 或 tools_not_called。
   - { "type": "regex", "pattern": "...", "weight": 1 }
         ↑ 同 contains 限制:只用在固定格式字面量(如 SHA / UUID / 路径模板)。
-- environment: 可选,对象。**评测环境的"已就绪"声明**,LLM 看到后跳过环境探测直接进工作流。
+- environment: 可选,对象。**仅作 prompt 上下文的题设环境声明**,不会修改 PATH、创建文件或物化 fixture。
   字段:
     - cli_available: string[],已在 PATH 上的 CLI(如 ["node", "git", "code-host"])
     - files_available: string[],已存在的文件/脚本(如 ["~/.req-tool-api.json", "$SKILL_DIR/scripts/x.js"])
     - notes: string,自由文本兜底(如"DevAPI 凭证有效,工号 testuser001")
   原则:
-    凡是 skill 跑起来需要的环境(凭证文件 / 业务 CLI / 自带脚本 / API token 等),
-    都写到这里,而不是在 mock 里 mock 它们的探测命令。这让 mock 只关注业务调用本身。
+    只有用例明确把某项环境能力作为题设前提时才写。需要读取真实内容的文件不能放在
+    files_available 里冒充 fixture:应放进 sample.cwd 下的真实 fixture,或由 mock 返回内容。
 - mocksStrict: **必填且必须设为 true**(只要 sample 配了 mocks)。
   原因:mocksStrict=false 时,LLM 调到没匹配 mock 的命令会**透传到真 shell**,
   既可能真调外部接口产生副作用,也可能因二进制不存在(如 mcporter)报噪声错误污染评测信号。
@@ -223,8 +225,8 @@ const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据
            - 对(✅):写一条宽 mock:\`{tool:"Bash", match:{command_glob:"ls *"},
              return:{stdout:"<模拟目录列表>", exit:0}}\` — \`command_glob\` 用 \`*\` 兜底各种
              ls 参数变体(\`ls\` / \`ls -la\` / \`ls -d\` / \`ls /xx\` 全命中)
-       (c) 单纯"已就绪"声明(凭证文件 / 业务 CLI 是否安装)还是走 \`environment\` 字段,
-           不需要 LLM 真调命令检查 — environment 字段就是告诉 LLM "这些不用检查"。
+       (c) 题设明确声明可用的凭证 / CLI 可写进 \`environment\`,让 LLM 不做可用性探测。
+           该字段只进 prompt,不会真的安装 CLI、创建凭证文件或改变 runtime。
        (d) **intent-level mock(文件搜索/读取类操作)** — LLM 搜代码时会自由选择 Bash grep、
            Grep 工具、Glob+Read 组合、甚至 Agent 子代理,逐个枚举工具写 mock 不可持续。
            正确做法:用 \`tool: "*"\` + \`input_contains: "关键词"\` 按意图匹配:
@@ -367,6 +369,37 @@ const SYSTEM_PROMPT = `你是一个评测用例生成器。你的任务是根据
 - 字符串字段（prompt / rubric / capability 等）内部如需引号，**必须用全角「」**而不是半角 \`""\`，避免漏转义破坏 JSON 解析
 - 例：错 → \`"prompt": "查询"Daily"标签..."\`（内部 \`"\` 未转义，JSON 解析失败）
        对 → \`"prompt": "查询「Daily」标签..."\`（全角引号，无转义压力）`;
+const MOCKLESS_SYSTEM_OVERRIDE = `
+
+## 目标执行器能力约束（最高优先级）
+
+目标执行器不支持可靠的工具调用拦截。本节覆盖上文所有要求 mocks、mock_hit、
+files_available 或正向工具调用断言的规则：
+
+- 不要输出 mocks、mocksStrict、environment 字段。
+- 不要输出 mock_hit、tools_called、tools_count_min、tool_input_contains、tool_output_contains。
+- 可以保留 tools_not_called、tools_count_max、tool_input_not_contains 这类负向安全约束。
+- 把必要的输入事实直接写进 context，把可判分结果写进 rubric 或输出内容断言。
+- 不要假装某个文件、CLI 或工具调用已经由 fixture 物化。`;
+function generationSystemPrompt(mockless) {
+    return mockless ? `${SYSTEM_PROMPT}${MOCKLESS_SYSTEM_OVERRIDE}` : SYSTEM_PROMPT;
+}
+const warnedAutoMocklessExecutors = new Set();
+function warnAutoMockless(executorName, noMock) {
+    if (noMock
+        || !executorName
+        || executorSupportsSampleMocks(executorName)
+        || warnedAutoMocklessExecutors.has(executorName))
+        return;
+    warnedAutoMocklessExecutors.add(executorName);
+    process.stderr.write(`[omk sample] 执行器「${executorName}」不支持工具调用拦截，`
+        + '已自动切换为无 mocks 用例。\n');
+}
+export function sampleGenerationUsesMocks(executorName, noMock = false) {
+    if (noMock)
+        return false;
+    return executorName ? executorSupportsSampleMocks(executorName) : true;
+}
 /**
  * 拼出送给 LLM 的 user prompt。抽出来便于单测验证 focus 是否真的注入了。
  *
@@ -390,9 +423,12 @@ ${skillContent}
 
 ${countLine}直接输出 JSON 数组。${focusBlock}${noMockBlock}`;
 }
-export async function generateSamples({ skillContent, count, model, executorName, focus, noMock }) {
-    const executor = createExecutor(executorName);
-    const prompt = buildSamplesPrompt({ skillContent, count, focus, noMock });
+export async function generateSamples({ skillContent, count, model, executorName, focus, noMock, executor: injectedExecutor, }) {
+    const executor = injectedExecutor ?? createExecutor(executorName);
+    const mockless = !sampleGenerationUsesMocks(executorName, noMock);
+    warnAutoMockless(executorName, noMock);
+    const prompt = buildSamplesPrompt({ skillContent, count, focus, noMock: mockless });
+    const system = generationSystemPrompt(mockless);
     // 生成场景比单次 eval 调用更重(LLM 要思考结构 + 输出大段 JSON),
     // 默认 120s 对长 skill + count >= 8 经常不够,这里用 5 分钟兜底。
     // lean=true 关掉 agent 工具循环 / skill 发现 — 生成只需要纯文本,不需要 Bash / Read 等工具。
@@ -405,7 +441,7 @@ export async function generateSamples({ skillContent, count, model, executorName
         const attemptPrompt = attempt === 1
             ? prompt
             : `${prompt}\n\n上一次输出解析失败:${lastErr}\n请严格按 JSON 规范输出(字符串内部用「」全角引号),只输出数组,不要包含其他文字。`;
-        const result = await executor({ model, system: SYSTEM_PROMPT, prompt: attemptPrompt, timeoutMs: 300_000, lean: true });
+        const result = await executor({ model, system, prompt: attemptPrompt, timeoutMs: 300_000, lean: true });
         totalCost += result.costUSD || 0;
         if (!result.ok) {
             lastErr = result.error || 'unknown error';
@@ -435,12 +471,15 @@ export async function generateSamples({ skillContent, count, model, executorName
             continue;
         }
         // 通过校验,跳出循环继续后续 sanitize
-        return await finalizeSamples(samples, totalCost, skillContent);
+        return await finalizeSamples(samples, totalCost, {
+            skillContent,
+            mockless,
+        });
     }
     // 不可达 (循环里所有出口都 throw 或 return),保留是为了 TS 类型推断
     throw new Error('unreachable');
 }
-async function finalizeSamples(samples, costUSD, skillContent) {
+async function finalizeSamples(samples, costUSD, options) {
     // Validate required fields + sanitize metadata enums *at generator boundary*
     // (see sanitizeGeneratedSamples). skillContent is passed so the function can
     // strip "脑补"-style fact assertions whose tool name has no literal mention
@@ -448,7 +487,10 @@ async function finalizeSamples(samples, costUSD, skillContent) {
     // the data-security-review v1-v5 regen series (generator kept producing
     // tool_input_contains "WebFetch:语雀URL" even after 7 prompt iterations,
     // because LLM's "URL → fetch" training prior overrides instructional text).
-    const { stripped } = sanitizeGeneratedSamples(samples, { skillContent });
+    const { stripped } = sanitizeGeneratedSamples(samples, {
+        ...options,
+        migrateMocklessEnvironment: true,
+    });
     if (stripped.length > 0) {
         process.stderr.write(`[omk sample] LLM-output 含 ${stripped.length} 个非法元数据/断言字段，已剥离避免污染：\n  - ${stripped.join('\n  - ')}\n`);
     }
@@ -492,7 +534,7 @@ export function stratifyTraceSignals(items) {
  * production failures. The trace text feeds the *generator* only — never the
  * judge prompt — so judge-prompt isolation is unaffected.
  */
-export function buildSamplesFromTracesPrompt(items, count) {
+export function buildSamplesFromTracesPrompt(items, count, options = {}) {
     // 先按频次分层（合并重复 + 算占比 + 降序），让模型按「占比」分配配额，而非每信号一刀切。
     const stratified = stratifyTraceSignals(items);
     const sections = stratified.map((it, i) => {
@@ -515,13 +557,16 @@ export function buildSamplesFromTracesPrompt(items, count) {
     const countLine = typeof count === 'number'
         ? `共生成约 ${count} 条评测用例，按各信号的「占比」分配配额（高频多、低频少），覆盖整体失败分布。`
         : '按各信号「占比」分配：高频信号多生成、低频少生成，覆盖整体失败分布。';
+    const mocklessBlock = options.noMock
+        ? '\n\n目标执行器不支持工具调用拦截：不要生成 mocks、mocksStrict、environment 或正向工具调用断言；把 trace 证据写入 context 和 rubric。'
+        : '';
     return `${TRACE_GEN_INSTRUCTIONS}
 
 ## 观测到的失败信号（共 ${stratified.length} 个，已按出现频次降序）
 
 ${sections}
 
-${countLine}直接输出 JSON 数组。`;
+${countLine}直接输出 JSON 数组。${mocklessBlock}`;
 }
 /** Build a sanitize context string from trace evidence so finalizeSamples keeps
  *  tool-name assertions that reference tools actually seen in the traces (an empty
@@ -539,14 +584,17 @@ function traceSanitizeContext(items) {
  * stamps `provenance: 'production-trace'`. Output is meant to land in a review draft,
  * not the live dataset (the CLI enforces that).
  */
-export async function generateSamplesFromTraces({ items, count, model, executorName, executor: injectedExecutor, }) {
+export async function generateSamplesFromTraces({ items, count, model, executorName, noMock, executor: injectedExecutor, }) {
     if (items.length === 0)
         return { samples: [], costUSD: 0 };
     if (!injectedExecutor && !executorName) {
         throw new Error('executorName is required when no executor is injected');
     }
     const executor = injectedExecutor ?? createExecutor(executorName);
-    const prompt = buildSamplesFromTracesPrompt(items, count);
+    const mockless = !sampleGenerationUsesMocks(executorName, noMock);
+    warnAutoMockless(executorName, noMock);
+    const prompt = buildSamplesFromTracesPrompt(items, count, { noMock: mockless });
+    const system = generationSystemPrompt(mockless);
     const sanitizeContext = traceSanitizeContext(items);
     const PROVENANCE = 'production-trace';
     const MAX_ATTEMPTS = 3;
@@ -556,7 +604,7 @@ export async function generateSamplesFromTraces({ items, count, model, executorN
         const attemptPrompt = attempt === 1
             ? prompt
             : `${prompt}\n\n上一次输出解析失败:${lastErr}\n请严格按 JSON 规范输出(字符串内部用「」全角引号),只输出数组,不要包含其他文字。`;
-        const result = await executor({ model, system: SYSTEM_PROMPT, prompt: attemptPrompt, timeoutMs: 300_000, lean: true });
+        const result = await executor({ model, system, prompt: attemptPrompt, timeoutMs: 300_000, lean: true });
         totalCost += result.costUSD || 0;
         if (!result.ok) {
             lastErr = result.error || 'unknown error';
@@ -593,7 +641,10 @@ export async function generateSamplesFromTraces({ items, count, model, executorN
         // Stamp provenance before sanitize so it survives (it's a valid enum value).
         for (const s of samples)
             s.provenance = PROVENANCE;
-        return await finalizeSamples(samples, totalCost, sanitizeContext);
+        return await finalizeSamples(samples, totalCost, {
+            skillContent: sanitizeContext,
+            mockless,
+        });
     }
     throw new Error('unreachable');
 }
@@ -656,6 +707,78 @@ const TEXT_VALUE_TYPES = new Set([
 const TOOL_POSITIVE_TYPES = new Set([
     'tool_input_contains', 'tool_output_contains', 'mock_hit',
 ]);
+const MOCKLESS_POSITIVE_TOOL_TYPES = new Set([
+    'mock_hit',
+    'tools_called',
+    'tools_count_min',
+    'tool_input_contains',
+    'tool_output_contains',
+]);
+function stripMocklessPositiveToolAssertions(assertions, label, stripped) {
+    const kept = [];
+    for (const [index, assertion] of assertions.entries()) {
+        const assertionLabel = `${label}[${index}]`;
+        if (assertion?.type === 'assert-set' && Array.isArray(assertion.children)) {
+            const children = stripMocklessPositiveToolAssertions(assertion.children, `${assertionLabel}.children`, stripped);
+            if (children.length === 0) {
+                stripped.push(`${assertionLabel}.assert-set（无 mock 模式下没有可执行子断言）`);
+                continue;
+            }
+            assertion.children = children;
+            kept.push(assertion);
+            continue;
+        }
+        if (MOCKLESS_POSITIVE_TOOL_TYPES.has(assertion?.type)) {
+            stripped.push(`${assertionLabel}.${assertion.type}（目标执行器不支持 mocks，正向工具证据不可复现）`);
+            continue;
+        }
+        kept.push(assertion);
+    }
+    return kept;
+}
+function stripInvalidMockHitAssertions(assertions, mockKeys, label, stripped) {
+    const kept = [];
+    for (const [index, assertion] of assertions.entries()) {
+        const assertionLabel = `${label}[${index}]`;
+        if (assertion?.type === 'assert-set' && Array.isArray(assertion.children)) {
+            const children = stripInvalidMockHitAssertions(assertion.children, mockKeys, `${assertionLabel}.children`, stripped);
+            if (children.length === 0) {
+                stripped.push(`${assertionLabel}.assert-set（没有可执行子断言）`);
+                continue;
+            }
+            assertion.children = children;
+            kept.push(assertion);
+            continue;
+        }
+        if (assertion?.type === 'mock_hit'
+            && typeof assertion.value === 'string'
+            && !mockKeys.has(assertion.value)) {
+            stripped.push(`${assertionLabel}.mock_hit（未引用实际 mock：${assertion.value}）`);
+            continue;
+        }
+        kept.push(assertion);
+    }
+    return kept;
+}
+function environmentAsPromptContext(environment) {
+    if (!environment || typeof environment !== 'object' || Array.isArray(environment)) {
+        return null;
+    }
+    const env = environment;
+    const lines = ['题设环境声明（仅作上下文，不会在 cwd 物化）：'];
+    if (Array.isArray(env.cli_available)
+        && env.cli_available.every((item) => typeof item === 'string' && item.length > 0)) {
+        lines.push(`- 可用 CLI：${env.cli_available.join('、')}`);
+    }
+    if (Array.isArray(env.files_available)
+        && env.files_available.every((item) => typeof item === 'string' && item.length > 0)) {
+        lines.push(`- 题设引用路径（未物化）：${env.files_available.join('、')}`);
+    }
+    if (typeof env.notes === 'string' && env.notes.trim()) {
+        lines.push(`- 说明：${env.notes.trim()}`);
+    }
+    return lines.length > 1 ? lines.join('\n') : null;
+}
 function isAsciiTokenLike(v) {
     if (typeof v !== 'string')
         return false;
@@ -719,6 +842,40 @@ export function sanitizeGeneratedSamples(samples, opts = {}) {
             stripped.push(`samples[${i}].tripwire (${typeof s.tripwire})`);
             delete s.tripwire;
         }
+        if (opts.mockless) {
+            if (Array.isArray(s.mocks) && s.mocks.length > 0) {
+                stripped.push(`samples[${i}].mocks（目标执行器不支持工具调用拦截）`);
+            }
+            delete s.mocks;
+            if (s.mocksStrict !== undefined) {
+                stripped.push(`samples[${i}].mocksStrict（无 mocks）`);
+            }
+            delete s.mocksStrict;
+            if (s.environment !== undefined && !opts.preserveMocklessEnvironment) {
+                const environmentContext = opts.migrateMocklessEnvironment
+                    ? environmentAsPromptContext(s.environment)
+                    : null;
+                if (environmentContext) {
+                    const existingContext = typeof s.context === 'string' ? s.context.trim() : '';
+                    s.context = existingContext
+                        ? `${existingContext}\n\n${environmentContext}`
+                        : environmentContext;
+                    stripped.push(`samples[${i}].environment（已迁移到题设 context，未物化）`);
+                }
+                else {
+                    stripped.push(`samples[${i}].environment（未物化的环境声明不能充当 fixture）`);
+                }
+            }
+            if (!opts.preserveMocklessEnvironment) {
+                delete s.environment;
+            }
+            if (Array.isArray(s.assertions)) {
+                s.assertions = stripMocklessPositiveToolAssertions(s.assertions, `samples[${i}].assertions`, stripped);
+                if (s.assertions.length === 0) {
+                    delete s.assertions;
+                }
+            }
+        }
         // assertions 校验:loader 会拒掉两类无效断言 — 在 generator boundary 提前 strip,
         // 避免落盘的 sample 跑不动:
         //   1. tools_called / tools_not_called 的 values 必须非空
@@ -843,6 +1000,12 @@ export function sanitizeGeneratedSamples(samples, opts = {}) {
                     delete s.mocks;
             }
         }
+        if (Array.isArray(s.assertions)) {
+            s.assertions = stripInvalidMockHitAssertions(s.assertions, sampleMockReferenceKeys(s.mocks), `samples[${i}].assertions`, stripped);
+            if (s.assertions.length === 0) {
+                delete s.assertions;
+            }
+        }
         // mocksStrict 兜底:有 mocks 时强制 true。
         // SYSTEM_PROMPT 已要求 LLM 必填,但偶尔 LLM 漏填 — 在 generator boundary 修掉,
         // 避免运行时 mock 未命中透传到真 shell(报 mcporter not found 等噪声错误)。

package/dist/authoring/sample-fixer.d.ts

@@ -34,6 +34,8 @@ export interface FixSamplesOptions {
     }>;
     model: string;
     maxAttemptsPerSample?: number;
+    /** Target executor cannot intercept tool calls; strip mocks and dependent positive evidence. */
+    mockless?: boolean;
 }
 export interface FixSamplesResult {
     samples: Record<string, unknown>[];