python-yama 0.1.0__tar.gz → 0.2.0__tar.gz

{python_yama-0.1.0 → python_yama-0.2.0}/CLAUDE.md

@@ -37,6 +37,11 @@ Reproducible evaluation runner for tool-using Agent skills (see `README.md` and
   default `.yama/runs` / `.yama/reports/latest.html` locations) changes.
   Everything beyond that basic subset only needs to stay in
   `docs/agent-skill-test-framework-design.md`.
+- `README.zh-CN.md` is the Chinese mirror of `README.md` (English, also the
+  PyPI page). Whenever `README.md` changes, apply the same change to
+  `README.zh-CN.md` so the two never drift. Links inside `README.md` use
+  absolute GitHub URLs (relative links break on PyPI); `README.zh-CN.md` may
+  use relative links.
 - When changing the Case syntax (schema fields, mock/assertion/judge
   configuration, path conventions) or the Case execution logic (tool loop,
   step advancement, failure semantics, artifact layout, CLI behavior), update
@@ -59,6 +64,8 @@ Reproducible evaluation runner for tool-using Agent skills (see `README.md` and
 
 ## Releasing
 
+- The PyPI distribution is named `python-yama` (PyPI rejects `yama` as too
+  similar to `yaml`); the import package and CLI command are still `yama`.
 - The package version is derived from git tags by `hatch-vcs`; there is no
   `version` field in `pyproject.toml` to edit. Untagged commits build as
   `+g<hash>` dev versions, which PyPI rejects, so only tagged commits can be

python_yama-0.2.0/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.4
+Name: python-yama
+Version: 0.2.0
+Summary: A reproducible evaluation runner for tool-using Agent skills
+Project-URL: Repository, https://github.com/world-sim-dev/yama
+Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
+Author-email: Farmer Sun <podpodiumapp@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,evaluation,llm,skill,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Requires-Dist: fastapi>=0.139.0
+Requires-Dist: jinja2>=3.1
+Requires-Dist: litellm<2.0,>=1.80
+Requires-Dist: orjson>=3.11.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich<15.0,>=14.1
+Description-Content-Type: text/markdown
+
+# yama
+
+English | [简体中文](https://github.com/world-sim-dev/yama/blob/main/README.zh-CN.md)
+
+`yama` is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese) for the full Case schema and execution contract.
+
+## Quick start
+
+```bash
+# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## Writing a Case
+
+A Case is a YAML file organized as `context` (what the model sees) → `mocks` (how tool calls get executed) → `steps` (user turns sent in order, each with assertions) → `outcome` (the pass bar for the whole Case):
+
+```yaml
+context:
+  system_prompt: { default: true }     # use SYSTEM.md from the plugin root
+
+  tools:
+    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: Vintage Film }
+
+steps:
+  - id: request-directions
+    user: Give me a few creative directions
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: creative direction
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` is the tool schema sent to the model; `mocks.tools` is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
+- To use a Skill, declare it by name under `context.skills` and also declare `{builtin: skill}` under `context.tools`; the model reads the Skill body via `skill(name, file?)`.
+- To simulate command-line tools, declare `{builtin: bash}` and configure per-command output under `mocks.cli`.
+- `steps[].assert.hard` are deterministic checks (nine types, including `tool_called`, `tool_arguments`, and `assistant_contains`); add `assert.judge` for LLM scoring.
+
+For the full schema (every Skill/tool/mock form, the `bash` sandbox, message injection, judge configuration, and more), see the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese).
+
+## CLI usage
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| Flag | Meaning |
+| --- | --- |
+| `paths` (positional, repeatable) | Explicit Case YAML paths to run |
+| `--plugin-root PATH` | Use the given path as the single plugin root |
+| `--plugin NAME` (repeatable) | Select plugins by name from `yama.toml` |
+| `--all-plugins` | Run every plugin configured in `yama.toml` |
+| `--result-dir PATH` | Override the artifact root (default `<plugin_root>/.yama/runs`) |
+| `--report [PATH]` | Also generate a single-file HTML report (default `.yama/reports/latest.html`) |
+| `--list` | Only print the collected Cases, without running them |
+| `--no-artifacts` | Write no artifact files at all |
+| `--json` | Emit machine-readable JSON instead of Rich tables |
+
+`--plugin`, `--all-plugins`, and `--plugin-root` are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing `yama.toml` and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

python_yama-0.2.0/README.md

@@ -0,0 +1,74 @@
+# yama
+
+English | [简体中文](https://github.com/world-sim-dev/yama/blob/main/README.zh-CN.md)
+
+`yama` is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese) for the full Case schema and execution contract.
+
+## Quick start
+
+```bash
+# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## Writing a Case
+
+A Case is a YAML file organized as `context` (what the model sees) → `mocks` (how tool calls get executed) → `steps` (user turns sent in order, each with assertions) → `outcome` (the pass bar for the whole Case):
+
+```yaml
+context:
+  system_prompt: { default: true }     # use SYSTEM.md from the plugin root
+
+  tools:
+    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: Vintage Film }
+
+steps:
+  - id: request-directions
+    user: Give me a few creative directions
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: creative direction
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` is the tool schema sent to the model; `mocks.tools` is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
+- To use a Skill, declare it by name under `context.skills` and also declare `{builtin: skill}` under `context.tools`; the model reads the Skill body via `skill(name, file?)`.
+- To simulate command-line tools, declare `{builtin: bash}` and configure per-command output under `mocks.cli`.
+- `steps[].assert.hard` are deterministic checks (nine types, including `tool_called`, `tool_arguments`, and `assistant_contains`); add `assert.judge` for LLM scoring.
+
+For the full schema (every Skill/tool/mock form, the `bash` sandbox, message injection, judge configuration, and more), see the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese).
+
+## CLI usage
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| Flag | Meaning |
+| --- | --- |
+| `paths` (positional, repeatable) | Explicit Case YAML paths to run |
+| `--plugin-root PATH` | Use the given path as the single plugin root |
+| `--plugin NAME` (repeatable) | Select plugins by name from `yama.toml` |
+| `--all-plugins` | Run every plugin configured in `yama.toml` |
+| `--result-dir PATH` | Override the artifact root (default `<plugin_root>/.yama/runs`) |
+| `--report [PATH]` | Also generate a single-file HTML report (default `.yama/reports/latest.html`) |
+| `--list` | Only print the collected Cases, without running them |
+| `--no-artifacts` | Write no artifact files at all |
+| `--json` | Emit machine-readable JSON instead of Rich tables |
+
+`--plugin`, `--all-plugins`, and `--plugin-root` are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing `yama.toml` and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

python_yama-0.1.0/README.md → python_yama-0.2.0/README.zh-CN.md

@@ -1,5 +1,7 @@
 # yama
 
+[English](README.md) | 简体中文
+
 `yama` 是一个可复现的 Agent Skill 评测框架：用声明式 YAML Case 描述模型看到的 System Prompt、Skill metadata、Tool Schema 与多轮 User Message，通过 LiteLLM 统一调用 LLM 并驱动 Tool Loop，再对 Transcript 做确定性 Hard Check 或 LLM Judge 评分。完整 Case Schema 与执行契约见[设计文档](docs/agent-skill-test-framework-design.md)。
 
 ## 快速开始
@@ -63,7 +65,6 @@ uv run --project yama yama --plugin dingding-simple --report
 | `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
 | `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择 |
 | `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
-| `--response-script PATH` | 用脚本化 LLM 响应回放，不调用真实模型 |
 | `--result-dir PATH` | 覆盖产物根目录（默认 `<plugin_root>/.yama/runs`） |
 | `--report [PATH]` | 额外生成单文件 HTML 报告（默认 `.yama/reports/latest.html`） |
 | `--list` | 只打印匹配到的 Case，不运行 |

{python_yama-0.1.0 → python_yama-0.2.0}/docs/agent-skill-test-framework-design.md

@@ -141,6 +141,9 @@ cli.main
   `id(context.state)` 做 `ShellSession` 的 key）。
 - **`LLMClient`**：`LiteLLMClient` 走真实 Provider，`ScriptedLLMClient` 用于
   确定性测试/回放，两者共用同一套消息/工具转换逻辑 (`_litellm_call_arguments`)。
+  脚本响应可带 `expect`（`tools`/`last_role`/`last_user_contains`）对当次请求做
+  断言；请求没有工具时 litellm 参数会整体省略 `tools` 键，`expect: {tools: []}`
+  按空列表匹配（Judge 请求正是这种无工具请求）。
 
 ## 4. Workspace 与 Plugin
 
@@ -214,7 +217,6 @@ yama --plugin-root ./dingding-simple
 | `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
 | `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择，一次可传多次 |
 | `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
-| `--response-script PATH` | 用 `ScriptedLLMClient` 回放一份确定性响应脚本，不调用真实模型 |
 | `--result-dir PATH` | 覆盖产物根目录（默认每个 Case 写到其所属 `plugin_root/.yama/runs`） |
 | `--report [PATH]` | 额外写一份单文件 HTML 报告；省略路径时默认写到 `<第一个 result_root 的上级>/reports/latest.html` |
 | `--list` | 只收集并打印 `case_key` 列表，不运行 |
@@ -1295,10 +1297,21 @@ mocks:
 持久 bash 子进程绑定。这个路径是 `plugin_root/case_key/run_index` 的纯函数，Python
 Handler 可以在物化发生之前就用包根导出的 `workspace_dir(context)`（定义在
 `tools/bash/fs.py`）预先计算它；首次物化时同一路径也会写入共享 state 的
-`__fs__.workspace_dir`，供没有 `ToolMockContext` 的 Python Checker 读取。run 结束（成功、断言失败或异常）时框架在 `runner.py` 的
-`finally` 块里保证调用 `cleanup_session` 杀掉这个进程，工作目录本身不自动清理，
-方便事后排查。因为是同一个真实进程贯穿整个 run，`cd`、`export`、写文件等状态天然
-延续到下一次 Tool Call，不需要额外的状态字典维护 cwd/环境。
+`__fs__.workspace_dir`，供没有 `ToolMockContext` 的 Python Checker 读取。
+
+物化时机以每 run 共享 state 里的 `__fs__` 标记为准，而不是"目录是否已存在"
+（`fs.py:_get_session`）：run 的第一次 `bash` 调用总会执行 `materialize_workspace`
+（即使 Python Handler 已提前创建了该目录），把当前 Case 配置声明的 `files`、
+`.mockbin/` 假命令和编译后的 `mocks.cli` 规则文件全部写到最新状态；run 中途因超时/
+会话死亡而重建 shell 会话时则不再重复物化，模型已写入的文件得以保留。此外
+`YamaRunner.run_case` 在一个 Case 的第一轮 run 开始前会整体删除
+`.yama/fs-runs/<sanitized-case-key>/`（`runner.py:_clean_case_dirs`）——run 编号每次
+执行都从 1 重新计数，不先清掉上一次执行遗留的同名工作目录，本次执行就会复用过期的
+mock 脚本与旧文件，结果不可复现。run 结束（成功、断言失败或异常）时框架在
+`runner.py` 的 `finally` 块里保证调用 `cleanup_session` 杀掉持久 shell 进程，
+工作目录本身保留到下一次执行该 Case 前才被清掉，方便事后排查。因为是同一个真实
+进程贯穿整个 run，`cd`、`export`、写文件等状态天然延续到下一次 Tool Call，不需要
+额外的状态字典维护 cwd/环境。
 
 **命令执行：持久 shell + marker 协议。** 每次 Tool Call 把命令包装成如下 payload
 写入这个 shell 的 stdin：
@@ -1919,7 +1932,11 @@ Registry 的执行顺序固定为：
 `request_index`、`call_index`、`handler_kind`（`"python"` 或 `"declarative"`）、
 `callable`（Python Handler 的点分路径，声明式 Mock 为 `None`）、`match_failed`、
 `arguments`、`result`、`is_error`、`duration_ms`、`state_diff`（`{op: add|remove|replace,
-path, value|old}` 形式的差异列表）。Run 结束额外保存 `mock-state.json`；Python
+path, value|old}` 形式的差异列表）。`is_error` 的判定是语义化的
+（`mocks.py:result_is_error`，Mock Event 与 `ToolCallRecord.is_error` 共用同一函数）：
+结果携带**非 null** 的 `error` 字段，或是 bash 形状的 `{stdout, stderr, exit_code}`
+信封且 `exit_code != 0` 时才算错误；成功结果里字面的 `error: null` 不算错误，
+`ToolMockError`/`match_failed` 路径则始终算错误。Run 结束额外保存 `mock-state.json`；Python
 Checker 从同一个最终 state 读取业务结果。Handler 源码本身不复制到 Judge 输入，
 也不发送给被测模型。
 
@@ -1981,7 +1998,7 @@ async def run_step(case, step, messages, registry, run_index, requests):
             context = ToolMockContext(..., call_index=registry.next_call_index(call.name))
             result = await registry.call(call.name, call.arguments, context)
             records.append(ToolCallRecord(..., call=call, result=result,
-                                           is_error=isinstance(result, dict) and "error" in result))
+                                           is_error=result_is_error(result)))  # 语义化判定，见第 9 章
             messages.append({
                 "role": "tool",
                 "tool_call_id": call.id,
@@ -2106,6 +2123,13 @@ class ToolCallRecord:
 └── result.json                          # RunResult.to_dict() + generated_at + traceback?
 ```
 
+写产物前，`YamaRunner.run_case` 会先整体删除该 Case 的产物目录
+`<result_root>/<sanitize(case_key)>/`（`runner.py:_clean_case_dirs`，与 8.4.3 节的
+`.yama/fs-runs/<case>` 清理同一处）：run 编号每次执行都从 1 重新计数，若本次
+`execution.repeat` 比上次小，不清理就会留下过期的 `run-002` 等目录，人工或脚本
+扫描 `.yama/runs` 时会把旧执行误当成本次结果。`--no-artifacts` 时不写也不删任何
+产物目录（用户显式选择不落盘，不应顺带删掉历史产物），但 fs-runs 工作目录照常清理。
+
 `--report` 触发的单文件 HTML 报告（`report.py:write_html_report`）是 CLI 层面单独
 生成的汇总产物，**不属于**上面这份逐 Case/逐 run 的产物列表，也不是按 Plugin 拆分的
 多个文件——一次 CLI 调用只写一个 HTML 文件，汇总本次涉及的所有 Case（见第 13 节）。
@@ -2383,6 +2407,13 @@ Details
 `--json` 输出机器可读的 `[CaseResult.to_dict(), ...]` 数组；`--list` 只打印
 `case_key` 列表，不运行任何 Case。
 
+`cli.py:main` 的错误出口按层归类（退出码都是 2）：`ConfigurationError`/
+`CollectionError`/`ValueError` 打印 `Collection error: ...`（Case 还没跑起来，
+问题出在工作区配置或 Case 收集/解析）；`LLMError` 单独打印 `LLM error: ...`——
+此时 Case 往往已经收集甚至执行过（例如脚本回放响应未被消费、Provider 配置缺失），
+把它标成 Collection error 会把排查引到错误的层级。run 内部的 LLM 异常仍由
+`YamaRunner._run_once` 捕获进 `RunResult.error`，不走这个出口。
+
 一个 Plugin 的 Skill、Tool、Case、产物目录不会泄漏到另一个 Plugin——这一点由
 "每个 Case 归属唯一 Plugin Root、产物写在该 Plugin 自己的 `.yama/` 下"这一实现
 天然保证，不依赖某种显式的隔离机制。

{python_yama-0.1.0 → python_yama-0.2.0}/skills/eval-plugin-with-yama/SKILL.md

@@ -105,23 +105,38 @@ outcome:
 完整 Schema（多模态消息、`message_inject`、`mocks.cli` 命令树、`mocks.fs` 沙箱、
 Python Handler / Checker、Judge 配置）见 [references/case-schema.md](references/case-schema.md)。
 
+## 安装
+
+yama 在 PyPI 上的发行名是 `python-yama`（安装后的命令行入口仍叫 `yama`），要求
+Python ≥ 3.12。推荐装成全局命令行工具：
+
+```bash
+uv tool install python-yama    # 或 pipx install python-yama / pip install python-yama
+yama --help                    # 验证安装成功
+```
+
+已安装过的话，用 `uv tool upgrade python-yama` 升级到最新版。
+
 ## 运行
 
+以下命令假定 `yama` 已按上一节安装、可直接调用：
+
 ```bash
 # 在 Plugin 根目录直接运行（默认收集 __evals__/cases/**/*.yaml）
 cd <plugin-root>
-OPENAI_API_KEY=... uv run --project yama yama
+OPENAI_API_KEY=... yama
 
 # 在 Workspace 根目录（存在 yama.toml）按名字选 Plugin，并生成 HTML 报告
-uv run --project yama yama --plugin <name> --report
+yama --plugin <name> --report
 
 # 只跑一个 Case；只列出会收集到哪些 Case
-uv run --project yama yama __evals__/cases/foo.yaml
-uv run --project yama yama --list
+yama __evals__/cases/foo.yaml
+yama --list
 ```
 
-退出码：`0` 全部通过；`1` 有 Case 失败；`2` Collection error 或没收集到任何 Case；
-`130` 手动中断。
+退出码：`0` 全部通过；`1` 有 Case 失败；`2` Collection error、LLM error
+（stderr 分别打印 `Collection error: ...` / `LLM error: ...`）或没收集到任何
+Case；`130` 手动中断。
 
 ## 排查不通过的用例
 
@@ -132,6 +147,7 @@ uv run --project yama yama --list
 | 现象 | 层级 | 去哪看 |
 | --- | --- | --- |
 | stderr 打印 `Collection error: ...`，退出码 2 | 用例根本没跑：YAML/Schema/路径/配置错误 | 错误消息本身，对照上面「硬规则」 |
+| stderr 打印 `LLM error: ...`，退出码 2 | LLM 层错误（Provider 配置缺失、脚本回放响应未消费），Case 可能已经收集甚至执行过 | 错误消息本身；已写出的产物目录 |
 | 退出码 1，某 Run 标红且带 error | run 级基础设施失败（未 Mock 的 Tool 被调用、sequence 用尽、Step 超时、Handler 抛异常、LLM 请求失败） | `result.json` 的 `runs[].error` / `traceback` |
 | 退出码 1，某 Step 的 hard check 失败 | 断言失败（模型行为不符合预期，或断言写错） | 终端展开的 `expected/actual/reason`；`hard-checks.json` |
 | hard check 全过但 Case 失败 | Judge 不达标 | `judge-result.json` 各维度 `score/reason/evidence` |

{python_yama-0.1.0 → python_yama-0.2.0}/skills/eval-plugin-with-yama/references/case-schema.md

@@ -235,7 +235,8 @@ mocks:
 ```
 
 - 工作目录：`<plugin_root>/.yama/fs-runs/<case-key>/run-00N/`，run 结束**不清理**，
-  供排查。持久 bash 会话贯穿整个 run，`cd`/`export`/写文件状态自然延续。
+  供排查；下一次执行同一 Case 时才整体清掉重建（改 mock 配置后重跑不会复用旧
+  `.mockbin` 脚本）。持久 bash 会话贯穿整个 run，`cd`/`export`/写文件状态自然延续。
 - Python Handler 可用 `from yama import workspace_dir` 提前算出该路径
   （`workspace_dir(context)`，纯函数，物化前即可调用）；首次 bash 调用后同一路径
   写入共享 state 的 `__fs__.workspace_dir`，供没有 context 的 Python Checker 读取。

{python_yama-0.1.0 → python_yama-0.2.0}/skills/eval-plugin-with-yama/references/debugging.md

@@ -14,13 +14,15 @@
 每次 run 写入 `<plugin_root>/.yama/runs/<目录名>/run-{NNN}/`，目录名 =
 `case_key` 把 `/` 换成 `-` 并去掉 `.yaml` 扩展名（如
 `__evals__/cases/image-led-mv.yaml` → `__evals__-cases-image-led-mv`）。
-`--result-dir` 可覆盖根目录；`--no-artifacts` 则完全不写。
+`--result-dir` 可覆盖根目录；`--no-artifacts` 则完全不写（也不删历史产物）。
+每次执行开始时会先整体清掉该 Case 的产物目录再重写，所以目录下看到的
+`run-00N` 一定全部来自最近一次执行，不会混入上一次执行的残留。
 
 | 文件 | 内容 | 什么时候看 |
 | --- | --- | --- |
 | `result.json` | run 是否通过、`error`/`traceback`、全量数据汇总 | **永远先看**：error 非空 = run 级错误 |
 | `hard-checks.json` | `{step_id: [每条断言的 type/passed/label/expected/actual/reason/detail]}` | Step 断言失败时 |
-| `mock-events.json` | 每次 Tool 调用：`tool/arguments/result/is_error/match_failed/state_diff/duration_ms` | 怀疑 Mock 匹配 / Tool 行为时 |
+| `mock-events.json` | 每次 Tool 调用：`tool/arguments/result/is_error/match_failed/state_diff/duration_ms`。`is_error` 是语义判定：结果带**非 null** `error`，或 bash 信封 `exit_code != 0`；成功结果里的 `error: null` 不算错误 | 怀疑 Mock 匹配 / Tool 行为时 |
 | `mock-state.json` | run 结束时的最终共享 state | 有状态 Handler / Python Checker 断言失败时 |
 | `requests/request-00N.json` | 每次真实 LLM 调用的完整入参（messages/tools）与响应；请求失败时带 `error` 字段 | 怀疑「模型到底看到了什么」时 |
 | `transcript.json` | 每个 Step 结束后的完整对话快照（含注入消息、Tool Result） | 复盘模型行为时 |
@@ -34,7 +36,10 @@
 
 ## 第 1 层：Collection error（退出码 2）
 
-用例在收集/解析阶段就被拒绝，一条也没运行。错误消息即答案，常见对照：
+用例在收集/解析阶段就被拒绝，一条也没运行。若 stderr 打印的是
+`LLM error: ...`（退出码同样是 2），问题不在收集层而在 LLM 层
+（如 Provider 配置缺失、脚本回放响应未消费），不要按下表排查。
+错误消息即答案，常见对照：
 
 | 错误消息（关键词） | 原因与修法 |
 | --- | --- |
@@ -130,6 +135,8 @@ Step 全部失败项。先回答一个问题：**是模型行为错了，还是
 
 - 沙箱工作目录保留在 `<plugin_root>/.yama/fs-runs/<case-key>/run-00N/`，run 结束
   不清理——直接进去看模型写了什么文件、`.mockbin/` 里编译出了哪些假命令。
+  该目录在**下一次执行同一 Case 时**才被整体清掉重建，因此排查时看到的内容
+  总是最近一次执行的；改了 `mocks.cli`/`mocks.fs` 配置后重跑不会复用旧脚本。
 - `command not found`（exit 127）三种可能：命令没在 `mocks.cli` 声明；模型用了
   绝对路径/带 `/` 的调用（绕过 PATH shadow，Mock 不生效，要求模型用裸命令名）；
   命令名拼错。
@@ -137,6 +144,8 @@ Step 全部失败项。先回答一个问题：**是模型行为错了，还是
   「未声明命令一律 not found」对内建不成立。
 - 模型「没意识到」命令失败：发给模型的 Tool Result 只有 `stdout + stderr` 拼接文本，
   `exit_code` 不可见——mock 的报错信息要写进 `stderr` 文本才对模型有意义。
+  （框架侧则看得见：非零 `exit_code` 会让该调用在 `mock-events.json` 里标记
+  `is_error: true`，可用来快速过滤失败调用。）
 - 链式命令输出无人工换行（`a && b` 的两段 stdout 直接相连）；需要分行就在每段
   `stdout` 末尾自带 `\n`。
 - `BASH_TIMEOUT`：单条命令超过 `timeout_seconds`（默认 30s），会话被重启且 cwd 重置。
@@ -146,8 +155,6 @@ Step 全部失败项。先回答一个问题：**是模型行为错了，还是
 
 ## 隔离变量的调试手段
 
-- 只跑出问题的 Case：`uv run --project yama yama __evals__/cases/foo.yaml`。
-- 怀疑框架/断言本身而非模型：`--response-script <path>` 用 `ScriptedLLMClient` 回放
-  固定 LLM 响应，让 run 完全确定，再单独验证断言与 Mock 行为。
+- 只跑出问题的 Case：`yama __evals__/cases/foo.yaml`。
 - 结果不稳定：`execution.repeat` 调大看 `pass_rate`，每次 run 的产物目录独立
   （`run-001`、`run-002`…），可横向对比 transcript 找分叉点。

{python_yama-0.1.0 → python_yama-0.2.0}/tests/test_cli.py

@@ -7,8 +7,10 @@ import os
 import tempfile
 import unittest
 from pathlib import Path
+from unittest.mock import patch
 
 from yama.cli import DEFAULT_REPORT, _parser, main
+from yama.llm import LLMError, ScriptedLLMClient
 
 
 def write_plugin(root: Path, *, case_names: list[str] | None = None) -> Path:
@@ -32,15 +34,14 @@ steps:
     return root
 
 
-def write_response_script(root: Path, *, count: int = 1) -> Path:
-    fixtures_dir = root / "__evals__" / "fixtures"
-    fixtures_dir.mkdir(parents=True, exist_ok=True)
-    script_path = fixtures_dir / "responses.yaml"
-    responses = "\n".join(
-        ['  - content: "hi there, friend!"' for _ in range(count)]
+def scripted_llm(*, count: int = 1):
+    """Patch the CLI's LLM client with deterministic scripted responses."""
+    return patch(
+        "yama.cli.LiteLLMClient",
+        lambda: ScriptedLLMClient(
+            [{"content": "hi there, friend!"} for _ in range(count)]
+        ),
     )
-    script_path.write_text(f"responses:\n{responses}\n", encoding="utf-8")
-    return script_path
 
 
 @contextlib.contextmanager
@@ -60,7 +61,6 @@ class ParserTests(unittest.TestCase):
         self.assertIsNone(args.plugin_root)
         self.assertIsNone(args.plugin)
         self.assertFalse(args.all_plugins)
-        self.assertIsNone(args.response_script)
         self.assertIsNone(args.result_dir)
         self.assertIsNone(args.report)
         self.assertFalse(args.list)
@@ -102,15 +102,12 @@ class MainEndToEndTests(unittest.TestCase):
     def test_successful_run_with_scripted_responses_and_json_output(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            script = write_response_script(plugin_root)
             stdout = io.StringIO()
-            with contextlib.redirect_stdout(stdout):
+            with scripted_llm(), contextlib.redirect_stdout(stdout):
                 code = main(
                     [
                         "--plugin-root",
                         str(plugin_root),
-                        "--response-script",
-                        str(script),
                         "--no-artifacts",
                         "--json",
                     ]
@@ -123,15 +120,12 @@ class MainEndToEndTests(unittest.TestCase):
     def test_text_output_renders_results_table(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            script = write_response_script(plugin_root)
             stdout = io.StringIO()
-            with contextlib.redirect_stdout(stdout):
+            with scripted_llm(), contextlib.redirect_stdout(stdout):
                 code = main(
                     [
                         "--plugin-root",
                         str(plugin_root),
-                        "--response-script",
-                        str(script),
                         "--no-artifacts",
                     ]
                 )
@@ -141,16 +135,13 @@ class MainEndToEndTests(unittest.TestCase):
     def test_report_flag_writes_html_report(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            script = write_response_script(plugin_root)
             report_path = Path(tmp) / "report.html"
             stdout = io.StringIO()
-            with contextlib.redirect_stdout(stdout):
+            with scripted_llm(), contextlib.redirect_stdout(stdout):
                 code = main(
                     [
                         "--plugin-root",
                         str(plugin_root),
-                        "--response-script",
-                        str(script),
                         "--no-artifacts",
                         "--report",
                         str(report_path),
@@ -162,16 +153,13 @@ class MainEndToEndTests(unittest.TestCase):
     def test_default_report_path_is_derived_from_result_root(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            script = write_response_script(plugin_root)
             result_dir = Path(tmp) / "results"
             stdout = io.StringIO()
-            with contextlib.redirect_stdout(stdout):
+            with scripted_llm(), contextlib.redirect_stdout(stdout):
                 code = main(
                     [
                         "--plugin-root",
                         str(plugin_root),
-                        "--response-script",
-                        str(script),
                         "--result-dir",
                         str(result_dir),
                         "--no-artifacts",
@@ -190,10 +178,24 @@ class MainEndToEndTests(unittest.TestCase):
             self.assertEqual(code, 2)
             self.assertIn("Collection error", stderr.getvalue())
 
+    def test_llm_error_is_reported_as_llm_error(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            plugin_root = write_plugin(Path(tmp) / "plugin")
+            stderr = io.StringIO()
+
+            def raise_llm_error():
+                raise LLMError("scripted LLM has 1 unconsumed responses")
+
+            with patch("yama.cli.LiteLLMClient", raise_llm_error), \
+                    contextlib.redirect_stderr(stderr):
+                code = main(["--plugin-root", str(plugin_root), "--no-artifacts"])
+            self.assertEqual(code, 2)
+            self.assertIn("LLM error", stderr.getvalue())
+            self.assertNotIn("Collection error", stderr.getvalue())
+
     def test_keyboard_interrupt_returns_130(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            from unittest.mock import patch
 
             def _raise(coro):
                 coro.close()
@@ -206,14 +208,11 @@ class MainEndToEndTests(unittest.TestCase):
     def test_default_target_is_current_workspace_root(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             plugin_root = write_plugin(Path(tmp) / "plugin")
-            script = write_response_script(plugin_root)
             with chdir(plugin_root):
                 stdout = io.StringIO()
-                with contextlib.redirect_stdout(stdout):
+                with scripted_llm(), contextlib.redirect_stdout(stdout):
                     code = main(
                         [
-                            "--response-script",
-                            str(script),
                             "--no-artifacts",
                             "--json",
                         ]
@@ -225,19 +224,16 @@ class MainEndToEndTests(unittest.TestCase):
     def test_all_plugins_and_named_plugin_selection(self) -> None:
         with tempfile.TemporaryDirectory() as tmp:
             workspace_root = Path(tmp)
-            plugin_root = write_plugin(workspace_root / "demo")
-            script = write_response_script(plugin_root)
+            write_plugin(workspace_root / "demo")
             (workspace_root / "yama.toml").write_text(
                 '[plugins.demo]\npath = "demo"\n', encoding="utf-8"
             )
             with chdir(workspace_root):
                 stdout = io.StringIO()
-                with contextlib.redirect_stdout(stdout):
+                with scripted_llm(), contextlib.redirect_stdout(stdout):
                     code = main(
                         [
                             "--all-plugins",
-                            "--response-script",
-                            str(script),
                             "--no-artifacts",
                             "--json",
                         ]
@@ -247,13 +243,11 @@ class MainEndToEndTests(unittest.TestCase):
                 self.assertEqual(len(payload), 1)
 
                 stdout2 = io.StringIO()
-                with contextlib.redirect_stdout(stdout2):
+                with scripted_llm(), contextlib.redirect_stdout(stdout2):
                     code2 = main(
                         [
                             "--plugin",
                             "demo",
-                            "--response-script",
-                            str(script),
                             "--no-artifacts",
                             "--json",
                         ]
@@ -276,17 +270,14 @@ class MainEndToEndTests(unittest.TestCase):
             plugin_root = write_plugin(
                 Path(tmp) / "plugin", case_names=["one", "two"]
             )
-            script = write_response_script(plugin_root)
             only_one = plugin_root / "__evals__" / "cases" / "one.yaml"
             stdout = io.StringIO()
-            with contextlib.redirect_stdout(stdout):
+            with scripted_llm(), contextlib.redirect_stdout(stdout):
                 code = main(
                     [
                         str(only_one),
                         "--plugin-root",
                         str(plugin_root),
-                        "--response-script",
-                        str(script),
                         "--no-artifacts",
                         "--json",
                     ]