python-yama 0.1.0__py3-none-any.whl → 0.2.0__py3-none-any.whl

python_yama-0.2.0.dist-info/METADATA

@@ -0,0 +1,99 @@
+Metadata-Version: 2.4
+Name: python-yama
+Version: 0.2.0
+Summary: A reproducible evaluation runner for tool-using Agent skills
+Project-URL: Repository, https://github.com/world-sim-dev/yama
+Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
+Author-email: Farmer Sun <podpodiumapp@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,evaluation,llm,skill,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Requires-Dist: fastapi>=0.139.0
+Requires-Dist: jinja2>=3.1
+Requires-Dist: litellm<2.0,>=1.80
+Requires-Dist: orjson>=3.11.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich<15.0,>=14.1
+Description-Content-Type: text/markdown
+
+# yama
+
+English | [简体中文](https://github.com/world-sim-dev/yama/blob/main/README.zh-CN.md)
+
+`yama` is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese) for the full Case schema and execution contract.
+
+## Quick start
+
+```bash
+# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## Writing a Case
+
+A Case is a YAML file organized as `context` (what the model sees) → `mocks` (how tool calls get executed) → `steps` (user turns sent in order, each with assertions) → `outcome` (the pass bar for the whole Case):
+
+```yaml
+context:
+  system_prompt: { default: true }     # use SYSTEM.md from the plugin root
+
+  tools:
+    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: Vintage Film }
+
+steps:
+  - id: request-directions
+    user: Give me a few creative directions
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: creative direction
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` is the tool schema sent to the model; `mocks.tools` is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
+- To use a Skill, declare it by name under `context.skills` and also declare `{builtin: skill}` under `context.tools`; the model reads the Skill body via `skill(name, file?)`.
+- To simulate command-line tools, declare `{builtin: bash}` and configure per-command output under `mocks.cli`.
+- `steps[].assert.hard` are deterministic checks (nine types, including `tool_called`, `tool_arguments`, and `assistant_contains`); add `assert.judge` for LLM scoring.
+
+For the full schema (every Skill/tool/mock form, the `bash` sandbox, message injection, judge configuration, and more), see the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese).
+
+## CLI usage
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| Flag | Meaning |
+| --- | --- |
+| `paths` (positional, repeatable) | Explicit Case YAML paths to run |
+| `--plugin-root PATH` | Use the given path as the single plugin root |
+| `--plugin NAME` (repeatable) | Select plugins by name from `yama.toml` |
+| `--all-plugins` | Run every plugin configured in `yama.toml` |
+| `--result-dir PATH` | Override the artifact root (default `<plugin_root>/.yama/runs`) |
+| `--report [PATH]` | Also generate a single-file HTML report (default `.yama/reports/latest.html`) |
+| `--list` | Only print the collected Cases, without running them |
+| `--no-artifacts` | Write no artifact files at all |
+| `--json` | Emit machine-readable JSON instead of Rich tables |
+
+`--plugin`, `--all-plugins`, and `--plugin-root` are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing `yama.toml` and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

{python_yama-0.1.0.dist-info → python_yama-0.2.0.dist-info}/RECORD

@@ -1,12 +1,12 @@
 yama/__init__.py,sha256=UYhoZTSu1HiMT54LOZCdvLKMQ_L0qec8RT8jDfde_cY,1156
 yama/__main__.py,sha256=k1ocEWawweo1qCJWNFAAvyxz3tcY13dzvCenHszij30,48
 yama/assertions.py,sha256=0x1r82W2dtAY1mTk4LC86VulD5Q9blS7QeRCTysb4kE,21736
-yama/cli.py,sha256=0RsIDhv767oJ42-1FfkAxttJXqardA2UG33bSHDQH6w,11718
-yama/llm.py,sha256=RrgU2xO8J_AfiXwOJn1Dmb3OzjbwWFpW7mpQ6dlHYiQ,13352
-yama/mocks.py,sha256=v94SFa5VyTB3cvMMcuHMkiSL8HS962-0OmY5iWq9mxs,14256
+yama/cli.py,sha256=RJcffqSX2DTUU8i6PB2ZyPMXuACCpBuj_Ifm-uQ4j9g,11736
+yama/llm.py,sha256=XqGE4aKPOimy9n3oNF5rUvqWdJlpi0EVzMlD-FBim6A,13570
+yama/mocks.py,sha256=YyiU6Y7NgttxHYJBsqGPILKYlTsM4IMTIilMNRkUuzs,14973
 yama/models.py,sha256=-ry0QUI2e0ialL6A9NUEAM0QpwCQg8oXEfPBXn3CvNI,3964
-yama/report.py,sha256=qnLpp8yA2TdZw1UWHN_XtJWMVa0v9Z7f0ez9z9HViY0,37563
-yama/runner.py,sha256=GRg3VWZBw4bupDW6qMQO7TjE2VHjwo1hV8HYYw0_Mks,20661
+yama/report.py,sha256=bjp_aev2KHx3i2pqYRGE1VHTuBmiakXgbqYjN3RAGEg,37558
+yama/runner.py,sha256=dJi-hxXxpUawZ3g14Hk9NFZy7HQNvcGpxpxvj8QKbX8,21694
 yama/workspace.py,sha256=rHQ9kQUkAo32gJpqRf7pSd5JhBc05xgxA1wPRC4bOQ8,3618
 yama/loaders/__init__.py,sha256=DIwfa8QlIlKt6NpFMZJ_OcqbJPPrmKLBzd9_e-BCuBs,174
 yama/loaders/case.py,sha256=gas5J9jtPuaqo43IO5gk-4uVJ49JWc7sNMXFI8I_7AA,14255
@@ -17,11 +17,11 @@ yama/tools/base.py,sha256=ONFuGzTSWVsA0k4-UUREncqAHmyQI2uHpVOU2Dh_XiQ,446
 yama/tools/bash/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 yama/tools/bash/_cli_shim.py,sha256=f3Rom5UipW0dA0KRYavqIa9HJDKT336ssDZGWeVmkvo,2494
 yama/tools/bash/bash.py,sha256=4GlFvmqTUS4k6DgJ5Io91YF5fB6wF6e3e7p2P-pmbEQ,10358
-yama/tools/bash/fs.py,sha256=CMPPAEfubt2kkaccxQaIYriJ497J2_CDfZm5l3evnaY,20674
+yama/tools/bash/fs.py,sha256=ncLZf4_gErRP3HHXinLEQeN6chrf7JaSiaTuadVl8MY,21314
 yama/tools/skill/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 yama/tools/skill/skill.py,sha256=P2rDz6VqRLfhiabWjOU3ZBaWLw30Eywf3o_OxVfFbPI,1992
-python_yama-0.1.0.dist-info/METADATA,sha256=klpVw1LaW-EsoGJQtvzH950HF4vLA5QmOehOX2sxWPI,4299
-python_yama-0.1.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
-python_yama-0.1.0.dist-info/entry_points.txt,sha256=N5se3y1FflUcDU3TmUrH2Tu9dzsrBBo7sUP-Gu_2cVs,39
-python_yama-0.1.0.dist-info/licenses/LICENSE,sha256=cMAJJAs7R1lemJPPc7qMNv4w_sqn5QncFf4aXKYyEc4,1067
-python_yama-0.1.0.dist-info/RECORD,,
+python_yama-0.2.0.dist-info/METADATA,sha256=TuKyzkSIHUvjSVNTBj6YYErZjK91mznGMs6gFBFLCmg,4598
+python_yama-0.2.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+python_yama-0.2.0.dist-info/entry_points.txt,sha256=N5se3y1FflUcDU3TmUrH2Tu9dzsrBBo7sUP-Gu_2cVs,39
+python_yama-0.2.0.dist-info/licenses/LICENSE,sha256=cMAJJAs7R1lemJPPc7qMNv4w_sqn5QncFf4aXKYyEc4,1067
+python_yama-0.2.0.dist-info/RECORD,,

yama/cli.py

@@ -19,7 +19,7 @@ from .workspace import (
     load_workspace_config,
     resolve_plugin_root,
 )
-from .llm import LLMError, LiteLLMClient, ScriptedLLMClient
+from .llm import LLMError, LiteLLMClient
 from .loaders import CollectionError, resolve_case
 from .report import write_html_report
 from .runner import YamaRunner
@@ -190,9 +190,6 @@ def _parser() -> argparse.ArgumentParser:
     parser.add_argument(
         "--all-plugins", action="store_true", help="Run all configured plugins"
     )
-    parser.add_argument(
-        "--response-script", type=Path, help="Use deterministic scripted LLM responses"
-    )
     parser.add_argument("--result-dir", type=Path, help="Override artifact directory")
     parser.add_argument(
         "--report",
@@ -255,11 +252,7 @@ async def _run(args: argparse.Namespace) -> int:
         Console(stderr=True).print("[bold red]No eval cases collected.[/bold red]")
         return 2
 
-    llm = (
-        ScriptedLLMClient.from_file(args.response_script.resolve())
-        if args.response_script
-        else LiteLLMClient()
-    )
+    llm = LiteLLMClient()
     all_results = []
     result_roots = []
     for case in collected:
@@ -275,8 +268,6 @@ async def _run(args: argparse.Namespace) -> int:
             write_artifacts=not args.no_artifacts,
         )
         all_results.append(await runner.run_case(case))
-    if isinstance(llm, ScriptedLLMClient):
-        llm.assert_consumed()
     report_target = args.report
     if report_target == DEFAULT_REPORT:
         report_target = result_roots[0].parent / "reports" / "latest.html"
@@ -309,10 +300,18 @@ def main(argv: list[str] | None = None) -> int:
     args = _parser().parse_args(argv)
     try:
         return asyncio.run(_run(args))
-    except (ConfigurationError, CollectionError, LLMError, ValueError) as exc:
+    except (ConfigurationError, CollectionError, ValueError) as exc:
         Console(stderr=True).print(
             Text.assemble(("Collection error: ", "bold red"), (str(exc), "red"))
         )
         return 2
+    except LLMError as exc:
+        # Not a collection problem: cases were already collected (and possibly
+        # executed) when the LLM layer failed -- label it accordingly so the
+        # user debugs the right layer.
+        Console(stderr=True).print(
+            Text.assemble(("LLM error: ", "bold red"), (str(exc), "red"))
+        )
+        return 2
     except KeyboardInterrupt:
         return 130

yama/llm.py

@@ -337,7 +337,10 @@ class ScriptedLLMClient(LLMClient):
         if not isinstance(expectation, dict):
             raise LLMError("script expect must be an object")
         if "tools" in expectation:
-            actual = [tool["function"]["name"] for tool in request["tools"]]
+            # A request with no tools omits the "tools" key entirely (see
+            # _litellm_call_arguments), so expect: {tools: []} must read it
+            # as an empty list instead of raising KeyError.
+            actual = [tool["function"]["name"] for tool in request.get("tools", [])]
             if actual != expectation["tools"]:
                 raise LLMError(
                     f"script expected tools {expectation['tools']}, got {actual}"

yama/mocks.py

@@ -178,6 +178,27 @@ def _matches(actual: Any, expected: Any) -> bool:
 _MISSING = object()
 
 
+def result_is_error(result: Any) -> bool:
+    """Semantic error status of a tool result envelope.
+
+    A result is an error only when it carries a non-null ``error`` payload,
+    or when it is a bash-style ``{stdout, stderr, exit_code}`` envelope with a
+    non-zero exit code. A literal ``error: null`` field is a successful result.
+    """
+    if not isinstance(result, dict):
+        return False
+    if result.get("error") is not None:
+        return True
+    if (
+        isinstance(result.get("stdout"), str)
+        and isinstance(result.get("stderr"), str)
+        and isinstance(result.get("exit_code"), int)
+        and not isinstance(result.get("exit_code"), bool)
+    ):
+        return result["exit_code"] != 0
+    return False
+
+
 def _state_diff(before: Any, after: Any, path: str = "$") -> list[dict[str, Any]]:
     if before == after:
         return []
@@ -353,7 +374,7 @@ class MockToolRegistry:
             "match_failed": match_failed,
             "arguments": arguments,
             "result": result,
-            "is_error": is_error or (isinstance(result, dict) and "error" in result),
+            "is_error": is_error or result_is_error(result),
             "duration_ms": round((time.perf_counter() - started) * 1000, 3),
             "state_diff": _state_diff(before, after),
         }

yama/report.py

@@ -418,7 +418,7 @@ def _render_step(
         "</span>"
         "</summary>"
         '<div class="step-body">'
-        '<details class="subsection timeline-section" open>'
+        '<details class="subsection timeline-section">'
         '<summary><span>Execution timeline</span>'
         f'<span class="muted">{len(events)} events</span></summary>'
         f'<div class="timeline">{"".join(events)}</div></details>'

yama/runner.py

@@ -14,9 +14,15 @@ from typing import Any
 import yaml
 
 from .assertions import run_hard_assertions, run_judge
-from .tools.bash.fs import cleanup_session
+from .tools.bash.fs import case_workspace_root, cleanup_session
 from .llm import LLMClient
-from .mocks import MockToolRegistry, ToolMockContext, _matches, load_python_callable
+from .mocks import (
+    MockToolRegistry,
+    ToolMockContext,
+    _matches,
+    load_python_callable,
+    result_is_error,
+)
 from .models import ResolvedCase, StepResult, ToolCallRecord, sanitize_case_key
 
 
@@ -109,6 +115,7 @@ class YamaRunner:
         repeat = int(case.execution.get("repeat", 1))
         if repeat < 1:
             raise ValueError("execution.repeat must be at least 1")
+        self._clean_case_dirs(case)
         runs = [
             await self._run_once(case, run_index) for run_index in range(1, repeat + 1)
         ]
@@ -351,7 +358,7 @@ class YamaRunner:
                     call_index=call_index,
                     call=call,
                     result=result,
-                    is_error=isinstance(result, dict) and "error" in result,
+                    is_error=result_is_error(result),
                 )
                 records.append(record)
                 messages.append(
@@ -452,12 +459,32 @@ class YamaRunner:
                     return False
         return True
 
+    def _clean_case_dirs(self, case: ResolvedCase) -> None:
+        """Drop leftovers of earlier executions before the first run starts.
+
+        Both the bash-sandbox workspaces (`.yama/fs-runs/<case>`) and the run
+        artifacts (`.yama/runs/<case>`) are keyed by run index, which restarts
+        at 1 on every execution -- without this wipe a later execution would
+        reuse stale workspaces (old .mockbin scripts, files written by a
+        previous model) and leave orphaned run-00N artifact dirs behind when
+        execution.repeat shrinks.
+        """
+        fs_root = case_workspace_root(case.plugin_root, case.case_key)
+        if fs_root.exists():
+            shutil.rmtree(fs_root)
+        if self.write_artifacts:
+            case_dir = self._case_artifact_root(case)
+            if case_dir.exists():
+                shutil.rmtree(case_dir)
+
+    def _case_artifact_root(self, case: ResolvedCase) -> Path:
+        root = self.result_root or case.plugin_root / ".yama" / "runs"
+        return root / sanitize_case_key(case.case_key)
+
     def _artifact_dir(self, case: ResolvedCase, run_index: int) -> Path | None:
         if not self.write_artifacts:
             return None
-        root = self.result_root or case.plugin_root / ".yama" / "runs"
-        key = sanitize_case_key(case.case_key)
-        return root / key / f"run-{run_index:03d}"
+        return self._case_artifact_root(case) / f"run-{run_index:03d}"
 
     def _write_artifacts(self, case: ResolvedCase, run: RunResult) -> None:
         assert run.artifact_dir is not None

yama/tools/bash/fs.py

@@ -463,6 +463,15 @@ class ShellSession:
                 pass
 
 
+def case_workspace_root(plugin_root: Path, case_key: str) -> Path:
+    """Parent directory of all bash-sandbox workspaces of one case.
+
+    The runner wipes it before a case's first run so a fresh execution never
+    reuses stale workspaces from an earlier one.
+    """
+    return plugin_root / ".yama" / "fs-runs" / sanitize_case_key(case_key)
+
+
 def workspace_dir(context: ToolMockContext) -> Path:
     """Bash-sandbox workspace directory of the current run.
 
@@ -471,10 +480,7 @@ def workspace_dir(context: ToolMockContext) -> Path:
     first `bash` call actually materializes the directory.
     """
     return (
-        context.plugin_root
-        / ".yama"
-        / "fs-runs"
-        / sanitize_case_key(context.case_key)
+        case_workspace_root(context.plugin_root, context.case_key)
         / f"run-{context.run_index:03d}"
     )
 
@@ -485,7 +491,12 @@ async def _get_session(
     session = _SESSIONS.get(id(context.state))
     if session is None or session.closed:
         workspace = workspace_dir(context)
-        if not workspace.exists():
+        # Keyed on the per-run state marker, not directory existence: a
+        # leftover directory from an earlier execution (or one pre-created by
+        # a python handler) must still get this run's files and .mockbin
+        # scripts, while a session restart mid-run must not clobber what the
+        # model already wrote.
+        if "__fs__" not in context.state:
             materialize_workspace(workspace, fs_config)
         session = ShellSession(workspace, fs_config)
         await session.start()

python_yama-0.1.0.dist-info/METADATA

@@ -1,98 +0,0 @@
-Metadata-Version: 2.4
-Name: python-yama
-Version: 0.1.0
-Summary: A reproducible evaluation runner for tool-using Agent skills
-Project-URL: Repository, https://github.com/world-sim-dev/yama
-Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
-Author-email: Farmer Sun <podpodiumapp@gmail.com>
-License-Expression: MIT
-License-File: LICENSE
-Keywords: agent,evaluation,llm,skill,testing
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Software Development :: Testing
-Requires-Python: >=3.12
-Requires-Dist: fastapi>=0.139.0
-Requires-Dist: jinja2>=3.1
-Requires-Dist: litellm<2.0,>=1.80
-Requires-Dist: orjson>=3.11.9
-Requires-Dist: pyyaml>=6.0
-Requires-Dist: rich<15.0,>=14.1
-Description-Content-Type: text/markdown
-
-# yama
-
-`yama` 是一个可复现的 Agent Skill 评测框架：用声明式 YAML Case 描述模型看到的 System Prompt、Skill metadata、Tool Schema 与多轮 User Message，通过 LiteLLM 统一调用 LLM 并驱动 Tool Loop，再对 Transcript 做确定性 Hard Check 或 LLM Judge 评分。完整 Case Schema 与执行契约见[设计文档](docs/agent-skill-test-framework-design.md)。
-
-## 快速开始
-
-```bash
-# 在 Plugin 根目录下直接运行，默认收集 __evals__/cases/**/*.yaml
-cd dingding-simple
-OPENAI_API_KEY=... uv run --project yama yama
-
-# 在 Workspace 根目录（存在 yama.toml）按名字指定 Plugin
-OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
-```
-
-## 编写一个 Case
-
-一个 Case 是一份 YAML 文件，按 `context`（模型看到什么）→ `mocks`（Tool 怎么被执行）→ `steps`（依次发送的用户回合与断言）→ `outcome`（整个 Case 的通过门槛）的结构组织：
-
-```yaml
-context:
-  system_prompt: { default: true }     # 使用 Plugin 根目录下的 SYSTEM.md
-
-  tools:
-    - file: tools/read-dsl.yaml         # Tool Schema，相对 __evals__ 目录解析
-
-mocks:
-  tools:
-    read_dsl:
-      respond:
-        result:
-          visualStyle: { name: 复古胶片 }
-
-steps:
-  - id: request-directions
-    user: 给我几个创意方向
-    assert:
-      hard:
-        - tool_called: { name: read_dsl }
-        - assistant_contains: 创意方向
-
-outcome:
-  require:
-    hard_checks: all_pass
-```
-
-- `context.tools` 是发给模型的 Tool Schema，`mocks.tools` 是 Runner 收到 Tool Call 后如何返回结果；两者的 Tool 名字集合必须完全一致。
-- 需要 Skill 时在 `context.skills` 中按名字声明，并在 `context.tools` 中同时声明 `{builtin: skill}`；模型通过 `skill(name, file?)` 读取正文。
-- 需要模拟命令行工具时声明 `{builtin: bash}`，在 `mocks.cli` 中按命令配置输出。
-- `steps[].assert.hard` 是确定性检查（`tool_called`、`tool_arguments`、`assistant_contains` 等九种类型）；还可以加 `assert.judge` 做 LLM 打分。
-
-完整 Schema（Skill/Tool/Mock 的全部写法、`bash` 沙箱、Message Injection、Judge 配置等）见[设计文档](docs/agent-skill-test-framework-design.md)。
-
-## 命令行使用
-
-```bash
-uv run --project yama yama --plugin dingding-simple --report
-```
-
-| 参数 | 含义 |
-| --- | --- |
-| `paths`（位置参数，可多个） | 显式指定要跑的 Case YAML 路径 |
-| `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
-| `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择 |
-| `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
-| `--response-script PATH` | 用脚本化 LLM 响应回放，不调用真实模型 |
-| `--result-dir PATH` | 覆盖产物根目录（默认 `<plugin_root>/.yama/runs`） |
-| `--report [PATH]` | 额外生成单文件 HTML 报告（默认 `.yama/reports/latest.html`） |
-| `--list` | 只打印匹配到的 Case，不运行 |
-| `--no-artifacts` | 不写任何产物文件 |
-| `--json` | 输出机器可读 JSON 而不是 Rich 表格 |
-
-`--plugin`/`--all-plugins`/`--plugin-root` 三者互斥；都不提供时默认用当前目录向上搜索到的 `yama.toml` 所在目录作为 Workspace Root（找不到则用 cwd），把该目录当作单一 Plugin Root 运行。