python-yama 0.1.0__py3-none-any.whl

python_yama-0.1.0.dist-info/METADATA

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: python-yama
+Version: 0.1.0
+Summary: A reproducible evaluation runner for tool-using Agent skills
+Project-URL: Repository, https://github.com/world-sim-dev/yama
+Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
+Author-email: Farmer Sun <podpodiumapp@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,evaluation,llm,skill,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Requires-Dist: fastapi>=0.139.0
+Requires-Dist: jinja2>=3.1
+Requires-Dist: litellm<2.0,>=1.80
+Requires-Dist: orjson>=3.11.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich<15.0,>=14.1
+Description-Content-Type: text/markdown
+
+# yama
+
+`yama` 是一个可复现的 Agent Skill 评测框架：用声明式 YAML Case 描述模型看到的 System Prompt、Skill metadata、Tool Schema 与多轮 User Message，通过 LiteLLM 统一调用 LLM 并驱动 Tool Loop，再对 Transcript 做确定性 Hard Check 或 LLM Judge 评分。完整 Case Schema 与执行契约见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 快速开始
+
+```bash
+# 在 Plugin 根目录下直接运行，默认收集 __evals__/cases/**/*.yaml
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# 在 Workspace 根目录（存在 yama.toml）按名字指定 Plugin
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## 编写一个 Case
+
+一个 Case 是一份 YAML 文件，按 `context`（模型看到什么）→ `mocks`（Tool 怎么被执行）→ `steps`（依次发送的用户回合与断言）→ `outcome`（整个 Case 的通过门槛）的结构组织：
+
+```yaml
+context:
+  system_prompt: { default: true }     # 使用 Plugin 根目录下的 SYSTEM.md
+
+  tools:
+    - file: tools/read-dsl.yaml         # Tool Schema，相对 __evals__ 目录解析
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: 复古胶片 }
+
+steps:
+  - id: request-directions
+    user: 给我几个创意方向
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: 创意方向
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` 是发给模型的 Tool Schema，`mocks.tools` 是 Runner 收到 Tool Call 后如何返回结果；两者的 Tool 名字集合必须完全一致。
+- 需要 Skill 时在 `context.skills` 中按名字声明，并在 `context.tools` 中同时声明 `{builtin: skill}`；模型通过 `skill(name, file?)` 读取正文。
+- 需要模拟命令行工具时声明 `{builtin: bash}`，在 `mocks.cli` 中按命令配置输出。
+- `steps[].assert.hard` 是确定性检查（`tool_called`、`tool_arguments`、`assistant_contains` 等九种类型）；还可以加 `assert.judge` 做 LLM 打分。
+
+完整 Schema（Skill/Tool/Mock 的全部写法、`bash` 沙箱、Message Injection、Judge 配置等）见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 命令行使用
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| 参数 | 含义 |
+| --- | --- |
+| `paths`（位置参数，可多个） | 显式指定要跑的 Case YAML 路径 |
+| `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
+| `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择 |
+| `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
+| `--response-script PATH` | 用脚本化 LLM 响应回放，不调用真实模型 |
+| `--result-dir PATH` | 覆盖产物根目录（默认 `<plugin_root>/.yama/runs`） |
+| `--report [PATH]` | 额外生成单文件 HTML 报告（默认 `.yama/reports/latest.html`） |
+| `--list` | 只打印匹配到的 Case，不运行 |
+| `--no-artifacts` | 不写任何产物文件 |
+| `--json` | 输出机器可读 JSON 而不是 Rich 表格 |
+
+`--plugin`/`--all-plugins`/`--plugin-root` 三者互斥；都不提供时默认用当前目录向上搜索到的 `yama.toml` 所在目录作为 Workspace Root（找不到则用 cwd），把该目录当作单一 Plugin Root 运行。

python_yama-0.1.0.dist-info/RECORD

@@ -0,0 +1,27 @@
+yama/__init__.py,sha256=UYhoZTSu1HiMT54LOZCdvLKMQ_L0qec8RT8jDfde_cY,1156
+yama/__main__.py,sha256=k1ocEWawweo1qCJWNFAAvyxz3tcY13dzvCenHszij30,48
+yama/assertions.py,sha256=0x1r82W2dtAY1mTk4LC86VulD5Q9blS7QeRCTysb4kE,21736
+yama/cli.py,sha256=0RsIDhv767oJ42-1FfkAxttJXqardA2UG33bSHDQH6w,11718
+yama/llm.py,sha256=RrgU2xO8J_AfiXwOJn1Dmb3OzjbwWFpW7mpQ6dlHYiQ,13352
+yama/mocks.py,sha256=v94SFa5VyTB3cvMMcuHMkiSL8HS962-0OmY5iWq9mxs,14256
+yama/models.py,sha256=-ry0QUI2e0ialL6A9NUEAM0QpwCQg8oXEfPBXn3CvNI,3964
+yama/report.py,sha256=qnLpp8yA2TdZw1UWHN_XtJWMVa0v9Z7f0ez9z9HViY0,37563
+yama/runner.py,sha256=GRg3VWZBw4bupDW6qMQO7TjE2VHjwo1hV8HYYw0_Mks,20661
+yama/workspace.py,sha256=rHQ9kQUkAo32gJpqRf7pSd5JhBc05xgxA1wPRC4bOQ8,3618
+yama/loaders/__init__.py,sha256=DIwfa8QlIlKt6NpFMZJ_OcqbJPPrmKLBzd9_e-BCuBs,174
+yama/loaders/case.py,sha256=gas5J9jtPuaqo43IO5gk-4uVJ49JWc7sNMXFI8I_7AA,14255
+yama/loaders/common.py,sha256=Fg71_vcnu5339QI_itvYPvRbk9LtBgEnzoA3t4Krb4U,754
+yama/loaders/tool_loader.py,sha256=BubcplVYKiJxgNsMKGRfiRiK1qVo0EvJfLOJzQEx9js,6388
+yama/tools/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+yama/tools/base.py,sha256=ONFuGzTSWVsA0k4-UUREncqAHmyQI2uHpVOU2Dh_XiQ,446
+yama/tools/bash/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+yama/tools/bash/_cli_shim.py,sha256=f3Rom5UipW0dA0KRYavqIa9HJDKT336ssDZGWeVmkvo,2494
+yama/tools/bash/bash.py,sha256=4GlFvmqTUS4k6DgJ5Io91YF5fB6wF6e3e7p2P-pmbEQ,10358
+yama/tools/bash/fs.py,sha256=CMPPAEfubt2kkaccxQaIYriJ497J2_CDfZm5l3evnaY,20674
+yama/tools/skill/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+yama/tools/skill/skill.py,sha256=P2rDz6VqRLfhiabWjOU3ZBaWLw30Eywf3o_OxVfFbPI,1992
+python_yama-0.1.0.dist-info/METADATA,sha256=klpVw1LaW-EsoGJQtvzH950HF4vLA5QmOehOX2sxWPI,4299
+python_yama-0.1.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+python_yama-0.1.0.dist-info/entry_points.txt,sha256=N5se3y1FflUcDU3TmUrH2Tu9dzsrBBo7sUP-Gu_2cVs,39
+python_yama-0.1.0.dist-info/licenses/LICENSE,sha256=cMAJJAs7R1lemJPPc7qMNv4w_sqn5QncFf4aXKYyEc4,1067
+python_yama-0.1.0.dist-info/RECORD,,

python_yama-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

python_yama-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+yama = yama.cli:main

python_yama-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Farmer Sun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

yama/__init__.py

@@ -0,0 +1,31 @@
+"""Agent Skill evaluation framework.
+
+The package root only exports the contract surface case authors need when
+writing `python` hard-assertion checkers and python tool-mock handlers:
+
+- checker signature types: `StepResult` (the `step_result` parameter),
+  `ToolCallRecord` / `ToolCall` (its `tool_calls` records), and
+  `CheckerResult` (the structured return value)
+- mock handler types: `ToolMockContext` (the `context` parameter) and
+  `ToolMockError` (expected business errors raised by handlers)
+- mock handler helpers: `workspace_dir` (bash-sandbox workspace of the
+  current run; checkers can read the same path from `state["__fs__"]`
+  once the first bash call materialized it)
+
+Framework internals (runner, loaders, llm, assertions, report, cli) are
+deliberately not re-exported; import them from their defining modules.
+"""
+
+from .mocks import ToolMockContext, ToolMockError
+from .models import CheckerResult, StepResult, ToolCall, ToolCallRecord
+from .tools.bash.fs import workspace_dir
+
+__all__ = [
+    "CheckerResult",
+    "StepResult",
+    "ToolCall",
+    "ToolCallRecord",
+    "ToolMockContext",
+    "ToolMockError",
+    "workspace_dir",
+]

yama/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())