python-yama 0.1.0__tar.gz

python_yama-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      # Listing any permission resets the rest to none, so contents: read
+      # must be granted back for checkout to fetch the (private) repo.
+      contents: read
+      # Required for PyPI Trusted Publishing (OIDC).
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # hatch-vcs derives the version from git tags; a shallow clone
+          # without tags would produce a wrong dev version.
+          fetch-depth: 0
+      - uses: astral-sh/setup-uv@v5
+      - name: Build sdist and wheel
+        run: uv build
+      - name: Publish to PyPI
+        run: uv publish

python_yama-0.1.0/.gitignore

@@ -0,0 +1,51 @@
+# Python bytecode and caches
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# Build and packaging artifacts
+build/
+dist/
+*.egg-info/
+.eggs/
+*.egg
+
+# Test, coverage, lint, and type-check caches
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+.tox/
+.nox/
+.mypy_cache/
+.pyright/
+.ruff_cache/
+.hypothesis/
+
+# Local environment and secrets
+.env
+.env.*
+!.env.example
+
+# Logs and temporary files
+*.log
+*.tmp
+*.swp
+*~
+
+# Editors and operating systems
+.idea/
+.vscode/
+.DS_Store
+Thumbs.db
+
+# Claude Code local state (settings.json itself is shared/committed)
+.claude/settings.local.json
+.claude/worktrees/

python_yama-0.1.0/CLAUDE.md

@@ -0,0 +1,71 @@
+# yama
+
+Reproducible evaluation runner for tool-using Agent skills (see `README.md` and
+`docs/agent-skill-test-framework-design.md` for the case schema and execution contract).
+
+## Testing
+
+- Overall unit test coverage must stay **at or above 80%**. Run:
+
+  ```bash
+  uv run pytest tests/ --cov=yama --cov-report=term-missing
+  ```
+
+  Check the `TOTAL` line at the end of the report. When adding or changing code in
+  `yama/`, add or update tests in `tests/` so the total does not drop below 80%.
+- Prefer direct unit tests of pure functions/classes (`yama/assertions.py`,
+  `yama/mocks.py`, `yama/workspace.py`, etc.) over building full end-to-end cases;
+  reserve end-to-end `ResolvedCase`/`YamaRunner` tests for behavior that only
+  emerges from the full pipeline.
+- `tests/test_framework.py` resolves its external `vidmuse-plugins` fixture repo by
+  walking up from the test file to find a `vidmuse-plugins` sibling directory
+  (not a fixed `parents[N]` depth), so it keeps working from nested git worktrees.
+
+## Documentation
+
+- When changing the implementation in `yama/` (case schema, execution contract,
+  assertions, mocks, workspace semantics, etc.), update
+  `docs/agent-skill-test-framework-design.md` in the same change so the design
+  doc never drifts from the code.
+- `README.md` only documents the minimal subset needed to write a basic Case
+  and run the CLI (the `context`/`mocks`/`steps`/`outcome` skeleton, and
+  `cli.py`'s argument parser). Update `README.md` in the same change whenever
+  any of that basic subset changes: CLI flags are added/removed/renamed
+  (`cli.py:_parser`), the minimal Case fields shown in the README example
+  change meaning or name, or a default path/convention referenced in the
+  README (default `SYSTEM.md`, `__evals__/cases/**/*.yaml` collection glob,
+  default `.yama/runs` / `.yama/reports/latest.html` locations) changes.
+  Everything beyond that basic subset only needs to stay in
+  `docs/agent-skill-test-framework-design.md`.
+- When changing the Case syntax (schema fields, mock/assertion/judge
+  configuration, path conventions) or the Case execution logic (tool loop,
+  step advancement, failure semantics, artifact layout, CLI behavior), update
+  `skills/eval-plugin-with-yama/` (`SKILL.md` and its `references/*.md`) in
+  the same change so the skill's authoring guidance and failure-debugging
+  guidance never drift from the actual behavior.
+
+## Communication
+
+- Speak to the user in Chinese (中文).
+
+## Workflow
+
+- Do not commit code unless the user explicitly asks you to.
+- Do not push to remote unless the user explicitly asks you to.
+- Do not proactively open a pull request. Only open a PR when the user
+  explicitly asks for one.
+- Do not create a git worktree unless the user explicitly allows it. By
+  default, develop directly on the current branch.
+
+## Releasing
+
+- The package version is derived from git tags by `hatch-vcs`; there is no
+  `version` field in `pyproject.toml` to edit. Untagged commits build as
+  `+g<hash>` dev versions, which PyPI rejects, so only tagged commits can be
+  published.
+- To release: `git tag v<X.Y.Z> && git push origin v<X.Y.Z>` (only when the
+  user explicitly asks to release). `.github/workflows/publish.yml` then
+  builds the tagged commit and publishes it to PyPI via OIDC trusted
+  publishing — no API tokens are involved.
+- Pre-1.0 versioning: bump minor (`0.2.0`) for new features or breaking
+  changes (e.g. Case schema changes), patch (`0.1.1`) for bug fixes.

python_yama-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Farmer Sun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

python_yama-0.1.0/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: python-yama
+Version: 0.1.0
+Summary: A reproducible evaluation runner for tool-using Agent skills
+Project-URL: Repository, https://github.com/world-sim-dev/yama
+Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
+Author-email: Farmer Sun <podpodiumapp@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,evaluation,llm,skill,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Requires-Dist: fastapi>=0.139.0
+Requires-Dist: jinja2>=3.1
+Requires-Dist: litellm<2.0,>=1.80
+Requires-Dist: orjson>=3.11.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich<15.0,>=14.1
+Description-Content-Type: text/markdown
+
+# yama
+
+`yama` 是一个可复现的 Agent Skill 评测框架：用声明式 YAML Case 描述模型看到的 System Prompt、Skill metadata、Tool Schema 与多轮 User Message，通过 LiteLLM 统一调用 LLM 并驱动 Tool Loop，再对 Transcript 做确定性 Hard Check 或 LLM Judge 评分。完整 Case Schema 与执行契约见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 快速开始
+
+```bash
+# 在 Plugin 根目录下直接运行，默认收集 __evals__/cases/**/*.yaml
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# 在 Workspace 根目录（存在 yama.toml）按名字指定 Plugin
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## 编写一个 Case
+
+一个 Case 是一份 YAML 文件，按 `context`（模型看到什么）→ `mocks`（Tool 怎么被执行）→ `steps`（依次发送的用户回合与断言）→ `outcome`（整个 Case 的通过门槛）的结构组织：
+
+```yaml
+context:
+  system_prompt: { default: true }     # 使用 Plugin 根目录下的 SYSTEM.md
+
+  tools:
+    - file: tools/read-dsl.yaml         # Tool Schema，相对 __evals__ 目录解析
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: 复古胶片 }
+
+steps:
+  - id: request-directions
+    user: 给我几个创意方向
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: 创意方向
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` 是发给模型的 Tool Schema，`mocks.tools` 是 Runner 收到 Tool Call 后如何返回结果；两者的 Tool 名字集合必须完全一致。
+- 需要 Skill 时在 `context.skills` 中按名字声明，并在 `context.tools` 中同时声明 `{builtin: skill}`；模型通过 `skill(name, file?)` 读取正文。
+- 需要模拟命令行工具时声明 `{builtin: bash}`，在 `mocks.cli` 中按命令配置输出。
+- `steps[].assert.hard` 是确定性检查（`tool_called`、`tool_arguments`、`assistant_contains` 等九种类型）；还可以加 `assert.judge` 做 LLM 打分。
+
+完整 Schema（Skill/Tool/Mock 的全部写法、`bash` 沙箱、Message Injection、Judge 配置等）见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 命令行使用
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| 参数 | 含义 |
+| --- | --- |
+| `paths`（位置参数，可多个） | 显式指定要跑的 Case YAML 路径 |
+| `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
+| `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择 |
+| `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
+| `--response-script PATH` | 用脚本化 LLM 响应回放，不调用真实模型 |
+| `--result-dir PATH` | 覆盖产物根目录（默认 `<plugin_root>/.yama/runs`） |
+| `--report [PATH]` | 额外生成单文件 HTML 报告（默认 `.yama/reports/latest.html`） |
+| `--list` | 只打印匹配到的 Case，不运行 |
+| `--no-artifacts` | 不写任何产物文件 |
+| `--json` | 输出机器可读 JSON 而不是 Rich 表格 |
+
+`--plugin`/`--all-plugins`/`--plugin-root` 三者互斥；都不提供时默认用当前目录向上搜索到的 `yama.toml` 所在目录作为 Workspace Root（找不到则用 cwd），把该目录当作单一 Plugin Root 运行。

python_yama-0.1.0/README.md

@@ -0,0 +1,73 @@
+# yama
+
+`yama` 是一个可复现的 Agent Skill 评测框架：用声明式 YAML Case 描述模型看到的 System Prompt、Skill metadata、Tool Schema 与多轮 User Message，通过 LiteLLM 统一调用 LLM 并驱动 Tool Loop，再对 Transcript 做确定性 Hard Check 或 LLM Judge 评分。完整 Case Schema 与执行契约见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 快速开始
+
+```bash
+# 在 Plugin 根目录下直接运行，默认收集 __evals__/cases/**/*.yaml
+cd dingding-simple
+OPENAI_API_KEY=... uv run --project yama yama
+
+# 在 Workspace 根目录（存在 yama.toml）按名字指定 Plugin
+OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
+```
+
+## 编写一个 Case
+
+一个 Case 是一份 YAML 文件，按 `context`（模型看到什么）→ `mocks`（Tool 怎么被执行）→ `steps`（依次发送的用户回合与断言）→ `outcome`（整个 Case 的通过门槛）的结构组织：
+
+```yaml
+context:
+  system_prompt: { default: true }     # 使用 Plugin 根目录下的 SYSTEM.md
+
+  tools:
+    - file: tools/read-dsl.yaml         # Tool Schema，相对 __evals__ 目录解析
+
+mocks:
+  tools:
+    read_dsl:
+      respond:
+        result:
+          visualStyle: { name: 复古胶片 }
+
+steps:
+  - id: request-directions
+    user: 给我几个创意方向
+    assert:
+      hard:
+        - tool_called: { name: read_dsl }
+        - assistant_contains: 创意方向
+
+outcome:
+  require:
+    hard_checks: all_pass
+```
+
+- `context.tools` 是发给模型的 Tool Schema，`mocks.tools` 是 Runner 收到 Tool Call 后如何返回结果；两者的 Tool 名字集合必须完全一致。
+- 需要 Skill 时在 `context.skills` 中按名字声明，并在 `context.tools` 中同时声明 `{builtin: skill}`；模型通过 `skill(name, file?)` 读取正文。
+- 需要模拟命令行工具时声明 `{builtin: bash}`，在 `mocks.cli` 中按命令配置输出。
+- `steps[].assert.hard` 是确定性检查（`tool_called`、`tool_arguments`、`assistant_contains` 等九种类型）；还可以加 `assert.judge` 做 LLM 打分。
+
+完整 Schema（Skill/Tool/Mock 的全部写法、`bash` 沙箱、Message Injection、Judge 配置等）见[设计文档](docs/agent-skill-test-framework-design.md)。
+
+## 命令行使用
+
+```bash
+uv run --project yama yama --plugin dingding-simple --report
+```
+
+| 参数 | 含义 |
+| --- | --- |
+| `paths`（位置参数，可多个） | 显式指定要跑的 Case YAML 路径 |
+| `--plugin-root PATH` | 以指定路径作为单一 Plugin Root |
+| `--plugin NAME`（可重复） | 按 `yama.toml` 中的 Plugin 名字选择 |
+| `--all-plugins` | 运行 `yama.toml` 中配置的全部 Plugin |
+| `--response-script PATH` | 用脚本化 LLM 响应回放，不调用真实模型 |
+| `--result-dir PATH` | 覆盖产物根目录（默认 `<plugin_root>/.yama/runs`） |
+| `--report [PATH]` | 额外生成单文件 HTML 报告（默认 `.yama/reports/latest.html`） |
+| `--list` | 只打印匹配到的 Case，不运行 |
+| `--no-artifacts` | 不写任何产物文件 |
+| `--json` | 输出机器可读 JSON 而不是 Rich 表格 |
+
+`--plugin`/`--all-plugins`/`--plugin-root` 三者互斥；都不提供时默认用当前目录向上搜索到的 `yama.toml` 所在目录作为 Workspace Root（找不到则用 cwd），把该目录当作单一 Plugin Root 运行。