vibeval 0.1.0__tar.gz

vibeval-0.1.0/.gitignore

@@ -0,0 +1,26 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+
+# Virtual environment
+.venv/
+
+# pytest
+.pytest_cache/
+
+
+# vibeval results (generated at runtime)
+.vibeval/
+**/results/
+**/comparisons/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+
+# OS
+.DS_Store

vibeval-0.1.0/CLAUDE.md

@@ -0,0 +1,96 @@
+# vibeval 项目开发指南
+
+## 项目概述
+
+vibeval (Vibe Coding Eval) 是一个 AI 应用评测框架，由两部分组成：
+- **vibeval CLI** (`src/vibeval/`) — 评测工具（judge、compare、simulate、report 等）
+- **Claude Code Plugin** (`plugin/`) — VibeCoding 工作流（analyze、design、generate、run、update）
+
+## 核心原则
+
+### 1. 协议为本
+
+`plugin/skills/protocol/references/` 下的协议文档是整个项目的核心设定（Source of Truth）。所有 plugin 命令、CLI 代码、文档都必须围绕和遵循协议。当任何地方的内容与协议矛盾时，以协议为准。编写文档或命令时，避免重复协议中已有的定义，改为引用协议文件。
+
+协议文件：
+- `00-philosophy.md` — 评测哲学（信息不对称 + 全局视野）
+- `01-overview.md` — 目录结构、统一 turn 模型
+- `02-dataset.md` — 数据集格式
+- `03-judge-spec.md` — 评判规格（rule/llm、target、所有字段定义）
+- `04-result.md` — 结果格式（trace turns/steps）
+- `05-comparison.md` — 横评格式
+
+### 2. 语言无关
+
+所有设计必须避免语言耦合。vibeval CLI 提供通用的、可通过命令行直接执行的功能，不需要在代码层面做依赖。用户的测试代码不 import vibeval 包，而是通过 subprocess 调用 CLI（如 `vibeval simulate`），或直接按协议格式生成文件。CLI 既服务开发者，也服务 VibeCoding Agent。
+
+### 3. CLI Help 是命令文档的唯一来源
+
+CLI 的每个命令必须保持完整、最新的 `--help` 描述（包括用途、参数说明、使用示例）。Plugin 的命令文档和 SKILL.md 中不重复 CLI 的参数细节，而是提示通过 `vibeval --help` / `vibeval <command> --help` 查看。这样 CLI 和文档永远不会不同步。
+
+### 4. 测试目录的区分
+
+`tests/` 目录下是 vibeval CLI 工具和代码自身的单元测试、集成测试。`examples/` 下的项目（如 `meeting_app/`）是独立的示例应用，它们有自己的测试（在 `examples/meeting_app/tests/vibeval/` 下），演示的是用户使用 vibeval 的完整流程。两者不要混淆。
+
+## 开发约定
+
+### 运行测试
+
+```bash
+# vibeval 自身的测试
+python -m pytest tests/ -v
+
+# 示例应用的测试（独立运行）
+cd examples/meeting_app/tests/vibeval/meeting_summary/tests
+python -m pytest test_summarizer.py -v
+
+# 示例应用的 judge
+cd examples/meeting_app
+vibeval judge meeting_summary latest
+```
+
+### 修改协议
+
+修改协议文件后，检查以下是否需要同步更新：
+- `src/vibeval/` 中的代码实现（rules.py、llm.py、judge.py、compare.py、result.py）
+- `plugin/commands/` 中的命令文档（应引用协议而非重复内容）
+- `plugin/skills/protocol/SKILL.md`（Quick Reference 摘要）
+- CLI 的 `--help` 描述
+
+### 修改 CLI
+
+新增或修改 CLI 命令时：
+- 在 `cli.py` 中提供完整的 `help` 和 `description` 文本
+- 不需要同步更新 plugin 文档中的命令细节——plugin 文档应引导查看 `vibeval --help`
+- 添加对应的测试到 `tests/`
+
+### 版本管理
+
+项目版本在三处维护，必须保持一致：
+- `pyproject.toml` — `version` 字段
+- `plugin/.claude-plugin/plugin.json` — `version` 字段
+- `src/vibeval/__init__.py` — `__version__` 变量
+
+发版或变更版本号时，三处同时更新。
+
+### 修改 Plugin
+
+修改 plugin 命令或 skill 时：
+- 基础概念定义（数据格式、字段、规则名等）一律引用 `references/` 中的协议文件，不内联重复
+- 操作指导（如何设计、如何生成）可以在命令文档中展开，但引用协议作为定义来源
+- CLI 命令的参数和用法，引导使用 `vibeval --help` 查看
+
+### 项目结构
+
+```
+vibeval/
+├── src/vibeval/            # CLI 工具实现
+├── plugin/                # Claude Code 插件
+│   ├── commands/           # /vibeval-analyze, /vibeval-design, /vibeval-generate, /vibeval-run, /vibeval-update
+│   └── skills/protocol/    # 数据协议（Source of Truth）
+├── examples/              # 独立的示例应用
+├── tests/                 # vibeval 自身的测试
+├── CLAUDE.md
+├── README.md
+└── pyproject.toml
+```

vibeval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SandyKid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vibeval-0.1.0/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.4
+Name: vibeval
+Version: 0.1.0
+Summary: vibeval (Vibe Coding Eval) — AI application testing framework
+Project-URL: Homepage, https://github.com/SandyKidYao/vibeval
+Project-URL: Repository, https://github.com/SandyKidYao/vibeval
+Project-URL: Issues, https://github.com/SandyKidYao/vibeval/issues
+Author-email: SandyKid <yupengyao912@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# vibeval — Vibe Coding Eval
+
+AI 应用的快速评测框架。只需安装 Claude Code，即可完成从代码分析、测试生成到评估的全闭环。
+
+## 解决什么问题
+
+传统软件测试框架无法评估 AI 输出的质量；传统 AI 评测平台依赖数据集构建，跟不上功能迭代速度。vibeval 在两者之间取得平衡：
+
+- 通过 VibeCoding 分析你的代码，快速生成合成数据和测试用例
+- 确定性规则 + LLM 语义评判，双重评估
+- 跨版本横评，追踪质量变化
+- 语言无关：生成的测试代码适配你的项目框架，不依赖 vibeval 包
+
+## 前置要求
+
+- [Claude Code](https://claude.ai/code)
+- Python 3.10+
+
+## 安装
+
+```bash
+# 安装 vibeval CLI
+pip install vibeval
+
+# 安装 Claude Code 插件（在 Claude Code 中执行）
+/install-plugin /path/to/vibeval/plugin
+```
+
+## 使用方式
+
+在 Claude Code 中执行，全程不需要离开对话：
+
+```
+/vibeval-analyze  →  /vibeval-design  →  /vibeval-generate  →  /vibeval-run
+                                                      ↑
+                          代码变更 → /vibeval-update ──────┘
+```
+
+### 分析代码
+
+```
+/vibeval-analyze meeting_summary src/services/
+```
+
+分析源码，识别 AI 调用点、数据流、Mock 点，给出可评估性改进建议。
+
+### 设计测试
+
+```
+/vibeval-design meeting_summary
+```
+
+设计合成数据规格、评判标准、测试结构。每一步产出可编辑的中间文件，可以 review 和修改。
+
+### 生成代码和数据
+
+```
+/vibeval-generate meeting_summary
+```
+
+生成完整的测试套件（合成数据 + 测试代码），使用你项目的测试框架（pytest/vitest/jest/...）。
+
+### 运行 + 评估
+
+```
+/vibeval-run meeting_summary
+```
+
+一步完成：执行测试 → 评估 → 诊断分析 → 生成报告。
+
+### 代码变更后更新
+
+```
+/vibeval-update meeting_summary
+```
+
+检测代码变更，增量更新测试套件和数据。
+
+### 跨版本对比
+
+```bash
+# 统计对比
+vibeval diff meeting_summary run_a run_b
+
+# LLM 深度横评
+vibeval compare meeting_summary run_a run_b
+```
+
+### 可视化报告
+
+```bash
+vibeval report meeting_summary latest --open
+```
+
+生成自包含的 HTML 报告，涵盖测试设计、数据、过程 trace 和评判结果。
+
+### 更多命令
+
+```bash
+vibeval --help
+```
+
+## License
+
+MIT

vibeval-0.1.0/README.md

@@ -0,0 +1,105 @@
+# vibeval — Vibe Coding Eval
+
+AI 应用的快速评测框架。只需安装 Claude Code，即可完成从代码分析、测试生成到评估的全闭环。
+
+## 解决什么问题
+
+传统软件测试框架无法评估 AI 输出的质量；传统 AI 评测平台依赖数据集构建，跟不上功能迭代速度。vibeval 在两者之间取得平衡：
+
+- 通过 VibeCoding 分析你的代码，快速生成合成数据和测试用例
+- 确定性规则 + LLM 语义评判，双重评估
+- 跨版本横评，追踪质量变化
+- 语言无关：生成的测试代码适配你的项目框架，不依赖 vibeval 包
+
+## 前置要求
+
+- [Claude Code](https://claude.ai/code)
+- Python 3.10+
+
+## 安装
+
+```bash
+# 安装 vibeval CLI
+pip install vibeval
+
+# 安装 Claude Code 插件（在 Claude Code 中执行）
+/install-plugin /path/to/vibeval/plugin
+```
+
+## 使用方式
+
+在 Claude Code 中执行，全程不需要离开对话：
+
+```
+/vibeval-analyze  →  /vibeval-design  →  /vibeval-generate  →  /vibeval-run
+                                                      ↑
+                          代码变更 → /vibeval-update ──────┘
+```
+
+### 分析代码
+
+```
+/vibeval-analyze meeting_summary src/services/
+```
+
+分析源码，识别 AI 调用点、数据流、Mock 点，给出可评估性改进建议。
+
+### 设计测试
+
+```
+/vibeval-design meeting_summary
+```
+
+设计合成数据规格、评判标准、测试结构。每一步产出可编辑的中间文件，可以 review 和修改。
+
+### 生成代码和数据
+
+```
+/vibeval-generate meeting_summary
+```
+
+生成完整的测试套件（合成数据 + 测试代码），使用你项目的测试框架（pytest/vitest/jest/...）。
+
+### 运行 + 评估
+
+```
+/vibeval-run meeting_summary
+```
+
+一步完成：执行测试 → 评估 → 诊断分析 → 生成报告。
+
+### 代码变更后更新
+
+```
+/vibeval-update meeting_summary
+```
+
+检测代码变更，增量更新测试套件和数据。
+
+### 跨版本对比
+
+```bash
+# 统计对比
+vibeval diff meeting_summary run_a run_b
+
+# LLM 深度横评
+vibeval compare meeting_summary run_a run_b
+```
+
+### 可视化报告
+
+```bash
+vibeval report meeting_summary latest --open
+```
+
+生成自包含的 HTML 报告，涵盖测试设计、数据、过程 trace 和评判结果。
+
+### 更多命令
+
+```bash
+vibeval --help
+```
+
+## License
+
+MIT

vibeval-0.1.0/examples/meeting_app/.vibeval.yml

@@ -0,0 +1 @@
+vibeval_root: tests/vibeval

vibeval-0.1.0/examples/meeting_app/llm_client.py

@@ -0,0 +1,24 @@
+"""LLM client — wraps the actual LLM API call.
+
+In production this calls OpenAI/Anthropic. In tests this gets mocked.
+"""
+
+from __future__ import annotations
+
+
+def chat(prompt: str, system: str = "") -> str:
+    """Send a prompt to the LLM and return the response text."""
+    # In a real app, this would be:
+    #   client = openai.OpenAI()
+    #   resp = client.chat.completions.create(
+    #       model="gpt-4o",
+    #       messages=[
+    #           {"role": "system", "content": system},
+    #           {"role": "user", "content": prompt},
+    #       ],
+    #   )
+    #   return resp.choices[0].message.content
+    raise RuntimeError(
+        "LLM client not configured. "
+        "Set OPENAI_API_KEY or mock this in tests."
+    )

vibeval-0.1.0/examples/meeting_app/summarizer.py

@@ -0,0 +1,58 @@
+"""Meeting summarizer — the core AI pipeline.
+
+Pipeline:
+  1. Fetch transcript from API
+  2. Call LLM to generate summary
+  3. Call LLM to extract action items
+  4. Return structured result
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from . import llm_client, transcript_api
+
+
+def summarize_meeting(meeting_id: str) -> dict[str, Any]:
+    """Run the full meeting summarization pipeline."""
+
+    # Step 1: Fetch transcript
+    transcript = transcript_api.fetch_transcript(meeting_id)
+    speakers = transcript.get("speakers", [])
+    text = transcript.get("text", "")
+
+    # Step 2: Generate summary
+    summary_prompt = (
+        f"Summarize the following meeting transcript.\n"
+        f"Speakers: {', '.join(speakers)}\n\n"
+        f"Transcript:\n{text}\n\n"
+        f"Requirements:\n"
+        f"- Include ALL mentioned times, dates, and deadlines\n"
+        f"- Attribute key points to the correct speaker\n"
+        f"- Be concise but complete"
+    )
+    summary = llm_client.chat(
+        prompt=summary_prompt,
+        system="You are a meeting summarizer. Be thorough and accurate.",
+    )
+
+    # Step 3: Extract action items
+    actions_prompt = (
+        f"Extract action items from this meeting summary:\n\n"
+        f"{summary}\n\n"
+        f"Format each as: - [Owner] Action (deadline if mentioned)"
+    )
+    action_items = llm_client.chat(
+        prompt=actions_prompt,
+        system="You extract action items from meeting summaries.",
+    )
+
+    # Step 4: Return structured result
+    return {
+        "meeting_id": meeting_id,
+        "speakers": speakers,
+        "summary": summary,
+        "action_items": action_items,
+        "transcript_length": len(text),
+    }

vibeval-0.1.0/examples/meeting_app/tests/vibeval/meeting_summary/datasets/meeting_summaries/manifest.yaml

@@ -0,0 +1,31 @@
+name: meeting_summaries
+description: "Synthetic meeting transcripts for testing the summarization pipeline"
+version: "1"
+tags:
+  - meeting
+  - summarization
+
+judge_specs:
+  - method: rule
+    rule: contains_all
+    args:
+      field: "outputs.summary"
+      values_from: "expected_times"
+    weight: gate
+
+  - method: rule
+    rule: length_between
+    args:
+      field: "outputs.action_items"
+      min: 10
+      max: 5000
+
+  - method: rule
+    rule: max_steps
+    args:
+      max: 20
+
+  - method: rule
+    rule: max_turns
+    args:
+      max: 1

vibeval-0.1.0/examples/meeting_app/tests/vibeval/meeting_summary/datasets/meeting_summaries/planning.json

@@ -0,0 +1,8 @@
+{
+  "_id": "planning",
+  "meeting_id": "mtg-plan-001",
+  "_tags": ["planning", "deadlines"],
+  "speakers": ["Diana", "Eve"],
+  "text": "Diana: Let's go over the project timeline. Phase 1 needs to be completed by June 30th. The kickoff was on January 15th.\n\nEve: Right. The mid-project review is scheduled for April 15th at 10:00 AM. We also have weekly syncs every Tuesday at 3:30 PM.\n\nDiana: The final demo is on July 5th. We need to submit the report by end of day July 10th.\n\nEve: Got it. I'll set up the sprint planning for next Monday at 9:30 AM.",
+  "expected_times": ["June 30th", "January 15th", "April 15th", "10:00 AM", "Tuesday", "3:30 PM", "July 5th", "July 10th", "Monday", "9:30 AM"]
+}

vibeval-0.1.0/examples/meeting_app/tests/vibeval/meeting_summary/datasets/meeting_summaries/standup.json

@@ -0,0 +1,8 @@
+{
+  "_id": "standup",
+  "meeting_id": "mtg-standup-001",
+  "_tags": ["standup", "multiple-times"],
+  "speakers": ["Alice", "Bob", "Charlie"],
+  "text": "Alice: Good morning everyone. It's 9:00 AM, let's start our standup. Bob, what did you work on yesterday?\n\nBob: Yesterday I finished the API refactoring that was due by March 15th. Today I'll work on the database migration, which needs to be done before the 3:00 PM deployment window.\n\nAlice: Great. Charlie?\n\nCharlie: I have a meeting with the client at 2:30 PM today to discuss the Q2 roadmap. The proposal deadline is April 1st. I'll also review Bob's PR before 11:00 AM.\n\nAlice: Perfect. Remember the all-hands is at 4:00 PM today. Let's wrap up by 9:15 AM.",
+  "expected_times": ["9:00 AM", "March 15th", "3:00 PM", "2:30 PM", "April 1st", "11:00 AM", "4:00 PM", "9:15 AM"]
+}

vibeval-0.1.0/examples/meeting_app/tests/vibeval/meeting_summary/tests/conftest.py

@@ -0,0 +1,110 @@
+"""Pytest configuration for meeting_app vibeval tests.
+
+[Generated by VibeCoding] This conftest:
+1. Loads vibeval datasets
+2. Provides a fixture that creates tracked mock wrappers
+3. Saves vibeval-format results after each test
+"""
+
+import json
+import time
+from pathlib import Path
+from typing import Any
+from unittest.mock import patch
+
+import pytest
+import yaml
+
+
+# --- vibeval dataset loading ---
+
+FEATURE_DIR = Path(__file__).parent.parent  # tests/vibeval/meeting_summary/
+DATASETS_DIR = FEATURE_DIR / "datasets"
+RESULTS_DIR = FEATURE_DIR / "results"
+
+
+def load_dataset(name: str) -> dict[str, Any]:
+    """Load a dataset manifest + items."""
+    ds_dir = DATASETS_DIR / name
+    manifest = yaml.safe_load((ds_dir / "manifest.yaml").read_text())
+
+    items = {}
+    for f in sorted(ds_dir.iterdir()):
+        if f.name.startswith("manifest"):
+            continue
+        if f.suffix == ".json":
+            data = json.loads(f.read_text())
+            item_id = data.pop("_id", f.stem)
+            items[item_id] = data
+
+    return {"manifest": manifest, "items": items}
+
+
+MEETING_DS = load_dataset("meeting_summaries")
+
+
+@pytest.fixture(params=list(MEETING_DS["items"].keys()))
+def meeting_item(request):
+    """Parametrized fixture: yields each dataset item."""
+    item_id = request.param
+    return item_id, MEETING_DS["items"][item_id]
+
+
+# --- vibeval result saving ---
+
+class VibevalResultCollector:
+    """Collects turn-based trace and saves vibeval-format results (v0.2)."""
+
+    def __init__(self, test_name: str, dataset: str, item_id: str, item_data: dict):
+        self.test_name = test_name
+        self.dataset = dataset
+        self.item_id = item_id
+        self.item_data = item_data
+        self.turns: list[dict] = []
+        self.outputs: dict = {}
+        self.start_time = time.time()
+        # Current turn being built
+        self._current_input: dict = {}
+        self._current_steps: list[dict] = []
+
+    def begin_turn(self, input_data: dict):
+        """Start a new turn with the given input."""
+        self._current_input = input_data
+        self._current_steps = []
+
+    def step(self, step_type: str, data: dict):
+        """Record a processing step in the current turn."""
+        self._current_steps.append({
+            "type": step_type,
+            "data": data,
+            "timestamp": time.time(),
+        })
+
+    def end_turn(self, output_data: dict):
+        """Complete the current turn with the given output."""
+        self.turns.append({
+            "turn": len(self.turns) + 1,
+            "input": self._current_input,
+            "steps": self._current_steps,
+            "output": output_data,
+        })
+
+    def save(self, run_id: str = "latest"):
+        duration = time.time() - self.start_time
+        result = {
+            "test_name": self.test_name,
+            "dataset": self.dataset,
+            "item_id": self.item_id,
+            "judge_results": [],
+            "trace": {"turns": self.turns},
+            "inputs": {k: v for k, v in self.item_data.items() if not k.startswith("expected")},
+            "outputs": self.outputs,
+            "timestamp": self.start_time,
+            "duration": duration,
+        }
+
+        run_dir = RESULTS_DIR / run_id
+        run_dir.mkdir(parents=True, exist_ok=True)
+        path = run_dir / f"{self.test_name}__{self.item_id}.result.json"
+        path.write_text(json.dumps(result, indent=2, default=str, ensure_ascii=False))
+        return path

vibeval-0.1.0/examples/meeting_app/tests/vibeval/meeting_summary/tests/pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+testpaths = .
+pythonpath = ../../../../..