voxarena 0.1.0__tar.gz

voxarena-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Simkeyur
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

voxarena-0.1.0/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: voxarena
+Version: 0.1.0
+Summary: An evaluation arena for realtime voice agents.
+Author: VoxArena contributors
+License: MIT License
+        
+        Copyright (c) 2026 Simkeyur
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/simkeyur/vox-arena
+Project-URL: Issues, https://github.com/simkeyur/vox-arena/issues
+Keywords: voice-agents,realtime-llm,evaluation,benchmarking,gemini-live,openai-realtime,pipecat
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.110.0
+Requires-Dist: uvicorn>=0.28.0
+Requires-Dist: pydantic>=2.6.0
+Requires-Dist: pydantic-settings>=2.2.0
+Requires-Dist: pipecat-ai[google,openai]>=0.5.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: loguru>=0.7.2
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="ui/src/assets/logo-dark.png" />
+    <img src="ui/src/assets/logo.png" alt="VoxArena" width="220" />
+  </picture>
+</p>
+
+<p align="center"><em>An evaluation arena for realtime voice agents.</em></p>
+
+<p align="center">
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
+[![Built with Pipecat](https://img.shields.io/badge/built%20with-pipecat-9cf.svg)](https://github.com/pipecat-ai/pipecat)
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](#contributing)
+
+</p>
+
+VoxArena is a reproducible benchmarking harness for realtime voice agents. Run the same scripted conversation across Gemini Live, OpenAI Realtime, and other [Pipecat](https://github.com/pipecat-ai/pipecat)-supported providers — and compare them apples-to-apples on latency, tool-call accuracy, and hallucinations.
+
+Drop it into your CI pipeline, your dev loop, or the bundled control panel.
+
+---
+
+## 🚀 CI & Pipeline Integration
+
+VoxArena ships a `voxarena` CLI designed for headless use in your build pipeline. It returns a non-zero exit code when metrics fall below thresholds you define, and emits JUnit XML for native CI reporting.
+
+```bash
+pip install voxarena
+
+voxarena run \
+  --provider gemini \
+  --script ./script/utterances.yaml \
+  --min-tool-accuracy 0.9 \
+  --max-hallucinations 0 \
+  --max-avg-ttfa-ms 1500 \
+  --output result.json \
+  --junit voxarena.xml
+# exit 0 if every threshold passes, 1 otherwise
+```
+
+### Compare two providers in one shot
+
+```bash
+voxarena compare \
+  --gemini-model gemini-3.1-flash-live-preview \
+  --openai-model gpt-realtime-2 \
+  --num-turns 5 \
+  --min-tool-accuracy 0.9 \
+  --output compare.json
+```
+
+### GitHub Actions
+
+```yaml
+- name: Voice agent regression check
+  env:
+    GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+  run: |
+    pip install voxarena
+    voxarena run --provider gemini \
+      --min-tool-accuracy 0.92 --max-hallucinations 0 \
+      --junit voxarena.xml --quiet
+
+- uses: mikepenz/action-junit-report@v4
+  if: always()
+  with:
+    report_paths: voxarena.xml
+```
+
+### Subcommands
+
+| Command | What it does |
+| --- | --- |
+| `voxarena run` | Single-provider scripted run; exits 0/1 against thresholds. |
+| `voxarena compare` | Runs Gemini and OpenAI in parallel against the same script. |
+| `voxarena report` | Generates a markdown comparison report from past runs. |
+
+Run `voxarena <command> --help` for the full flag set.
+
+---
+
+## Features
+
+- 🎙️ **Provider-agnostic agent** — one Pipecat pipeline drives every provider; swap models without re-implementing your agent
+- 🔁 **Scripted conversations** — multi-turn YAML scripts with pre-recorded WAV inputs and expected tool calls / response content
+- 📊 **Automated scoring** — tool-call correctness, response matching, hallucination counts, time-to-first-audio, interruption-stop latency
+- 🆚 **Side-by-side comparisons** — run multiple providers in parallel against the same script
+- 🗄️ **Persistent run history** — JSON manifests on disk, indexed in SQLite
+- 🖥️ **Web control panel** — React UI to launch runs, watch live status, browse results, and edit scripts
+- 🧩 **Extensible** — add a new provider by implementing one adapter class
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A["Recorded WAVs<br/>script/audio/*.wav"] --> B["Injection Harness<br/>voxarena/harness.py"]
+    B --> C
+
+    subgraph C ["Pipecat Pipeline"]
+        direction LR
+        C1["Audio Injector"] --> C2["Provider Adapter"]
+        C2 --> C3["Audio Capture"]
+        C3 --> C4["Metrics Collector"]
+    end
+
+    C2 <--> D{{"Provider Backend"}}
+    D --> D1["Gemini Live"]
+    D --> D2["OpenAI Realtime"]
+    D --> D3["...future providers"]
+
+    C4 --> E["Run Manifest<br/>results/PROVIDER/RUN_ID/manifest.json"]
+    E --> F[("SQLite Index<br/>runs.db")]
+
+    F <--> G["voxarena CLI<br/>+ FastAPI Backend"]
+    G <--> H["React Control Panel<br/>ui/"]
+
+    style D1 fill:#4285F4,color:#fff,stroke:#333
+    style D2 fill:#10A37F,color:#fff,stroke:#333
+    style D3 fill:#999,color:#fff,stroke:#333
+    style F fill:#f5f5f5,stroke:#333
+    style H fill:#fff7da,stroke:#333
+```
+
+## Local Dev (with UI)
+
+```bash
+git clone https://github.com/simkeyur/vox-arena.git
+cd vox-arena
+cp .env.example .env  # add GOOGLE_API_KEY / OPENAI_API_KEY
+
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e .
+
+uvicorn voxarena.main:app --reload --port 8000
+```
+
+Then in another terminal:
+
+```bash
+cd ui && npm install && npm run dev
+```
+
+Open the control panel at `http://localhost:5173`.
+
+## Bring Your Own Agent
+
+The demo ships with the "Saffron Leaf" restaurant agent so you can run end-to-end on day one. To evaluate your own:
+
+1. Replace the system prompt and tool schemas in `voxarena/agent.py`
+2. Implement (or stub) your tools in `voxarena/tools.py`
+3. Re-record `script/audio/*.wav` and update `script/utterances.yaml` to reflect your real workload
+4. Run the arena as normal — every provider gets scored against your scripts
+
+## Scripted Conversations
+
+Conversations live in [`script/utterances.yaml`](script/utterances.yaml). Each turn pairs an utterance id with an `expect` block describing the correct tool call and/or response content:
+
+```yaml
+- id: u04
+  text: "Are you open on Sundays?"
+  expect:
+    tool: get_hours
+    args:
+      day: sunday
+    response_contains:
+      - "closed"
+```
+
+The harness plays `script/audio/{id}.wav` into the pipeline and scores the agent's actual tool calls and transcript against `expect`.
+
+## Configuration
+
+| Variable | Description |
+| --- | --- |
+| `GOOGLE_API_KEY` / `OPENAI_API_KEY` | Provider credentials |
+| `GEMINI_MODEL` / `OPENAI_MODEL` | Realtime model under test |
+| `GEMINI_EVAL_MODEL` / `OPENAI_EVAL_MODEL` | Cheaper text models for grading |
+| `PORT` | FastAPI server port |
+| `BASE_DIR` | Override workdir (CLI: `--workdir`) |
+
+## Contributing
+
+To add a new provider: implement an adapter in `voxarena/providers/` following the pattern in `gemini.py` / `openai.py`, wire it into `voxarena/harness.py` and `voxarena/config.py`, and open a PR.
+
+For bugs and feature requests, please open an issue.
+
+## License
+
+[MIT](LICENSE).

voxarena-0.1.0/README.md

@@ -0,0 +1,191 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="ui/src/assets/logo-dark.png" />
+    <img src="ui/src/assets/logo.png" alt="VoxArena" width="220" />
+  </picture>
+</p>
+
+<p align="center"><em>An evaluation arena for realtime voice agents.</em></p>
+
+<p align="center">
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
+[![Built with Pipecat](https://img.shields.io/badge/built%20with-pipecat-9cf.svg)](https://github.com/pipecat-ai/pipecat)
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](#contributing)
+
+</p>
+
+VoxArena is a reproducible benchmarking harness for realtime voice agents. Run the same scripted conversation across Gemini Live, OpenAI Realtime, and other [Pipecat](https://github.com/pipecat-ai/pipecat)-supported providers — and compare them apples-to-apples on latency, tool-call accuracy, and hallucinations.
+
+Drop it into your CI pipeline, your dev loop, or the bundled control panel.
+
+---
+
+## 🚀 CI & Pipeline Integration
+
+VoxArena ships a `voxarena` CLI designed for headless use in your build pipeline. It returns a non-zero exit code when metrics fall below thresholds you define, and emits JUnit XML for native CI reporting.
+
+```bash
+pip install voxarena
+
+voxarena run \
+  --provider gemini \
+  --script ./script/utterances.yaml \
+  --min-tool-accuracy 0.9 \
+  --max-hallucinations 0 \
+  --max-avg-ttfa-ms 1500 \
+  --output result.json \
+  --junit voxarena.xml
+# exit 0 if every threshold passes, 1 otherwise
+```
+
+### Compare two providers in one shot
+
+```bash
+voxarena compare \
+  --gemini-model gemini-3.1-flash-live-preview \
+  --openai-model gpt-realtime-2 \
+  --num-turns 5 \
+  --min-tool-accuracy 0.9 \
+  --output compare.json
+```
+
+### GitHub Actions
+
+```yaml
+- name: Voice agent regression check
+  env:
+    GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
+  run: |
+    pip install voxarena
+    voxarena run --provider gemini \
+      --min-tool-accuracy 0.92 --max-hallucinations 0 \
+      --junit voxarena.xml --quiet
+
+- uses: mikepenz/action-junit-report@v4
+  if: always()
+  with:
+    report_paths: voxarena.xml
+```
+
+### Subcommands
+
+| Command | What it does |
+| --- | --- |
+| `voxarena run` | Single-provider scripted run; exits 0/1 against thresholds. |
+| `voxarena compare` | Runs Gemini and OpenAI in parallel against the same script. |
+| `voxarena report` | Generates a markdown comparison report from past runs. |
+
+Run `voxarena <command> --help` for the full flag set.
+
+---
+
+## Features
+
+- 🎙️ **Provider-agnostic agent** — one Pipecat pipeline drives every provider; swap models without re-implementing your agent
+- 🔁 **Scripted conversations** — multi-turn YAML scripts with pre-recorded WAV inputs and expected tool calls / response content
+- 📊 **Automated scoring** — tool-call correctness, response matching, hallucination counts, time-to-first-audio, interruption-stop latency
+- 🆚 **Side-by-side comparisons** — run multiple providers in parallel against the same script
+- 🗄️ **Persistent run history** — JSON manifests on disk, indexed in SQLite
+- 🖥️ **Web control panel** — React UI to launch runs, watch live status, browse results, and edit scripts
+- 🧩 **Extensible** — add a new provider by implementing one adapter class
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A["Recorded WAVs<br/>script/audio/*.wav"] --> B["Injection Harness<br/>voxarena/harness.py"]
+    B --> C
+
+    subgraph C ["Pipecat Pipeline"]
+        direction LR
+        C1["Audio Injector"] --> C2["Provider Adapter"]
+        C2 --> C3["Audio Capture"]
+        C3 --> C4["Metrics Collector"]
+    end
+
+    C2 <--> D{{"Provider Backend"}}
+    D --> D1["Gemini Live"]
+    D --> D2["OpenAI Realtime"]
+    D --> D3["...future providers"]
+
+    C4 --> E["Run Manifest<br/>results/PROVIDER/RUN_ID/manifest.json"]
+    E --> F[("SQLite Index<br/>runs.db")]
+
+    F <--> G["voxarena CLI<br/>+ FastAPI Backend"]
+    G <--> H["React Control Panel<br/>ui/"]
+
+    style D1 fill:#4285F4,color:#fff,stroke:#333
+    style D2 fill:#10A37F,color:#fff,stroke:#333
+    style D3 fill:#999,color:#fff,stroke:#333
+    style F fill:#f5f5f5,stroke:#333
+    style H fill:#fff7da,stroke:#333
+```
+
+## Local Dev (with UI)
+
+```bash
+git clone https://github.com/simkeyur/vox-arena.git
+cd vox-arena
+cp .env.example .env  # add GOOGLE_API_KEY / OPENAI_API_KEY
+
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e .
+
+uvicorn voxarena.main:app --reload --port 8000
+```
+
+Then in another terminal:
+
+```bash
+cd ui && npm install && npm run dev
+```
+
+Open the control panel at `http://localhost:5173`.
+
+## Bring Your Own Agent
+
+The demo ships with the "Saffron Leaf" restaurant agent so you can run end-to-end on day one. To evaluate your own:
+
+1. Replace the system prompt and tool schemas in `voxarena/agent.py`
+2. Implement (or stub) your tools in `voxarena/tools.py`
+3. Re-record `script/audio/*.wav` and update `script/utterances.yaml` to reflect your real workload
+4. Run the arena as normal — every provider gets scored against your scripts
+
+## Scripted Conversations
+
+Conversations live in [`script/utterances.yaml`](script/utterances.yaml). Each turn pairs an utterance id with an `expect` block describing the correct tool call and/or response content:
+
+```yaml
+- id: u04
+  text: "Are you open on Sundays?"
+  expect:
+    tool: get_hours
+    args:
+      day: sunday
+    response_contains:
+      - "closed"
+```
+
+The harness plays `script/audio/{id}.wav` into the pipeline and scores the agent's actual tool calls and transcript against `expect`.
+
+## Configuration
+
+| Variable | Description |
+| --- | --- |
+| `GOOGLE_API_KEY` / `OPENAI_API_KEY` | Provider credentials |
+| `GEMINI_MODEL` / `OPENAI_MODEL` | Realtime model under test |
+| `GEMINI_EVAL_MODEL` / `OPENAI_EVAL_MODEL` | Cheaper text models for grading |
+| `PORT` | FastAPI server port |
+| `BASE_DIR` | Override workdir (CLI: `--workdir`) |
+
+## Contributing
+
+To add a new provider: implement an adapter in `voxarena/providers/` following the pattern in `gemini.py` / `openai.py`, wire it into `voxarena/harness.py` and `voxarena/config.py`, and open a PR.
+
+For bugs and feature requests, please open an issue.
+
+## License
+
+[MIT](LICENSE).

voxarena-0.1.0/pyproject.toml

@@ -0,0 +1,54 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "voxarena"
+version = "0.1.0"
+description = "An evaluation arena for realtime voice agents."
+readme = "README.md"
+requires-python = ">=3.11"
+license = { file = "LICENSE" }
+authors = [{ name = "VoxArena contributors" }]
+keywords = [
+    "voice-agents",
+    "realtime-llm",
+    "evaluation",
+    "benchmarking",
+    "gemini-live",
+    "openai-realtime",
+    "pipecat",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "fastapi>=0.110.0",
+    "uvicorn>=0.28.0",
+    "pydantic>=2.6.0",
+    "pydantic-settings>=2.2.0",
+    "pipecat-ai[openai,google]>=0.5.0",
+    "python-dotenv>=1.0.1",
+    "pyyaml>=6.0.1",
+    "loguru>=0.7.2",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0.0"]
+
+[project.scripts]
+voxarena = "voxarena.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/simkeyur/vox-arena"
+Issues = "https://github.com/simkeyur/vox-arena/issues"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["voxarena*"]

voxarena-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

voxarena-0.1.0/voxarena/agent.py

@@ -0,0 +1,91 @@
+"""Agent definition — system prompt + tool schemas.
+
+The bundled defaults ship the "Saffron Leaf" restaurant assistant under
+``voxarena/data/saffron_leaf/`` so VoxArena runs end-to-end out of the box.
+
+To evaluate your own agent, point ``data_dir`` at a directory containing a
+``system_prompt.txt`` and edit ``voxarena/tools.py`` to register your tools.
+"""
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from typing import Any, Dict, List, Optional
+
+from voxarena.tools import TOOL_SCHEMAS
+
+DEFAULT_DATA_DIR = os.path.join(os.path.dirname(__file__), "data", "saffron_leaf")
+
+
+class Agent:
+    """A versioned, hashable bundle of (system prompt + tool schemas).
+
+    Args:
+        prompt_version: Manifest tag for the prompt revision.
+        tool_schema_version: Manifest tag for the tool schema revision.
+        data_dir: Directory containing ``system_prompt.txt``. Defaults to the
+            bundled Saffron Leaf example agent.
+    """
+
+    def __init__(
+        self,
+        prompt_version: str = "v1.0",
+        tool_schema_version: str = "v1.0",
+        data_dir: Optional[str] = None,
+    ):
+        self.prompt_version = prompt_version
+        self.tool_schema_version = tool_schema_version
+        self.data_dir = data_dir or DEFAULT_DATA_DIR
+
+        self.prompt_path = os.path.join(self.data_dir, "system_prompt.txt")
+        self.system_prompt = self._load_system_prompt()
+
+        self.tool_schemas = TOOL_SCHEMAS
+
+        self.prompt_hash = self._sha256(self.system_prompt)
+        self.tool_schema_hash = self._sha256(json.dumps(self.tool_schemas, sort_keys=True))
+
+    def _load_system_prompt(self) -> str:
+        if not os.path.exists(self.prompt_path):
+            raise FileNotFoundError(f"System prompt file not found at {self.prompt_path}")
+        with open(self.prompt_path, "r") as f:
+            return f.read().strip()
+
+    @staticmethod
+    def _sha256(content: str) -> str:
+        return hashlib.sha256(content.encode("utf-8")).hexdigest()
+
+    def get_agent_metadata(self) -> Dict[str, Any]:
+        """Versioning and checksum info recorded in manifests."""
+        return {
+            "prompt_version": self.prompt_version,
+            "prompt_hash": self.prompt_hash,
+            "tool_schema_version": self.tool_schema_version,
+            "tool_schema_hash": self.tool_schema_hash,
+        }
+
+    def get_openai_tools(self) -> List[Dict[str, Any]]:
+        return [
+            {
+                "type": "function",
+                "name": s["name"],
+                "description": s["description"],
+                "parameters": s["parameters"],
+            }
+            for s in self.tool_schemas
+        ]
+
+    def get_gemini_tools(self) -> List[Dict[str, Any]]:
+        return [
+            {
+                "function_declarations": [
+                    {
+                        "name": s["name"],
+                        "description": s["description"],
+                        "parameters": s["parameters"],
+                    }
+                ]
+            }
+            for s in self.tool_schemas
+        ]