indicorderbench 0.1.0__tar.gz

indicorderbench-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+.venv/
+__pycache__/
+*.pyc
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+results/
+demo-out/
+packs/*/clips/**/*.wav
+!packs/*/clips/manifest.json
+.superpowers/
+.DS_Store

indicorderbench-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

indicorderbench-0.1.0/CHANGELOG.md

@@ -0,0 +1,36 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## [Unreleased]
+
+### Changed
+- README rewritten for the public repository: comparison table, report screenshot,
+  install from git until the PyPI release.
+
+## [0.1.0] - 2026-10-01
+
+First release.
+
+### Added
+- Pydantic schemas for menus, scenarios with acceptable end states, and results.
+- Sandbox order backend with a timestamped tool-call trace and JSON-schema tool definitions.
+- Deterministic checker producing field-level expected/actual rows.
+- Scripted caller with clarification rules, confirm/closing/fallback/nudge behaviour and
+  simulator-validity tracking.
+- Runner with per-turn and per-trial timeouts, outcome classification
+  (pass / fail / simulator_invalid / infra_error), pass^k, Wilson intervals and latency
+  percentiles.
+- In-process and HTTP turn adapters; HTTP sandbox backend server.
+- Rule-based reference agent with switchable bugs (`ignore_corrections`, `drop_modifiers`,
+  `double_submit`, `ignore_cancellation`, `quantity_default_one`).
+- Starter pack: 42 scenarios (21 English, 21 Hinglish) across five failure categories,
+  including two `demo`-tagged correction scenarios; bundled in the wheel.
+- JSON, JUnit and single-file HTML reports; baseline comparison with regression exit code.
+- Audio: WAV helpers, noise/gain/telephone perturbations, Sarvam bulbul TTS synthesis,
+  Sarvam saaras STT, oracle transcriber, clip manifest, and an offline placeholder TTS
+  (`--provider silence`) so audio-modality runs work without API keys.
+- `examples/http_agent_shim.py`: the reference agent behind the HTTP turn protocol.
+- `iob` CLI: validate, ls, run, report, compare, demo, synth, perturb, serve-backend;
+  `iob run` takes `--allow-infra`, `--backend-host`, `--backend-port` and `--backend-url`.

indicorderbench-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,98 @@
+# Contributing
+
+Thanks for helping make voice ordering agents measurably better. Small contributions are
+the most useful ones right now.
+
+## Good first contributions
+
+| Area | What helps | Where |
+|---|---|---|
+| Language review | A native Hindi/Hinglish speaker reviewing the 21 Hinglish scenarios for naturalness and regional phrasing | `packs/starter/REVIEW.md`, set `review_status: reviewed` |
+| New scenarios | Real failures you have seen an agent make, as a scenario with acceptable end states | `docs/scenarios.md` |
+| New language packs | Telugu and Tamil are next; copy `packs/starter`, translate, add aliases, get a native review | `packs/<lang>/` |
+| Adapters | LiveKit, Pipecat, Vapi, Retell: a turn endpoint or a native adapter | `docs/adapters.md`, `src/indicorderbench/adapters/` |
+| Reports | Better HTML, a Markdown summary for PR comments, a JSON schema file | `src/indicorderbench/report/` |
+| Reference agents | An LLM-backed reference agent using `OrderBackend.tool_specs()` | `src/indicorderbench/agents/` |
+
+## Development setup
+
+```bash
+git clone https://github.com/Lingikaushikreddy/indicorderbench.git indicorderbench
+cd indicorderbench
+uv sync --all-extras --dev
+uv run pytest -q
+uv run iob demo --out demo-out
+```
+
+Before opening a pull request:
+
+```bash
+uv run ruff format src tests
+uv run ruff check src tests
+uv run mypy
+uv run pytest -q
+uv run iob validate packs/starter
+```
+
+CI runs the same commands on Python 3.11, 3.12 and 3.13.
+
+## Pull request checklist
+
+- One change per PR, with a test that failed before the change.
+- Scenario changes keep `tests/test_starter_pack.py` green (the bug-isolation contract).
+- No new core dependencies. Audio-only dependencies go under the `audio` extra.
+- Public functions have type hints; `mypy --strict` passes.
+- `CHANGELOG.md` gets a line under "Unreleased".
+
+## Release checklist
+
+Releases are published by `.github/workflows/release.yml` through PyPI Trusted Publishing
+when a `v*` tag is pushed; the tag must equal the version in `pyproject.toml` and
+`src/indicorderbench/__init__.py`, and `CHANGELOG.md` must have a dated section for it.
+Before tagging, run these from a clean checkout of the release commit.
+
+1. Validate the bundled pack and run the checks above:
+
+   ```bash
+   uv run iob validate packs/starter
+   ```
+
+2. Build the sdist and the wheel:
+
+   ```bash
+   rm -rf dist && uv build
+   ```
+
+3. Confirm the wheel carries no audio. `hatch_build.py` bundles `packs/starter` without
+   `*.wav`, so clips from a local `iob synth` must never appear. This prints nothing:
+
+   ```bash
+   python -m zipfile -l dist/indicorderbench-*.whl | grep -i '\.wav'
+   ```
+
+4. Install the wheel into a fresh environment and run the installed CLI in an empty
+   directory, so it uses the bundled pack rather than the repository's:
+
+   ```bash
+   wheel="$(pwd)/$(ls dist/*.whl)"
+   cd "$(mktemp -d)"
+   uv venv && uv pip install "$wheel"
+   .venv/bin/iob validate starter
+   .venv/bin/iob demo --out demo-out
+   ```
+
+## Recordings and consent
+
+If you contribute human recordings, include written consent from each speaker for
+publication under this repository's license. Removing a name from a file does not grant
+permission to publish it. Synthesised clips must name the provider, voice and model in the
+manifest.
+
+## Reporting results
+
+If you publish numbers produced with this benchmark, follow `docs/methodology.md` section 9:
+always include trial counts, invalid and infra counts, pack hash and agent configuration.
+
+## License
+
+By contributing you agree that your contributions are licensed under the Apache License 2.0.

indicorderbench-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

indicorderbench-0.1.0/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.5
+Name: indicorderbench
+Version: 0.1.0
+Summary: Catches multilingual voice agents placing the wrong order. A τ-bench style benchmark for voice ordering in Indian languages.
+Project-URL: Homepage, https://github.com/Lingikaushikreddy/indicorderbench
+Project-URL: Repository, https://github.com/Lingikaushikreddy/indicorderbench
+Project-URL: Issues, https://github.com/Lingikaushikreddy/indicorderbench/issues
+Project-URL: Changelog, https://github.com/Lingikaushikreddy/indicorderbench/blob/main/CHANGELOG.md
+Author: Kaushik Reddy Lingireddy
+License: Apache-2.0
+License-File: LICENSE
+Keywords: benchmark,evaluation,hinglish,indic,livekit,tool-calling,voice-agents
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: jinja2>=3.1
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: typer>=0.12
+Provides-Extra: audio
+Requires-Dist: numpy>=1.26; extra == 'audio'
+Requires-Dist: soundfile>=0.12; extra == 'audio'
+Description-Content-Type: text/markdown
+
+<h1 align="center">IndicOrderBench</h1>
+
+<p align="center">
+  <strong>Your voice agent sounds convincing. Did it actually place the correct order?</strong><br/>
+  An open-source benchmark and CI gate that catches voice ordering agents committing the wrong order,<br/>
+  in English and Hinglish, by checking the order they <em>submitted</em>, not the words they said.
+</p>
+
+<p align="center">
+  <a href="https://github.com/Lingikaushikreddy/indicorderbench/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Lingikaushikreddy/indicorderbench/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="LICENSE"><img alt="License: Apache-2.0" src="https://img.shields.io/badge/license-Apache--2.0-blue.svg"></a>
+  <img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-3776AB.svg">
+  <img alt="No API keys needed for the demo" src="https://img.shields.io/badge/demo-no%20API%20keys-success.svg">
+  <img alt="Languages: English, Hinglish" src="https://img.shields.io/badge/languages-English%20%7C%20Hinglish-orange.svg">
+</p>
+
+<p align="center">
+  <a href="#quick-start">Quick start</a> ·
+  <a href="#connect-your-agent">Connect your agent</a> ·
+  <a href="#what-it-measures">What it measures</a> ·
+  <a href="#how-it-compares">How it compares</a> ·
+  <a href="docs/methodology.md">Methodology</a> ·
+  <a href="docs/adapters.md">Adapters</a> ·
+  <a href="docs/scenarios.md">Write scenarios</a>
+</p>
+
+---
+
+## The problem in one call
+
+```
+caller:  Do paneer wrap... actually ek hi karo. Bina pyaaz. Aur ek mango lassi.
+agent:   2 Paneer Wrap add kar diya. Paneer Wrap 1 kar diya. Paneer Wrap No onion kar diya.
+         1 Mango Lassi add kar diya. Aur kuch?
+caller:  Bas itna hi, thanks.
+agent:   Aapka order: 1 Paneer Wrap, No onion; 1 Mango Lassi. Order place ho gaya, shukriya!
+```
+
+The agent *read back* one wrap without onion. Here is what it *committed* to the order system:
+
+| Field                | Expected | Actual     | Result |
+|----------------------|----------|------------|--------|
+| Submitted orders     | 1        | 1          | Pass   |
+| Paneer Wrap quantity | 1        | 2          | Fail   |
+| Paneer Wrap · Onion  | No onion | With onion | Fail   |
+| Mango Lassi quantity | 1        | 1          | Pass   |
+
+Transcript-based testing and LLM judges would have passed this call. IndicOrderBench fails it,
+because it compares the **committed backend state** against the scenario's acceptable end states.
+The table is real output from `iob demo`.
+
+<p align="center">
+  <img src="docs/images/report-demo-scenario.png" alt="IndicOrderBench HTML report: order check table, conversation and tool-call trace for a failed Hinglish correction scenario" width="900">
+</p>
+
+## Why this exists
+
+Voice ordering agents for restaurants, quick commerce and pharmacies are shipping on
+LiveKit, Pipecat, Vapi, Retell and custom stacks. Their hardest failures are not accent or
+latency. They are **mid-sentence corrections** ("two... actually one"), **modifiers**
+("no onion"), **cancellations**, and **duplicate submissions**, especially in code-mixed
+speech like Hinglish, where the number words and the negations do not look like English.
+
+Existing tools judge the conversation. Nobody was checking the cart. IndicOrderBench is a
+small, deterministic, reproducible benchmark for exactly that question, with a CI exit code
+so an agent change that starts placing wrong orders fails the build before it reaches a customer.
+
+## Quick start
+
+```bash
+pip install indicorderbench          # PyPI release coming; until then install from git:
+pip install "git+https://github.com/Lingikaushikreddy/indicorderbench.git"
+
+iob demo --out demo-out              # buggy vs fixed reference agent, no network, no keys
+iob run starter --agent builtin:correct --tag smoke --trials 3
+```
+
+`iob run` writes `results.json` (every transcript, tool call and backend snapshot),
+`junit.xml` (for any CI system) and a single-file `report.html` that works from `file://`.
+Open [docs/demo/fixed/report.html](docs/demo/fixed/report.html) and
+[docs/demo/buggy/report.html](docs/demo/buggy/report.html) to see both sides of the demo.
+
+## Connect your agent
+
+Three ways. Payloads, timeouts and the sandbox backend API are in [docs/adapters.md](docs/adapters.md).
+
+**In-process Python.** Any agent whose logic you can import. The sandbox backend gives you
+JSON-schema tool definitions to hand to your LLM:
+
+```python
+# my_agent.py
+from indicorderbench.adapters.protocol import AgentReply, CallerUtterance, SessionInfo
+from indicorderbench.backend.state import OrderBackend
+
+class MyAgent:
+    def __init__(self, backend: OrderBackend, session: SessionInfo) -> None:
+        self.backend = backend                  # add_item, update_line, submit_order, ...
+        self.tools = OrderBackend.tool_specs()  # function-calling schemas for your LLM
+
+    async def handle(self, u: CallerUtterance) -> AgentReply:
+        text = u.text or transcribe(u.audio_bytes)
+        ...  # your LLM loop; run tool calls with self.backend.call(name, args)
+        return AgentReply(text="Added two paneer wraps. Anything else?")
+
+def make_agent(backend, session):
+    return MyAgent(backend, session)
+```
+
+```bash
+iob run starter --agent python:my_agent:make_agent --trials 3
+```
+
+**HTTP.** Your agent runs anywhere it can reach the sandbox backend: it receives each caller
+turn by HTTP and calls the backend's tools by HTTP. `examples/http_agent_shim.py` is a complete,
+tested example:
+
+```bash
+python examples/http_agent_shim.py --port 8900
+iob run starter --agent http:http://127.0.0.1:8900/iob --tag smoke
+# remote agent: iob run ... --backend-host 0.0.0.0 --backend-port 8765 --backend-url http://<your host>:8765
+```
+
+**CI gate.** Fail the build when ordering accuracy drops below a threshold or regresses
+against the baseline you committed from the last good run:
+
+```bash
+iob run starter --agent http:http://127.0.0.1:8900/iob --tag smoke \
+    --baseline baselines/smoke.json --fail-under 0.9 --max-regression 0.0 --allow-infra 0
+```
+
+Exit codes: `0` ok, `1` infra or usage error, `2` accuracy below threshold or regression.
+A ready-made GitHub Actions workflow is in [examples/ci/github-actions.yml](examples/ci/github-actions.yml).
+
+## What it measures
+
+| Outcome | Meaning | In the pass rate? |
+|---|---|---|
+| `pass` | committed state matches an acceptable end state | yes |
+| `fail` | caller was valid, infrastructure worked, order is wrong or never placed | yes |
+| `simulator_invalid` | the agent asked something the script could not answer; the agent is not blamed | no, reported separately |
+| `infra_error` | adapter raised, timed out, or a clip was missing | no, reported separately |
+
+- **Pass rate and pass^k** (τ-bench's repeated-run metric) by language and by failure
+  category, with Wilson 95% intervals. `--trials 3` tells you whether the agent is right
+  three times in a row, not just once.
+- **Five failure categories:** quantity, modifier, mid-order correction, cancellation,
+  duplicate submission.
+- **Per-turn latency** (p50 / p95) and a timestamped **tool-call trace**, including refused calls.
+- **Reproducibility fields** in every `results.json`: benchmark version, pack content hash,
+  agent spec, trials, modality, timeouts, host and Python version. `iob compare` refuses to
+  compare runs from different packs.
+
+Validity rules, metric definitions and how to publish numbers responsibly:
+[docs/methodology.md](docs/methodology.md).
+
+## How it compares
+
+| | IndicOrderBench | VoiceTest | EVA-Bench | VoiceAgentBench | τ-bench |
+|---|---|---|---|---|---|
+| Scores the committed order state | **yes** | no (LLM judge on transcript) | partly (composite metrics) | no (tool-call correctness on static queries) | yes (text only) |
+| Live, multi-turn agent under test | yes | yes | yes | no | yes |
+| Indian languages | **English + Hinglish, more planned** | no | no | 7 Indic languages | no |
+| Simulator-invalid as a separate outcome | yes | no | yes | n/a | no |
+| Deterministic pass/fail, no judge model | **yes** | no | no | yes | yes |
+| Runs offline with no API keys | **yes** | no | no | no | no |
+| CI exit code and baseline regression gate | **yes** | partly | no | no | no |
+
+Links: [VoiceTest](https://github.com/voicetestdev/voicetest) ·
+[EVA-Bench](https://huggingface.co/blog/ServiceNow-AI/eva) ·
+[VoiceAgentBench](https://arxiv.org/abs/2510.07978) ·
+[τ-bench](https://arxiv.org/abs/2406.12045). IndicOrderBench borrows the end-state
+comparison and pass^k from τ-bench and the simulator-validation idea from EVA-Bench, and
+narrows the task to ordering so the verdict can be deterministic.
+
+## The starter pack
+
+42 scenarios: 21 English (en-IN) and 21 romanised Hinglish (hi-en), four per failure category
+per language plus two `demo`-tagged correction scenarios. Ten are tagged `smoke` for a
+30-second gate. Every scenario lists its acceptable end states and carries clarification rules,
+so an agent that asks "how many?" before acting gets a fair run. A rule-based reference agent
+with five switchable bugs validates the pack: the correct agent passes 42/42, and each bug
+fails exactly its own category.
+
+Audio: `iob synth starter --provider sarvam` generates caller clips with Sarvam bulbul
+(set `SARVAM_API_KEY`); `iob perturb` derives noisy, quiet or telephone-band variants;
+`iob run --modality audio` sends only audio. `--provider silence` makes placeholder clips so
+the whole audio pipeline runs offline (pipeline tests, not evaluations).
+
+Write your own scenarios or a new language pack: [docs/scenarios.md](docs/scenarios.md).
+
+## Honest limits
+
+- The runner is **turn-based**: no barge-in or overlapping speech is measured.
+- The Hinglish scenarios are marked **unreviewed** until native speakers review them
+  (see `packs/starter/REVIEW.md`). Treat Hinglish numbers as provisional until then.
+- No LLM-based reference agent, generative caller or failure explanations in v0.1. The
+  interfaces exist: `OrderBackend.tool_specs()`, `AgentAdapter`, `Transcriber`.
+- No LiveKit or Pipecat native adapter yet; the HTTP turn protocol is the integration point.
+
+## Roadmap
+
+- Native-speaker review of the Hinglish pack, then Telugu and Tamil packs.
+- A LiveKit Agents adapter and a Pipecat adapter.
+- A generative caller for unscripted clarifications.
+- An LLM reference agent, to publish measured ordering accuracy for real STT + LLM + TTS stacks.
+- PyPI release.
+
+If you build or run voice ordering agents and want a category or a language covered, open an
+issue with a real failure you have seen. That is the most useful contribution right now.
+
+## Contributing and development
+
+Language review, scenarios, adapters and report improvements are all welcome and small.
+See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+```bash
+uv sync --all-extras --dev
+uv run pytest -q          # ~300 tests, no network
+uv run iob validate starter
+```
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE).
+
+<sub>Keywords: voice agent testing, voice AI evaluation, conversational AI QA, speech agent benchmark, LLM agent benchmark, tool calling evaluation, function calling accuracy, order accuracy, Hinglish, Hindi, Indian languages, Indic NLP, LiveKit agents, Pipecat, Vapi, Retell, Sarvam, τ-bench, pass^k, regression testing for voice agents, CI for AI agents.</sub>