evalrun 0.4.0__py3-none-any.whl

evalrun-0.4.0.dist-info/METADATA

@@ -0,0 +1,268 @@
+Metadata-Version: 2.4
+Name: evalrun
+Version: 0.4.0
+Summary: Provider-agnostic evaluation and regression-testing toolkit for tool-using AI agents.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: requests>=2.28.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: python-frontmatter>=1.0.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: langfuse>=2.0.0
+Requires-Dist: fastmcp>=0.1.0
+Dynamic: license-file
+
+# evalrun
+
+> **Quality Score $\neq$ Release Decision.** An LLM evaluator can rate an itinerary 95/100 while an independent auditor blocks it for hard financial violations.
+
+`evalrun` is a local-first toolkit for testing AI agents. You bring the agent, model, and API key; `evalrun` runs the checks on your machine, saves the evidence locally, and tells you whether the result should be released.
+
+The toolkit works with hosted APIs and models running on your own computer. It does not host models, provide API credits, provision GPUs, or require a hosted account.
+
+---
+
+## 🌟 Key Capabilities
+
+- **3-Tier Release Gatekeeping**:
+  - **Evaluator Thresholds**: Qualitative dimension scoring (0–100) via LLM judges.
+  - **Independent Auditor Gate**: Hard financial, policy, and math validation (blocks releases on unbudgeted items or currency hallucinations).
+  - **Baseline Regression Gate**: Automatically detects score drops ($\Delta \text{score}$) against stored baselines.
+- **Local-First & Multi-Model**: Compatible with hosted APIs (OpenAI GPT-5.6, NVIDIA NIM, Gemini) and local model servers (vLLM, Ollama, LM Studio) on `http://localhost:8000/v1`.
+- **Standalone HTML Review Reports**: Interactive local HTML report with failure quick-jump bars, auto-opened failing cards, search/filter controls, visual score progress bars, and raw model output inspection.
+- **CI Exit Code Contract**:
+  - `0`: All scenarios passed evaluator, auditor gate passed, and no regression detected.
+  - `1`: Release blocked due to evaluator threshold failure, auditor violation, or baseline score regression.
+  - `2`: Runtime error, missing file, or invalid configuration.
+
+---
+
+## 🚀 Quick Start
+
+### 1. Installation
+
+```bash
+git clone https://github.com/imshivamb/agent-eval-platform.git
+cd agent-eval-platform
+pip install -e .
+```
+
+If your shell says `evalrun: command not found`, activate the virtual environment and install the repository first:
+
+```bash
+source .venv/bin/activate
+python -m pip install -e .
+rehash  # zsh only, refreshes the command cache
+```
+
+### 2. Zero-Cost Offline Demo
+
+Try EvalRun without an API key or network request:
+
+```bash
+evalrun demo
+open results/demo/report.html  # macOS
+```
+
+### 3. Single Scenario Run (Hosted Model)
+
+```bash
+export OPENAI_API_KEY="sk-proj-..."
+
+evalrun run \
+  --scenario evals/scenarios/travel-agent/budget-constrained-itinerary.md \
+  --agent agents.travel:TravelPlanningAgent \
+  --model gpt-5.6-terra \
+  --judge-model gpt-5.6-terra \
+  --output results/run-001
+```
+
+### Choosing a model endpoint
+
+EvalRun does not lock you to one model provider. The `--base-url` value is the address where the model accepts OpenAI-compatible requests:
+
+| Provider | Base URL example |
+| --- | --- |
+| OpenAI-compatible hosted service | `https://api.openai.com/v1` |
+| Gemini | `https://generativelanguage.googleapis.com/v1beta/openai/` |
+| OpenRouter | `https://openrouter.ai/api/v1` |
+| Local vLLM, Ollama, or LM Studio | `http://localhost:8000/v1` |
+
+The judge normally uses the same endpoint and API key as the target model. You only need `--judge-base-url` or `--judge-api-key` when the judge is hosted somewhere different. The endpoint URL is not a credential; it simply tells EvalRun where to send the request.
+
+For example, a Gemini run can be written as:
+
+```bash
+export GEMINI_API_KEY="your-key"
+
+evalrun run \
+  --scenario evals/scenarios/travel-agent/budget-constrained-itinerary.md \
+  --agent agents.travel:TravelPlanningAgent \
+  --model gemini-3.7-flash \
+  --base-url https://generativelanguage.googleapis.com/v1beta/openai/ \
+  --api-key "$GEMINI_API_KEY" \
+  --judge-model gemini-3.7-flash \
+  --output results/gemini-run
+```
+
+### 4. Local Model Server Run (vLLM / Ollama)
+
+```bash
+# Users host their own local OpenAI-compatible server at http://localhost:8000/v1
+evalrun run \
+  --scenario evals/scenarios/travel-agent/budget-constrained-itinerary.md \
+  --agent agents.travel:TravelPlanningAgent \
+  --model qwen2.5-72b-instruct \
+  --base-url http://localhost:8000/v1 \
+  --api-key EMPTY \
+  --judge-model gpt-5.6-terra \
+  --output results/local-run
+```
+
+### 5. Guided Local Web UI
+
+Launch the zero-dependency local web interface:
+
+```bash
+evalrun ui --port 8501
+```
+
+Open `http://127.0.0.1:8501` in your browser to configure endpoints, select scenarios, run evaluations, view pass/fail/block verdicts, and launch interactive HTML reports.
+
+---
+
+## 🐍 Python SDK Usage
+
+Integrate `evalrun` programmatically into Python automation pipelines:
+
+```python
+from framework.sdk import evaluate, compare
+
+# 1. Execute Benchmark Evaluation
+results = evaluate(
+    scenario="evals/scenarios/travel-agent/budget-constrained-itinerary.md",
+    agent="agents.travel:TravelPlanningAgent",
+    model="gpt-5.6-terra",
+    judge_model="gpt-5.6-terra",
+    base_url="http://localhost:8000/v1",
+)
+
+# 2. Compare Candidate Results against Baseline
+report = compare(
+    candidate_results=results,
+    baseline="results/run-001",
+    max_overall_drop=5.0,
+)
+
+if report.release_blocked:
+    print(f"RELEASE BLOCKED: {report.summary['blocked_reason']}")
+```
+
+---
+
+## 📊 Baseline Regression Testing
+
+Compare a candidate prompt, model version, or code change against a prior baseline run:
+
+```bash
+evalrun run \
+  --scenario evals/scenarios/travel-agent/budget-constrained-itinerary.md \
+  --agent agents.travel:TravelPlanningAgent \
+  --model gpt-5.6-terra \
+  --judge-model gpt-5.6-terra \
+  --baseline results/run-001 \
+  --max-regression 5.0 \
+  --max-dimension-regression 10.0 \
+  --output results/candidate-run
+```
+
+### Generated Artifacts
+
+- `manifest.json`: Execution metadata with redacted API credentials.
+- `regression_report.json`: Machine-readable score deltas ($\Delta \text{score}$) and gate decisions.
+- `report.html`: Standalone interactive HTML report for human inspection.
+
+---
+
+## Distribution and deployment
+
+There is no central service to deploy for the current product. The recommended path is:
+
+1. Publish the repository on GitHub.
+2. Add tagged releases and a clear quick-start guide.
+3. Users install it locally with `pip install -e .` from a clone.
+4. Later publish the package to PyPI so users can run `pip install evalrun`.
+5. Host documentation on GitHub Pages if useful; the evaluation engine itself remains local.
+6. Users provide their own hosted-model API keys or run their own local model server.
+
+The local UI is intended for local use, not public internet deployment. A shared hosted deployment would be a separate project requiring authentication, secret management, isolation, and hosted execution.
+
+### Credential promise
+
+EvalRun does not provide, collect, or store model credentials. A key is read by the local process and passed to the selected model endpoint for that run. Keys are never written to `manifest.json`, `report.html`, or `regression_report.json`; reports contain only `[REDACTED]`. Prefer environment variables such as `OPENAI_API_KEY`, `GEMINI_API_KEY`, or `OPENROUTER_API_KEY`. Avoid putting real keys directly in shell commands because your terminal may save command history.
+
+## How a run works
+
+```text
+Choose a scenario
+      ↓
+Run your agent with your chosen model
+      ↓
+Score the result against the criteria
+      ↓
+Run the independent auditor
+      ↓
+Compare with a previous run, if supplied
+      ↓
+Write JSON and HTML evidence
+      ↓
+Return PASS or BLOCK
+```
+
+The command line is useful for repeatable runs, the Python SDK is useful inside scripts and pipelines, and the local browser interface is useful when you prefer a form. All three use the same evaluation engine.
+
+## Add your own evaluation scenario
+
+Scenarios are ordinary Markdown files. Copy [`templates/scenario_template.md`](templates/scenario_template.md), edit the prompt and criteria, and save the file under a folder such as `evals/scenarios/my-domain/my-scenario.md`.
+
+Each scenario defines:
+
+- the request sent to the agent;
+- hard constraints that must not be violated;
+- the expected behavior;
+- the dimensions to score;
+- what counts as a pass or failure.
+
+Run one custom scenario with `--scenario`:
+
+```bash
+evalrun run \
+  --scenario evals/scenarios/my-domain/my-scenario.md \
+  --agent my_agent:MyAgent \
+  --model gemini-3.7-flash \
+  --base-url https://generativelanguage.googleapis.com/v1beta/openai/ \
+  --api-key "$GEMINI_API_KEY" \
+  --judge-model gemini-3.7-flash \
+  --output results/my-scenario
+```
+
+Or place multiple `.md` files in a folder and use `--suite path/to/folder`.
+
+## 🏗️ Architecture & Documentation
+
+For detailed system component diagrams, dual-pass auditor sequence flows, and release gate decision trees, see [`docs/architecture.md`](file:///Users/shivam/Projects/AI/agent-eval-platform/docs/architecture.md).
+
+- Hosted Models Guide: [`docs/quickstart-hosted.md`](file:///Users/shivam/Projects/AI/agent-eval-platform/docs/quickstart-hosted.md)
+- Local Models Guide: [`docs/quickstart-local.md`](file:///Users/shivam/Projects/AI/agent-eval-platform/docs/quickstart-local.md)
+- CLI Specification: [`docs/phase4-local-cli-design.md`](file:///Users/shivam/Projects/AI/agent-eval-platform/docs/phase4-local-cli-design.md)
+- Regression Engine Design: [`docs/phase5-regression-gates-design.md`](file:///Users/shivam/Projects/AI/agent-eval-platform/docs/phase5-regression-gates-design.md)
+
+---
+
+## ⚠️ Limitations & Reproducibility Guidelines
+
+- **Judge Variance**: LLM judge evaluations can exhibit non-zero variance. For baseline regression testing, fix model versions and set deterministic sampling parameters where available.
+- **Local Model Requirements**: Local evaluation throughput depends on server VRAM and concurrency settings. Ensure your local server handles parallel requests cleanly.
+- **Credential Security**: Credentials in `manifest.json` and `regression_report.json` are automatically redacted into `"[REDACTED]"`. Never commit unredacted API keys.

evalrun-0.4.0.dist-info/RECORD

@@ -0,0 +1,100 @@
+agents/__init__.py,sha256=d--Be1pW03iwvln-9Oz6IEN0w_pGHW4jGa8as7s_JIk,248
+agents/base.py,sha256=rZ56Uah1k16iUWVjpwZ4UCQc85cRdNL3lD0-dGE0VMQ,550
+agents/auditor/__init__.py,sha256=BF5VNlhZ5ZZOkv9EZjG_YLEpFSP9S6ExDleauRp0ADw,367
+agents/auditor/budget_auditor.py,sha256=7zknzQQbqzmuBC_qyaECAe-GTvOViWk4dJ5SouKvWiM,3720
+agents/auditor/parser.py,sha256=cBViw59CCOxz1yM4cZm2dfN5S-OVcBP5RCValZCZcLI,5859
+agents/auditor/prompts.py,sha256=ss7LKHdH3Dwbfj-0CTsSDIOEP1lBTdURujxIWLyd-k8,3073
+agents/auditor/schema.py,sha256=lRePzbr8etfOGrZAHFzH9HqTtfeedEWK_O1uBEEdmAE,2260
+agents/reflection/__init__.py,sha256=V0cqZwu-2PgNFQzuTfGm1Kawupwy9yYyHgAC_EU3DmM,85
+agents/reflection/agent.py,sha256=oqjU2Wp57wVQ6mImNIiQe2mJVcJ0lVoZHqlPldesHYk,2479
+agents/reflection/prompts.py,sha256=kEwlBOfdZLSTFmU8WgfzvwanRzwyvACHWBiWcRnZbMI,1439
+agents/research/__init__.py,sha256=XjSpY9_A7Qr6Q0W947xp1QPv1rc8Um_57OwB0rafpP4,131
+agents/research/agent.py,sha256=bpyCHW0kkZ4BWVS2K9BsgKnq0UXc5LVOHebr7CVjwTg,1295
+agents/research/planner.py,sha256=-IeYxGY1sqgGrswvyMhVuLpxoXKZmBVMBOnxyGVMVQ8,1889
+agents/research/prompts.py,sha256=xPJgsAccHS_dYOSu3Yjjk0LXgEoigBXRtE5lyRoihAM,851
+agents/support/__init__.py,sha256=XIcVllA_AqwZ5N9lH4Wpnk0AiFXcTacEDg8HSFLLmr8,109
+agents/support/triage_agent.py,sha256=I9In6PYNBZyUu0pcKvBIxatmVHTNUwkGBx-LwFqh6Ec,1531
+agents/travel/__init__.py,sha256=RI8awH0GtmEVd24GO5uidK0VvXu5azdQ-a8T4AEHwx8,254
+agents/travel/agent.py,sha256=C6nvDMUq92BMlAnweQC-1aBmhRusV74vDl8-C87LDtQ,17612
+agents/travel/prompts.py,sha256=y2WK0ouC_eO3_dYa6haXjYPzn9Qoxysgz230YVvj0XQ,1834
+agents/travel/session.py,sha256=Zha_5cFTmtW3F05mnnJRTQCAC65sMwZ3VHsCVUmcNko,3726
+cli/__init__.py,sha256=nyDigv65BJqXf5iizLyHca8b6drvFXDEnxo63GKbbas,170
+cli/demo.py,sha256=kkj-nQJ9DUgccco02hauoCZXq4jUd_cN-1oevzEo0To,2192
+cli/formatter.py,sha256=ljfJr9pnXwBVdgU1E_-TlzWvJ2MjNAfIXJME6f6AsfE,3935
+cli/html_reporter.py,sha256=GwzLiTUlNFqR0VguAbrIoJKW0ECDZUM-vbDXuCXksP4,25086
+cli/main.py,sha256=rpyXbtce6157_PHBj7B5AMoY_1iB5T7ShVOQ2a8Dj-g,15786
+cli/progress.py,sha256=XPvmIiMPCGRxcgQEmu6Vng3P1PzyIqw7WfFfKuixBps,1284
+cli/resolver.py,sha256=O4dsC0nb-cP4GCJzK2qo-jk43U7y9qeoxwZ1oMgjdOI,3754
+evalrun-0.4.0.dist-info/licenses/LICENSE,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+framework/__init__.py,sha256=elFon9ugWqcdvdDw4wONfWOtfN_TkT7SCxdpFTK6Ing,1505
+framework/exceptions.py,sha256=Y6A2hjD5u2_OKF2KIfTLTVtUJ0gGOgcRcVt4-QZz7VM,1026
+framework/models.py,sha256=zd_l8AJxpPxwLEYqOpPr1iBEDhXVd3qUtcPl3kW_iDQ,1902
+framework/parser.py,sha256=AG-htozoexjOwwkYyzpBuQR3rAW99rYn5_IOGFlhpAg,1563
+framework/sdk.py,sha256=H30QbGhIbZDPc38vI3HGT1H2lwFRQYX2S2Odcw--heU,5684
+framework/utils.py,sha256=YxwLPXtzAzArb5-0F1NF5Hxm6WzepAZfilBcPG90Y34,1335
+framework/core/__init__.py,sha256=FMRWwu22MB2S0nR1oX_8RRLjQTmA-u2Ttg43GlrMYoU,485
+framework/core/adapters.py,sha256=gYGDMwE-LbWJIrvLFXTilq_CbhtSPSGgLtDzYNW-gc8,4764
+framework/core/contracts.py,sha256=KjDCphztYrZlAq7o7YLRGRveJ9KumbrwKCNhhFsSKYY,2903
+framework/core/suite.py,sha256=R9EMoyqsMNnKcBFj3ZXtp7qe-aeYKs4xWbkJ4soCb7c,1623
+framework/evaluation/__init__.py,sha256=KjkPL80haNd7hUFBFYMBkB31V3F0uu864nZYS5leqc8,801
+framework/evaluation/base.py,sha256=zRNvQ8rRrWVX5VfgDPMbRdeTAqXRHwW5p-29TtCg33s,867
+framework/evaluation/dimensions.py,sha256=jdfgCKXicWsNg0Y-Tidcrnbm7UckdjKOXsfTi4chat0,250
+framework/evaluation/engine.py,sha256=6rPNUlBsUBeRso62IFtN3Refyl334hqhlAiZGYCRNjg,3893
+framework/evaluation/runner.py,sha256=uGxWuyGnHcx2IjUNgiD-BbF06xFHOLnVKIgcrCs09QY,16036
+framework/evaluation/testing.py,sha256=vmSpNrQMhXnd2M-l0JjXR5SHBQEfJKhaiya6QIScrCI,907
+framework/evaluation/evaluators/__init__.py,sha256=X80klkpKl5p1wDTSwuytXwTDqcxBjDM0sZhHKVSVYzU,479
+framework/evaluation/evaluators/adaptability.py,sha256=9oNGE_qEwbZYImibyUKzUGPYNCVdlRVrno0qsMERx0U,898
+framework/evaluation/evaluators/base_llm.py,sha256=EK8bSho3LvTYbB8tbsmIMIoGq1gS9p5X7BdZCHn0REQ,3081
+framework/evaluation/evaluators/constraint.py,sha256=pvkHVZmb85H0QaB4xL0mho3ov1tcC0kmT7hfg0JUoK4,892
+framework/evaluation/evaluators/information_accuracy.py,sha256=Lv0Bb0KXNMP2_yl_Kh7s7AVq3aNLBn7OzIJX9Ph0Vn0,1562
+framework/evaluation/evaluators/personalization.py,sha256=ISHcUN13S5pxT2JU0vmSgc0XdnGg1rsgDU0AtEqcmV4,931
+framework/evaluation/evaluators/planning.py,sha256=9Kx0pLvrmaj05wVxlq_MNAKwHWRIBAaM3DwvIAN55aU,897
+framework/evaluation/evaluators/support.py,sha256=dqT0khLnhbmlDoooMcufZbhATKThTJYc-xEFWbrY-Ug,3118
+framework/evaluation/prompts/__init__.py,sha256=aa4gxNX77ds3UwDeExifMO2857n5dBbkXUq7NrssSJg,549
+framework/evaluation/prompts/adaptability.py,sha256=8GC9uDd0lgd3gJgaIhIlNa3aZKyQd30zEq8iP1YCsQU,3239
+framework/evaluation/prompts/base.py,sha256=YwGagUdKVv1fKXmlR9GJVla_6ocYoDQIVtoly1hMH9U,1662
+framework/evaluation/prompts/constraint.py,sha256=Yf1THuImU-YKpkAQQHgZkIDmAC7ua_yLHD2j-Wh5Cwk,1254
+framework/evaluation/prompts/information_accuracy.py,sha256=kjAHUKLrOSnnnD4HJ3PvoPKEdxQEVrF8DhJBSwauzxA,4080
+framework/evaluation/prompts/personalization.py,sha256=_gyKMs4kS3pBjG8DPHD2lpukw5GWw0FSo6eb0OEXeOM,3377
+framework/evaluation/prompts/planning.py,sha256=KnSRCQjtVBWvI7bqrEr3gg70-opHQcXipnISRZSEIMA,3391
+framework/llms/__init__.py,sha256=9pgFMd1MO97Cjbo-RvKdOMl859NV3MNGThfJz0aJR3c,368
+framework/llms/base.py,sha256=SQnpgoK8g5v7k3dVSJ3caZnShhWHtObN5Zp07tuPVts,870
+framework/llms/factory.py,sha256=Nq1TKKcCkvK4mWEe2J0B5L8Z7IJPVy0FDa_0PdSvWR4,1153
+framework/llms/gemini.py,sha256=tza5D4h4vx4wMAzicMLDnbWb6z_ZOOPg1AAN3abN3Kc,3109
+framework/llms/mock.py,sha256=Yp3M4bg9XOu4XcI9I8pVqWqdSNUPscMoMP_1ntewpnw,921
+framework/llms/openai.py,sha256=BGtnEmXuYBJmbA0dHV_W96tI-y7tWB-oJYSMecKDbEI,3475
+framework/llms/openai_compatible.py,sha256=zMjBKGuTgT6vNhv9wvz1LlbjqwYkfJkzCLRcQCgDxco,5333
+framework/mcp/__init__.py,sha256=31aISAqLDwAASTtp7QRsc4tbiVchRYu7GO5n2r8wRYQ,551
+framework/mcp/client.py,sha256=dKVGRoi9iXXZkUzFic6SrQyrO_EuKwPuq8DT6sQlGIU,2576
+framework/mcp/constraints.py,sha256=QyWUKNVN1OYp8dNlQKvwLRN1-rvGU6pxjAejohbBQ3U,4402
+framework/mcp/revision_summary.py,sha256=X9lJA3C_o6Uq03ZuNxgWfEYtHnJ0TfIjL1GNCov58dE,4600
+framework/mcp/server.py,sha256=7sWwGc49su1o4-PchA1r3Z6rKGL5sIJE3hHPYyxHd9U,1410
+framework/memory/__init__.py,sha256=DRmERPlTFg5GvPnOSc0XAd0OCGcJRRSqfIFHzuTOG-o,105
+framework/memory/base.py,sha256=G5c_dzV6nvGxvTQ3Rf3fHWkvCKvLKyXZq2ZWd-yggHs,474
+framework/parsers/__init__.py,sha256=ElWVaNALXXgv28dBL4L4OirLdvX2xQ73qDfCWob8Q-o,442
+framework/parsers/frontmatter.py,sha256=09wJenN6vcLnUqgsUJ-yjApVJ9o2qTLCz3eBvcRs0fE,959
+framework/parsers/mapper.py,sha256=keiKEfEaUxTun9n_pIIRljz7mDPSMRPkLbwKjaQICoE,2376
+framework/parsers/markdown.py,sha256=PkbAY3rvOF0a0-IMtRZEgdrKdWoqzVCw4XiKI1Td1_k,4316
+framework/parsers/transformers.py,sha256=PPFjv8C8S7q6iFiwuoJ-kZ8eiegJarWBD6Wc-MwOq90,3546
+framework/profiles/__init__.py,sha256=_uo6Pckt8gjSOVCiEQ8TxdALg8LMdwlVNXJwXah0rZI,842
+framework/profiles/registry.py,sha256=QG3w2-KmWRWdaNmXvpD8DD304AuIKQKF3aJM2diWQIs,3679
+framework/profiles/support.py,sha256=Ay1NnR-s_ATRkWzFvk-0pX4iqi0gQK3P8cLNlyV6BHI,853
+framework/profiles/travel.py,sha256=IVz3N8h88jCKNrH1Zjph3T9hNshBLtjJNSjSrMblia4,2232
+framework/regression/__init__.py,sha256=e4eEOZg6nHZ3r0oV3qQjvJqQVoKa7xY1AcbWRiZRZOg,392
+framework/regression/comparator.py,sha256=ozSUNBQAZbbh-3KI0SaIdDJJFyjwi0ia5NaigQMSOqI,10646
+framework/regression/loader.py,sha256=aTOJEtbze-WfYttugQGXuv2VJQBmnsJ0ohObvv2Qz4I,6111
+framework/verification/__init__.py,sha256=6Y9k7OU13lx0p3MmNXQ4fl3kJ08m8sDUcxojW12I1-k,458
+framework/verification/base.py,sha256=hdNWPYaPZysTrL5poS0akyAxLtnIGtcx5WRfCoUnZsc,861
+framework/verification/extractor.py,sha256=MMKbK8WpnO7U-z2KeUDfGmyRzxxd8A8bTU49ODJZLg8,3548
+framework/verification/local.py,sha256=7JJrSIdkkO-k3n0ffbJn1VXkt4o5m4WeL1MOTwllVtQ,4691
+framework/verification/models.py,sha256=8eRL6UfX4rYdzX-WRPZVOmpazpF4564lbOxQUGeZanw,3556
+framework/verification/pipeline.py,sha256=XQOyO7JYQ_kUswsS9Y51SdUj5w28eTcN72SROBwpFF4,1248
+framework/verification/prompts.py,sha256=HV-BqZ-donfYbBF5I6BJZd2gjQg37Q-NoMptjLbu3f4,1440
+framework/verification/utils.py,sha256=LiYtT5Ft-wE3b9Ge0w0YO8AuZ9IB1LZqoV77htQEpMY,1443
+ui/__init__.py,sha256=ngbjUI_X8nA_lsdEx0VzvyJSG5SlRxdhMSVKzNkxrBc,47
+ui/server.py,sha256=v0ljSOo6BnXPKPlLa24iVgx5c_K7LEEHkTe2ehg4QwU,10954
+evalrun-0.4.0.dist-info/METADATA,sha256=69C9cajBRn5wQcWtkhyGH2MOcIZ27-V9_CrEkJtMHpY,10361
+evalrun-0.4.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+evalrun-0.4.0.dist-info/entry_points.txt,sha256=UIXWZUtcjtFuFD9v54VS6meRdbHcEKhINU8nZ3UNU1c,42
+evalrun-0.4.0.dist-info/top_level.txt,sha256=vsUSUEXYBjHS6azZmZe9-0ZcSChVK5i17tzCi0eS2BM,24
+evalrun-0.4.0.dist-info/RECORD,,

evalrun-0.4.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

evalrun-0.4.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+evalrun = cli.main:main

evalrun-0.4.0.dist-info/top_level.txt

@@ -0,0 +1,4 @@
+agents
+cli
+framework
+ui

framework/__init__.py

@@ -0,0 +1,70 @@
+"""Generic evaluation framework for AI agents."""
+
+from .models import (
+    Benchmark,
+    AgentOutput,
+    DimensionScore,
+    EvaluationResult,
+    EvaluationProfile,
+    ParsedSections,
+    ParsedBenchmark,
+)
+from .parser import parse_benchmark, parse_benchmark_text
+from .evaluation import (
+    BaseEvaluator,
+    BaseLLMEvaluator,
+    EvaluationEngine,
+    BenchmarkRunner,
+    ConstraintEvaluator,
+    PlanningQualityEvaluator,
+    PersonalizationEvaluator,
+    AdaptabilityEvaluator,
+    InformationAccuracyEvaluator,
+    DummyEvaluator,
+    CONSTRAINT_SATISFACTION,
+    PLANNING_QUALITY,
+    INFORMATION_ACCURACY,
+    PERSONALIZATION,
+    ADAPTABILITY,
+)
+from .llms import (
+    BaseLLM,
+    Message,
+    MockLLM,
+    OpenAILLM,
+    GeminiLLM,
+    create_llm,
+)
+from .profiles import (
+    TRAVEL_PROFILE,
+    TRAVEL_ROUTE_OPTIMIZATION_PROFILE,
+    TRAVEL_REMOTE_WORKER_TIMEZONES_PROFILE,
+    TRAVEL_MID_TRIP_REPLANNING_PROFILE,
+    TRAVEL_INFORMATION_GATHERING_UNCERTAINTY_PROFILE,
+    PROFILE_REGISTRY,
+)
+from .verification import (
+    ClaimType,
+    VerificationStatus,
+    Claim,
+    Evidence,
+    VerificationReport,
+    BaseVerifier,
+    LocalKnowledgeBaseVerifier,
+    ClaimExtractor,
+    VerificationPipeline,
+)
+from .memory import BaseSessionMemory
+from .sdk import evaluate, compare
+
+__all__ = [
+    "evaluate",
+    "compare",
+    "Benchmark",
+    "AgentOutput",
+    "DimensionScore",
+    "EvaluationResult",
+    "EvaluationProfile",
+    "BenchmarkRunner",
+    "BaseSessionMemory",
+]

framework/core/__init__.py

@@ -0,0 +1,17 @@
+"""Generic evaluation core module exports."""
+
+from framework.core.contracts import Scenario, RunTrace, to_scenario, to_benchmark
+from framework.core.adapters import AgentAdapter, PythonAgentAdapter, HttpAgentAdapter, CliAgentAdapter
+from framework.core.suite import EvaluationSuite
+
+__all__ = [
+    "Scenario",
+    "RunTrace",
+    "to_scenario",
+    "to_benchmark",
+    "AgentAdapter",
+    "PythonAgentAdapter",
+    "HttpAgentAdapter",
+    "CliAgentAdapter",
+    "EvaluationSuite",
+]

framework/core/adapters.py

@@ -0,0 +1,118 @@
+"""Agent adapters for wrapping internal Python agents, HTTP endpoints, or CLI binaries."""
+
+import json
+import subprocess
+import requests
+from abc import ABC, abstractmethod
+from typing import Any, Dict, Optional
+
+from framework.core.contracts import Scenario
+from framework.models import AgentOutput
+
+
+class AgentAdapter(ABC):
+    """Abstract adapter interface decoupling agent evaluation from specific implementations."""
+
+    @abstractmethod
+    def run(self, scenario: Any, **kwargs) -> AgentOutput:
+        """Executes the wrapped agent on a scenario or prompt string and returns standardized AgentOutput."""
+        pass
+
+
+class PythonAgentAdapter(AgentAdapter):
+    """Adapter wrapping any in-process Python agent object (e.g. TravelPlanningAgent)."""
+
+    def __init__(self, agent: Any, agent_id: Optional[str] = None):
+        self.agent = agent
+        self.agent_id = agent_id or getattr(agent, "agent_name", agent.__class__.__name__)
+
+    def run(self, scenario: Any, **kwargs) -> AgentOutput:
+        """Invokes the underlying Python agent instance."""
+        prompt_text = scenario if isinstance(scenario, str) else getattr(scenario, "prompt", str(scenario))
+        if hasattr(self.agent, "run"):
+            result = self.agent.run(prompt_text, **kwargs)
+            if isinstance(result, AgentOutput):
+                return result
+            elif isinstance(result, str):
+                return AgentOutput(content=result, metadata={"adapter": "PythonAgentAdapter"})
+            else:
+                return AgentOutput(content=str(result), metadata={"adapter": "PythonAgentAdapter"})
+        else:
+            raise AttributeError(f"Wrapped agent object {self.agent} has no 'run' method.")
+
+
+class HttpAgentAdapter(AgentAdapter):
+    """Adapter wrapping an external HTTP/REST API agent endpoint."""
+
+    def __init__(self, endpoint_url: str, headers: Optional[Dict[str, str]] = None, timeout: float = 120.0):
+        self.endpoint_url = endpoint_url
+        self.headers = headers or {"Content-Type": "application/json"}
+        self.timeout = timeout
+
+    def run(self, scenario: Any, **kwargs) -> AgentOutput:
+        if isinstance(scenario, str):
+            prompt_text = scenario
+            scenario_id = "scenario-001"
+            constraints = {}
+        else:
+            prompt_text = getattr(scenario, "prompt", str(scenario))
+            scenario_id = getattr(scenario, "id", "scenario-001")
+            constraints = getattr(scenario, "constraints", {})
+
+        payload = {
+            "scenario_id": scenario_id,
+            "prompt": prompt_text,
+            "constraints": constraints,
+        }
+        try:
+            resp = requests.post(self.endpoint_url, json=payload, headers=self.headers, timeout=self.timeout)
+            resp.raise_for_status()
+        except requests.exceptions.Timeout as e:
+            raise TimeoutError(f"HTTP Agent endpoint timed out after {self.timeout}s: {e}") from e
+
+        data = resp.json()
+        content = data.get("content") or data.get("output") or resp.text
+        metadata = data.get("metadata", {})
+        metadata["adapter"] = "HttpAgentAdapter"
+        return AgentOutput(content=content, metadata=metadata)
+
+
+class CliAgentAdapter(AgentAdapter):
+    """Adapter wrapping a CLI command-line binary agent."""
+
+    def __init__(self, command: Any, timeout: float = 120.0):
+        if isinstance(command, str):
+            import shlex
+            self.command_args = shlex.split(command)
+        else:
+            self.command_args = command
+        self.timeout = timeout
+
+    def run(self, scenario: Any, **kwargs) -> AgentOutput:
+        prompt_text = scenario if isinstance(scenario, str) else getattr(scenario, "prompt", str(scenario))
+        try:
+            proc = subprocess.run(
+                self.command_args,
+                input=prompt_text,
+                text=True,
+                capture_output=True,
+                timeout=self.timeout,
+            )
+        except subprocess.TimeoutExpired as e:
+            raise TimeoutError(f"CLI Agent command timed out after {self.timeout}s: {e}") from e
+
+        if proc.returncode != 0:
+            raise RuntimeError(f"CLI Agent command failed (code {proc.returncode}): {proc.stderr}")
+
+        stdout_text = proc.stdout.strip()
+        try:
+            parsed = json.loads(stdout_text)
+            if isinstance(parsed, dict) and "content" in parsed:
+                content = parsed["content"]
+                metadata = parsed.get("metadata", {})
+                metadata["adapter"] = "CliAgentAdapter"
+                return AgentOutput(content=content, metadata=metadata)
+        except Exception:
+            pass
+
+        return AgentOutput(content=stdout_text, metadata={"adapter": "CliAgentAdapter"})