multivon-eval 0.1.0__tar.gz

multivon_eval-0.1.0/LICENSE

@@ -0,0 +1,185 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship made available under
+      the License, as indicated by a copyright notice that is included in
+      or attached to the work (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and the Derivative Works thereof.
+
+      "Contribution" shall mean, as submitted to the Licensor for inclusion
+      in the Work by the copyright owner or by an individual or Legal Entity
+      authorized to submit on behalf of the copyright owner. For the purposes
+      of this definition, "submitted" means any form of electronic, verbal,
+      or written communication sent to the Licensor or its representatives,
+      including but not limited to communication on electronic mailing lists,
+      source code control systems, and issue tracking systems that are managed
+      by, or on behalf of, the Licensor for the purpose of discussing and
+      improving the Work, but excluding communication that is conspicuously
+      marked or designated in writing by the copyright owner as "Not a
+      Contribution."
+
+      "Contributor" shall mean Licensor and any Legal Entity on behalf of
+      whom a Contribution has been received by the Licensor and included
+      within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by the combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a cross-claim
+      or counterclaim in a lawsuit) alleging that the Work or any
+      Contribution embodied within the Work constitutes direct or contributory
+      patent infringement, then any patent licenses granted to You under
+      this License for that Work shall terminate as of the date such
+      litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, You must include a readable copy of the
+          attribution notices contained within such NOTICE file, in
+          at least one of the following places: within a NOTICE text
+          file distributed as part of the Derivative Works; within
+          the Source form or documentation, if provided along with the
+          Derivative Works; or, within a display generated by the
+          Derivative Works, if and wherever such third-party notices
+          normally appear. The contents of the NOTICE file are for
+          informational purposes only and do not modify the License.
+          You may add Your own attribution notices within Derivative
+          Works that You distribute, alongside or in addition to the
+          NOTICE text from the Work, provided that such additional
+          attribution notices cannot be construed as modifying the License.
+
+      You may add Your own license statement for Your modifications and
+      may provide additional grant of rights to use, copy, modify, merge,
+      publish, distribute, sublicense, and/or sell copies of the
+      Derivative Works, and to permit persons to whom the Derivative Works
+      are furnished to do so, subject to the following conditions described
+      in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or reproducing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or exemplary damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or all other
+      commercial damages or losses), even if such Contributor has been
+      advised of the possibility of such damages.
+
+   9. Accepting Warranty or Liability. While redistributing the Work or
+      Derivative Works thereof, You may choose to offer, and charge a fee
+      for, acceptance of support, warranty, indemnity, or other liability
+      obligations and/or rights consistent with this License. However, in
+      accepting such obligations, You may offer such obligations only on
+      Your own behalf and on Your sole responsibility, not on behalf of
+      any other Contributor, and only if You agree to indemnify, defend,
+      and hold each Contributor harmless for any liability incurred by,
+      or claims asserted against, such Contributor by reason of your
+      accepting any warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 Multivon (https://multivon.ai)
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

multivon_eval-0.1.0/PKG-INFO

@@ -0,0 +1,387 @@
+Metadata-Version: 2.4
+Name: multivon-eval
+Version: 0.1.0
+Summary: AI evaluation for teams that ship models to production
+License: Apache-2.0
+Project-URL: Homepage, https://multivon.ai
+Project-URL: Repository, https://github.com/multivon-ai/multivon-eval
+Project-URL: Documentation, https://evaldocs.multivon.ai
+Project-URL: Bug Tracker, https://github.com/multivon-ai/multivon-eval/issues
+Keywords: llm,evaluation,ai,evals,rag,agents,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anthropic>=0.40.0
+Requires-Dist: openai>=1.50.0
+Requires-Dist: jsonschema>=4.20.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: ruff>=0.3.0; extra == "dev"
+Requires-Dist: build>=1.0.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Dynamic: license-file
+
+# multivon-eval
+
+![Python](https://img.shields.io/badge/python-3.10+-blue.svg)
+![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)
+![PyPI](https://img.shields.io/pypi/v/multivon-eval.svg)
+
+**[Documentation](https://evaldocs.multivon.ai)** · [Website](https://multivon.ai) · [PyPI](https://pypi.org/project/multivon-eval)
+
+**AI evaluation for teams that ship models to production.**
+
+Run structured evals over your AI outputs — from simple string checks to LLM-as-judge scoring to agent trace validation — with a clean Python API, beautiful terminal reports, and CI/CD integration out of the box.
+
+---
+
+```python
+from multivon_eval import EvalSuite, EvalCase, Relevance, Faithfulness, NotEmpty
+
+suite = EvalSuite("Support Bot Eval")
+suite.add_cases([
+    EvalCase(
+        input="How do I reset my password?",
+        context="Users can reset their password by clicking 'Forgot Password' on the login page.",
+    ),
+])
+suite.add_evaluators(NotEmpty(), Relevance(), Faithfulness())
+report = suite.run(my_model_fn)
+```
+
+```
+─────────────────────── Support Bot Eval ───────────────────────
+  #  Input                      Output                   Score  Status    Latency
+  1  How do I reset my pas...   Click 'Forgot Passwor…   0.92   PASS      843ms
+
+                           By Evaluator
+  Evaluator       Avg Score    Pass Rate
+  not_empty          1.00        100%
+  relevance          0.88         88%
+  faithfulness       0.87         87%
+
+╭─────────────────────── Summary ───────────────────────╮
+│ Total: 1   Passed: 1   Failed: 0   Pass Rate: 100%   │
+╰────────────────────────────────────────────────────────╯
+```
+
+---
+
+## Why multivon-eval
+
+Every team building AI products hits the same problem: **how do you know if your model is getting better or worse?**
+
+Existing tools have real limitations:
+- **DeepEval** — powerful but LLM-as-judge for everything is expensive, slow, and hard to audit
+- **RAGAS** — excellent, but RAG-only
+- **Promptfoo** — YAML-driven, feels rigid for Python teams
+
+`multivon-eval` is different in three ways:
+
+**QAG scoring** — Instead of asking a judge "rate this 1-10", we generate yes/no questions about the output and score by the fraction answered correctly. Binary questions are easier for LLMs to get right, fully auditable, and cheaper.
+
+**Agent-native** — Built-in evaluators for tool call accuracy, plan quality, step faithfulness, and task completion. Covers agent traces from any framework (LangChain, LlamaIndex, custom).
+
+**Four tiers** — Deterministic (free, instant), LLM-judge (QAG), agent-trace, and conversation evaluators. Mix and match; pay for LLM calls only where it matters.
+
+---
+
+## Install
+
+```bash
+pip install multivon-eval
+```
+
+```bash
+cp .env.example .env
+# Add ANTHROPIC_API_KEY and/or OPENAI_API_KEY
+```
+
+---
+
+## Core concepts
+
+### `EvalCase` — A test case
+
+```python
+from multivon_eval import EvalCase
+
+case = EvalCase(
+    input="What caused the 2008 financial crisis?",          # required
+    expected_output="Subprime mortgage collapse...",          # for ExactMatch, Contains
+    context="The 2008 crisis was triggered by...",           # for Faithfulness, Hallucination
+    tags=["finance", "history"],                             # for filtering reports
+    metadata={"source": "test_set_v2", "difficulty": "hard"},
+)
+```
+
+For agent evals:
+
+```python
+from multivon_eval import EvalCase, AgentStep, ToolCall
+
+case = EvalCase(
+    input="search for recent AI papers and summarize",
+    agent_trace=[
+        AgentStep(tool_calls=[ToolCall(name="search", arguments={"query": "AI papers 2025"})]),
+        AgentStep(tool_calls=[ToolCall(name="summarize")]),
+    ],
+    expected_tool_calls=["search", "summarize"],
+)
+```
+
+### Evaluators
+
+Four tiers — pick what fits your use case.
+
+#### Tier 1: Deterministic (free, instant, no LLM needed)
+
+| Evaluator | What it checks |
+|-----------|---------------|
+| `NotEmpty` | Response is non-empty |
+| `ExactMatch` | Response matches `expected_output` exactly |
+| `Contains(substrings)` | Response contains all required strings |
+| `RegexMatch(pattern)` | Response matches a regex pattern |
+| `JSONSchemaEval(schema)` | Response is valid JSON matching a schema |
+| `WordCount(min, max)` | Word count within range |
+| `Latency(max_ms)` | Response time under limit |
+| `BLEU(n)` | BLEU-n score vs expected output |
+| `ROUGE` | ROUGE-L F1 vs expected output |
+| `StartsWith(prefix)` | Response starts with prefix |
+
+#### Tier 2: LLM-as-judge (QAG scoring)
+
+| Evaluator | What it measures | Requires `context` |
+|-----------|-----------------|-------------------|
+| `Faithfulness` | Response is grounded in context | Yes |
+| `Hallucination` | Response doesn't invent facts | Yes |
+| `Relevance` | Response addresses the question | No |
+| `Coherence` | Response is clear and well-structured | No |
+| `Toxicity` | Response is safe and non-harmful | No |
+| `Bias` | Response is free of demographic bias | No |
+| `Summarization` | Summary captures key points faithfully | Yes |
+| `AnswerAccuracy` | Factual correctness vs expected | No |
+| `ContextPrecision` | Relevant context retrieved | Yes |
+| `ContextRecall` | All needed context retrieved | Yes |
+| `CustomRubric` | Your own yes/no criteria | Optional |
+| `GEval` | Holistic numeric quality score | Optional |
+
+```python
+from multivon_eval import Faithfulness, CustomRubric
+
+CustomRubric(
+    name="support_quality",
+    criteria=[
+        ("Does the response acknowledge the customer's problem?", True),
+        ("Does the response provide a concrete next step?", True),
+        ("Does the response use apologetic or defensive language?", False),
+    ],
+    threshold=0.8,
+)
+```
+
+#### Tier 3: Agent trace evaluators
+
+| Evaluator | What it checks |
+|-----------|---------------|
+| `ToolCallAccuracy` | Expected tools called (ordered or unordered) |
+| `ToolArgumentAccuracy` | Quality of tool arguments |
+| `PlanQuality` | Plan logic, completeness, efficiency |
+| `TaskCompletion` | Final output satisfies the task goal |
+| `StepFaithfulness` | Each step follows logically from prior |
+
+```python
+from multivon_eval import ToolCallAccuracy
+
+ToolCallAccuracy(require_order=True)  # strict ordering
+ToolCallAccuracy(require_order=False) # set match (default)
+```
+
+#### Tier 4: Conversation evaluators
+
+| Evaluator | What it checks |
+|-----------|---------------|
+| `ConversationRelevance` | Each response stays on topic |
+| `KnowledgeRetention` | Model remembers earlier context |
+| `ConversationCompleteness` | Conversation resolves the original goal |
+| `TurnConsistency` | No contradictions across turns |
+
+```python
+from multivon_eval import EvalCase
+
+case = EvalCase(
+    input="Is this product available in blue?",
+    conversation=[
+        {"role": "user", "content": "I need a new laptop"},
+        {"role": "assistant", "content": "I can help you find a laptop. What's your budget?"},
+        {"role": "user", "content": "Around $1000"},
+        {"role": "assistant", "content": "Here are some options around $1000..."},
+    ],
+)
+```
+
+### `EvalSuite` — The runner
+
+```python
+from multivon_eval import EvalSuite
+
+suite = EvalSuite("My Eval", model_id="gpt-4o")
+suite.add_cases(cases)
+suite.add_evaluators(NotEmpty(), Relevance(), Faithfulness(threshold=0.7))
+
+# Serial
+report = suite.run(model_fn, verbose=True, fail_threshold=0.8)
+
+# Parallel (thread-based)
+report = suite.run(model_fn, workers=8)
+
+# Async
+import asyncio
+report = asyncio.run(suite.run_async(model_fn, concurrency=10))
+```
+
+### Loading datasets
+
+```python
+from multivon_eval import load
+
+cases = load("tests/dataset.jsonl")  # auto-detects format
+cases = load("tests/dataset.csv")
+```
+
+**JSONL format:**
+```json
+{"input": "What is the capital of France?", "expected_output": "Paris", "tags": ["factual"]}
+{"input": "Summarize this document.", "context": "Document text here...", "tags": ["summarization"]}
+```
+
+**CSV format:**
+```
+input,expected_output,context,tags
+What is 2+2?,4,,math
+Summarize this.,,Long text here,summarization
+```
+
+### Exporting results
+
+```python
+report.save_json("results.json")
+report.save_csv("results.csv")
+```
+
+---
+
+## CLI
+
+```bash
+multivon-eval run eval.py
+multivon-eval report results.json
+```
+
+---
+
+## CI/CD integration
+
+```python
+# eval.py
+report = suite.run(model_fn, fail_threshold=0.85)  # exits 1 if < 85% pass
+```
+
+```yaml
+# .github/workflows/eval.yml
+- name: Run evals
+  run: python eval.py
+  env:
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+---
+
+## Architecture
+
+```
+EvalSuite.run(model_fn)
+     │
+     ├── for each EvalCase:
+     │     ├── call model_fn(case.input) → output
+     │     └── for each Evaluator:
+     │           ├── Deterministic → no LLM, instant
+     │           ├── LLM Judge → QAG yes/no questions → fraction score
+     │           ├── Agent → trace inspection + LLM judge
+     │           └── Conversation → multi-turn analysis
+     │
+     └── EvalReport
+           ├── CaseResult × N
+           ├── per-evaluator scores
+           ├── terminal report (rich)
+           └── export → JSON / CSV
+```
+
+**Judge model:** Configured via `JUDGE_MODEL` and `JUDGE_PROVIDER` env vars. Defaults to `claude-sonnet-4-6`. The model under test and the judge model can be different providers.
+
+---
+
+## Examples
+
+| File | What it shows |
+|------|--------------|
+| [`basic_eval.py`](examples/basic_eval.py) | Deterministic evaluators, no LLM judge |
+| [`rag_eval.py`](examples/rag_eval.py) | Faithfulness + hallucination for RAG systems |
+| [`ci_eval.py`](examples/ci_eval.py) | CI/CD integration with pass threshold |
+
+---
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+---
+
+## Roadmap
+
+- [x] Deterministic evaluators (BLEU, ROUGE, regex, JSON schema, latency)
+- [x] LLM-as-judge with QAG scoring
+- [x] Agent trace evaluators (tool call accuracy, plan quality)
+- [x] Conversation evaluators
+- [x] Parallel + async runners
+- [x] CLI (`multivon-eval run`, `multivon-eval report`)
+- [ ] HTML report export
+- [ ] Pytest plugin (`@eval_case` decorator)
+- [ ] Model comparison mode — diff two models on same cases
+- [ ] Eval versioning — track scores over time
+
+---
+
+## Contributing
+
+Issues and PRs welcome.
+
+**Small changes** (docs, bug fixes): open a PR directly.
+**Large changes** (new evaluators, architecture): open an issue first.
+
+```bash
+git clone https://github.com/multivon-ai/multivon-eval
+cd llm-evals
+pip install -e ".[dev]"
+pytest tests/
+```
+
+---
+
+## License
+
+MIT — built by [Multivon](https://multivon.ai)