rag-benchmarking 1.0.0rc1__tar.gz

rag_benchmarking-1.0.0rc1/LICENSE

@@ -0,0 +1,192 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2025 Ajay Pundhir
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+

rag_benchmarking-1.0.0rc1/PKG-INFO

@@ -0,0 +1,386 @@
+Metadata-Version: 2.4
+Name: rag-benchmarking
+Version: 1.0.0rc1
+Summary: Framework-agnostic evaluation harness for RAG and agentic AI systems
+Author: Ajay Pundhir
+License: Apache-2.0
+Project-URL: Homepage, https://aiexponent.com
+Project-URL: Repository, https://github.com/aiexponenthq/rag-benchmarking
+Project-URL: Documentation, https://github.com/aiexponenthq/rag-benchmarking#readme
+Project-URL: Bug Tracker, https://github.com/aiexponenthq/rag-benchmarking/issues
+Keywords: rag,evaluation,benchmarking,llm,ai-governance,eu-ai-act,agentic-ai
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn>=0.30
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pydantic-settings>=2.2
+Requires-Dist: python-json-logger>=2.0.7
+Requires-Dist: qdrant-client>=1.9
+Requires-Dist: sentence-transformers>=3.0
+Requires-Dist: numpy>=1.26
+Requires-Dist: tqdm>=4.66
+Requires-Dist: requests>=2.31
+Requires-Dist: ragas>=0.1.9
+Requires-Dist: langchain-google-genai>=2.0.9
+Requires-Dist: FlagEmbedding>=1.2.11
+Provides-Extra: test
+Requires-Dist: pytest>=8.2; extra == "test"
+Requires-Dist: pytest-asyncio>=0.23; extra == "test"
+Requires-Dist: pytest-cov>=5.0; extra == "test"
+Requires-Dist: httpx>=0.27; extra == "test"
+Dynamic: license-file
+
+# RAG Benchmarking
+
+![Python](https://img.shields.io/badge/python-3.11+-blue.svg)
+![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)
+![Status](https://img.shields.io/badge/status-v1.0.0--rc1-orange.svg)
+![EU AI Act](https://img.shields.io/badge/EU%20AI%20Act-Article%2015-gold.svg)
+[![PyPI](https://img.shields.io/pypi/v/rag-benchmarking.svg)](https://pypi.org/project/rag-benchmarking/)
+
+**A framework-agnostic evaluation harness for RAG and agentic AI systems.**
+
+Bring your own RAG pipeline — LangChain, LlamaIndex, or custom — and benchmark it against standard classic and agentic-era metrics. Built for teams who need to prove their AI systems work before they ship, not hope they do.
+
+> Part of the [AiExponent](https://aiexponent.com) open source portfolio. Maps to EU AI Act Article 15 (accuracy requirements).
+
+---
+
+## Quick Start
+
+### Install
+
+```bash
+# Install from PyPI (recommended)
+pip install rag-benchmarking
+
+# Or install from source
+git clone https://github.com/aiexponenthq/rag-benchmarking.git
+cd rag-benchmarking
+pip install -e ".[test]"
+```
+
+### Evaluate your existing RAG system in 5 minutes
+
+```python
+from app.sdk.client import RagEval
+
+client = RagEval(api_url="http://localhost:5001", api_key="your-key")
+
+# LangChain integration
+from langchain.chains import RetrievalQA
+chain = RetrievalQA.from_chain_type(...)
+result = chain.invoke({"query": "What is RAG?"})
+sample = RagEval.from_langchain(result)
+
+# LlamaIndex integration
+engine = index.as_query_engine()
+response = engine.query("What is RAG?")
+sample = RagEval.from_llamaindex(response, "What is RAG?")
+
+# Or any dict with question / contexts / answer
+sample = {
+    "question": "What is RAG?",
+    "contexts": ["RAG stands for Retrieval-Augmented Generation."],
+    "answer": "RAG combines retrieval with LLM generation.",
+}
+
+report = client.evaluate([sample], metrics=["faithfulness", "answer_relevancy"])
+print(report["scores"])
+# {"faithfulness": 0.92, "answer_relevancy": 0.88}
+```
+
+### Start the evaluation server
+
+```bash
+# With Docker Compose
+docker compose up
+
+# Or directly
+uvicorn app.main:app --port 5001
+```
+
+### Interactive API docs
+
+Once the server is running, the full OpenAPI reference is available at:
+
+```
+http://localhost:5001/docs
+```
+
+---
+
+## LLM Backend for Evaluation
+
+Several metrics (faithfulness, context_precision, context_recall, agent_faithfulness, tool_call_accuracy, retrieval_necessity) use an LLM as a judge. The harness supports **Gemini** (recommended) and **OpenAI**.
+
+```bash
+# Set in .env:
+LLM_PROVIDER=gemini
+GEMINI_API_KEY=your-gemini-key
+
+# Or OpenAI:
+LLM_PROVIDER=openai
+OPENAI_API_KEY=your-openai-key
+```
+
+**Determinism:** Judge calls run at `temperature=0.0` to minimise variance across evaluation runs. For CI/CD integration, run evaluations at least twice and flag changes beyond a ±0.05 threshold rather than asserting exact scores.
+
+**Cost guidance:** A full evaluation pass (all classic metrics) on 50 samples costs approximately $0.05–$0.15 with Gemini Flash or GPT-4o-mini. Source attribution accuracy is deterministic and costs nothing.
+
+---
+
+## Metrics
+
+### Classic RAG Metrics
+
+| Metric | What it measures | Requires |
+|---|---|---|
+| `faithfulness` | Are all claims in the answer supported by context? | question, contexts, answer |
+| `answer_relevancy` | Does the answer address the question? | question, answer |
+| `context_precision` | Are retrieved chunks relevant to the query? | + ground_truth |
+| `context_recall` | Does context contain enough to answer correctly? | + ground_truth |
+| `precision_at_k` | Fraction of top-K retrieved docs that are relevant | + retrieved_doc_ids, relevant_doc_ids |
+| `recall_at_k` | Fraction of relevant docs found in top-K | + retrieved_doc_ids, relevant_doc_ids |
+| `mrr` | Reciprocal rank of first relevant doc | + retrieved_doc_ids, relevant_doc_ids |
+| `ndcg_at_k` | Rank-weighted retrieval quality | + retrieved_doc_ids, relevant_doc_ids |
+
+### Agentic-Era Metrics
+
+For multi-step agents, tool-using systems, and autonomous RAG pipelines:
+
+| Metric | What it measures | LLM needed? |
+|---|---|---|
+| `source_attribution_accuracy` | Did the agent cite sources it actually retrieved? | No (deterministic) |
+| `agent_faithfulness` | Is every reasoning step faithful to retrieved sources? | Yes |
+| `tool_call_accuracy` | Did the agent choose the right tool at the right time? | Yes |
+| `retrieval_necessity` | Was retrieval actually needed for this query? | Yes |
+
+### Metric Groups
+
+| Group | Metrics included |
+|---|---|
+| `classic` | faithfulness, answer_relevancy |
+| `retrieval` | precision_at_k, recall_at_k, mrr, ndcg_at_k |
+| `agentic_v1` | source_attribution_accuracy, retrieval_necessity, agent_faithfulness, tool_call_accuracy |
+| `agentic_v2` | multihop_faithfulness, agent_trajectory_efficiency, reasoning_hallucination, context_coherence_across_turns |
+| `full` | all classic + retrieval + agentic_v1 metrics |
+
+```python
+# Use a pre-defined group instead of listing metrics individually
+report = client.evaluate(samples, metric_group="classic")
+```
+
+---
+
+## API Reference
+
+### REST API
+
+```bash
+# Classic RAG evaluation
+curl -X POST http://localhost:5001/v1/evaluate \
+  -H "X-API-Key: your-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "samples": [{
+      "question": "What is RAG?",
+      "contexts": ["RAG is Retrieval-Augmented Generation."],
+      "answer": "RAG combines retrieval with LLM generation."
+    }],
+    "metrics": ["faithfulness", "answer_relevancy"]
+  }'
+
+# Agentic trace evaluation
+curl -X POST http://localhost:5001/v1/evaluate/agent \
+  -H "X-API-Key: your-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "trace": {
+      "question": "What is the GPAI deadline?",
+      "final_answer": "GPAI obligations apply from August 2025.",
+      "tool_calls": [{
+        "tool_name": "retrieve",
+        "tool_input": {"query": "GPAI deadline"},
+        "tool_output": "Article 53 obligations apply from August 2025.",
+        "step_index": 0
+      }]
+    },
+    "metrics": ["source_attribution_accuracy", "tool_call_accuracy"]
+  }'
+
+# List and compare runs
+curl http://localhost:5001/v1/runs -H "X-API-Key: your-key"
+curl -X POST http://localhost:5001/v1/runs/compare \
+  -H "X-API-Key: your-key" \
+  -d '["run-id-a", "run-id-b"]'
+```
+
+### Python SDK
+
+```python
+from app.sdk.client import RagEval
+
+client = RagEval(api_url="http://localhost:5001", api_key="your-key")
+
+# Classic evaluation
+report = client.evaluate(samples, metrics=["faithfulness"])
+report = client.evaluate(samples, metric_group="agentic_v1")
+
+# Agentic trace
+report = client.evaluate_agent(trace_dict, metrics=["agent_faithfulness"])
+
+# Run history and comparison
+runs = client.list_runs()
+comparison = client.compare_runs(["run-a", "run-b"])
+```
+
+---
+
+## Architecture
+
+```mermaid
+flowchart TD
+    A["Your RAG System\n(LangChain / LlamaIndex / Custom)"]
+    B["SDK Adapters\nRagEval.from_langchain()\nRagEval.from_llamaindex()"]
+    C["EvalSample / AgentTrace\nharness/schemas.py"]
+    D["EvaluationRunner\nharness/runner.py"]
+    E["RAGAS Metrics\nfaithfulness · answer_relevancy\ncontext_precision · context_recall"]
+    F["Retrieval Metrics\nPrecision@K · Recall@K · MRR · NDCG"]
+    G["Agentic Metrics\nagent_faithfulness · tool_call_accuracy\nretrieval_necessity · source_attribution"]
+    H["BenchmarkReport"]
+    I["SQLite ResultStore\nharness/result_store.py"]
+    J["REST API\n/v1/evaluate · /v1/evaluate/agent\n/v1/runs · /v1/runs/compare"]
+
+    A --> B --> C --> D
+    D --> E
+    D --> F
+    D --> G
+    E --> H
+    F --> H
+    G --> H
+    H --> I --> J
+```
+
+### Plug-in contract
+
+Any RAG system implements one method to integrate:
+
+```python
+class MyRAG:
+    def run(self, question: str, contexts_override=None) -> dict:
+        result = self.chain.invoke({"query": question})
+        return {
+            "answer": result["result"],
+            "contexts": [d.page_content for d in result["source_documents"]],
+            "retrieved_doc_ids": [d.metadata.get("id") for d in result["source_documents"]],
+        }
+```
+
+---
+
+## Configuration
+
+Copy `.env.example` to `.env` and set:
+
+```bash
+# LLM provider for faithfulness judge
+LLM_PROVIDER=gemini          # or openai
+GEMINI_API_KEY=...
+OPENAI_API_KEY=...
+
+# Vector store (built-in RAG pipeline only)
+QDRANT_URL=https://your-cluster.qdrant.io
+QDRANT_API_KEY=...
+
+# API authentication
+API_KEY=your-secret-key
+ENFORCE_API_KEY=true
+```
+
+---
+
+## Project Structure
+
+```
+src/
+  harness/                   # Framework-agnostic evaluation harness
+    schemas.py               # EvalSample, AgentTrace, BenchmarkReport, RunConfig
+    protocol.py              # RAGEvaluable Protocol — the plug-in contract
+    runner.py                # EvaluationRunner — orchestrates metrics
+    result_store.py          # SQLite persistence for BenchmarkReport
+  app/
+    api/                     # FastAPI endpoints
+    eval/
+      ragas_runner.py        # RAGAS classic metrics
+      retrieval_metrics.py   # Precision@K, Recall@K, MRR, NDCG
+      faithfulness.py        # Claim-decomposition faithfulness (LLM-as-judge)
+      agentic_metrics.py     # source_attribution_accuracy (deterministic)
+      agentic_llm_metrics.py # LLM-as-judge agentic metrics
+    sdk/                     # Python SDK (RagEval client)
+    engine/                  # Built-in RAG pipeline (optional)
+
+data/
+  golden/qa.jsonl            # 50-sample golden dataset (10 domains)
+
+tests/
+  unit/                      # Unit tests (no LLM, no network)
+  integration/               # SDK integration tests
+  e2e/                       # Full API endpoint tests
+```
+
+---
+
+## EU AI Act Context
+
+Maps to **Article 15** — Accuracy, Robustness and Cybersecurity for High-Risk AI Systems. Systematic RAG evaluation implements the technical testing requirements for demonstrating accuracy under Article 15.
+
+---
+
+## Known Limitations
+
+- Benchmark datasets are English-only; no multilingual evaluation support.
+- Custom dataset integration requires manual formatting to the JSONL schema.
+- Accuracy metrics only — latency and throughput are not measured.
+- LLM-as-judge metrics depend on the quality of the configured judge model.
+- Rate limiting is in-memory and resets on server restart.
+
+---
+
+## Version Compatibility
+
+| Dependency | Tested version |
+|---|---|
+| Python | 3.11, 3.12 |
+| RAGAS | 0.4.x |
+| LangChain | ≥ 0.1 |
+| LlamaIndex | ≥ 0.10 |
+| FastAPI | ≥ 0.110 |
+
+---
+
+## Further Reading
+
+- [DEPLOYMENT.md](DEPLOYMENT.md) — Docker, Kubernetes, security configuration
+- [CONTRIBUTING.md](CONTRIBUTING.md) — how to contribute, commit conventions, test guidelines
+- [SECURITY.md](SECURITY.md) — vulnerability reporting policy
+- [CHANGELOG.md](CHANGELOG.md) — version history
+
+---
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). Apache 2.0 licensed.
+
+Built by [AiExponent](https://aiexponent.com) — Building AI that deserves to be trusted.