ragcheck-eval 0.1.0__tar.gz

ragcheck_eval-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,23 @@
+---
+name: Bug report
+about: Something broken or behaving unexpectedly
+labels: bug
+---
+
+**What happened**
+
+**What you expected**
+
+**Reproduction**
+
+```bash
+# minimal command / config that triggers it
+```
+
+**Environment**
+- ragcheck version:
+- Python version:
+- Judge provider/model (if metric-related):
+- Judge prompt version (from the report JSON):
+
+**Report snippet** (if applicable — the relevant `MetricResult` from your report JSON, with `judge_model` and `prompt_version`)

ragcheck_eval-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,16 @@
+---
+name: Feature request
+about: Propose a metric, adapter, or capability
+labels: enhancement
+---
+
+**Problem it solves**
+What can't you measure or do today?
+
+**Proposed behavior**
+CLI/config/API sketch if you have one.
+
+**Alternatives considered**
+
+**Scope check**
+RAGCheck deliberately excludes web UIs, hosted services, and non-eval features (see README "Explicit non-goals"). If this touches those, say why it should be an exception.

ragcheck_eval-0.1.0/.github/workflows/ci.yaml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - run: pip install -e ".[dev]"
+      - run: ruff check .
+      - run: mypy ragcheck/
+      - run: pytest -q

ragcheck_eval-0.1.0/.github/workflows/smoke-eval.yaml

@@ -0,0 +1,51 @@
+# Live smoke eval on merges to main: runs the toy example against a real judge
+# and regression-checks it against the committed baseline (dogfooding
+# `ragcheck compare`). Skips gracefully when no API key secret is configured.
+name: smoke-eval
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    env:
+      GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+      RAGCHECK_MODEL: ${{ vars.RAGCHECK_MODEL || 'meta-llama/llama-4-scout-17b-16e-instruct' }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Check for API key
+        id: gate
+        run: |
+          if [ -n "$GROQ_API_KEY" ]; then
+            echo "run=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "No GROQ_API_KEY secret configured - skipping live smoke eval."
+          fi
+      - uses: actions/setup-python@v5
+        if: steps.gate.outputs.run == 'true'
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Run toy eval
+        if: steps.gate.outputs.run == 'true'
+        run: |
+          pip install -e .
+          ragcheck run examples/toy_config_groq.yaml --yes
+      - name: Regression check against baseline
+        if: steps.gate.outputs.run == 'true'
+        run: |
+          if [ -f examples/smoke_baseline.json ]; then
+            ragcheck compare examples/smoke_baseline.json ragcheck_output/toy-demo-groq.json \
+              --fail-if "faithfulness<-0.1" --fail-if "hit_rate@3<-0.1"
+          else
+            echo "No baseline committed yet - skipping compare."
+          fi
+      - name: Upload report
+        if: steps.gate.outputs.run == 'true'
+        uses: actions/upload-artifact@v4
+        with:
+          name: smoke-report
+          path: ragcheck_output/

ragcheck_eval-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+build/
+.venv/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+*.sqlite
+ragcheck_output/
+benchmarks/corpus/data/
+benchmarks/results/
+.DS_Store

ragcheck_eval-0.1.0/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+## 0.1.0 (unreleased)
+
+Initial release.
+
+- `RAGAdapter` / `FunctionAdapter` / `LangChainAdapter` pipeline wrappers
+- Retrieval metrics: `hit_rate@k`, `mrr`, `context_precision`, `context_recall`
+- Generation metrics: `faithfulness`, `answer_relevance`, `citation_accuracy`
+- Robustness metrics: `refusal_calibration`, `paraphrase_consistency`
+- Judge validation: `ragcheck validate-judge` (Cohen's kappa, confusion matrix vs. human labels)
+- Synthetic datasets: `ragcheck generate-dataset` (difficulty tiers, unanswerables, paraphrase groups, resumable)
+- SQLite judgment cache keyed on judge model + prompt version + inputs
+- Versioned judge prompts (markdown + YAML frontmatter)
+- Judge providers: Anthropic, Groq; parallel judging with rate-limit backoff
+- JSON + self-contained HTML reports with worst-failing-sample drilldowns
+- `ragcheck compare` regression diffing with `--fail-if` thresholds (CI-friendly exit codes)
+- Reproducible 4-architecture benchmark (naive / hybrid / reranked / agentic RAG) on SEC 10-Ks
+- Cost guard: large LLM-judged runs require `--yes`

ragcheck_eval-0.1.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,7 @@
+# Code of Conduct
+
+This project follows the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+
+In short: be respectful, be constructive, assume good faith. Harassment, personal attacks, and derailing behavior are not tolerated.
+
+Report unacceptable behavior by opening a confidential issue or emailing the maintainer. All reports are reviewed and handled with discretion.

ragcheck_eval-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,36 @@
+# Contributing to RAGCheck
+
+Thanks for considering a contribution. This project values honest metrics and small, reviewable changes.
+
+## Setup
+
+```bash
+git clone https://github.com/Ekta-Shah/ragcheck && cd ragcheck
+python3.10+ -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Before you open a PR
+
+```bash
+ruff check .        # lint (line length 100)
+mypy ragcheck/      # types are required on all public APIs
+pytest              # must stay green; no live API calls in unit tests
+```
+
+- **No live LLM calls in unit tests.** Judges are mocked (see `tests/conftest.py`). Live calls belong only in the smoke-eval workflow and `benchmarks/`.
+- **Every LLM-judged metric needs known-answer tests** — hand-constructed cases where the correct score is verifiable by inspection (aim for 3+).
+- **Judge prompts are versioned.** Changing a prompt's wording means bumping `version:` in its frontmatter; cached judgments key on the version, so old results stay traceable.
+- **Docstrings (Google style) + type hints** on all public classes and functions.
+- One logical change per commit; keep PRs focused.
+
+## What's welcome
+
+- New metrics (deterministic ones are the easiest entry point — see `ragcheck/metrics/retrieval/mrr.py` as a template)
+- Judge prompt improvements (with before/after validation via `ragcheck validate-judge`)
+- Adapters for other frameworks (LlamaIndex, Haystack) — open an issue first
+- Benchmark reproductions and corrections
+
+## Filing issues
+
+Use the issue templates. For metric-quality issues, include the judge model, prompt version, and a minimal failing sample — all three are recorded in every report.

ragcheck_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ekta Shah
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ragcheck_eval-0.1.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: ragcheck-eval
+Version: 0.1.0
+Summary: pytest for RAG systems - evaluate retrieval-augmented generation pipelines
+Author: Ekta Shah
+License: MIT License
+        
+        Copyright (c) 2026 Ekta Shah
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: anthropic>=0.40
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.5
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Provides-Extra: benchmarks
+Requires-Dist: matplotlib>=3.8; extra == 'benchmarks'
+Requires-Dist: rank-bm25>=0.2; extra == 'benchmarks'
+Requires-Dist: sentence-transformers>=3.0; extra == 'benchmarks'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# RAGCheck
+
+[![CI](https://github.com/Ekta-Shah/ragcheck/actions/workflows/ci.yaml/badge.svg)](https://github.com/Ekta-Shah/ragcheck/actions/workflows/ci.yaml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
+
+**pytest for RAG systems.** Wrap your retrieval-augmented generation pipeline in a one-method adapter and get quality scores, a failure taxonomy, cost/latency profiles, and CI-friendly regression checks — with a judge you can actually verify.
+
+```bash
+pip install ragcheck-eval   # imports and CLI are `ragcheck`
+ragcheck demo               # no API key needed - see the whole workflow in 10 seconds
+```
+
+The demo runs a canned pipeline with two planted failures — faithfulness catches the hallucinated claim, refusal calibration flags the invented answer to an unanswerable question, and the HTML report shows both with their retrieved contexts. ([Or open the quickstart in Colab.](https://colab.research.google.com/github/Ekta-Shah/ragcheck/blob/main/examples/quickstart.ipynb))
+
+For real runs:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...  # or GROQ_API_KEY for open-weight judges
+ragcheck run examples/toy_config.yaml
+```
+
+Real output from the bundled toy pipeline (keyword retrieval over a 10-doc corpus):
+
+```text
+                    RAGCheck - toy-demo-groq
+┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Metric       ┃ Score ┃ Samples ┃ Judge                        ┃
+┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ hit_rate@3   │ 1.000 │      10 │ -                            │
+│ faithfulness │ 1.000 │      10 │ llama-3.3-70b-versatile (v1) │
+└──────────────┴───────┴─────────┴──────────────────────────────┘
+pipeline tokens: {'input_tokens': 1410, 'output_tokens': 184}  |
+judge tokens: {'input_tokens': 4338, 'output_tokens': 274}  |  cache: 0 hits / 26 misses
+```
+
+Alongside the terminal scorecard you get a JSON report and a **self-contained HTML report** — scorecard, judge-validation status, latency/cost, and the worst-failing samples with question, answer, and retrieved context inline.
+
+## Why another RAG eval tool?
+
+Existing tools score your pipeline with an LLM judge and stop there. RAGCheck is built around the questions that actually block shipping:
+
+- **Can you trust the judge?** `ragcheck validate-judge labels.jsonl --metric faithfulness` measures LLM-judge vs. human agreement (Cohen's kappa + confusion matrix) *before* you trust judged metrics. On our own labels it caught a real miscalibration: the default pass threshold scored κ=0.40; the corrected threshold scores κ=1.00. Reports display "validated at κ=0.XX" — or a visible "not validated" flag.
+- **Does your system know when to say "I don't know"?** `refusal_calibration` reports the false-answer rate (hallucinated answers to unanswerable questions) and over-refusal rate separately.
+- **Is it robust?** `paraphrase_consistency` judges pairwise answer agreement within paraphrase groups — catching pipelines that only work on one phrasing.
+- **No eval set?** `ragcheck generate-dataset corpus_dir/` builds one from your documents: easy/medium/hard tiers, unanswerable questions, paraphrase groups, chunk-level provenance. Cached and resumable.
+- **What does quality cost?** The benchmark harness produces a cost-quality frontier across architectures.
+- **Did this PR make it worse?** `ragcheck compare old.json new.json --fail-if "faithfulness<-0.05"` exits non-zero on regression and prints a markdown diff for PR comments.
+
+## Compared to existing tools
+
+| Capability | RAGCheck | RAGAS | DeepEval | TruLens |
+|---|:---:|:---:|:---:|:---:|
+| Core RAG metrics (faithfulness, context precision/recall, relevance) | ✅ | ✅ | ✅ | ✅ |
+| Built-in judge validation vs. human labels (Cohen's κ) | ✅ | ❌ | ❌ | ❌ |
+| Refusal calibration (false-answer + over-refusal rates) | ✅ | ❌ | ❌ | ❌ |
+| Paraphrase-consistency robustness testing | ✅ | ❌ | ❌ | ❌ |
+| Persistent judgment cache keyed on judge model + prompt version | ✅ | ❌ | partial | ❌ |
+| Report diffing with CI exit codes | ✅ | ❌ | ✅ | ❌ |
+| Runs entirely local — no dashboard, account, or service | ✅ | ✅ | partial | partial |
+
+*Comparison reflects our reading of each tool's docs at the time of writing; corrections welcome via issues.*
+
+## How it works
+
+Wrap any pipeline in a `RAGAdapter` (or just a function):
+
+```python
+from ragcheck import RAGResponse, RetrievedChunk
+from ragcheck.adapters import FunctionAdapter
+
+def my_pipeline(question: str) -> RAGResponse:
+    chunks = my_retriever(question)
+    answer = my_generator(question, chunks)
+    return RAGResponse(
+        answer=answer,
+        retrieved_chunks=[RetrievedChunk(content=c.text, source_id=c.id) for c in chunks],
+    )
+
+adapter = FunctionAdapter(my_pipeline)
+```
+
+LangChain users: `LangChainAdapter(retriever, chain)` wraps a retriever + chain directly.
+
+Point a YAML config at your adapter, dataset, and metrics, then `ragcheck run config.yaml`. Every LLM judgment is cached in SQLite (keyed on judge model, prompt version, and inputs) so re-runs on unchanged data are free — and every judged result records exactly which judge and prompt produced it.
+
+## Benchmark: which RAG architecture?
+
+Four pipelines — naive dense, hybrid (BM25+dense RRF), cross-encoder reranked, and agentic (query decomposition) — on 8 SEC 10-K filings, identical chunking/generator, retrieval as the only variable. Preliminary 12-sample free-tier run (differences under ~0.2 are noise; full 300-500 sample run pending):
+
+| Pipeline | faithfulness | refusal_calibration | paraphrase_consistency | tok/q | retrieval p50 |
+|---|---:|---:|---:|---:|---:|
+| naive (dense) | 0.950 | 0.583 | 0.700 | 1535 | 43 ms |
+| hybrid (BM25+dense RRF) | 0.958 | 0.500 | **1.000** | 1448 | 75 ms |
+| reranked (cross-encoder) | 0.958 | 0.500 | **1.000** | 1511 | 359 ms |
+| agentic (decomposition) | 0.958 | **0.750** | 0.450 | 2683 | 6602 ms |
+
+Early hypothesis worth noticing: agentic RAG refuses unanswerables best but answers paraphrases least consistently, at 1.75× the cost — exactly the trade-offs single-metric evals miss. Methodology, frontier chart, and honest limitations: [benchmarks/](benchmarks/README.md).
+
+## Design decisions
+
+- **Judge validation is a feature, not a footnote.** Judged metrics are untrustworthy by default; the κ workflow makes trust measurable, and reports flag unvalidated judges.
+- **Deterministic first.** Where a metric can be computed without an LLM (hit_rate, MRR), it is. LLM judging is reserved for what actually needs it.
+- **Everything cached, everything versioned.** Judgments are cached on (judge model, prompt version, inputs); prompts live as versioned markdown. Interrupted dataset generation resumes for free; changing a prompt never silently reuses stale verdicts.
+- **CI is the target environment.** JSON reports, exit codes, markdown diffs, an env-gated smoke workflow, and a cost guard (`--yes` required above a configurable sample count).
+- **No UI.** The self-contained HTML report is the only visual output. No dashboard, no server, no account.
+
+## Docs
+
+[Quickstart](docs/quickstart.md) · [Metrics reference](docs/metrics.md) · [Judge validation](docs/judge-validation.md) · [CI integration](docs/ci-integration.md)
+
+## Status
+
+| Area | Status |
+|---|---|
+| Zero-key demo (`ragcheck demo`) + Colab quickstart notebook | ✅ |
+| Adapters: `FunctionAdapter`, `LangChainAdapter` | ✅ |
+| 9 metrics: retrieval (4) · generation (3) · robustness (2) | ✅ |
+| Judge validation (`validate-judge`, Cohen's κ) | ✅ |
+| Synthetic datasets (`generate-dataset`) | ✅ |
+| JSON + self-contained HTML reports | ✅ |
+| Regression diffing (`compare --fail-if`) | ✅ |
+| CI workflows (lint/type/test + env-gated live smoke eval) | ✅ |
+| 4-architecture benchmark harness | ✅ |
+| 4-pipeline benchmark: preliminary 12-sample results | ✅ |
+| Full benchmark run (300-500 samples) | 🔜 needs funded API budget |
+| PyPI release (`ragcheck-eval` 0.1.0) | 🔜 with benchmark results |
+| LlamaIndex/Haystack adapters | ❌ not in v0.1 (issues welcome) |
+
+Explicit non-goals: web UI/dashboards, hosted services, fine-tuning or embedding-model evaluation.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest            # unit tests (no API calls - judges are mocked)
+ruff check .
+mypy ragcheck/
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+MIT

ragcheck_eval-0.1.0/README.md

@@ -0,0 +1,145 @@
+# RAGCheck
+
+[![CI](https://github.com/Ekta-Shah/ragcheck/actions/workflows/ci.yaml/badge.svg)](https://github.com/Ekta-Shah/ragcheck/actions/workflows/ci.yaml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
+
+**pytest for RAG systems.** Wrap your retrieval-augmented generation pipeline in a one-method adapter and get quality scores, a failure taxonomy, cost/latency profiles, and CI-friendly regression checks — with a judge you can actually verify.
+
+```bash
+pip install ragcheck-eval   # imports and CLI are `ragcheck`
+ragcheck demo               # no API key needed - see the whole workflow in 10 seconds
+```
+
+The demo runs a canned pipeline with two planted failures — faithfulness catches the hallucinated claim, refusal calibration flags the invented answer to an unanswerable question, and the HTML report shows both with their retrieved contexts. ([Or open the quickstart in Colab.](https://colab.research.google.com/github/Ekta-Shah/ragcheck/blob/main/examples/quickstart.ipynb))
+
+For real runs:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...  # or GROQ_API_KEY for open-weight judges
+ragcheck run examples/toy_config.yaml
+```
+
+Real output from the bundled toy pipeline (keyword retrieval over a 10-doc corpus):
+
+```text
+                    RAGCheck - toy-demo-groq
+┏━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Metric       ┃ Score ┃ Samples ┃ Judge                        ┃
+┡━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ hit_rate@3   │ 1.000 │      10 │ -                            │
+│ faithfulness │ 1.000 │      10 │ llama-3.3-70b-versatile (v1) │
+└──────────────┴───────┴─────────┴──────────────────────────────┘
+pipeline tokens: {'input_tokens': 1410, 'output_tokens': 184}  |
+judge tokens: {'input_tokens': 4338, 'output_tokens': 274}  |  cache: 0 hits / 26 misses
+```
+
+Alongside the terminal scorecard you get a JSON report and a **self-contained HTML report** — scorecard, judge-validation status, latency/cost, and the worst-failing samples with question, answer, and retrieved context inline.
+
+## Why another RAG eval tool?
+
+Existing tools score your pipeline with an LLM judge and stop there. RAGCheck is built around the questions that actually block shipping:
+
+- **Can you trust the judge?** `ragcheck validate-judge labels.jsonl --metric faithfulness` measures LLM-judge vs. human agreement (Cohen's kappa + confusion matrix) *before* you trust judged metrics. On our own labels it caught a real miscalibration: the default pass threshold scored κ=0.40; the corrected threshold scores κ=1.00. Reports display "validated at κ=0.XX" — or a visible "not validated" flag.
+- **Does your system know when to say "I don't know"?** `refusal_calibration` reports the false-answer rate (hallucinated answers to unanswerable questions) and over-refusal rate separately.
+- **Is it robust?** `paraphrase_consistency` judges pairwise answer agreement within paraphrase groups — catching pipelines that only work on one phrasing.
+- **No eval set?** `ragcheck generate-dataset corpus_dir/` builds one from your documents: easy/medium/hard tiers, unanswerable questions, paraphrase groups, chunk-level provenance. Cached and resumable.
+- **What does quality cost?** The benchmark harness produces a cost-quality frontier across architectures.
+- **Did this PR make it worse?** `ragcheck compare old.json new.json --fail-if "faithfulness<-0.05"` exits non-zero on regression and prints a markdown diff for PR comments.
+
+## Compared to existing tools
+
+| Capability | RAGCheck | RAGAS | DeepEval | TruLens |
+|---|:---:|:---:|:---:|:---:|
+| Core RAG metrics (faithfulness, context precision/recall, relevance) | ✅ | ✅ | ✅ | ✅ |
+| Built-in judge validation vs. human labels (Cohen's κ) | ✅ | ❌ | ❌ | ❌ |
+| Refusal calibration (false-answer + over-refusal rates) | ✅ | ❌ | ❌ | ❌ |
+| Paraphrase-consistency robustness testing | ✅ | ❌ | ❌ | ❌ |
+| Persistent judgment cache keyed on judge model + prompt version | ✅ | ❌ | partial | ❌ |
+| Report diffing with CI exit codes | ✅ | ❌ | ✅ | ❌ |
+| Runs entirely local — no dashboard, account, or service | ✅ | ✅ | partial | partial |
+
+*Comparison reflects our reading of each tool's docs at the time of writing; corrections welcome via issues.*
+
+## How it works
+
+Wrap any pipeline in a `RAGAdapter` (or just a function):
+
+```python
+from ragcheck import RAGResponse, RetrievedChunk
+from ragcheck.adapters import FunctionAdapter
+
+def my_pipeline(question: str) -> RAGResponse:
+    chunks = my_retriever(question)
+    answer = my_generator(question, chunks)
+    return RAGResponse(
+        answer=answer,
+        retrieved_chunks=[RetrievedChunk(content=c.text, source_id=c.id) for c in chunks],
+    )
+
+adapter = FunctionAdapter(my_pipeline)
+```
+
+LangChain users: `LangChainAdapter(retriever, chain)` wraps a retriever + chain directly.
+
+Point a YAML config at your adapter, dataset, and metrics, then `ragcheck run config.yaml`. Every LLM judgment is cached in SQLite (keyed on judge model, prompt version, and inputs) so re-runs on unchanged data are free — and every judged result records exactly which judge and prompt produced it.
+
+## Benchmark: which RAG architecture?
+
+Four pipelines — naive dense, hybrid (BM25+dense RRF), cross-encoder reranked, and agentic (query decomposition) — on 8 SEC 10-K filings, identical chunking/generator, retrieval as the only variable. Preliminary 12-sample free-tier run (differences under ~0.2 are noise; full 300-500 sample run pending):
+
+| Pipeline | faithfulness | refusal_calibration | paraphrase_consistency | tok/q | retrieval p50 |
+|---|---:|---:|---:|---:|---:|
+| naive (dense) | 0.950 | 0.583 | 0.700 | 1535 | 43 ms |
+| hybrid (BM25+dense RRF) | 0.958 | 0.500 | **1.000** | 1448 | 75 ms |
+| reranked (cross-encoder) | 0.958 | 0.500 | **1.000** | 1511 | 359 ms |
+| agentic (decomposition) | 0.958 | **0.750** | 0.450 | 2683 | 6602 ms |
+
+Early hypothesis worth noticing: agentic RAG refuses unanswerables best but answers paraphrases least consistently, at 1.75× the cost — exactly the trade-offs single-metric evals miss. Methodology, frontier chart, and honest limitations: [benchmarks/](benchmarks/README.md).
+
+## Design decisions
+
+- **Judge validation is a feature, not a footnote.** Judged metrics are untrustworthy by default; the κ workflow makes trust measurable, and reports flag unvalidated judges.
+- **Deterministic first.** Where a metric can be computed without an LLM (hit_rate, MRR), it is. LLM judging is reserved for what actually needs it.
+- **Everything cached, everything versioned.** Judgments are cached on (judge model, prompt version, inputs); prompts live as versioned markdown. Interrupted dataset generation resumes for free; changing a prompt never silently reuses stale verdicts.
+- **CI is the target environment.** JSON reports, exit codes, markdown diffs, an env-gated smoke workflow, and a cost guard (`--yes` required above a configurable sample count).
+- **No UI.** The self-contained HTML report is the only visual output. No dashboard, no server, no account.
+
+## Docs
+
+[Quickstart](docs/quickstart.md) · [Metrics reference](docs/metrics.md) · [Judge validation](docs/judge-validation.md) · [CI integration](docs/ci-integration.md)
+
+## Status
+
+| Area | Status |
+|---|---|
+| Zero-key demo (`ragcheck demo`) + Colab quickstart notebook | ✅ |
+| Adapters: `FunctionAdapter`, `LangChainAdapter` | ✅ |
+| 9 metrics: retrieval (4) · generation (3) · robustness (2) | ✅ |
+| Judge validation (`validate-judge`, Cohen's κ) | ✅ |
+| Synthetic datasets (`generate-dataset`) | ✅ |
+| JSON + self-contained HTML reports | ✅ |
+| Regression diffing (`compare --fail-if`) | ✅ |
+| CI workflows (lint/type/test + env-gated live smoke eval) | ✅ |
+| 4-architecture benchmark harness | ✅ |
+| 4-pipeline benchmark: preliminary 12-sample results | ✅ |
+| Full benchmark run (300-500 samples) | 🔜 needs funded API budget |
+| PyPI release (`ragcheck-eval` 0.1.0) | 🔜 with benchmark results |
+| LlamaIndex/Haystack adapters | ❌ not in v0.1 (issues welcome) |
+
+Explicit non-goals: web UI/dashboards, hosted services, fine-tuning or embedding-model evaluation.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest            # unit tests (no API calls - judges are mocked)
+ruff check .
+mypy ragcheck/
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+MIT

ragcheck_eval-0.1.0/benchmarks/README.md

@@ -0,0 +1,70 @@
+# RAGCheck Architecture Benchmark
+
+Reproducible comparison of RAG retrieval architectures on a financial-filings corpus. Everything except the retrieval strategy is held constant: same chunking (1200 chars, 200 overlap), same embedding model (`all-MiniLM-L6-v2`), same generator model and prompt.
+
+> **Status: preliminary results below (12 samples, free-tier run).** All 4 pipelines and the metric suite run end-to-end. The full run (300-500 samples) needs a funded API budget and will replace these numbers; at n=12, differences under ~0.2 are noise.
+
+## Pipelines
+
+| Pipeline | Retrieval |
+|---|---|
+| `naive_rag` | Dense (MiniLM cosine) top-5 |
+| `hybrid_rag` | BM25 + dense top-20 each, reciprocal rank fusion (k=60), top-5 |
+| `reranked_rag` | Dense top-20 → cross-encoder (`ms-marco-MiniLM-L-6-v2`) rerank → top-5 |
+| `agentic_rag` | LLM query decomposition → iterative dense retrieval (max 3 rounds) → synthesis |
+
+All four share the same chunking, embeddings, generator model, and generation prompt (which requires `[source_id]` citations) - the retrieval strategy is the only variable. Agentic's decomposition/sufficiency LLM calls are charged to its cost per query. Generator/judge models are selected with `RAGCHECK_BENCH_MODEL` / `RAGCHECK_JUDGE_MODEL`.
+
+## Corpus
+
+Latest 10-K filings of Apple, Microsoft, NVIDIA, Alphabet, Amazon, Meta, Tesla, and JPMorgan, fetched from SEC EDGAR and converted to plain text (~3.8 MB). Raw data is gitignored; the fetch script is deterministic.
+
+## Reproduce
+
+```bash
+pip install -e ".[benchmarks]"
+export GROQ_API_KEY=...
+export RAGCHECK_BENCH_MODEL=meta-llama/llama-4-scout-17b-16e-instruct
+python benchmarks/corpus/fetch_corpus.py                    # ~4 MB from EDGAR
+ragcheck generate-dataset benchmarks/corpus/data --n 400 \
+  --paraphrase-groups 20 --out benchmarks/synthetic_dataset.jsonl \
+  --provider groq --model $RAGCHECK_BENCH_MODEL             # resumable
+python benchmarks/run_benchmark.py                          # full; --quick for 20 samples
+```
+
+Outputs land in `benchmarks/results/`: per-pipeline JSON reports, `comparison.json`/`comparison.md`, and `cost_quality_frontier.png` (cost/query vs. faithfulness). `--pipelines`, `--metrics`, and `--n` subset a run for cheap smoke tests.
+
+## Preliminary results (4 pipelines, 12 stratified samples, 2026-07-06)
+
+Generator + judge: `meta-llama/llama-4-scout-17b-16e-instruct`. Samples include 2 unanswerable questions and one full paraphrase group. Chunk-level judged metrics (context precision/recall, citation accuracy) were excluded from this budget-constrained run.
+
+| Pipeline | hit_rate@5 | mrr | faithfulness | answer_relevance | refusal_calibration | paraphrase_consistency | tok/q | $/q | retrieval p50 |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| naive_rag | 0.500 | 0.450 | 0.950 | 0.917 | 0.583 | 0.700 | 1535 | $0.00018 | 43 ms |
+| hybrid_rag | 0.400 | 0.245 | 0.958 | 0.938 | 0.500 | **1.000** | 1448 | $0.00017 | 75 ms |
+| reranked_rag | 0.500 | 0.183 | 0.958 | 0.917 | 0.500 | **1.000** | 1511 | $0.00018 | 359 ms |
+| agentic_rag | 0.500 | 0.450 | 0.958 | 0.854 | **0.750** | 0.450 | 2683 | $0.00032 | 6602 ms |
+
+![Cost-quality frontier](cost_quality_frontier.png)
+
+**Preliminary reads (all within noise at this sample size, offered as hypotheses for the full run):**
+- **Agentic RAG is the refusal-calibration leader but the consistency laggard** - its multi-round retrieval gathers enough to decline unanswerables more often (0.75 vs 0.50-0.58), but different decompositions per paraphrase produce different answers (0.45 consistency). It also costs 1.75x tokens/query and ~90x retrieval latency.
+- **Hybrid and reranked are perfectly paraphrase-consistent** on this subset - stabler retrieval under rephrasing.
+- **Faithfulness is uniformly high (~0.95)** across architectures: the shared generator rarely asserts beyond its context; retrieval quality, robustness, and cost are where architectures actually differ.
+
+## Early results (2 pipelines, 16 samples, 2026-07-02)
+
+| Pipeline | hit_rate@5 | faithfulness | tokens/query | retrieval p50 (ms) |
+|---|---:|---:|---:|---:|
+| `naive_rag` (dense) | 0.438 | 1.000 | 1666 | 40 |
+| `hybrid_rag` (BM25 + dense, RRF) | **0.750** | 0.948 | 1555 | 47 |
+
+**Read:** on 10-K filings — dense-heavy with exact figures, entity names, and legal terms — adding BM25 lexical matching via reciprocal rank fusion lifts hit_rate@5 from 0.44 to 0.75 for ~7 ms extra retrieval latency. Both pipelines stay highly faithful *to what they retrieve*; the naive pipeline's perfect faithfulness with poor retrieval illustrates why retrieval and generation metrics must be read together (a pipeline can faithfully report that the answer isn't in its badly-retrieved context).
+
+## Honest limitations
+
+- **Small sample (16).** Differences of a few points are noise at this size.
+- **QA pairs are single-chunk and auto-generated.** Each question is grounded in exactly one chunk, and `hit_rate` counts only that chunk as relevant - a retrieved *duplicate* passage with the same fact counts as a miss. Some generated questions lean on filing boilerplate rather than substantive financials.
+- **No judge validation yet.** Faithfulness scores come from an unvalidated LLM judge; the judge-vs-human agreement module is Phase 2.
+- **Corpus text extraction is rough.** HTML tables flatten into text streams; some chunks are noisy.
+- **10-K filings shift over time.** The fetch script pulls each company's *latest* 10-K, so exact numbers will drift as new filings land; re-generate the dataset after re-fetching.

ragcheck_eval-0.1.0/benchmarks/benchmark_dataset.jsonl

@@ -0,0 +1,16 @@
+{"question": "When was Ms. Hood appointed Executive Vice President and Chief Financial Officer of Microsoft?", "ground_truth_answer": "July 2013", "relevant_source_ids": ["MSFT_10K_0085"], "difficulty": "easy"}
+{"question": "What type of breaches provides evidence of an external environment hostile to information security, according to MSFT's 10K filing?", "ground_truth_answer": "high-profile data breaches", "relevant_source_ids": ["MSFT_10K_0117"], "difficulty": "easy"}
+{"question": "What international standard for information security management does NVDA follow?", "ground_truth_answer": "ISO 27001", "relevant_source_ids": ["NVDA_10K_0194"], "difficulty": "easy"}
+{"question": "What percentage of NVIDIA's outstanding common stock would trigger Microsoft's first and last rights of refusal to purchase the stock?", "ground_truth_answer": "30%", "relevant_source_ids": ["NVDA_10K_0193"], "difficulty": "easy"}
+{"question": "What was the balance of other long-term liabilities for NVDA as of January 25, 2026?", "ground_truth_answer": "$7,306", "relevant_source_ids": ["NVDA_10K_0314"], "difficulty": "easy"}
+{"question": "What is the date listed for the signature of John O. Dabiri in the NVDA 10-K filing?", "ground_truth_answer": "February 25, 2026", "relevant_source_ids": ["NVDA_10K_0362"], "difficulty": "easy"}
+{"question": "What is the date of the Officer\u2019s Certificate of the Registrant mentioned in 4.12?", "ground_truth_answer": "August 4, 2016", "relevant_source_ids": ["AAPL_10K_0211"], "difficulty": "easy"}
+{"question": "In what year did the USG publish the AI Diffusion IFR in the Federal Register, according to NVDA's 10-K filing?", "ground_truth_answer": "2025", "relevant_source_ids": ["NVDA_10K_0160"], "difficulty": "easy"}
+{"question": "What operating system do Microsoft's server software applications support?", "ground_truth_answer": "Windows Server", "relevant_source_ids": ["MSFT_10K_0056"], "difficulty": "easy"}
+{"question": "What entity might NVIDIA need to notify regarding details of some of its Trustworthy AI processes?", "ground_truth_answer": "European Commission", "relevant_source_ids": ["NVDA_10K_0175"], "difficulty": "easy"}
+{"question": "What was the effective tax rate of Apple as of September 27, 2025?", "ground_truth_answer": "15.6%", "relevant_source_ids": ["AAPL_10K_0171"], "difficulty": "easy"}
+{"question": "What is the name of the AI-powered business and productivity solutions platform mentioned in the MSFT 10K excerpt?", "ground_truth_answer": "Microsoft 365 Commercial", "relevant_source_ids": ["MSFT_10K_0050"], "difficulty": "easy"}
+{"question": "What department's list is mentioned in the NVDA 10K excerpt as having restricted parties?", "ground_truth_answer": "U.S. Department of Commerce", "relevant_source_ids": ["NVDA_10K_0149"], "difficulty": "easy"}
+{"question": "What is the name of the independent registered public accounting firm that audited NVDA's internal control over financial reporting as of January 25, 2026?", "ground_truth_answer": "PricewaterhouseCoopers LLP", "relevant_source_ids": ["NVDA_10K_0244"], "difficulty": "easy"}
+{"question": "What date was the Officer\u2019s Certificate of the Registrant issued that included forms of global notes representing the 0.875% Notes due 2025 and 1.375% Notes due 2029 for AAPL?", "ground_truth_answer": "May 24, 2017", "relevant_source_ids": ["AAPL_10K_0212"], "difficulty": "easy"}
+{"question": "What method does AAPL use to recognize revenue for product-related bundled services?", "ground_truth_answer": "straight-line basis", "relevant_source_ids": ["AAPL_10K_0154"], "difficulty": "easy"}