retrieval-eval-gate 0.1.0__tar.gz

retrieval_eval_gate-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 jinwovo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

retrieval_eval_gate-0.1.0/PKG-INFO

@@ -0,0 +1,376 @@
+Metadata-Version: 2.4
+Name: retrieval-eval-gate
+Version: 0.1.0
+Summary: Retrieval evaluation that reports what it cannot resolve — intervals, significance, anytime-valid CI gates, and distribution-free guarantees. Zero dependencies, any retriever.
+Author-email: Jinwoo Kwon <kownjinwoo@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 jinwovo
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/jinwovo/rag-eval-gate
+Project-URL: Source, https://github.com/jinwovo/rag-eval-gate
+Project-URL: Issues, https://github.com/jinwovo/rag-eval-gate/issues
+Keywords: rag,retrieval,evaluation,information-retrieval,benchmark,conformal-prediction,anytime-valid,confidence-sequence,ci-gate,beir
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: scipy>=1.10; extra == "dev"
+Dynamic: license-file
+
+# rag-eval-gate
+
+**Retrieval evaluation that tells you what it can't resolve.**
+
+[![ci](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml)
+[![python](https://img.shields.io/badge/python-3.10%20%E2%80%93%203.13-blue)](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml)
+[![dependencies](https://img.shields.io/badge/dependencies-none-brightgreen)](#why-zero-dependencies)
+[![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)
+
+> `Recall@5 = 1.00` on 10 queries is really **`[0.69, 1.00]`**.
+> That `+0.30` lift over BM25 has a p-value floor of **0.25** — no outcome of that
+> comparison could ever have been significant.
+> Both of those are computable **from the published table alone.**
+
+## → [Check a number right now, nothing to install](https://jinwovo.github.io/rag-eval-gate/)
+
+Type in the figure and the query count. It returns the interval that figure supports and,
+for a claimed improvement, whether *any* outcome of that comparison could have reached
+significance. The arithmetic runs in the page; nothing is sent anywhere.
+
+Or from a terminal:
+
+```bash
+pip install retrieval-eval-gate      # the PyPI name; `rag-eval-gate` there is someone else's
+
+rag-eval-gate audit -n 10 -v 1.00 -b 0.70     # audit a table you are reading
+rag-eval-gate power gold.jsonl                # what your own gold set can resolve
+```
+
+Zero dependencies. Works with any retriever in any language. Runs as a GitHub Action.
+The command is `rag-eval-gate`; the distribution is `retrieval-eval-gate` because the shorter
+name on PyPI belongs to an unrelated project. Both spellings work as the command.
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/anytime-dark.png">
+  <img src="docs/anytime-light.png" alt="Left: a fixed-N 95% confidence interval, checked after every query, misses the true score in 17% of runs at 50 queries and 33% at 300, against 0.5-1.7% for an anytime-valid confidence sequence. Right: a sequential gate spends 31 of its 300-query budget on a clearly broken system and 54 on a clearly good one, but the full budget on one sitting exactly at the threshold.">
+</picture>
+
+---
+
+## Three things your eval is probably not telling you
+
+**A perfect score is not a perfect score.** Ten queries, one relevant document each,
+`Recall@5 = 1.00` — that is ten successes out of ten, and the exact binomial interval
+bottoms out at `(α/2)^(1/n) = 0.69`. The headline is 31 points wide.
+
+**Some comparisons cannot be won.** Your hybrid retriever beats BM25 by `+0.30 Recall@5`.
+On ten queries that is three queries flipping, seven tying — and a paired randomization
+test on three differing pairs has a smallest attainable two-sided p of `2/2³ = 0.25`. Not
+"not significant": *unable* to be, whatever the result.
+
+**Your interval breaks when you look at it.** A 95% interval promises that **one** look at
+a **finished** sample contains the truth 95% of the time. Watching a CI log is not that.
+Over 800 simulated evaluation streams, inspection starting only at n = 30 where the normal
+approximation is defensible:
+
+| metric | queries | fixed-N interval | confidence sequence |
+|---|:---:|:---:|:---:|
+| Recall@5 (0/1) | 300 | **34.0%** | 1.9% |
+| reciprocal rank (discrete, skewed) | 300 | **32.8%** | 2.6% |
+| nDCG@10 (graded) | 300 | **35.1%** | 1.6% |
+| groundedness judge (skewed high) | 300 | **30.0%** | 0.5% |
+
+*Runs where the true mean escapes a nominal 95% interval at some point while it is watched.*
+The fixed-N column grows with stream length — the longer you look, the more chances to be
+wrong. That is the law of the iterated logarithm, and it is why the right-hand column exists.
+
+## Audit a table in five seconds
+
+No gold set, no retriever, no data — the reported means and the query count are enough:
+
+```console
+$ rag-eval-gate audit -n 10 -m recall@5 -v 1.00 -b 0.70
+
+recall@5 = 1.000 on 10 queries
+  that is 10/10 successes
+
+  95% interval   [0.692, 1.000]   width 0.308   (exact binomial)
+  The reported figure is 31 points wide. Quoting it to two decimal places implies a
+  precision 10 queries cannot deliver.
+
+Claimed lift +0.300 over 0.700
+  note: the two means determine it exactly: 3 of 10 queries differ
+
+  smallest attainable p   0.2500
+
+  UNRESOLVABLE. Under every possible overlap of the two systems' successes,
+  a paired test on this many differing queries has a floor above 0.05, so no
+  outcome of that comparison could have reached significance. This is a fact
+  about the sample size, not about the systems.
+```
+
+It exits non-zero when the comparison could not have been significant, so it works as a
+check and not only as something to read. The interval is exact for a 0/1 metric; for a
+graded one the per-query spread is unknown, so it reports the widest interval the mean
+allows — a `[0,1]` variable with mean *m* has variance at most *m*(1−*m*).
+
+The same thing runs in a browser at
+[jinwovo.github.io/rag-eval-gate](https://jinwovo.github.io/rag-eval-gate/), with a
+shareable link for the result.
+
+## Quickstart
+
+```bash
+pip install retrieval-eval-gate
+
+# 1. What can this gold set even measure?  (no retriever needed)
+rag-eval-gate power gold.jsonl
+
+# 2. Score it, with intervals and significance
+rag-eval-gate eval gold.jsonl --modes bm25,vector,hybrid
+
+# 3. Gate a pull request
+rag-eval-gate eval gold.jsonl --gate --min-mrr10 0.85
+```
+
+A gold set is JSONL:
+
+```json
+{"query": "how do I rotate a signing key", "relevant_doc_ids": ["kms-rotation"]}
+```
+
+…or the TREC qrels you already have (`--gold qrels/test.tsv --queries-file queries.jsonl`).
+
+## Works with whatever you built
+
+Three ways in, so the language and shape of your retriever are irrelevant.
+
+**HTTP** — a URL template and a small field map. No assumption about the route or schema:
+
+```bash
+rag-eval-gate eval gold.jsonl \
+  --url-template 'https://search.internal/v2/query?text={query}&strategy={mode}' \
+  --results-path 'data.hits' --id-field 'document_id' --score-field 'relevance'
+```
+
+**A command** — anything that prints JSON to stdout. Go, Rust, a notebook, a shell script:
+
+```bash
+rag-eval-gate eval gold.jsonl \
+  --backend command --command './retrieve --q {query} --mode {mode} --json'
+```
+
+**A TREC run file** — rank the corpus once, offline, on whatever hardware; evaluate as often
+as you like with nothing running:
+
+```bash
+rag-eval-gate eval qrels/test.tsv --backend run-file --run-file runs/bm25.trec
+```
+
+## Four gate policies
+
+```bash
+rag-eval-gate eval gold.jsonl --gate --gate-policy sequential
+```
+
+| policy | fails when | use it when |
+|---|---|---|
+| `point` *(default)* | the mean falls below the threshold | always — the absolute floor |
+| `ci-lower` | the 95% lower bound falls below it | your gold set is big enough that the interval is narrower than the safety margin |
+| `regression` | *also* on a significant paired drop against a recorded run | you have a green baseline to compare against |
+| `sequential` | the anytime-valid verdict is `fail`, or the budget runs out `undecided` | queries cost money or minutes |
+
+`regression` is the sensitive one. An absolute threshold only notices a regression once the
+mean crosses a line someone guessed; a paired test compares the **same queries** before and
+after, so a real drop on two of them is caught while the mean still clears the line.
+
+`sequential` is the cheap one. Because a confidence sequence is valid at every sample size,
+stopping as soon as the verdict is settled is a decision rather than a peek:
+
+| the system's true Recall@5 | verdict | queries scored (of 300) | saved |
+|:---:|:---:|:---:|:---:|
+| 0.98 | pass | 54 | **82%** |
+| 0.95 | pass | 89 | **70%** |
+| 0.85 (exactly at the line) | `undecided` | 293 | 2% |
+| 0.60 | fail | 31 | **90%** |
+
+`undecided` fails. The budget ran out before the evidence arrived, which is a fact about the
+gold set rather than a clean bill of health — and a gate that treats "we could not tell" as
+a pass has quietly stopped gating.
+
+## As a GitHub Action
+
+```yaml
+- uses: jinwovo/rag-eval-gate@v1
+  with:
+    gold-file: eval/gold.jsonl
+    url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
+    gate-policy: sequential
+    min-mrr10: "0.85"
+```
+
+The step summary gets the interval table, the significance verdicts, what the gold set can
+resolve, and a ✅/❌.
+
+## What comes out
+
+```
+queries=300  depth=10  intervals=95%
+
+mode                      recall@5             recall@10                mrr@10               ndcg@10
+--------------------------------------------------------------------------------------------------
+bm25              0.700 [0.35,0.93]    0.800 [0.44,0.97]    0.670 [0.35,0.90]    0.702 [0.38,0.90]
+hybrid            1.000 [0.69,1.00]    1.000 [0.69,1.00]    0.950 [0.70,1.00]    0.963 [0.78,1.00]
+
+paired randomization test vs bm25 (Holm-corrected within each metric)
+  recall@5   hybrid    delta=+0.300 p=0.2500 holm=0.2500  unresolvable (3/10 queries differ; floor p = 0.250)
+  mrr@10     hybrid    delta=+0.280 p=0.1250 holm=0.2500  unresolvable (4/10 queries differ; floor p = 0.125)
+
+resolution of this gold set (hybrid vs bm25, mrr@10):
+  n = 10 queries, per-query difference sd = 0.388
+  smallest detectable effect at 80% power: +0.344
+  queries needed to resolve +0.020: 2,957
+```
+
+`unresolvable` is the word this tool exists to print. It is not a softer way of saying "not
+significant" — it means the queries that differ are too few for *any* outcome to reach
+p < 0.05, which is a property of your labels and no amount of rerunning will change it.
+
+## The statistics, and why each one
+
+| | what | why not the obvious thing |
+|---|---|---|
+| **Exact binomial** (Clopper–Pearson) | intervals for Recall@k when each query has one relevant document | a bootstrap resamples a constant at 10/10 and returns `[1.00, 1.00]`, which reads as certainty |
+| **BCa bootstrap** | intervals for graded metrics | percentile intervals under-cover on skewed samples, and reciprocal rank is very skewed |
+| **Paired randomization test** <br><sub>Smucker, Allan & Carterette, CIKM 2007</sub> | mode comparisons | reciprocal rank is discrete, bounded and skewed; a t-test at n = 10 is not measuring what it claims |
+| **Holm–Bonferroni** | correction across a mode sweep | five challengers against one baseline is five chances at a false win; uncorrected 0.05 is about 0.23 |
+| **Betting confidence sequences** <br><sub>Waudby-Smith & Ramdas, JRSS-B 2024</sub> | the sequential gate | alpha spending needs the looks planned in advance; a CI log is inspected continuously |
+| **Conformal prediction, adaptive sets** <br><sub>Romano, Sesia & Candès, NeurIPS 2020</sub> | `calibrate`: how many passages to send an LLM | a fixed top-K over-spends on easy queries and truncates the hard ones |
+| **Risk-controlling prediction sets** <br><sub>Bates et al., JACM 2021</sub> | `calibrate`: when to abstain | a hand-picked threshold implies a claim about hallucination rate that was never checked |
+| **Prediction-powered inference** <br><sub>Angelopoulos et al., Science 2023</sub> | `ppi_mean`: a valid interval when only a biased LLM judge scored your data | averaging the judge gives a tight interval around the judge's belief, which covers the truth ~0% of the time |
+
+Every one is seeded, so a gate cannot flap because a resample drew differently.
+
+## Certified thresholds
+
+```bash
+rag-eval-gate calibrate gold.jsonl --alpha 0.10 --risk-alpha 0.05
+```
+
+Turns the two constants that decide what a RAG system costs and how often it can make things
+up into thresholds with finite-sample, distribution-free guarantees:
+
+```
+coverage   P(the passages sent to the LLM contain a relevant document)  >= 1 - alpha
+risk       P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta
+```
+
+Measured on 800 queries against a stand-in cross-encoder: **5.0 passages against a fixed 9
+carrying the identical promise — 44% fewer prompt tokens**, because the budget moves to
+where it is needed (3.2 passages when the answer ranked first, 8.4 when it ranked fourth or
+lower). Three splits — one picks the softmax temperature, one fits the quantile, one scores
+both having informed neither — because choosing the temperature on the data the quantile is
+fitted on breaks exchangeability, and the resulting under-coverage is invisible in every
+in-sample number.
+
+A certificate that abstains on everything is valid, useless, and labelled **degenerate**. A
+threshold search that stalls names the calibration size that would settle it.
+
+## When only an LLM has judged your data
+
+Your groundedness score is a judge model's opinion of the system it belongs to. Your
+relevance labels, at any real scale, were written by a model too. Both are cheap and neither
+is trustworthy on its own, and averaging thousands of them does not produce a 95% interval —
+it produces a very narrow interval around whatever the model believes.
+
+`ppi_mean` implements **prediction-powered inference** ([Angelopoulos et al., *Science*
+2023](https://www.science.org/doi/10.1126/science.adi6000), power-tuned as PPI++): hand-label
+a small sample, let the model predict everything, and subtract the bias measured on the
+labelled part.
+
+```python
+from rag_eval_gate import ppi_mean
+
+estimate = ppi_mean(hand_labels, judge_on_those, judge_on_everything_else)
+print(estimate.statement())
+# The judge reports 0.886. Measured against 50 hand labels it is optimistic by 0.081,
+# so the true value is 0.805 [0.781, 0.829] — an interval worth about 474 hand labels,
+# from 50.
+```
+
+Over 1,500 repetitions of the whole label-and-estimate cycle, 50 hand labels against 2,050
+judge-scored items:
+
+| judge behaviour | coverage: judge only | hand labels only | **PPI** | interval width | effective labels *(from 50)* |
+|---|:---:|:---:|:---:|:---:|:---:|
+| flattering, accurate | **0.0%** | 94.0% | **94.3%** | −67% | **474** |
+| flattering, noisy | **0.0%** | 94.2% | **93.5%** | −22% | 84 |
+| unbiased, noisy | 14.5% | 94.7% | **93.7%** | −22% | 85 |
+| uninformative | **0.0%** | 94.8% | **94.3%** | −1% | 51 |
+
+**Validity never depends on the judge being good** — the bias is measured, not assumed. And
+**a useless judge costs nothing**: λ is tuned to minimise variance, so noise gets λ ≈ 0 and
+the estimator falls back to the hand-label mean. That last row is the safety property, not a
+failure.
+
+## Why zero dependencies
+
+This runs inside a CI job that should not need a resolver, a wheel build or a lockfile to
+gate a pull request. The machinery it needs — a regularized incomplete beta, a BCa
+bootstrap, a martingale — is a few hundred readable lines, and `pip install retrieval-eval-gate`
+installs exactly one thing. The test suite cross-checks the hand-rolled special functions
+against SciPy, and skips those checks when SciPy is not installed.
+
+## Tests
+
+```bash
+pip install -e ".[dev]" && pytest
+```
+
+200 tests. Against closed forms (`(α/2)^(1/n)` at `k = n`, `2^(1-k)` for a uniform improvement, R's
+`p.adjust` worked example, the `(n+1)` conformal quantile correction), against an
+independent brute-force permutation reference written inside the test, and — for the
+guarantees — by simulating the whole calibrate-then-deploy cycle and counting how often each
+promise actually breaks.
+
+## Provenance
+
+Extracted from [jinwovo/recall](https://github.com/jinwovo/recall), a hybrid-search and
+grounded-RAG system, after its own eval harness was pointed at its own README and three of
+the headline claims failed. The design notes live there as ADRs 0011–0014.
+
+## License
+
+MIT.