retrieval-eval 0.1.0__tar.gz

retrieval_eval-0.1.0/.gitignore

@@ -0,0 +1,20 @@
+node_modules/
+dist/
+coverage/
+.coverage
+*.tsbuildinfo
+__pycache__/
+*.py[cod]
+.venv/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+build/
+*.egg-info/
+.turbo/
+.DS_Store
+.claude/
+
+# generated by scripts/sync-spec.mjs -- the canonical copy is /spec
+packages/retrieval-eval/spec/
+python/retrieval-eval/spec/

retrieval_eval-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

retrieval_eval-0.1.0/PKG-INFO

@@ -0,0 +1,372 @@
+Metadata-Version: 2.5
+Name: retrieval-eval
+Version: 0.1.0
+Summary: Your RAG test suite decays every time you re-chunk. Drift detection that tells you which relevance labels are still true after a chunker or corpus change and re-anchors the ones it can, plus deterministic retrieval metrics and a portable, TREC qrels-compatible judgment format.
+Project-URL: Homepage, https://github.com/aton-of-data/retrieval-eval
+Project-URL: Repository, https://github.com/aton-of-data/retrieval-eval
+Project-URL: Issues, https://github.com/aton-of-data/retrieval-eval/issues
+Project-URL: Changelog, https://github.com/aton-of-data/retrieval-eval/blob/main/CHANGELOG.md
+Author: retrieval-eval contributors
+License: Apache-2.0
+License-File: LICENSE
+Keywords: chunking,drift,embeddings,evaluation,golden-dataset,information-retrieval,label-drift,llm,map,mrr,ndcg,precision,qrels,rag,rag-evaluation,recall,regression-testing,reranking,retrieval,trec,vector-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: mypy>=1.13; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.3; extra == 'dev'
+Requires-Dist: ruff>=0.7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# retrieval-eval
+
+**Every re-chunk quietly invalidates part of your RAG test suite.<br>
+`retrieval-eval` tells you which labels are still true, and repairs the ones it can.**
+
+Label drift detection · deterministic retrieval metrics · a portable, qrels-compatible judgment format<br>
+Zero dependencies · no API key · TypeScript and Python
+
+[![CI](https://github.com/aton-of-data/retrieval-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/aton-of-data/retrieval-eval/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/retrieval-eval?logo=npm&color=cb3837)](https://www.npmjs.com/package/retrieval-eval)
+[![PyPI](https://img.shields.io/pypi/v/retrieval-eval?logo=pypi&logoColor=white&color=3775a9)](https://pypi.org/project/retrieval-eval/)
+[![zero dependencies](https://img.shields.io/badge/runtime%20deps-0-brightgreen)](#why-zero-dependencies)
+[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/aton-of-data/retrieval-eval/blob/main/LICENSE)
+
+</div>
+
+---
+
+A retrieval test suite rests on one assumption nobody writes down: that every label still points
+at the passage a human actually read. Labels are stored as chunk IDs, and chunk IDs are an
+artifact of your chunker. Change the chunk size, the splitter or the overlap, and the IDs move
+while the labels stay where they were. Nothing errors. The metrics keep computing, against ground
+truth that is partly no longer there.
+
+One command shows you how much:
+
+```bash
+retrieval-eval drift --judgments judgments.jsonl --corpus corpus.json
+```
+
+```
+  312 judgments · labeled @ fingerprint recursive/512
+  live corpus    @ fingerprint recursive/384
+
+  ok  188  VALID          chunk_id still present
+  !    47  RE-ANCHORABLE  text moved to a new chunk_id
+  !    24  MERGED         text absorbed into a coarser chunk
+  !    34  SPLIT          labeled text now spans 2+ chunks
+  x    19  ORPHANED       source text or document is gone
+
+  40% of your judgment set no longer matches the live corpus.
+  Any metric computed against it is measuring two changes at once.
+  71 recoverable automatically · 53 need a human
+
+  next    retrieval-eval drift --fix, to re-anchor the recoverable labels
+```
+
+You changed your chunker. Your `recall@5` moved 0.06. You credited the change.
+
+But 40% of your labels no longer point at the text a human actually read. Part of that movement
+was your test suite decaying, not your retriever improving, and no other evaluation tool will
+tell you which part.
+
+`retrieval-eval` separates them, then recovers the labels it can:
+
+```bash
+retrieval-eval drift --judgments judgments.jsonl --corpus corpus.json --fix
+#   fixed   re-anchored 71 label(s) in judgments.jsonl
+#   review  53 label(s) still need a human
+```
+
+`RE_ANCHORABLE` and `MERGED` are recoverable: the text a human judged is still there, either as
+its own chunk or inside a bigger one. `SPLIT` and `ORPHANED` are left alone on purpose, because
+guessing at them would fabricate ground truth, which is the exact failure this tool exists to
+expose.
+
+> **See it happen in 30 seconds.** [`examples/quickstart`](https://github.com/aton-of-data/retrieval-eval/tree/main/examples/quickstart) runs a corpus
+> whose text never changes, re-chunks it, and watches `recall@3` fall from 1.00 to 0.00 while
+> retrieval stays perfect, then brings it back with `--fix`. Six more runnable scenarios, the
+> integration for eight stacks, and complete CI jobs are in [`examples/`](https://github.com/aton-of-data/retrieval-eval/tree/main/examples).
+
+## Why this matters
+
+A golden set is the most expensive thing in a RAG project. Every label is a person reading a
+passage and deciding whether it answers a question. Yet it is the one artifact nothing checks.
+Retrievers get versioned, prompts get diffed, embedding models get benchmarked, and the labels
+underneath all of them are treated as permanent. They are coordinates into one particular
+chunking, and every chunking change moves the map.
+
+The damage is quiet, and it points the wrong way. A chunker that genuinely retrieves better can
+score worse, because its correct answers now live under IDs the labels have never seen. A real
+improvement and the decay it caused can cancel out and read as no change at all. Either way, the
+number you gate releases on is measuring two things at once, and nothing tells you how much of
+each.
+
+Teams that notice end up with three bad options: relabel from scratch, stop trusting the suite,
+or stop changing the chunker. `drift` is a fourth. It names exactly which labels moved, recovers
+every one whose text is provably still there, and sends only the rest to a human. After that,
+your metrics are measuring your retriever again.
+
+## Install
+
+```bash
+pip install retrieval-eval        # or: uvx retrieval-eval --help
+```
+
+Same name, same commands, same flags, same rendered output, same exit codes in both, so one CI
+step works for a polyglot team. Node 20.11 or newer, Python 3.10 or newer.
+
+## What it does
+
+**1. Keeps your labels true when the corpus changes.** Every chunk gets an identity derived from
+its content, not its position ([the one idea](#the-one-idea)), so a label remembers what a human
+read rather than where it used to sit. `drift` classifies every label as `VALID`,
+`RE_ANCHORABLE`, `MERGED`, `SPLIT` or `ORPHANED`, `--fix` repairs the two recoverable classes in
+place, and the command exits non-zero on decay, so a re-chunk cannot ship with a stale suite
+behind it. Run it before `score`, in CI, on every chunking change.
+
+**2. Measures retrieval, deterministically.** `precision@k`, `recall@k`, `ndcg@k`, `mrr@k`, `map@k`,
+`hit_rate@k`. Every metric carries its cutoff in its name, because a number that means
+something other than its name is how a report stops being trustworthy. No model call, no API key, no per-run cost, milliseconds per run.
+
+These are the metrics that localize a failure. High `recall@20` with low `recall@5` means
+retrieval found the answer and ranking buried it, so the reranker is the problem rather than the
+chunker. A single end-to-end score cannot make that distinction.
+
+```bash
+retrieval-eval score --judgments judgments.jsonl --run hits.jsonl -k 5 \
+  --baseline .retrieval-eval/baseline.json \
+  --gate recall@5:-0.02 \
+  --gate worst-stratum:recall@5:0.7
+```
+
+**3. Refuses to let an average hide a broken query class.**
+
+```
+  recall@3 by stratum
+  · legal                0.0000  ░░░░░░░░  n=2  worst
+  · billing              1.0000  ████████  n=3
+
+  gates
+  ok recall@3:0.5
+  x  worst-stratum:recall@3:0.7
+```
+
+Overall recall is 0.60 and passes an average gate. Legal queries retrieve nothing. That is why
+`stratum` is a field in the format, `per_stratum` is mandatory in the report, and worst-stratum
+gating is a first-class gate.
+
+**4. Speaks TREC qrels, so thirty years of tooling just works.**
+
+```bash
+retrieval-eval convert --judgments judgments.jsonl --to qrels | trec_eval -m all_trec - run.txt
+retrieval-eval convert --qrels beir/scifact/qrels/test.tsv --to judgments > judgments.jsonl
+```
+
+A judgments file is a strict superset of qrels, so `trec_eval`, `ir_measures`, `pytrec_eval`,
+BEIR and `ir_datasets` are all one conversion away in either direction.
+
+**5. Tells you when a judgment set is unsound**, not merely malformed:
+
+```
+  warning [no-human-labels] every label is synthetic. Without human labels you cannot measure
+          judge calibration, and synthetic labels quietly become ground truth
+  warning [mixed-fingerprints] judgments span 2 corpus fingerprints; run 'drift' first
+  warning [thin-stratum] stratum 'legal' has only 2 labels, too few to gate on
+  error   [no-positives] no judgment has relevance >= 1, so recall is undefined
+```
+
+## The one idea
+
+Give every chunk an identity derived from its **content**, not its position:
+
+```
+chunk_id = "c1:" + sha256(doc_uri ␟ doc_revision ␟ ordinal ␟ normalize(text) ␟ chunker_fingerprint)[0..16]
+#                    128 bits, written as 32 lowercase hex characters
+```
+
+A label then points at text a human read, so after a re-chunk the tool can say exactly which
+labels survived.
+
+The [spec](https://github.com/aton-of-data/retrieval-eval/tree/main/spec/) is two JSON Schemas and one hash definition, small enough to implement in an
+afternoon in any language. If it ever needs a framework, it has failed. It ships inside both
+packages, so a port never has to start from the GitHub UI.
+
+## What this removes
+
+Each row is a situation teams are already in.
+
+| Situation | What breaks today | What changes |
+|---|---|---|
+| Chunker changed and a metric moved | the whole movement is credited to the retriever | drift separates label decay from retrieval change |
+| Chunk size tuned, labels now point at stale IDs | the golden set is relabeled by hand, or quietly trusted anyway | `--fix` re-anchors every label whose text survived; only the rest need a human |
+| Golden set built in one evaluation tool, team adopts another | labels are rebuilt by hand or abandoned | one conversion, labels outlive the tool |
+| Public collection in TREC qrels, product written in TypeScript | no path into the ecosystem | one conversion, then scored locally |
+| Gate on an aggregate metric | a whole query class can fail silently | worst-stratum gate fails the build |
+| Judged metric gated on a single sample | a non-deterministic instrument reported as a point estimate | confidence interval, plus an `INDETERMINATE` verdict |
+| Synthetic labels accumulating over time | they become ground truth without anyone deciding that | `labeled_by` is required, and `validate` warns |
+| Evaluation platform priced per run | build gates get rationed to stay inside a tier | no model call, no API key, no per-run cost |
+| Evaluation tooling added to a CI image | a dependency tree makes the step slow or fragile | standard library only, in both languages |
+
+## Use it as a library
+
+```ts
+import { drift, score, chunkId, toQrels } from "retrieval-eval";
+
+const result = drift(judgments, corpus);
+if (result.summary.invalid_ratio > 0.1) throw new Error("golden set has decayed");
+
+const { metrics } = score(judgments, run, { k: 5 });
+metrics["recall@5"].value;
+```
+
+```python
+from retrieval_eval import drift, score, chunk_id, to_qrels
+
+result = drift(judgments, corpus)
+if result.summary.invalid_ratio > 0.1:
+    raise RuntimeError("golden set has decayed")
+
+scored = score(judgments, run, k=5)
+scored.metrics["recall@5"].value
+```
+
+## In CI
+
+```yaml
+- run: pip install retrieval-eval
+- run: retrieval-eval drift --judgments eval/judgments.jsonl --corpus eval/corpus.json
+- run: |
+    retrieval-eval score \
+      --judgments eval/judgments.jsonl --run eval/hits.jsonl -k 5 \
+      --baseline eval/baseline.json \
+      --gate recall@5:-0.02 --gate worst-stratum:recall@5:0.7 \
+      --out report.json
+```
+
+Exit `0` passed, exit `1` a gate failed or labels decayed, exit `2` bad invocation or unreadable
+input. `drift` exits non-zero on its own, so it works as a standalone check.
+
+GitLab, CircleCI, Jenkins, Azure Pipelines and pre-commit recipes are in [docs/ci.md](https://github.com/aton-of-data/retrieval-eval/blob/main/docs/ci.md).
+
+## Why this does not already exist
+
+Every evaluation tool scores against a golden set. None of them checks whether the golden set
+still describes the corpus. RAGAS, DeepEval, promptfoo and the IR toolkits all store a label as
+a pointer and trust it forever, because no shared format records which corpus revision or which
+chunker a label was made against. Without that, drift is not merely unreported, it is
+unmeasurable. The judgment format here adds exactly the fields that make it measurable, and
+`drift` is the algorithm that uses them.
+
+The metrics had a smaller gap of their own. Python has `pytrec_eval`, `ir_measures` and BEIR. On npm the package names you would reach for
+first, `ndcg`, `ir-measures`, `trec-eval` and `qrels`, are unpublished, and what does exist is
+thin: [`node-dcg`](https://www.npmjs.com/package/node-dcg) computes DCG alone and was last
+published in 2023, [`trec-eval-wrapper`](https://www.npmjs.com/package/trec-eval-wrapper) shells
+out to the `trec_eval` C binary rather than implementing anything and was last published in 2022,
+and the recent entries are coupled to something else, an MCP server or a recommender engine's own
+core package.
+
+So there are two gaps, in order of importance: **nothing in either ecosystem treats judgment
+drift as a measurable condition**, and **no maintained, standalone, dependency-free IR
+evaluation library exists for JavaScript**. The evidence, including the
+two claims this research falsified against its own earlier draft, is in
+[research/](https://github.com/aton-of-data/retrieval-eval/tree/main/research/).
+
+*(Registry state checked 2026-09-22. If this has changed, the claim is wrong and belongs in an
+issue. Accuracy about our own positioning is part of the point.)*
+
+## It is not a replacement for judged evaluation
+
+Retrieval quality is a large share of end-to-end RAG quality, not all of it. The rest is how well
+the model uses what it retrieved. This covers the retrieval half rigorously and deliberately does
+not claim to certify a RAG system.
+
+Use it alongside RAGAS, DeepEval, promptfoo, Phoenix or Braintrust rather than instead of them. It
+makes their golden sets portable and durable, which is why interoperability is the strategy rather
+than a feature.
+
+For the judged half, the report format carries the honesty those metrics need, and `summarize()`
+computes it: `n`, `stdev` and a confidence interval from repeated samples of the same judge, plus
+a third verdict, `INDETERMINATE`. LLM judges are non-deterministic even at temperature 0, so a
+`ci-lower` gate on a single sample says "I do not know" rather than guessing.
+
+```python
+report.metrics["faithfulness"] = summarize([0.8, 1.0, 0.6, 0.9, 0.7])
+#  value 0.8000, stdev 0.1581, 95% interval [0.6037, 0.9963]
+#  the mean clears a 0.8 floor; the interval says you cannot tell yet
+```
+
+[`examples/judged-metrics`](https://github.com/aton-of-data/retrieval-eval/tree/main/examples/judged-metrics) runs that end to end in both languages.
+
+## Why zero dependencies
+
+Both packages use only their standard library: `sha256` and JSON. A measurement tool that drags in
+a dependency tree becomes a thing you skip installing in CI. It also keeps the spec implementable
+anywhere, which is the point of a spec.
+
+## Status
+
+**0.1.0, alpha.** The formats and the metric mathematics are what we intend to keep. The CLI
+surface may still move. The `c1:` and `t1:` hash prefixes exist so identity can be versioned
+without breaking existing judgment files.
+
+What is already load-bearing, and enforced on every commit:
+
+- **Parity.** Both implementations run the same `spec/fixtures` through both CLIs and must agree
+  on the numbers, the rendered output, the help text and the exit codes. Not two test suites that
+  each pass, one job that runs both and diffs the result.
+- **Determinism.** Same inputs, same output, every run. No network, no model, no clock in the
+  numbers. Across the two implementations: rendered output and judgments files are byte-identical,
+  and `--json` agrees as data rather than as bytes, because JSON writes a whole number as `1` in
+  JavaScript and `1.0` in Python. Compare reports by parsing them, not by checksum.
+- **A quickstart that is executed, not described.** CI runs the README's own first-run path on a
+  clean checkout on Linux and macOS, and asserts the demo still demonstrates its claim.
+
+## Repo layout
+
+```
+spec/                        the contract: 2 schemas, 1 hash definition, shared fixtures
+  fixtures/                  conformance cases BOTH implementations must pass
+packages/retrieval-eval/     TypeScript (tsup · vitest · biome)
+python/retrieval-eval/       Python     (hatchling · pytest · ruff · mypy --strict)
+docs/                        CLI reference, CI recipes, interop, troubleshooting
+examples/                    runnable scenarios, stack integrations, pipeline jobs
+  frameworks/                LangChain, LlamaIndex, Haystack, Chroma, Qdrant, pgvector, RAGAS, DeepEval
+  pipelines/                 Jenkins, GitHub Actions, GitLab, Azure, CircleCI, pre-commit
+research/                    the evidence behind the design
+```
+
+Python and TypeScript are peers, not ports. Both read the same `spec/fixtures` and must agree to
+the twelfth decimal, which is what makes "parity by spec" a claim rather than a slogan.
+
+## Docs
+
+| | |
+|---|---|
+| [docs/cli.md](https://github.com/aton-of-data/retrieval-eval/blob/main/docs/cli.md) | every command, flag, exit code and output mode |
+| [docs/ci.md](https://github.com/aton-of-data/retrieval-eval/blob/main/docs/ci.md) | pipeline recipes and how to keep a baseline |
+| [docs/interop.md](https://github.com/aton-of-data/retrieval-eval/blob/main/docs/interop.md) | qrels, BEIR, `ir_datasets`, and the tools you already run |
+| [docs/troubleshooting.md](https://github.com/aton-of-data/retrieval-eval/blob/main/docs/troubleshooting.md) | every error and warning, and what to do about it |
+| [spec/](https://github.com/aton-of-data/retrieval-eval/tree/main/spec/) | the formats, the hash, the fixtures |
+| [examples/](https://github.com/aton-of-data/retrieval-eval/tree/main/examples) | seven runnable scenarios, eight stack integrations, seven pipeline jobs |
+| [research/](https://github.com/aton-of-data/retrieval-eval/tree/main/research/) | the problem, the market gap, and the impact argument |
+| [CONTRIBUTING.md](https://github.com/aton-of-data/retrieval-eval/blob/main/CONTRIBUTING.md) | add a metric, add an adapter, port the spec |
+
+## License
+
+Apache-2.0