retrieval-eval 0.1.0

package/package.json

@@ -0,0 +1,74 @@
+{
+  "name": "retrieval-eval",
+  "version": "0.1.0",
+  "description": "Your RAG test suite decays every time you re-chunk. Drift detection that tells you which relevance labels are still true after a chunker or corpus change and re-anchors the ones it can, plus deterministic retrieval metrics and a portable, TREC qrels-compatible judgment format. Zero dependencies, no API key.",
+  "keywords": [
+    "rag",
+    "drift",
+    "label-drift",
+    "chunking",
+    "retrieval",
+    "evaluation",
+    "eval",
+    "rag-evaluation",
+    "ndcg",
+    "recall",
+    "precision",
+    "mrr",
+    "map",
+    "qrels",
+    "trec",
+    "information-retrieval",
+    "golden-dataset",
+    "regression-testing",
+    "ci",
+    "llm",
+    "vector-search",
+    "embeddings",
+    "reranking",
+    "benchmark",
+    "typescript"
+  ],
+  "license": "Apache-2.0",
+  "type": "module",
+  "bin": {
+    "retrieval-eval": "./dist/cli.js"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": ["dist", "spec", "README.md", "LICENSE"],
+  "engines": {
+    "node": ">=20.11"
+  },
+  "sideEffects": false,
+  "scripts": {
+    "build": "node ../../scripts/sync-spec.mjs && tsup",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "typecheck": "tsc --noEmit",
+    "test:coverage": "vitest run --coverage"
+  },
+  "devDependencies": {
+    "@types/node": "^22.9.0",
+    "@vitest/coverage-v8": "^2.1.4",
+    "tsup": "^8.3.5",
+    "typescript": "^5.6.3",
+    "vitest": "^2.1.4"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/aton-of-data/retrieval-eval.git",
+    "directory": "packages/retrieval-eval"
+  },
+  "homepage": "https://github.com/aton-of-data/retrieval-eval#readme",
+  "bugs": "https://github.com/aton-of-data/retrieval-eval/issues",
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  }
+}

package/spec/README.md

@@ -0,0 +1,31 @@
+# The retrieval-eval spec
+
+Two file formats and one hash definition. Small on purpose: if implementing this takes more than
+an afternoon in any language, the spec has failed.
+
+| File | Purpose |
+|---|---|
+| [`judgments.schema.json`](judgments.schema.json) | relevance labels that survive a re-chunk |
+| [`report.schema.json`](report.schema.json) | results any CI or dashboard can ingest |
+| [`chunk-id.md`](chunk-id.md) | the content-addressed chunk identity |
+| [`drift.md`](drift.md) | the five label statuses and how they are resolved |
+| [`fixtures/`](fixtures/) | conformance cases every implementation must pass, and [their format](fixtures/README.md) |
+
+## Design rules
+
+1. **JSONL for data, JSON for reports.** Streamable, greppable, diffable in code review.
+2. **A strict superset of TREC qrels.** Any judgments file converts losslessly to
+   `query_id 0 doc_id relevance`, so `trec_eval`, `ir_measures`, BEIR and `ir_datasets` all work.
+3. **Zero required dependencies.** SHA-256 and JSON only, both in every standard library.
+4. **Unknown fields are preserved, never rejected.** Tools must round-trip fields they don't know.
+5. **One canonical way to write a judgments file.** Fields in schema order, unknown fields last in
+   their original order, compact JSON, one object per line. Two implementations writing the same
+   labels produce the same bytes, so a rewrite shows a diff of what changed rather than of the
+   whole file.
+6. **Every implementation passes `fixtures/`.** That is what makes a conformance claim mean
+   something: machine-readable, runnable black-box, stable over time. Their shape is documented
+   in [`fixtures/README.md`](fixtures/README.md), because a normative test suite nobody can read
+   without reverse-engineering it is not a spec.
+7. **Byte-identical applies to judgments files and rendered output.** A JSON report agrees as
+   *data*: JavaScript writes a whole number as `1` and Python writes `1.0`, so reports are
+   compared by parsing, which is what `scripts/check_parity.py` does and says.

package/spec/chunk-id.md

@@ -0,0 +1,51 @@
+# Chunk identity
+
+A relevance label must point at *text*, not at "position 7 of whatever the chunker produced that
+day." Everything `drift` can do follows from that.
+
+## Definition
+
+```
+chunk_id = "c1:" + lowercase_hex( SHA256( canonical )[0..16] )     # 128 bits, 32 hex chars
+```
+
+`canonical` is the UTF-8 encoding of these five fields joined by `U+001F` (unit separator):
+
+```
+doc_uri ␟ doc_revision ␟ ordinal ␟ normalize(text) ␟ chunker_fingerprint
+```
+
+| Field | Notes |
+|---|---|
+| `doc_uri` | canonical and stable (`notion://page/abc`, `s3://bucket/key`). Not a local path. |
+| `doc_revision` | source etag/version, or `normalize(text)`'s own hash when the source has none |
+| `ordinal` | 0-based chunk index within the document, base-10, no padding |
+| `text` | the chunk body, normalized as below |
+| `chunker_fingerprint` | opaque, stable string identifying chunker + settings (e.g. `recursive/512/64`) |
+
+## `normalize(text)`
+
+Exactly three steps, in order. Any deviation breaks cross-language agreement:
+
+1. Unicode **NFC** normalization.
+2. Line endings `\r\n` and `\r` → `\n`.
+3. Strip leading and trailing whitespace from the **whole string** (not per line).
+
+Deliberately *not* done: case folding, punctuation stripping, internal whitespace collapsing.
+Those lose information a retrieval system legitimately depends on.
+
+## `text_sha`
+
+```
+text_sha = "t1:" + lowercase_hex( SHA256( normalize(text) )[0..16] )
+```
+
+Recorded on a judgment so a label is **self-describing**: `drift` can re-anchor it after a
+re-chunk without storing the text itself. Store `chunk_text` too when you want `SPLIT` detection
+and can accept the size; use `text_sha` alone when the corpus is sensitive.
+
+## Why SHA-256 and not BLAKE3
+
+BLAKE3 is faster and would be the better hash. SHA-256 is in the standard library of every
+language that matters, which keeps implementations dependency-free, worth more than throughput
+for files this size. The `c1:`/`t1:` prefixes exist so a future `c2:` can switch.

package/spec/drift.md

@@ -0,0 +1,49 @@
+# Judgment drift
+
+A relevance label is a statement about *text a human read*. When the corpus is re-chunked, the
+chunk ids move but the text mostly does not, so most labels are still sound, and the ones that
+are not need to be identified rather than silently kept.
+
+`drift` classifies every label against a live corpus.
+
+| Status | Meaning | Recoverable? |
+|---|---|---|
+| `VALID` | The labeled `chunk_id` is still present in the corpus. | already fine |
+| `RE_ANCHORABLE` | The id is stale, but the exact labeled text is still a live chunk. | yes, `--fix` |
+| `MERGED` | The labeled text was absorbed into a coarser chunk. | yes, `--fix` |
+| `SPLIT` | The labeled text now spans two or more live chunks. | no, needs re-judgment |
+| `ORPHANED` | The text, or its document, is gone. | no, needs re-judgment |
+
+## Resolution order
+
+1. `chunk_id` present in the corpus → `VALID`.
+2. `text_sha` matches a live chunk exactly → `RE_ANCHORABLE`. Cheapest and most precise, so it
+   is tried first. A match in the same `doc_uri` is preferred over one elsewhere.
+3. Two or more live chunks are substrings of `chunk_text` → `SPLIT`, reporting every covering id.
+4. `chunk_text` is a substring of exactly one live chunk → `MERGED`, re-anchoring to it. When
+   several chunks contain it, the **shortest** wins: it is the tightest evidence.
+5. Otherwise → `ORPHANED`.
+
+Steps 3 and 4 need `chunk_text` on the judgment. With only `text_sha`, re-anchoring still works
+but splits and merges are indistinguishable from orphans, which is the tradeoff for not storing
+corpus text in your labels.
+
+## Why MERGED re-anchors but SPLIT does not
+
+`MERGED` is a sound inference: a chunk containing text a human judged relevant still contains
+the answer, so the coarser chunk is relevant too. It is reported separately from an exact match
+because the new chunk carries extra material, which dilutes precision-style metrics.
+
+`SPLIT` is not sound. The labeled text broke into several chunks and there is no way to know
+which of them carries the answer: possibly only one, possibly all. Assigning the label to all
+of them would inflate recall; picking one would be a guess. So `drift --fix` leaves it, and says
+so.
+
+**`--fix` never invents ground truth.** That rule is worth more than the convenience of a fully
+green report.
+
+## invalid_ratio
+
+The share of labels that are not `VALID`. It counts recoverable labels too, because a metric
+computed before you re-anchor them *is* wrong. The CLI prints the recoverable/needs-a-human
+split alongside it so the number is actionable rather than merely alarming.

package/spec/fixtures/README.md

@@ -0,0 +1,48 @@
+# Conformance fixtures
+
+Every implementation must pass these. That is what makes "parity by spec, not by port" a claim
+rather than a slogan, so the fixtures are normative and this file describes their shape.
+
+Each directory is one case. Inputs use the wire formats the spec already defines; `expected.json`
+holds the answers, and **its shape is per-case rather than fixed**, because the cases check
+different things: some check numbers, some check a classification, some check a refusal.
+
+| File | Meaning |
+|---|---|
+| `judgments.jsonl` | input, per [`judgments.schema.json`](../judgments.schema.json) |
+| `run.jsonl` | input, one `{"query_id", "ranking"}` object per line |
+| `corpus.json` | input, a corpus snapshot: `{"corpus_fingerprint", "chunker_fingerprint", "chunks"}` |
+| `*.tsv`, `*.qrels` | input, TREC or BEIR qrels |
+| `expected.json` | the answers, in the shape the table below gives |
+
+`expected.json` may carry a `note` in any case. It is prose for a human reader, never asserted.
+
+## What each case asserts
+
+| Case | Asserts |
+|---|---|
+| `basic` | `metrics` at `k`, `metrics_k1` at k=1, `per_query` detail, `qrels_lines` |
+| `chunk-id` | vectors for `normalize`, `text_sha` and `chunk_id` |
+| `drift` | per-label `statuses`, `summary` counts, `reanchor` and `split_into` targets |
+| `merge` | `statuses` and `summary` for a re-chunk that merges paragraphs |
+| `qrels` | the judgments a `beir` and a `trec` file convert to, and the `qrels_lines` back out |
+| `strata` | `metrics`, `per_stratum` scores, and `worst_stratum` |
+| `summarize` | `cases`, each with `samples` and the expected measurement |
+| `no-positives` | `queries_without_positives`, and the metrics over what remains |
+| `nothing-scored` | that no metric is emitted and the verdict is `INDETERMINATE` |
+| `duplicate-ranking` | that a repeated key counts once, plus the runs that are refused outright |
+| `unsound-judgments` | that a set `validate` rejects cannot be scored to a `PASS` |
+| `unsorted-queries` | the order validation issues are emitted in |
+
+## Rules a fixture follows
+
+1. **Numbers are given to twelve decimal places.** Both implementations round to 1e-12, so a
+   fixture that agrees to fewer digits is not testing agreement.
+2. **Inputs are the published formats.** A fixture never invents a file type an implementation
+   would not otherwise read.
+3. **A refusal is an answer.** Where the right behaviour is to reject the input, the fixture
+   carries the input file and the exact message, because two implementations that reject the
+   same file with different words are still two tools.
+4. **A case earns its place by catching something.** `unsorted-queries` exists because every
+   other fixture happened to have sorted query ids, which hid a real divergence between the two
+   implementations.

package/spec/fixtures/basic/expected.json

@@ -0,0 +1,43 @@
+{
+  "k": 3,
+  "metrics": {
+    "precision@3": 0.333333333333,
+    "recall@3": 0.5,
+    "ndcg@3": 0.329500902401,
+    "mrr@3": 0.25,
+    "map@3": 0.291666666667,
+    "hit_rate@3": 0.5
+  },
+  "metrics_k1": {
+    "precision@1": 0.0,
+    "recall@1": 0.0,
+    "ndcg@1": 0.0,
+    "mrr@1": 0.0,
+    "map@1": 0.0,
+    "hit_rate@1": 0.0
+  },
+  "per_query": {
+    "q1": {
+      "precision": 0.666666666667,
+      "recall": 1.0,
+      "ndcg": 0.659001804802,
+      "mrr": 0.5,
+      "ap": 0.583333333333,
+      "hit_rate": 1.0
+    },
+    "q2": {
+      "precision": 0.0,
+      "recall": 0.0,
+      "ndcg": 0.0,
+      "mrr": 0.0,
+      "ap": 0.0,
+      "hit_rate": 0.0
+    }
+  },
+  "qrels_lines": [
+    "q1 0 d1 2",
+    "q1 0 d2 1",
+    "q1 0 d3 0",
+    "q2 0 d5 1"
+  ]
+}

package/spec/fixtures/basic/judgments.jsonl

@@ -0,0 +1,4 @@
+{"query_id": "q1", "query": "query q1", "doc_uri": "d1", "relevance": 2, "labeled_by": "human:fixture"}
+{"query_id": "q1", "query": "query q1", "doc_uri": "d2", "relevance": 1, "labeled_by": "human:fixture"}
+{"query_id": "q1", "query": "query q1", "doc_uri": "d3", "relevance": 0, "labeled_by": "human:fixture"}
+{"query_id": "q2", "query": "query q2", "doc_uri": "d5", "relevance": 1, "labeled_by": "human:fixture"}

package/spec/fixtures/basic/run.jsonl

@@ -0,0 +1,2 @@
+{"query_id": "q1", "ranking": ["d3", "d1", "d2"]}
+{"query_id": "q2", "ranking": ["d6", "d7"]}

package/spec/fixtures/chunk-id/expected.json

@@ -0,0 +1,32 @@
+{
+  "normalize": [
+    {
+      "in": "  hello \r\n world  ",
+      "out": "hello \n world"
+    },
+    {
+      "in": "café",
+      "out": "café"
+    },
+    {
+      "in": "\n\nx\n\n",
+      "out": "x"
+    }
+  ],
+  "text_sha": [
+    {
+      "text": "hello world",
+      "out": "t1:b94d27b9934d3e08a52e52d7da7dabfa"
+    }
+  ],
+  "chunk_id": [
+    {
+      "doc_uri": "s3://b/k",
+      "doc_revision": "v1",
+      "ordinal": 0,
+      "text": "hello world",
+      "chunker_fingerprint": "recursive/512/64",
+      "out": "c1:06095518ff131adc3d109bfcf1521efa"
+    }
+  ]
+}

package/spec/fixtures/drift/corpus.json

@@ -0,0 +1,38 @@
+{
+  "corpus_fingerprint": "recursive/384/64",
+  "chunker_fingerprint": "recursive/384/64",
+  "chunks": [
+    {
+      "chunk_id": "c1:926073213714b1c617790e335bb0fabc",
+      "doc_uri": "notion://page/9f2a",
+      "doc_revision": "rev-7",
+      "ordinal": 0,
+      "text": "Annual plans may be refunded within 30 days of renewal.",
+      "text_sha": "t1:2bd6e363ab6a85703fabe2546b4ab2f8"
+    },
+    {
+      "chunk_id": "c1:4e556228a30930f72ad70a7f040abb44",
+      "doc_uri": "notion://page/9f2a",
+      "doc_revision": "rev-7",
+      "ordinal": 1,
+      "text": "Monthly plans are non-refundable after the billing date.",
+      "text_sha": "t1:943718a0217044cdb953c232f163f931"
+    },
+    {
+      "chunk_id": "c1:a4d7f269f473170d90c13b7e1af802c8",
+      "doc_uri": "notion://page/9f2a",
+      "doc_revision": "rev-7",
+      "ordinal": 2,
+      "text": "Enterprise agreements are governed by the master service agreement.",
+      "text_sha": "t1:a7202b2d8254f135bd469512898ab347"
+    },
+    {
+      "chunk_id": "c1:26d9eb35e49d5b7b76645144e439a694",
+      "doc_uri": "notion://page/9f2a",
+      "doc_revision": "rev-7",
+      "ordinal": 3,
+      "text": "Refund eligibility for enterprise plans is negotiated per contract.",
+      "text_sha": "t1:e3abd792d087bf155aecd300d9c6b311"
+    }
+  ]
+}

package/spec/fixtures/drift/expected.json

@@ -0,0 +1,22 @@
+{
+  "statuses": {
+    "q1": "VALID",
+    "q2": "RE_ANCHORABLE",
+    "q3": "SPLIT",
+    "q4": "ORPHANED"
+  },
+  "summary": {
+    "valid": 1,
+    "re_anchorable": 1,
+    "merged": 0,
+    "split": 1,
+    "orphaned": 1,
+    "invalid_ratio": 0.75
+  },
+  "reanchor": {
+    "q2": "c1:4e556228a30930f72ad70a7f040abb44"
+  },
+  "split_into": {
+    "q3": ["c1:a4d7f269f473170d90c13b7e1af802c8", "c1:26d9eb35e49d5b7b76645144e439a694"]
+  }
+}

package/spec/fixtures/drift/judgments.jsonl

@@ -0,0 +1,4 @@
+{"query_id": "q1", "query": "refund window annual", "doc_uri": "notion://page/9f2a", "chunk_id": "c1:926073213714b1c617790e335bb0fabc", "text_sha": "t1:2bd6e363ab6a85703fabe2546b4ab2f8", "relevance": 2, "corpus_fingerprint": "recursive/384/64", "labeled_by": "human:ana"}
+{"query_id": "q2", "query": "monthly refund", "doc_uri": "notion://page/9f2a", "chunk_id": "c1:a718502bc7696d9dad49d30a6d5f3ee5", "text_sha": "t1:943718a0217044cdb953c232f163f931", "relevance": 1, "corpus_fingerprint": "recursive/512/64", "labeled_by": "human:ana"}
+{"query_id": "q3", "query": "enterprise refund terms", "doc_uri": "notion://page/9f2a", "chunk_id": "c1:3dfd8e297d137dc87cdceca26f5fa07c", "text_sha": "t1:0973fc56d32bfc9d45473dac002029c6", "chunk_text": "Enterprise agreements are governed by the master service agreement.\n\nRefund eligibility for enterprise plans is negotiated per contract.", "relevance": 2, "corpus_fingerprint": "recursive/512/64", "labeled_by": "human:ana"}
+{"query_id": "q4", "query": "removed policy", "doc_uri": "notion://page/deleted", "chunk_id": "c1:41a0ba2c26579c92ffb5486781200d7b", "text_sha": "t1:4b6f2850b5f76bdae351b46d52358a5c", "relevance": 1, "corpus_fingerprint": "recursive/512/64", "labeled_by": "human:ana"}

package/spec/fixtures/duplicate-ranking/expected.json

@@ -0,0 +1,56 @@
+{
+  "k": 5,
+  "note": "A retriever that returns the same key twice must not be paid twice for it. q1 returns d1 five times and finds one of two relevant documents, so recall is 0.5, not 2.5. Counted naively every repeat scores as a fresh hit and recall, map and ndcg all rise above 1.0, which can carry a failing run past a gate. Only the first occurrence counts, and the report names the queries it happened to rather than correcting them silently.",
+  "queries_with_duplicates": [
+    "q1",
+    "q2"
+  ],
+  "deduped_rankings": {
+    "q1": [
+      "d1"
+    ],
+    "q2": [
+      "d9",
+      "d3"
+    ]
+  },
+  "metrics": {
+    "precision@5": 0.2,
+    "recall@5": 0.75,
+    "ndcg@5": 0.72858220535,
+    "mrr@5": 0.75,
+    "map@5": 0.5,
+    "hit_rate@5": 1.0
+  },
+  "verdict": "PASS",
+  "reason": "2 queries repeated a key in their ranking; only the first occurrence of each was counted",
+  "rejected": {
+    "note": "These are malformed rather than merely unsound, so parse_run refuses them outright. Two entries for one query mean the file no longer says what that query's ranking is, and picking one silently would be a guess.",
+    "cases": [
+      {
+        "file": "run-duplicate-query.jsonl",
+        "message": "run:2: duplicate entry for query q1, already on line 1"
+      },
+      {
+        "file": "run-non-string-key.jsonl",
+        "message": "run:1: ranking[1] must be a non-empty string"
+      }
+    ]
+  },
+  "per_query": {
+    "q1": {
+      "note": "d1 five times: one of two relevant documents found, so recall is 0.5",
+      "recall": 0.5,
+      "precision": 0.2,
+      "mrr": 1.0,
+      "ap": 0.5
+    },
+    "q2": {
+      "note": "d9,d3 twice each: the one relevant document found at rank 2",
+      "recall": 1.0,
+      "precision": 0.2,
+      "mrr": 0.5,
+      "ap": 0.5
+    }
+  }
+}

package/spec/fixtures/duplicate-ranking/judgments.jsonl

@@ -0,0 +1,3 @@
+{"query_id": "q1", "query": "how do I reset my password", "doc_uri": "d1", "relevance": 2, "labeled_by": "human:fixture"}
+{"query_id": "q1", "query": "how do I reset my password", "doc_uri": "d2", "relevance": 1, "labeled_by": "human:fixture"}
+{"query_id": "q2", "query": "where is my invoice", "doc_uri": "d3", "relevance": 2, "labeled_by": "human:fixture"}

package/spec/fixtures/duplicate-ranking/run-duplicate-query.jsonl

@@ -0,0 +1,2 @@
+{"query_id": "q1", "ranking": ["d1"]}
+{"query_id": "q1", "ranking": ["d2"]}

package/spec/fixtures/duplicate-ranking/run-non-string-key.jsonl

@@ -0,0 +1 @@
+{"query_id": "q1", "ranking": ["d1", 42]}

package/spec/fixtures/duplicate-ranking/run.jsonl

@@ -0,0 +1,2 @@
+{"query_id": "q1", "ranking": ["d1", "d1", "d1", "d1", "d1"]}
+{"query_id": "q2", "ranking": ["d9", "d3", "d9", "d3"]}

package/spec/fixtures/merge/corpus.json

@@ -0,0 +1,22 @@
+{
+  "corpus_fingerprint": "recursive/384",
+  "chunker_fingerprint": "recursive/384",
+  "chunks": [
+    {
+      "chunk_id": "c1:d577264d22f91060a5abc8d3df2b1cea",
+      "doc_uri": "wiki://refunds",
+      "doc_revision": "r3",
+      "ordinal": 0,
+      "text": "Annual plans may be refunded within 30 days of renewal.\n\nMonthly plans are non-refundable after the billing date.",
+      "text_sha": "t1:fe457555142c231ba419124e398f4b84"
+    },
+    {
+      "chunk_id": "c1:59b4853f2ca27466c9d2931c73ae489c",
+      "doc_uri": "wiki://refunds",
+      "doc_revision": "r3",
+      "ordinal": 1,
+      "text": "Enterprise refunds are negotiated per contract.",
+      "text_sha": "t1:4029941ea9e639a37fc53f410246171f"
+    }
+  ]
+}

package/spec/fixtures/merge/expected.json

@@ -0,0 +1,21 @@
+{
+  "statuses": {
+    "m1": "MERGED",
+    "m2": "MERGED",
+    "m3": "RE_ANCHORABLE"
+  },
+  "summary": {
+    "valid": 0,
+    "re_anchorable": 1,
+    "merged": 2,
+    "split": 0,
+    "orphaned": 0,
+    "invalid_ratio": 1.0
+  },
+  "reanchor": {
+    "m1": "c1:d577264d22f91060a5abc8d3df2b1cea",
+    "m2": "c1:d577264d22f91060a5abc8d3df2b1cea",
+    "m3": "c1:59b4853f2ca27466c9d2931c73ae489c"
+  },
+  "note": "Both m1 and m2 re-anchor to the same coarser chunk: it now contains both paragraphs."
+}

package/spec/fixtures/merge/judgments.jsonl

@@ -0,0 +1,3 @@
+{"query_id": "m1", "query": "annual plan refund window", "doc_uri": "wiki://refunds", "chunk_id": "c1:5e7383036261e5a707b24dcfa6f591d0", "text_sha": "t1:2bd6e363ab6a85703fabe2546b4ab2f8", "chunk_text": "Annual plans may be refunded within 30 days of renewal.", "relevance": 2, "corpus_fingerprint": "recursive/512", "labeled_by": "human:ana"}
+{"query_id": "m2", "query": "are monthly plans refundable", "doc_uri": "wiki://refunds", "chunk_id": "c1:92f79d5e1a7e764d0493fd9808a4a883", "text_sha": "t1:943718a0217044cdb953c232f163f931", "chunk_text": "Monthly plans are non-refundable after the billing date.", "relevance": 2, "corpus_fingerprint": "recursive/512", "labeled_by": "human:ana"}
+{"query_id": "m3", "query": "enterprise refund policy", "doc_uri": "wiki://refunds", "chunk_id": "c1:c3e4013ec7956f184dcfbdf05a971e26", "text_sha": "t1:4029941ea9e639a37fc53f410246171f", "chunk_text": "Enterprise refunds are negotiated per contract.", "relevance": 1, "corpus_fingerprint": "recursive/512", "labeled_by": "human:ana"}

package/spec/fixtures/no-positives/expected.json

@@ -0,0 +1,17 @@
+{
+  "k": 3,
+  "note": "q2 has no label at relevance >= 1, so it is excluded from the averages rather than scored zero. Recall, nDCG, MRR and AP are undefined when there is nothing relevant to find.",
+  "queries": 2,
+  "queries_scored": 1,
+  "queries_without_positives": [
+    "q2"
+  ],
+  "metrics": {
+    "precision@3": 0.666666666667,
+    "recall@3": 1.0,
+    "ndcg@3": 0.963940433317,
+    "mrr@3": 1.0,
+    "map@3": 0.833333333333,
+    "hit_rate@3": 1.0
+  }
+}

package/spec/fixtures/no-positives/judgments.jsonl

@@ -0,0 +1,4 @@
+{"query_id": "q1", "query": "answerable", "doc_uri": "d1", "relevance": 2, "labeled_by": "human:fixture"}
+{"query_id": "q1", "query": "answerable", "doc_uri": "d2", "relevance": 1, "labeled_by": "human:fixture"}
+{"query_id": "q2", "query": "nothing in the corpus answers this", "doc_uri": "d5", "relevance": 0, "labeled_by": "human:fixture"}
+{"query_id": "q2", "query": "nothing in the corpus answers this", "doc_uri": "d6", "relevance": 0, "labeled_by": "human:fixture"}

package/spec/fixtures/no-positives/run.jsonl

@@ -0,0 +1,2 @@
+{"query_id": "q1", "ranking": ["d1", "d9", "d2"]}
+{"query_id": "q2", "ranking": ["d5", "d6", "d9"]}

package/spec/fixtures/nothing-scored/expected.json

@@ -0,0 +1,9 @@
+{
+  "note": "Every label is below the threshold, so the averages are undefined. The report omits the metrics and the verdict is INDETERMINATE. Emitting 0 would fail an absolute gate over an empty set.",
+  "threshold": 2,
+  "queries": 1,
+  "queries_scored": 0,
+  "status": "INDETERMINATE",
+  "reason": "no query had a label at relevance >= 2, so no metric was computed",
+  "gate_reason": "recall@3:0.5: no query was scored, so 'recall@3' was not computed"
+}

package/spec/fixtures/nothing-scored/judgments.jsonl

@@ -0,0 +1 @@
+{"query_id": "q1", "query": "only a weak label", "doc_uri": "d1", "relevance": 1, "labeled_by": "human:fixture", "stratum": "billing"}

package/spec/fixtures/nothing-scored/run.jsonl

@@ -0,0 +1 @@
+{"query_id": "q1", "ranking": ["d1"]}

package/spec/fixtures/qrels/beir.tsv

@@ -0,0 +1,4 @@
+query-id	corpus-id	score
+q1	d3	1
+q1	d7	2
+q2	d4	1

package/spec/fixtures/qrels/expected.json

@@ -0,0 +1,14 @@
+{
+  "note": "Both files describe the same labels. The BEIR form is 3 columns with a header; the TREC form is 4 columns. A parser that accepts only one of them cannot read the collections it claims to read.",
+  "beir": [
+    { "query_id": "q1", "doc_uri": "d3", "relevance": 1 },
+    { "query_id": "q1", "doc_uri": "d7", "relevance": 2 },
+    { "query_id": "q2", "doc_uri": "d4", "relevance": 1 }
+  ],
+  "trec": [
+    { "query_id": "q1", "doc_uri": "d3", "relevance": 1 },
+    { "query_id": "q1", "doc_uri": "d7", "relevance": 2 },
+    { "query_id": "q2", "doc_uri": "d4", "relevance": 0 }
+  ],
+  "qrels_lines": ["q1 0 d3 1", "q1 0 d7 2", "q2 0 d4 1"]
+}

package/spec/fixtures/qrels/trec.qrels

@@ -0,0 +1,4 @@
+# a TREC collection, with a comment and a negative grade
+q1 0 d3 1
+q1 0 d7 2
+q2 0 d4 -1