pyterrier-tar 0.1.0__tar.gz

pyterrier_tar-0.1.0/BENCHMARKING.md

@@ -0,0 +1,309 @@
+# Reproducible replay benchmarks
+
+The public experiment API runs recorded histories, saves per-topic results,
+and reports uncertainty and failures. The input ranking is fixed before labels
+are attached. This workflow does not train a ranker or select a stopping rule.
+
+## Run and resume an experiment
+
+```python
+from pathlib import Path
+import pyterrier_tar as tar
+
+# trajectory has qid, docno, rank, label and covers the full collection.
+data = tar.ReplayDataset("my-collection", "recorded-ranker", trajectory)
+rules = [
+    tar.RuleSpec("IP-P", lambda c: tar.PoissonPoint(c.target, c.collection_size)),
+    tar.RuleSpec("50 irrelevant", lambda c: tar.ConsecutiveIrrelevant(50),
+                 config={"count": 50}),
+]
+output = Path("artifacts/my-experiment")
+results = tar.run_experiment([data], rules, targets=[.8, .9, .95],
+                             seeds=[0], output=output)
+summary = tar.experiment_summary(results)
+summary.to_csv(output / "summary.csv", index=False)
+(output / "REPORT.md").write_text(tar.experiment_report(results))
+```
+
+A factory receives `RuleContext(trajectory, target, seed, collection_size)`
+and returns a fresh stopping rule. Built-in classifications are inferred from
+the actual rule class using the catalogue, never from the display name:
+
+```python
+tar.RuleSpec("Oracle", lambda c: tar.Oracle(c.target))
+tar.RuleSpec("IP-H", lambda c: tar.IPHyperbolic(c.target, c.collection_size))
+```
+
+Custom classes, including subclasses of built-in rules, require `promise=`:
+`heuristic`, `estimator`, `certificate`, or `offline bound`. This is metadata,
+not an automatically verified guarantee. Existing explicit classifications
+remain supported, but a classification conflicting with a known built-in is
+rejected. An absent `config` defaults to an empty dictionary.
+
+Repetition is a separate setting: `repetitions="once"` uses the first supplied
+seed; `repetitions="seeds"` uses every supplied seed. When omitted, certificates
+default to `"seeds"` and other rules to `"once"`, preserving the earlier run
+counts. For example, `tar.RuleSpec("QBCB", make_qbcb, repetitions="once")`
+runs one control draw while retaining its certificate classification. A custom
+stochastic estimator can use `promise="estimator", repetitions="seeds"`.
+Factories must draw from `context.rng()`, not global randomness. It returns a
+generator keyed by topic, target and seed, so draws are independent across all
+three; `default_rng(context.seed)` alone repeats one stream for every topic and
+target. The ranker is not part of the key, so compared rankings of one topic
+share a draw. Varying ranker seeds
+should be separate named rankings, not extra topics. Resolved `promise` and
+`repetitions` are recorded on every result row; a factory cannot change its
+classification between contexts. Completed tasks resume without rebuilding
+their rule or resampling controls.
+
+`manifest.json` binds the parameters, input-frame hashes, package source hash,
+dependency versions and factory source (when available). An input hash covers
+column names, row order and values, not how pandas stores them, so a run saved
+under one pandas version or platform verifies under another. Include external
+factory parameters and implementation versions in `RuleSpec.config`. Put
+score snapshots and other frame inputs in `ReplayDataset.artifacts` and their
+provenance in `metadata`; the manifest hashes those too. A factory closure is
+not automatically a complete record of its external dependencies.
+
+Completed tasks are atomically written under `tasks/`, with a result checksum.
+`results.csv` is a regenerable export. Repeating the call resumes completed
+tasks; changing the manifest requires a new directory. Failures leave completed
+tasks intact. Only one writer may use a directory. After a process is killed,
+verify that it has stopped before removing its `.writer.lock` directory.
+Manifests use schema 3. A run saved with an earlier schema needs a new output
+directory, and its inputs can no longer be verified for plot-only regeneration.
+Existing result CSVs remain readable by the reporting and plotting APIs.
+
+For a partial history, pass **full-pool qrels including negatives**:
+
+```python
+data = tar.ReplayDataset("collection", "partial-history", history.trajectory,
+                        qrels=full_pool_qrels, control=history.control,
+                        artifacts={"scores": history.score_snapshots,
+                                   "rounds": history.rounds},
+                        metadata=history.metadata)
+```
+
+Omitting qrels declares the trajectory to be the entire collection. The runner
+cannot infer unseen relevant documents. Full-pool qrels provide the denominator
+and collection size; they do not supply missing review events to a rule. Rules
+that need unobserved scores or sampling metadata still require those inputs.
+
+The runner uses `stop_report` once and retains that prefix, avoiding a second
+stochastic decision. Rules must follow the package's prefix-replay contract.
+It obtains screened identities from `rule.control`, `rule.control_documents`,
+and the dataset's baseline controls, then counts their union with the prefix.
+Custom rules charging controls must expose their identities through these APIs.
+
+## Read uncertainty and failures
+
+`experiment_summary` supplies target attainment, Wilson 95% intervals, mean
+review fraction with bootstrap intervals, minimum recall, mean/maximum recall
+shortfall, and counts of actual firings, full-review fallbacks, exhausted partial
+histories, and zero-relevant observations. `worst_topics` retains the topic,
+target and seed of individual failures. `experiment_report` writes Markdown.
+
+For certificate rules, each summary row is **one topic across independent control
+draws**. It separates certification frequency from achieved-target frequency:
+reviewing everything can reach the target without producing a certificate.
+No pooled interval treats many draws on a few topics as many independent topics.
+For once-only non-certificate rules, intervals use distinct topics, with
+duplicates rejected. Explicitly repeated non-certificate rules are summarised
+across seeds within each topic and labelled `seed`, not certificate coverage.
+
+Review intervals use a seeded percentile bootstrap of the same observational
+units; one observation produces no cost interval. Each summary row has its own
+generator, so an interval is the same whether a rule is summarised alone or
+alongside other rules and rankers. These descriptive intervals
+do not adjust for selecting among methods or targets. Topic independence remains
+an assumption. Increase the control draws to estimate coverage precisely; 20
+draws per topic are only a coarse screen. Zero-relevant topics are counted and
+retained with the metrics' recall convention of 1, so inspect their count.
+
+## Compare rankings on the same pool
+
+```bash
+python examples/benchmark_rankings.py --years 2017 --limit 3 --waterloo \
+  --draws 100 --targets .9 --output artifacts/clef-development
+```
+
+This runs IP-P, ConsecutiveIrrelevant, ReviewHalf and QBCB on released AutoTAR,
+BM25 using the topic title, and three seeded random rankings. It writes a
+manifest, resumable tasks, results, summaries, failure rows, Markdown, and PNGs.
+Remove `--limit` and omit `--years` to evaluate all three collections. A limited
+run is a development check, not a new benchmark conclusion.
+
+`--waterloo` adds the released Waterloo B ranking for CLEF 2017, pinned to the
+same source revision and SHA-256 as the existing Kneedle parity check. This
+supplies a second recorded review ranking; it is completed against the same
+pool as AutoTAR. It is not a new training run or a reproduction of the original
+ranking-generation procedure. The releases share 17 topic IDs; this option
+restricts **all** rankers to those shared topics before applying `--limit`,
+and records excluded AutoTAR topics in each dataset's manifest metadata.
+
+`bm25_ranking` is a specified in-memory baseline: lowercase Unicode word tokens,
+no stemming or stopword removal, unique query terms, k1=1.2, b=.75, and
+`log(1 + (N-df+.5)/(df+.5))` IDF. It claims no Terrier scoring parity. Missing-text
+documents remain in the pool with score zero. Random rankings discard labels
+and are stable under input-row permutations. Both are generated before qrels
+are joined. `complete_rankings(pool, ranking)` appends unranked documents by
+docno and rejects foreign documents or absent topics.
+
+To add another recorded learning trajectory, pass `--recorded histories.csv`.
+The CSV needs `dataset,ranker,qid,docno,rank`, with dataset names such as
+`clef2017` and topic-prefixed document IDs matching the CLEF adapter. Each named
+ranker must cover every selected topic for its dataset. Missing documents are
+appended, and the same full pool is used for every ranker. The package does not
+invent an additional learned ranking if one is unavailable.
+
+`examples/results_table.py` delegates to the public runner: `--checkpoint-dir`
+enables resume, and it writes `RESULTS_UNCERTAINTY.md` beside `RESULTS.md` with
+intervals and failures. `RESULTS.md` was regenerated this way. Its certificate
+rows use control draws from `context.rng()`, so they differ by sampling noise
+from the first table; every estimator and heuristic row is unchanged.
+
+## Import review histories
+
+Portable CSV events have `qid,docno,rank,label,round`. `rank` records review
+order; `round` records batch membership, starting from zero or another
+non-negative integer. Extra columns, including recorded sampling probabilities,
+survive. Optional score CSVs have `qid,docno,index,score`, where `index` is the
+number reviewed when the model produced that snapshot.
+
+```python
+history = tar.read_review_history("events.csv", scores="scores.csv")
+# Or import existing DataFrames:
+history = tar.import_review_history(events, score_snapshots=scores)
+```
+
+The TARexp adapter follows the pinned
+[Ledger format](https://github.com/eugene-yang/tarexp/blob/d23724a73175cfacbaa95dbaf160a34bc17a1ac0/tarexp/ledger.py)
+and [workflow score timing](https://github.com/eugene-yang/tarexp/blob/d23724a73175cfacbaa95dbaf160a34bc17a1ac0/tarexp/workflow.py).
+For an already loaded ledger and saved score mapping:
+
+```python
+history = tar.import_tarexp(ledger, docnos=original_dataset_docnos,
+                           qid="topic", saved_scores=saved_scores)
+```
+
+It also accepts the numeric N-by-2 ledger record array. Round -1 becomes a
+separate control frame, round 0 is the seed batch, and NaN rows remain
+unreviewed. Binary `predict_proba` arrays use column 1. Mapping keys are the
+completed round numbers; snapshots become cumulative review indices.
+`docnos` **must follow the original dataset row order**.
+
+For a native TARexp `it_N` directory, with TARexp installed:
+
+```python
+history = tar.read_tarexp("saved_run/it_10", docnos=original_dataset_docnos,
+                         qid="topic", trusted=True)
+```
+
+Native TARexp files are gzipped Python pickles and may execute code; only load
+trusted checkpoints. The importer loads the ledger and workflow configuration.
+Portable CSV import needs neither pickle nor TARexp installed.
+
+TARexp records batch membership, not within-batch review order. Its imported
+order is original dataset row order inside each batch. Document-level stopping
+inside such a batch is not a reconstructed historical decision. For Quant,
+use the preserved endpoints and model snapshots:
+
+```python
+rule = tar.QuantCI(.9, min_documents=1,
+                   score_snapshots=history.score_snapshots,
+                   review_checkpoints=history.rounds)
+```
+
+This replaces fixed-size checkpoint spacing, while retaining `min_documents`.
+No future snapshot is used at an earlier checkpoint. Missing score rounds are
+listed in metadata; Quant uses its documented latest-available-snapshot policy,
+which is not per-round model parity when snapshots are missing. Scores remain
+recorded scores: importing values in [0,1] does not establish calibration.
+Quant's ranking must cover the full scored collection; a partial history or a
+history with omitted control documents does not suffice for its denominator.
+
+## Plot diagnostics
+
+For an executable walkthrough with inline figures, open
+[demo_publication_figures.ipynb](examples/notebooks/demo_publication_figures.ipynb).
+It uses the saved development run, verifies its inputs, and exposes the dataset,
+target, topic and export settings near the top.
+For fresh rankings and stopping decisions across ten topics by default, use
+[demo_multitopic_rankers_stopping.ipynb](examples/notebooks/demo_multitopic_rankers_stopping.ipynb).
+Set `N_TOPICS` to another count or `None` for the full year. `PLOT_QID` and
+`DIAGNOSTIC_RULE` select only the paired diagnostic; the overview uses all topics.
+
+Install the optional `plot` extra. For publication figures, select one dataset
+and target, then use the panel constructors and explicit export function:
+
+```python
+fig = tar.figure_recall_effort(topic_trajectory, topic_results, qrels=full_pool_qrels)
+tar.save_figure(fig, "figures/recall_effort")
+fig = tar.figure_target_misses(selected_results, n=15)
+tar.save_figure(fig, "figures/target_misses")
+fig = tar.figure_control_cost(selected_results)
+tar.save_figure(fig, "figures/control_cost")
+# All topics and methods for one ranker, dataset and target:
+ranker_results = selected_results.loc[selected_results.ranker == "BM25"]
+fig = tar.figure_stopping_outcomes(ranker_results)
+tar.save_figure(fig, "figures/all_topics")
+# One topic and method, separating ranked-prefix and total metrics:
+fig = tar.figure_stopping_diagnostic(
+    topic_trajectory, topic_results.loc[topic_results.rule == "QBCB"], qrels=full_pool_qrels)
+tar.save_figure(fig, "figures/qbcb_diagnostic")
+```
+
+Figures are 180 mm wide with 8–10 pt type, a white background, percentage axes,
+shared panel scales, and legends in a separate strip. Cost components have
+distinct colours and a hatch pattern. Each stopping rule has its own
+combination of marker colour and shape: the benchmark's four rules keep fixed
+ones, and any other rule takes one set by its name, so it looks the same
+across figures. Crosses are reserved for non-firing terminations.
+PDF exports embed TrueType fonts, SVG retains editable text, and PNG uses
+600 dpi. `save_figure` takes an output stem and appends each format, so
+`figures/cost_target0.9` exports `cost_target0.9.pdf`. These are journal-neutral defaults; journal-specific sizing can be
+applied to the returned Figure. `save_figure` preserves the requested physical
+width instead of cropping the canvas to a different size. Matplotlib's
+[constrained layout](https://matplotlib.org/stable/users/explain/axes/constrainedlayout_guide.html)
+allocates space for labels and panels; rendered layout tests check for clipped
+or overlapping text. The lower-level `plot_*` functions still return axes and
+accept an existing `ax=` for custom compositions.
+
+Recall panels separate the stopping rules, so nearby outcomes cannot hide a
+different method. The gain curve is a step function using offline labels. Stop
+markers now use only prefix cost and prefix recall, so they lie on the curve.
+Crosses mark non-firing terminations. Larger markers indicate coincident
+outcomes without adding overlapping text; coordinates are never jittered.
+For total cost and recall including controls, use `figure_stopping_outcomes`:
+it shows every topic/seed outcome in a panel per method, with no ranking curve.
+An open diamond shows one mean for each topic with repeated outcomes, regardless
+of its draw count. Panel titles distinguish topics from outcomes; exact
+coincidences may overlap. Means are not confidence intervals and a mean above
+the target does not imply that all its draws succeeded. The paired diagnostic
+puts ranked-prefix stops on the left and total outcomes on the right.
+These visual explanations do not change
+the stopping decisions or give a rule access to future labels.
+
+The cost chart separates rankers, and stacks prefix cost and *additional*
+controls to avoid double counting. It averages repeated draws within topics
+before averaging across topics. Open circles show individual topic mean total
+costs, not confidence intervals; slight vertical offsets keep topics distinct
+without altering cost values. Failure panels select the largest individual
+misses globally before grouping by ranker, retaining topic and control-draw
+identity. Their caption reports the selection and shortfall definition.
+
+Existing results can be re-rendered without running an experiment:
+
+```bash
+python examples/render_benchmark_figures.py --input artifacts/clef-validation
+```
+
+This refreshes the existing figures, adds an all-topic outcome figure per ranker
+and a paired diagnostic, and writes PDF/SVG versions, `FIGURE_CAPTIONS.md`
+and `FIGURE_PROVENANCE.json`. It verifies the stored result hash when available
+and the reconstructed AutoTAR trajectory against the experiment manifest.
+Numerical results, task checkpoints and the original manifest remain unchanged.
+Other recall rankings can use `--trajectory` with the original full input CSV;
+its hash must match the saved input. `--dataset`, `--target`, `--ranker`, `--qid`
+and `--output` select the cohort and destination. The publication formatting
+does not turn a limited development run into a general scientific result.

pyterrier_tar-0.1.0/CHANGELOG.md

@@ -0,0 +1,143 @@
+# Changelog
+
+## 0.1.0
+
+### Benchmark workflows
+
+* `figure_stopping_outcomes` compares total screening effort and recall across
+  topics in separate method panels, showing individual outcomes and one mean
+  per repeated topic. `figure_stopping_diagnostic` separates ranked-prefix
+  stops from total outcomes in two panels. Recall-curve markers now use prefix
+  cost and recall, so they lie on the curve; total outcomes include controls
+  only in their separate scatter panels. Fresh-ranking notebooks expose both
+  views, with ten topics by default in the multi-topic example.
+
+* `RuleSpec` infers built-in classifications from the catalogue and defaults
+  `config` to an empty dictionary. Custom classes require an explicit promise;
+  conflicting built-in classifications are rejected. `RuleContext.rng()` gives
+  each topic, target and seed its own generator. `repetitions="once"`
+  or `"seeds"` controls execution independently, with the previous repetition
+  counts as defaults. Repeated non-certificate methods are reported across
+  seeds within each topic. Manifests use schema 3, whose input hashes follow
+  values rather than pandas storage, so they verify across pandas versions and
+  accept array-valued columns such as `features`. Earlier manifests require a
+  fresh run directory; existing saved result tables can still be plotted and
+  summarised.
+
+* Publication figures separate rankers and stopping rules into labelled panels,
+  retain exact data coordinates, and keep legends outside the data area. PDF,
+  editable SVG and 600-dpi PNG exports use a 180-mm print width. Plot-only
+  regeneration verifies saved inputs and preserves the numerical experiment.
+  Every rule has its own marker colour and shape, export stems may contain
+  dots, and outcome columns reloaded as 0/1 are accepted.
+
+* Public `ReplayDataset`, `RuleSpec` and `run_experiment` APIs save per-topic
+  results, input/configuration hashes and atomic task checkpoints for resume.
+  Controls are counted once; exhausted partial histories remain distinct from
+  full-review fallbacks.
+* `experiment_summary`, `experiment_report` and `worst_topics` expose intervals,
+  recall shortfalls and termination counts. Certificate intervals remain
+  within-topic across draws, with certification frequency reported separately.
+  Each summary row seeds its own bootstrap, so its review interval does not
+  depend on which other rules or rankers are summarised with it.
+* Label-independent BM25/random baselines and completion of external rankings
+  support same-pool comparisons. `examples/benchmark_rankings.py` runs CLEF
+  comparisons and exports results and figures; `results_table.py` now uses the
+  shared runner and writes an uncertainty companion.
+* Portable CSV and native TARexp history import preserve batches, controls,
+  score timing and missing-history boundaries. Quant/QuantCI accept explicit
+  recorded review checkpoints.
+* Optional matplotlib diagnostics show gain curves/stops, target misses and
+  screening costs. Usage and evidence boundaries are in `BENCHMARKING.md`.
+
+### Rules
+
+* `CLEFTARDataset.complete_ranking` takes the pool to be every judged document
+  as well as every document with text. Ten CLEF topics judge documents the
+  archive has no text for (2,384 in CD009263, two of them relevant); it
+  rejected the released AutoTAR ranking for those topics, and would have left
+  such documents out of a completed ranking.
+* Every rule with a recall target takes `target_recall` first and requires
+  it. `IPHyperbolic` and `PoissonPoint` take `(target_recall, collection_size)`
+  like `CMHHeuristic`; `QBCB` and `TargetRecapture` no longer default the
+  target; `BaselineInclusionRate` takes `(baseline, target_recall,
+  collection_size)`. `Quant` and the EVPI variants list their real parameters.
+* `AutoStop` takes `collection_size`, which `strict_v2` now requires. It
+  previously used the sample size as the population, so its finite-population
+  correction shrank the variance towards zero and it stopped too early; at the
+  final checkpoint it equalled `loose`. `published_autostop_clef2017.py` now
+  checks the rule's public replay rather than its internal estimator.
+* Every match of a query across frames compares `qid` as text: qrels in
+  `stopping_metrics`, control sets, `QuantCI` snapshots, `SAFE`'s key papers
+  and seed counts, and `review_fraction`'s size mapping. A ranking keyed `1`
+  against qrels keyed `'1'` previously scored recall 1.0 on every topic, and a
+  control rule reported no controls and zero control cost. `stopping_metrics`
+  now raises when no ranked query appears in the qrels at all.
+* `SAFE` needs a `docno` column only when it has key papers.
+* `BetaBinomial`, `SAFE`, `stopping_metrics`, and `stopping_frontier` take an
+  optional `collection_size` (per query for `SAFE` and the metrics). Without it
+  they treat the trajectory as the whole collection, as before, which on a
+  truncated ranking understates what remains, lowers SAFE's minimum depth, and
+  inflates `review_fraction` and `loss_e`. The README states the contract.
+* `QBCB` and `TargetRecapture` say when a positive control is missing from the
+  ranking, instead of reporting only that the control set cannot certify.
+* Trajectories with a missing label are rejected instead of reading it as
+  irrelevant, and `Chao` rejects fractional capture counts instead of
+  truncating them. `Chao`'s documentation now says it uses classic Chao1,
+  bias-corrected only when `f2 = 0`.
+* `Kneedle` and `Budget` take `knee_distance`, defaulting to `'absolute'`: the
+  form every released TAR implementation uses, including the one behind the
+  86,243-effort figure. `'signed'` keeps Satopää et al.'s Kneedle. The two agree
+  on that run but diverge sharply on the CLEF AutoTAR rankings.
+* `IPHyperbolic` reproduces the released loop's empty-tail stop and its
+  dynamic relevant-document gate, so all nine CLEF 2017--19 settings now match
+  the published recall, cost, reliability, loss, and relative error. `tail`
+  selects the released expected-remaining formula (the default, which
+  understates the tail by `(1-b)^2`) or the integral of the fitted rate.
+* `SCAL` selects S-CAL's cutoff from the Horvitz-Thompson total over the whole
+  recorded sample, and charges the sample after the cutoff as review cost.
+  `control_documents()` lets `stopping_metrics` count it towards recall too.
+  It previously fired at the first checkpoint of a fully reviewed stratum.
+* `BaselineInclusionRate` no longer reports a fired stop at position 0 when its
+  pilot sample holds no relevant documents.
+* `BatchPrecision` counts a batch at the cutoff as low precision, as TARexp does.
+* `QuantCI` accepts recorded per-round `score_snapshots`, and raises when a
+  query is missing from them rather than reporting it as never certified.
+* `GRLStop.save` writes a configuration sidecar and stages both files before
+  publishing either; `load` restores the trained configuration, keeps this
+  instance's `target_recall` and `deterministic`, warns outside the trained
+  targets, and refuses metadata whose recorded digest does not match the policy.
+  `early_stopping` selects the released callback or a rollout-mean variant.
+* Round-off fixes: `ceil(.07 * 100)` no longer inflates a recall target, and
+  `BetaBinomial` no longer loses an allowed miss to floating point.
+
+### Usability
+
+* `catalogue()` lists every rule with its family, what it promises, what it
+  needs beyond a labelled trajectory, and its source; a test keeps it in step
+  with the exports.
+* A non-numeric `label` column now explains what a label should be instead of
+  surfacing pandas' parse error.
+
+### Evaluation
+
+* `RESULTS.md`, written by `examples/results_table.py`, reports every rule
+  that can run on the CLEF release: reliability and review at targets 0.8,
+  0.9, and 0.95 per collection, grouped by promise and not ranked.
+  `RESULTS_UNCERTAINTY.md` gives the intervals and largest shortfalls.
+* `reliability_summary` and `coverage_summary`, because the families promise
+  different things: an estimator's promise is across topics, a certificate's is
+  across repeated control draws. `EVALUATION.md` says which to use for each.
+* `published_kneedle_clef2017.py`, `published_point_process_clef2017.py`,
+  `published_autostop_clef2017.py`, and `published_baselines_clef.py` join the
+  parity scripts; all seven run weekly in CI and from `parity_published.ipynb`.
+
+### Layout
+
+* Each rule has its own file, grouped by what it reads: `rules/trajectory/`,
+  `rules/checkpoint/`, `rules/control/`, `rules/estimation/`. README documents
+  how to add one.
+* Five notebooks, each either a `demo_` or a `parity_`.
+* `GAPS.md` collects what the package does not implement or reproduce;
+  `EVALUATION.md` how to measure each family; `PUBLISHED_EVIDENCE.md` the parity claims and the
+  dated re-verification log.

pyterrier_tar-0.1.0/EVALUATION.md

@@ -0,0 +1,29 @@
+# How to evaluate a stopping rule
+
+Stopping rules in this package promise three different things, and a single
+leaderboard hides that. What a rule promises decides what would falsify it.
+
+| Family | Rules | Promise | The check |
+| --- | --- | --- | --- |
+| Heuristic | Kneedle, Budget, Rule2399, ReviewHalf, FixedRound, BatchPrecision, ConsecutiveIrrelevant, SAFE | none about recall: they detect a flattening gain curve or exhaust a budget | the cost/recall trade-off, and excess cost over the oracle depth |
+| Estimator | CMH, AnytimeCMH, IP-P, IP-H, Quant/QuantCI, AutoStop, SCAL, Chao, BetaBinomial | recall at or above the target **if its model holds** | `reliability_summary`: the share of topics reaching the target, with a Wilson interval, against the nominal level |
+| Certificate | QBCB, TargetRecapture | recall at or above the target with probability `1 - alpha` **over the draw of the control sample** | `coverage_summary`: the share of repeated draws that certified the target, for one review |
+
+The unit of repetition is the difference that matters. An estimator's promise
+is about topics, so it is measured across them. A certificate's promise is
+about the sample it drew, so it is measured by drawing again: coverage across
+replications of one review. Measuring a certificate across topics answers a
+question nobody asked.
+
+Costs are comparable only when the control screening is charged. `stop_report`
+and `stopping_metrics` already count it once; a certificate that costs more
+than a heuristic is buying a guarantee the heuristic never offers.
+
+The public `run_experiment` workflow applies these distinctions to saved runs.
+`experiment_summary` reports topic intervals for non-certificates and per-topic
+draw intervals for certificates, alongside shortfalls and non-firing terminations.
+`experiment_report` writes the tables and largest failures. See
+[BENCHMARKING.md](BENCHMARKING.md) for resume, input provenance, ranker comparisons,
+history import and plotting. Intervals are descriptive; full-review successes
+are counted separately from certifications, and zero-relevant observations are
+explicitly identified.

pyterrier_tar-0.1.0/GAPS.md

@@ -0,0 +1,37 @@
+# Known gaps
+
+What this package does not implement or cannot reproduce, and why. Parity
+claims live in [PUBLISHED_EVIDENCE.md](PUBLISHED_EVIDENCE.md); this file is the
+list of what is missing.
+
+## Published results that cannot be reproduced
+
+These need data the authors did not release. Nothing in the package should
+claim them.
+
+| Result | Missing input |
+| --- | --- |
+| QBCB's RCV1 figures | the 20% subset, seeds, active-learning trajectories, and control-sample identities |
+| Yang et al. (2021) heuristics on RCV1 | the saved review histories |
+| Adaptive AutoStop | the per-draw sampling distributions, which a trajectory cannot reconstruct |
+| Chao et al. Table 7 integer displays | the analysis and formatting code; raw aggregates do not round to the printed integers (57.566 prints as 57) |
+| The `Knee` rows of the point-process baseline table | the 2018 qrels are rebuilt in that repository from a PID list it does not ship; `published_baselines_clef.py` reports the difference instead of asserting it |
+| `TM` and `TM-adapted` rows of the same table | the released target method draws its target set with an unseeded `random.choice` |
+| RLStop and GRLStop paper results | no trained policies or result tables were released |
+
+## Methods named but not implemented
+
+| Name | State |
+| --- | --- |
+| Score-distribution stopping (Hollmann and Eickhoff) | not implemented; `QuantCI` over calibrated probabilities is the nearest rule |
+| The adapted target method of the point-process papers | not implemented; `TargetRecapture` is Cormack and Grossman's original target method |
+| TARexp-style Knee and Budget | TARexp takes the maximum slope ratio over every round split, measured in rounds. This package follows Cormack and Grossman's knee. An opt-in mode would be needed for TARexp-equivalent numbers. |
+| Adaptive AutoStop, a live S-CAL controller | out of scope: this package replays recorded trajectories and does not select documents |
+
+## Engineering and release
+
+| Gap | Notes |
+| --- | --- |
+| CLEF archives come from one mirror | `npai.science.uu.nl`, hash-pinned. The hashes protect the content, not its availability |
+| 9 of 119 SYNERGY datasets will not compose | missing source files or server errors at the pinned commit |
+| Parity runs weekly, archive audits by hand | the `published-parity` job downloads about 1 GB; the archive notebooks need artifacts you supply |