boost-skill-cli 1.0.374__tar.gz → 1.0.375__tar.gz

{boost_skill_cli-1.0.374/boost_skill_cli.egg-info → boost_skill_cli-1.0.375}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.374
+Version: 1.0.375
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.374 → boost_skill_cli-1.0.375}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.374'
-__version_tuple__ = version_tuple = (1, 0, 374)
+__version__ = version = '1.0.375'
+__version_tuple__ = version_tuple = (1, 0, 375)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.374 → boost_skill_cli-1.0.375/boost_skill_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.374
+Version: 1.0.375
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

boost_skill_cli-1.0.375/docs/roadmap/items/eval-corpus-is-one-strangers-repo.md

@@ -0,0 +1,83 @@
+---
+id: eval-corpus-is-one-strangers-repo
+board: code
+section: internals
+status: shipped
+category: Eval · Correctness
+complexity: S
+impact: High
+wow: 3
+note: one third-party repo is 62% of the gate's corpus, and if it disappears every PR goes red
+order: 84
+owner: loop/eval-corpus-availability
+pr: 428
+title: 62% of the required gate's corpus is a single third-party repository
+---
+<b>Pinning the eval corpus to commit SHAs fixed drift. It did not fix concentration, and cannot fix
+availability.</b> The twenty repositories in <code>tests/eval/taps.txt</code> resolve to 10,152
+entries, wildly unevenly: <code>sickn33/antigravity-awesome-skills</code> is <b>6,309 of them
+(62%)</b> and <code>affaan-m/ECC</code> another 1,616 (16%). Two strangers' repositories are
+<b>78%</b> of what this project measures retrieval quality against.
+
+<b>The failure mode is every open PR going red at once.</b>
+<code>ensure_eval_corpus.sh</code> runs under <code>set -euo pipefail</code> and a failed clone takes
+it with it &mdash; verified, not assumed: one unreachable repo in the list exits <b>1</b>, failing
+CI's <code>lint</code> job, a required context. A pinned SHA does not help, because a commit still
+has to be <em>fetchable</em>, and a repo that is deleted, renamed or made private takes its history
+along. Not hypothetical for twenty personal repos: one was already missing from the machine that
+measured this.
+
+<b>It also biases the numbers while everything is up.</b> 62% of the corpus being one publisher's
+house style is a sampling bias in every recall figure the project reports.
+
+<b>Options, none obviously right.</b> <em>Vendor</em> the needed files into the repo &mdash; no
+network, no third parties, but a corpus that no longer resembles a real tap. <em>Cache</em> the
+clones in CI keyed on the taps file, so a disappearance degrades to a stale corpus rather than a red
+gate, though a cold cache still fails. <em>Rebalance</em> so no single source exceeds some share,
+costing corpus size and drawing an arbitrary line. <em>Fail soft</em> on N missing repos, which keeps
+merges flowing but silently weakens the gate &mdash; the failure mode this line of work has been
+removing. Not proposed: dropping or lowering the gate.
+
+<b>Provenance.</b> The per-repo counts had to be printed to verify the pins in
+[[eval-corpus-was-not-actually-pinned]]; the distribution was the surprise.
+
+<b>Measured, and it eliminates one of the four options.</b> &ldquo;Fail soft on N missing repos&rdquo;
+reads like the pragmatic choice. It is the dangerous one, because a corpus that loses a repo does not
+get harder to pass &mdash; it gets easier. Same 91-query required set, one repo removed and everything
+else identical: <code>all 20 &rarr; 10,152 entries, <b>0.852 / 0.473 / 0.605 / 0.657</b></code><br>
+<code>minus sickn33 (62%) &rarr; 3,843, <b>0.885 / 0.593 / 0.711 / 0.746</b></code><br>
+<code>minus that and ECC (78%) &rarr; 2,227, <b>0.967 / 0.659 / 0.769 / 0.814</b></code><br>
+<code>minus LessUp (holds golden targets) &rarr; 9,682, <b>0.676 / 0.374 / 0.483 / 0.523</b></code><br>
+<code>floors &rarr; 0.780 / 0.400 / 0.520 / 0.580</code>
+
+So dropping a <em>scale</em> repo sails through all four floors having measured a third of the
+intended corpus, and dropping a repo that carries golden targets fails all four in a way
+indistinguishable from &ldquo;this PR broke retrieval&rdquo;. Fail-soft is unsafe in both directions.
+The same table also kills <em>rebalancing by trimming</em>: cutting the big repo to fix the 62% would
+inflate every published number. Diluting concentration means adding breadth, which is
+[[eval-corpus-is-96x-smaller-than-a-real-install]]'s problem, not this one.
+
+<b>What shipped.</b> The corpus is now <em>verifiable</em> and its unavailability
+<em>distinguishable</em>. Each <code>taps.txt</code> row carries an entry count beside its SHA, and
+<code>--ensure</code> refuses to leave a corpus that does not match &mdash; so the "quietly measured
+3,843 entries" case above cannot reach the scorer. Unreachable repos exit <b>75</b> (EX_TEMPFAIL)
+rather than 1, and every failing repo is named rather than just the first, so a third party's outage
+stops reading as a regression here. CI restores the clones from a cache keyed on the tap list's
+content, which is what actually stops one deleted repository reddening every open PR; only
+third-party bytes are cached, never anything this repo derives, and there are no
+<code>restore-keys</code> &mdash; a near-miss restore would carry taps this file no longer pins, and
+the index is built from <em>every configured tap</em>. That last hazard is now checked directly:
+a <code>BOOST_HOME</code> holding taps outside the list is a hard failure, which is exactly what
+[[local-boost-install-is-not-the-eval-corpus]] walked into from the other side.
+
+<b>A second, closer bug fell out of it.</b> The sentinel that makes a local <code>make eval</code>
+fast was <em>empty</em> &mdash; it meant &ldquo;some corpus was built here&rdquo;. So editing
+<code>taps.txt</code> (moving a pin, adding a repo) left the previous corpus in place and scored it
+against the new file's baseline, silently, for as long as the sentinel survived. It now carries the
+tap list's digest, so an edit is a cache miss.
+
+<b>What did not ship, and why.</b> The 62% itself. There is no honest way to fix it by removing
+anything, and adding breadth changes every published number and needs its own re-baselining. What is
+enforced instead is a ratchet: <code>MAX_SHARE</code> fails the audit if any single publisher goes
+above 65%, ~3 points over today's measured 62.1%. That stops it getting worse without pretending it
+is fixed.

{boost_skill_cli-1.0.374 → boost_skill_cli-1.0.375}/docs/roadmap.html

@@ -2646,8 +2646,9 @@ rank conventions. Measuring that is what exposed the name-collapse underneath it
       </article>
 
       <article class="cap rcard" id="eval-corpus-is-one-strangers-repo">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Eval · Correctness</span></div>
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Eval · Correctness</span></div>
         <h3>62% of the required gate's corpus is a single third-party repository</h3>
+        <details class="cardbody"><summary>Write-up</summary>
         <p><b>Pinning the eval corpus to commit SHAs fixed drift. It did not fix concentration, and cannot fix
 availability.</b> The twenty repositories in <code>tests/eval/taps.txt</code> resolve to 10,152
 entries, wildly unevenly: <code>sickn33/antigravity-awesome-skills</code> is <b>6,309 of them
@@ -2674,7 +2675,48 @@ merges flowing but silently weakens the gate &mdash; the failure mode this line
 removing. Not proposed: dropping or lowering the gate.
 
 <b>Provenance.</b> The per-repo counts had to be printed to verify the pins in
-[[eval-corpus-was-not-actually-pinned]]; the distribution was the surprise.</p>
+[[eval-corpus-was-not-actually-pinned]]; the distribution was the surprise.
+
+<b>Measured, and it eliminates one of the four options.</b> &ldquo;Fail soft on N missing repos&rdquo;
+reads like the pragmatic choice. It is the dangerous one, because a corpus that loses a repo does not
+get harder to pass &mdash; it gets easier. Same 91-query required set, one repo removed and everything
+else identical: <code>all 20 &rarr; 10,152 entries, <b>0.852 / 0.473 / 0.605 / 0.657</b></code><br>
+<code>minus sickn33 (62%) &rarr; 3,843, <b>0.885 / 0.593 / 0.711 / 0.746</b></code><br>
+<code>minus that and ECC (78%) &rarr; 2,227, <b>0.967 / 0.659 / 0.769 / 0.814</b></code><br>
+<code>minus LessUp (holds golden targets) &rarr; 9,682, <b>0.676 / 0.374 / 0.483 / 0.523</b></code><br>
+<code>floors &rarr; 0.780 / 0.400 / 0.520 / 0.580</code>
+
+So dropping a <em>scale</em> repo sails through all four floors having measured a third of the
+intended corpus, and dropping a repo that carries golden targets fails all four in a way
+indistinguishable from &ldquo;this PR broke retrieval&rdquo;. Fail-soft is unsafe in both directions.
+The same table also kills <em>rebalancing by trimming</em>: cutting the big repo to fix the 62% would
+inflate every published number. Diluting concentration means adding breadth, which is
+[[eval-corpus-is-96x-smaller-than-a-real-install]]'s problem, not this one.
+
+<b>What shipped.</b> The corpus is now <em>verifiable</em> and its unavailability
+<em>distinguishable</em>. Each <code>taps.txt</code> row carries an entry count beside its SHA, and
+<code>--ensure</code> refuses to leave a corpus that does not match &mdash; so the "quietly measured
+3,843 entries" case above cannot reach the scorer. Unreachable repos exit <b>75</b> (EX_TEMPFAIL)
+rather than 1, and every failing repo is named rather than just the first, so a third party's outage
+stops reading as a regression here. CI restores the clones from a cache keyed on the tap list's
+content, which is what actually stops one deleted repository reddening every open PR; only
+third-party bytes are cached, never anything this repo derives, and there are no
+<code>restore-keys</code> &mdash; a near-miss restore would carry taps this file no longer pins, and
+the index is built from <em>every configured tap</em>. That last hazard is now checked directly:
+a <code>BOOST_HOME</code> holding taps outside the list is a hard failure, which is exactly what
+[[local-boost-install-is-not-the-eval-corpus]] walked into from the other side.
+
+<b>A second, closer bug fell out of it.</b> The sentinel that makes a local <code>make eval</code>
+fast was <em>empty</em> &mdash; it meant &ldquo;some corpus was built here&rdquo;. So editing
+<code>taps.txt</code> (moving a pin, adding a repo) left the previous corpus in place and scored it
+against the new file's baseline, silently, for as long as the sentinel survived. It now carries the
+tap list's digest, so an edit is a cache miss.
+
+<b>What did not ship, and why.</b> The 62% itself. There is no honest way to fix it by removing
+anything, and adding breadth changes every published number and needs its own re-baselining. What is
+enforced instead is a ratchet: <code>MAX_SHARE</code> fails the audit if any single publisher goes
+above 65%, ~3 points over today's measured 62.1%. That stops it getting worse without pretending it
+is fixed.</p></details>
         <div class="meta">
           <span class="m">Complexity <b>S</b></span>
           <span class="m hi">Impact <b>High</b></span>

boost_skill_cli-1.0.375/scripts/ensure_eval_corpus.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Tap the pinned Tier 1 eval corpus (tests/eval/taps.txt) into $BOOST_HOME so
+# scripts/eval_retrieval.py has the repos its golden set grades against. Used by
+# CI's lint job and by `make eval`.
+#
+# Idempotent: a sentinel skips re-tapping so a local `make check` stays fast (and
+# offline) after the first run. Set FORCE=1 to re-tap; CI does, because a cached
+# corpus must still be re-verified rather than trusted. Honors $PYTHON (defaults
+# to python3) so make can pass its venv interpreter.
+#
+# THE SENTINEL CARRIES THE TAP LIST'S DIGEST, and that is not decoration. An
+# empty sentinel said "some corpus has been built here", so editing taps.txt —
+# moving a pin, adding a repo — left `make eval` scoring the OLD corpus against
+# the NEW file's baseline, silently, for as long as the sentinel survived. That
+# is the same class of bug as the unpinned list this whole area exists to fix,
+# one directory further along. Keying on the digest makes an edit a cache miss.
+#
+# The tap list carries a commit SHA and an entry count per repo, and the loop
+# that reads it lives in scripts/eval_corpus.py rather than here: the format
+# needs parsing, the pin needs a fetch-then-checkout, and the count needs
+# verifying — none of which a shell loop can be unit-tested on. This file keeps
+# what it is good at, the sentinel and the environment.
+set -euo pipefail
+
+root=$(cd "$(dirname "$0")/.." && pwd)
+py="${PYTHON:-python3}"
+home="${BOOST_HOME:-$HOME/.boost}"
+sentinel="$home/.eval-corpus-ready"
+taps="$root/tests/eval/taps.txt"
+
+digest=$("$py" -c 'import hashlib,sys
+print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest())' "$taps")
+
+if [ -z "${FORCE:-}" ] && [ -f "$sentinel" ] && [ "$(cat "$sentinel")" = "$digest" ]; then
+  echo "eval corpus already tapped for this taps.txt — skipping (FORCE=1 to re-tap)"
+  exit 0
+fi
+
+PYTHONPATH="$root" "$py" "$root/scripts/eval_corpus.py" --ensure
+
+# Only after a clean --ensure: `set -e` means a corpus that could not be
+# materialised, or that did not match its pinned counts, never gets a sentinel
+# and so is never skipped on the next run.
+mkdir -p "$home"
+printf '%s\n' "$digest" > "$sentinel"

boost_skill_cli-1.0.375/scripts/eval_corpus.py

@@ -0,0 +1,423 @@
+#!/usr/bin/env python3
+"""Materialise the Tier 1 eval corpus at the exact commits it was measured on.
+
+WHY THIS EXISTS. `tests/eval/taps.txt` used to pin repository NAMES. Tapping is
+a shallow clone of whatever the default branch points at, so the corpus the
+required `eval` gate scores against was a moving target: the list was written
+recording **743** entries and the same 20 repos later resolved to **3,843**, a
+5.2x growth nobody caused by editing a file. `affaan-m/ECC` alone is 1,616 of
+them, so one third-party repository can move the gate's number by itself.
+
+That is not academic. Measured on a clean 20-tap install, BM25 scores recall@10
+**0.912** against a floor of **0.85** — a margin of +0.062. Unpinned growth
+spends that margin quietly, and the first thing anyone would see is a required
+gate failing on a pull request that touched nothing to do with retrieval.
+
+So each row now carries a commit SHA, and this script checks each clone out at
+it. `boost tap` still does the cloning (the corpus must be built the way a user
+builds one); pinning is a step applied after, because the alternative — teaching
+`boost tap` to pin — would add a CLI surface for a test-harness problem.
+
+WHY A ROW ALSO CARRIES AN ENTRY COUNT. A SHA fixes the *tree*; it does not fix
+what this project makes of that tree, and it does nothing at all when the tree
+cannot be fetched. Both gaps end in the same place — the gate scoring a corpus
+that is not the one its floors were set on — and the direction of the error is
+the surprise. Measured over the 91-query required set, one repo removed and
+everything else identical:
+
+    all 20 repos           10,152 entries   0.852 / 0.473 / 0.605 / 0.657
+    minus sickn33 (62%)     3,843 entries   0.885 / 0.593 / 0.711 / 0.746
+    minus that and ECC      2,227 entries   0.967 / 0.659 / 0.769 / 0.814
+    minus LessUp (targets)  9,682 entries   0.676 / 0.374 / 0.483 / 0.523
+    floors                                  0.780 / 0.400 / 0.520 / 0.580
+
+Losing a *scale* repo makes the gate EASIER — all four metrics rise and the
+check goes green having measured a third of the intended corpus. Losing a repo
+that holds golden targets fails all four floors in a way indistinguishable from
+"this pull request broke retrieval". So "skip whichever repos are missing today"
+is unsafe in both directions, and the count is what makes the first case
+detectable at all: a corpus that does not match its pins stops the run before
+anything is scored.
+
+WHY UNAVAILABILITY EXITS 75. `ensure_eval_corpus.sh` runs under `set -euo
+pipefail` inside CI's `lint` job, a required context, so one deleted, renamed or
+privatised repository reddens every open pull request at once. That is a real
+risk for twenty personal repositories, not a hypothetical. Nothing here can keep
+a vanished repository available — CI caches the materialised corpus for that —
+but the failure can at least say what it is. EX_TEMPFAIL separates "a third
+party took their repo down" from "your ranker regressed", which today arrive as
+the same red check.
+
+Usage:
+  python3 scripts/eval_corpus.py --ensure    # tap, pin and verify every row
+  python3 scripts/eval_corpus.py --audit     # static checks, no network
+  python3 scripts/eval_corpus.py --relock    # re-measure the entry counts
+  python3 scripts/eval_corpus.py --list      # print "repo sha count" rows
+"""
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence, Tuple
+
+ROOT = Path(__file__).resolve().parent.parent
+TAPS = ROOT / "tests" / "eval" / "taps.txt"
+
+_SHA = re.compile(r"[0-9a-f]{40}")
+
+#: ``(owner/repo, pinned commit or None, pinned entry count or None)``.
+Row = Tuple[str, Optional[str], Optional[int]]
+
+#: A repository, or the commit a row pins, could not be fetched.
+UNAVAILABLE = "unavailable"
+#: The repository materialised, but into a different number of entries.
+DRIFT = "drift"
+
+# 75 is EX_TEMPFAIL from sysexits.h, and the distinction is the point: an
+# unreachable third-party repository is not a defect in the change under test,
+# and it must not arrive looking like one.
+EXIT_DRIFT = 1
+EXIT_UNAVAILABLE = 75
+
+# No single repository may hold more than this share of the corpus. Measured,
+# not chosen: sickn33/antigravity-awesome-skills is 62.1% of the 10,152 entries
+# today. So this is a ratchet against making the concentration worse — not a
+# claim that 62% is a healthy number, which it is not. Diluting it means adding
+# breadth, never dropping the big repo: the table above shows that dropping it
+# raises every metric, so "rebalancing" by trimming would flatter the gate.
+MAX_SHARE = 0.65
+
+
+class CorpusError(RuntimeError):
+    """A corpus problem that carries which KIND of problem it is.
+
+    The kind is the whole reason this class exists. A corpus can fail to be the
+    one the floors were measured on in two unrelated ways — someone else's
+    repository went away, or the tree we did get scans into a different number
+    of entries — and only the second says anything about this project. Collapsed
+    into one exit status they are indistinguishable, which is how a third
+    party's force-push comes to look like a retrieval regression.
+    """
+
+    def __init__(self, kind: str, repo: str, detail: str) -> None:
+        super().__init__("%s: %s" % (repo, detail))
+        self.kind = kind
+        self.repo = repo
+        self.detail = detail
+
+
+def parse_taps(text: str) -> List[Row]:
+    """Rows of ``owner/repo [sha [count]]``, comments and blanks dropped.
+
+    An absent SHA parses as ``None`` rather than an error so the format stays
+    backward compatible, but a SHA that is *present and malformed* is fatal: a
+    typo silently degrading to "unpinned" would reintroduce the exact drift this
+    file exists to stop, while still looking pinned to a reader. A count is
+    fatal on the same grounds, and requires a SHA — a count beside an unpinned
+    repo describes a tree that is free to change underneath it.
+    """
+    rows: List[Row] = []
+    for raw in text.splitlines():
+        line = raw.strip()
+        if not line or line.startswith("#"):
+            continue
+        parts = line.split()
+        repo = parts[0]
+        if len(parts) == 1:
+            rows.append((repo, None, None))
+            continue
+        sha = parts[1]
+        if not _SHA.fullmatch(sha):
+            raise SystemExit(
+                "tests/eval/taps.txt: %s is pinned to %r, which is not a "
+                "40-character commit SHA" % (repo, sha))
+        if len(parts) == 2:
+            rows.append((repo, sha, None))
+            continue
+        if len(parts) > 3:
+            raise SystemExit(
+                "tests/eval/taps.txt: %s has %d fields, expected at most 3 "
+                "(repo, sha, entry count)" % (repo, len(parts)))
+        count = parts[2]
+        if not count.isdigit():
+            raise SystemExit(
+                "tests/eval/taps.txt: %s records %r entries, which is not a "
+                "non-negative integer" % (repo, count))
+        rows.append((repo, sha, int(count)))
+    return rows
+
+
+def shares(rows: Sequence[Row]) -> List[Tuple[str, int, float]]:
+    """``(repo, count, share)`` for every counted row, largest first.
+
+    Rows with no recorded count are omitted rather than treated as zero: a
+    partially counted list should report the concentration of what it knows,
+    not a share diluted by rows it cannot see.
+    """
+    counted = [(repo, n) for repo, _sha, n in rows if n is not None]
+    total = sum(n for _repo, n in counted)
+    if not total:
+        return []
+    return sorted(((repo, n, n / total) for repo, n in counted),
+                  key=lambda row: (-row[1], row[0]))
+
+
+def check_concentration(rows: Sequence[Row]) -> Optional[str]:
+    """Return a message when one repository exceeds ``MAX_SHARE``, else ``None``.
+
+    Static — it reads the counts already in the file, so it costs no network and
+    runs anywhere. Concentration is a sampling bias in every recall figure this
+    project publishes, and the bias is invisible in a list of twenty names that
+    look equally weighted.
+    """
+    ranked = shares(rows)
+    if not ranked:
+        return None
+    repo, count, share = ranked[0]
+    if share <= MAX_SHARE:
+        return None
+    return ("%s is %d of %d entries (%.1f%%), over the %.0f%% ceiling. The "
+            "corpus would be mostly one publisher's house style, which biases "
+            "every recall figure this project reports. Add breadth rather than "
+            "dropping the big repo: a smaller corpus scores HIGHER, so trimming "
+            "to rebalance would flatter the gate."
+            % (repo, count, sum(n for _r, n, _s in ranked), share * 100,
+               MAX_SHARE * 100))
+
+
+def extra_taps(configured: Sequence[str], pinned: Sequence[str]) -> List[str]:
+    """Configured taps that this list does not pin, sorted.
+
+    The third way to score a corpus that is not the pinned one, and the easiest
+    to walk into: the index is built from ``catalog.all_entries()``, which is
+    every configured tap, not the twenty in this file. Point BOOST_HOME at a
+    real install — 445 taps and 71,655 entries on the machine this was written
+    on — and `make eval` reports numbers for that catalogue while every comment
+    in the tree says it measured twenty repos.
+    """
+    return sorted(set(configured) - set(pinned))
+
+
+def _run(path: Path, *args: str) -> subprocess.CompletedProcess:
+    return subprocess.run(["git", "-C", str(path), *args],
+                          capture_output=True, text=True)
+
+
+def has_commit(path: Path, sha: str) -> bool:
+    """True when ``sha`` names a commit already present in ``path``.
+
+    ``^{commit}`` matters: `cat-file -e` succeeds for any object, so a tree or
+    blob SHA would otherwise report as present and then fail at checkout.
+    """
+    return _run(path, "cat-file", "-e", "%s^{commit}" % sha).returncode == 0
+
+
+def _fetch(path: Path, sha: str) -> subprocess.CompletedProcess:
+    # Shallow: the corpus needs the tree at one commit, not the history to it.
+    return _run(path, "fetch", "--quiet", "--depth", "1", "origin", sha)
+
+
+def pin_clone(path: Path, sha: str) -> None:
+    """Check ``path`` out at ``sha``, fetching the commit first if it is absent.
+
+    Raises ``CorpusError(UNAVAILABLE, ...)``, which is the same class of event as
+    the whole repository being gone: the tree the floors were measured on is not
+    obtainable here and now, and that is not a statement about this project.
+    """
+    if not has_commit(path, sha):
+        _fetch(path, sha)
+    if not has_commit(path, sha):
+        raise CorpusError(
+            UNAVAILABLE, path.name,
+            "commit %s is not reachable — the pin in tests/eval/taps.txt is "
+            "stale, or the repository rewrote history" % sha)
+    res = _run(path, "checkout", "--quiet", "--detach", sha)
+    if res.returncode != 0:
+        raise CorpusError(UNAVAILABLE, path.name,
+                          "could not check out %s: %s" % (sha, res.stderr.strip()))
+
+
+def _materialise(rows: Sequence[Row], verify: bool = True
+                 ) -> Tuple[Dict[str, int], List[CorpusError]]:
+    """Tap, pin and rescan every row. Returns ``(counts, failures)``.
+
+    Every row is attempted even after one fails. Stopping at the first would
+    report "one repository is unreachable" when five are, and the difference
+    decides whether the answer is "re-run it" or "the list needs work".
+    """
+    sys.path.insert(0, str(ROOT))
+    from boost_cli.core import catalog, registry  # deferred: repo-root import shim
+
+    counts: Dict[str, int] = {}
+    failures: List[CorpusError] = []
+    for repo, sha, want in rows:
+        try:
+            try:
+                tap = registry.get(repo)
+            except Exception:  # not yet tapped; add it below
+                tap = registry.add(repo)
+            if sha:
+                pin_clone(tap.path, sha)
+        except CorpusError as exc:
+            # Re-name it: pin_clone only knows the clone directory, and the
+            # report has to say what taps.txt says so the reader can grep for it.
+            failures.append(CorpusError(exc.kind, repo, exc.detail))
+            print("  %-44s %s" % (repo, exc.detail))
+            continue
+        except Exception as exc:  # any failure to obtain the tree at all
+            failures.append(CorpusError(UNAVAILABLE, repo, str(exc)))
+            print("  %-44s could not be tapped: %s" % (repo, exc))
+            continue
+        # Always after pinning: `boost tap` built the cache from the default
+        # branch, so an unrebuilt cache would describe a tree we just replaced.
+        entries = catalog.rebuild_tap(tap)
+        counts[repo] = len(entries)
+        note = ""
+        if verify and want is not None and len(entries) != want:
+            failures.append(CorpusError(
+                DRIFT, repo,
+                "taps.txt records %d entries, this tree scans into %d"
+                % (want, len(entries))))
+            note = "  != %d pinned" % want
+        print("  %-44s %s  %5d entries%s"
+              % (repo, (sha or "unpinned")[:7], len(entries), note))
+
+    extras = extra_taps([t.name for t in registry.list_taps()],
+                        [r for r, _s, _n in rows])
+    if verify and extras:
+        failures.append(CorpusError(
+            DRIFT, "BOOST_HOME",
+            "%d tap(s) are configured here but not pinned by taps.txt, and the "
+            "index is built from ALL configured taps: %s"
+            % (len(extras), ", ".join(extras[:5])
+               + (", ..." if len(extras) > 5 else ""))))
+    return counts, failures
+
+
+def _report_failures(failures: Sequence[CorpusError], total_rows: int) -> int:
+    """Print the failures grouped by kind and return the exit code to use."""
+    unavailable = [f for f in failures if f.kind == UNAVAILABLE]
+    drift = [f for f in failures if f.kind == DRIFT]
+    if unavailable:
+        print("\nCORPUS UNAVAILABLE — this is not a retrieval regression.")
+        print("%d of %d pinned repositories could not be materialised:"
+              % (len(unavailable), total_rows))
+        for f in unavailable:
+            print("  %s — %s" % (f.repo, f.detail))
+        print("The gate cannot run, and scoring the repos that ARE reachable is\n"
+              "not a fallback: measured, dropping the largest repo alone moves\n"
+              "BM25 recall@10 0.852 -> 0.885 and hit@1 0.473 -> 0.593, so a\n"
+              "partial corpus clears the floors MORE easily than the real one.")
+    if drift:
+        print("\nCORPUS DRIFT — what materialised is not what taps.txt pins.")
+        for f in drift:
+            print("  %s — %s" % (f.repo, f.detail))
+        print("A count mismatch means the scanner changed or a pin moved: "
+              "re-measure with\n`--relock`, then regenerate "
+              "tests/eval/baseline.json. An unpinned tap means\nBOOST_HOME is "
+              "not a corpus this file describes — point it at a scratch\n"
+              "directory (`make eval` uses ./.eval-home).")
+    # Drift wins when both happen: it is the one that says something about this
+    # repository, and it is not fixed by waiting or re-running.
+    return EXIT_DRIFT if drift else EXIT_UNAVAILABLE
+
+
+def relock_text(text: str, counts: Dict[str, int]) -> str:
+    """Rewrite each pinned row's entry count, preserving everything else.
+
+    A whole-file rewrite would lose the header, which is where the reasoning
+    lives; this touches only rows it has a new count for. A row with no SHA is
+    left alone — a count beside an unpinned repo describes a tree free to change
+    underneath it, so it must never be written.
+    """
+    width = max([len(r) for r in counts] or [1])
+    out = []
+    for raw in text.splitlines():
+        line = raw.strip()
+        parts = line.split()
+        if not line or line.startswith("#") or len(parts) < 2 \
+                or parts[0] not in counts or not _SHA.fullmatch(parts[1]):
+            out.append(raw)
+            continue
+        out.append("%-*s %s %5d" % (width, parts[0], parts[1], counts[parts[0]]))
+    return "\n".join(out) + ("\n" if text.endswith("\n") else "")
+
+
+def _print_concentration(rows: Sequence[Row]) -> None:
+    ranked = shares(rows)
+    if not ranked:
+        return
+    total = sum(n for _r, n, _s in ranked)
+    top = ranked[0]
+    print("corpus: %d entries across %d taps" % (total, len(ranked)))
+    line = "concentration: %s is %.1f%%" % (top[0], top[2] * 100)
+    if len(ranked) > 1:
+        line += "; top two are %.1f%%" % ((top[2] + ranked[1][2]) * 100)
+    print(line)
+
+
+def _ensure(relock: bool = False) -> int:
+    rows = parse_taps(TAPS.read_text(encoding="utf-8"))
+    counts, failures = _materialise(rows, verify=not relock)
+    if failures:
+        return _report_failures(failures, len(rows))
+    if relock:
+        TAPS.write_text(relock_text(TAPS.read_text(encoding="utf-8"), counts),
+                        encoding="utf-8")
+        print("relocked %d rows in tests/eval/taps.txt" % len(counts))
+        rows = parse_taps(TAPS.read_text(encoding="utf-8"))
+    _print_concentration(rows)
+    problem = check_concentration(rows)
+    if problem:
+        print("\nCORPUS CONCENTRATION — %s" % problem)
+        return EXIT_DRIFT
+    return 0
+
+
+def _audit() -> int:
+    """Static checks over the shipped list — no network, no clones."""
+    rows = parse_taps(TAPS.read_text(encoding="utf-8"))
+    for repo, count, share in shares(rows):
+        print("  %-44s %5d entries  %5.1f%%" % (repo, count, share * 100))
+    _print_concentration(rows)
+    uncounted = [r for r, _s, n in rows if n is None]
+    if uncounted:
+        print("\nuncounted rows (run --relock): %s" % ", ".join(uncounted))
+        return EXIT_DRIFT
+    problem = check_concentration(rows)
+    if problem:
+        print("\nCORPUS CONCENTRATION — %s" % problem)
+        return EXIT_DRIFT
+    return 0
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    p = argparse.ArgumentParser(prog="eval_corpus.py", description=__doc__)
+    p.add_argument("--ensure", action="store_true",
+                   help="tap, pin and verify every row, then rebuild its cache")
+    p.add_argument("--relock", action="store_true",
+                   help="re-measure entry counts and write them into taps.txt")
+    p.add_argument("--audit", action="store_true",
+                   help="static concentration/count checks, no network")
+    p.add_argument("--list", action="store_true",
+                   help="print the parsed rows and exit")
+    args = p.parse_args(argv)
+    if args.list:
+        for repo, sha, count in parse_taps(TAPS.read_text(encoding="utf-8")):
+            print("%s %s %s" % (repo, sha or "", "" if count is None else count))
+        return 0
+    if args.audit:
+        return _audit()
+    if args.relock:
+        return _ensure(relock=True)
+    if args.ensure:
+        return _ensure()
+    p.error("provide --ensure, --relock, --audit or --list")
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())