boost-skill-cli 1.2.54__tar.gz → 1.2.55__tar.gz

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/CLAUDE.md

@@ -584,6 +584,26 @@ Consequences for code you write here:
   real install are repeats and are embedded once — so a progress total taken
   from row count would over-report the work.
 
+- **Dense reuse is decided at TWO levels, and the second one is why `chunks`
+  carries a `digest`.** A tap whose git commit is unchanged is skipped whole.
+  A tap whose commit *moved* is not therefore changed: `_split_by_digest`
+  compares each entry's `content` digest against the one stored beside its
+  chunks, and only the entries that really differ are deleted and re-embedded.
+  Measured on a real 464-tap install: 19 taps drifted in 6.85 h and re-embedded
+  112,081 chunks for 659 chunks of actual change (**170x**), and **10 of the
+  19 changed nothing boost indexes at all** — badge `.json`, star-history
+  `.svg`, CI `.yml` — for 40.0% of the bill. Three rules make it safe, and
+  each was a real bug waiting: an entry is reused only when **every** stored
+  chunk carries the current digest, because a half-written entry must not be
+  served as a mixture of two versions' vectors; `_prune_stale_entries` drops
+  rows for entries the catalog dropped, which whole-tap deletion used to sweep
+  for free; and a **missing digest is never a match**, because two absences
+  are two unknowns. The digest travels in a shard too — without it an imported
+  shard re-embeds on the next build, spending the CPU the download existed to
+  save. The lookup is scoped by tap through the `chunks_tap` index: reading
+  the whole table instead measured 3.75 s cold on a 657,587-chunk store, and
+  ran even on the build where nothing had changed.
+
 - **Nothing on the search path may count `chunks`.** `SELECT COUNT(*)` scans the
   `chunks_tap` covering index — 8,419 pages / 34.5 MB, measured 1.94 s — and
   both `ready()` and `status()` did it on every search, the latter only to word

{boost_skill_cli-1.2.54/boost_skill_cli.egg-info → boost_skill_cli-1.2.55}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.54
+Version: 1.2.55
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.54'
-__version_tuple__ = version_tuple = (1, 2, 54)
+__version__ = version = '1.2.55'
+__version_tuple__ = version_tuple = (1, 2, 55)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/boost_cli/commands/discovery.py

@@ -423,8 +423,29 @@ def cmd_reindex(argv):
                          "`boost reindex --dense --force`."
                          % len(dense_stats["reused"]))
             else:
-                out.ok("embedded %d passages (%s) into the dense vector store"
-                       % (dense_stats["chunks"], dense_stats["provider"]))
+                # `chunks` is the store TOTAL, not this run's work, so on an
+                # incremental run "embedded 657,587 passages" described an
+                # afternoon of CPU that did not happen. `added` is what this
+                # run actually paid for.
+                added = dense_stats.get("added", dense_stats["chunks"])
+                out.ok("embedded %d passage%s (%s) — the dense store holds %d"
+                       % (added, "" if added == 1 else "s",
+                          dense_stats["provider"], dense_stats["chunks"]))
+                # The saving the tap-level line cannot show: a tap appears in
+                # `reindexed` the moment its commit moves, so a run that reused
+                # every entry inside three moved taps otherwise reads as three
+                # taps fully re-embedded.
+                kept = dense_stats.get("reused_entries") or 0
+                if kept:
+                    out.info(out.role(
+                        "reused %d unchanged item%s inside %d moved tap%s — "
+                        "only %d item%s changed"
+                        % (kept, "" if kept == 1 else "s",
+                           len(dense_stats["reindexed"]),
+                           "" if len(dense_stats["reindexed"]) == 1 else "s",
+                           dense_stats.get("embedded_entries") or 0,
+                           "" if dense_stats.get("embedded_entries") == 1
+                           else "s"), "muted"))
             if dense_stats.get("quantized"):
                 out.ok("quantized %d existing vectors — dense search no longer "
                        "scans the whole store on every query"

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/boost_cli/core/dense.py

@@ -23,7 +23,13 @@ from ..errors import BoostError
 from . import catalog, embed, paths
 from .rag import Hit, chunk, entry_key, read_body
 
-INDEX_VERSION = 2
+# 3 -- `chunks.digest` carries each row's entry content digest, which is what
+#      lets reuse be decided per ENTRY rather than per tap. Bumping this is the
+#      migration: a v2 store has no such column, `_ensure_schema` cannot add one
+#      to an existing table (CREATE TABLE IF NOT EXISTS), and `build` already
+#      wipes on a version change. That costs one full re-embed, once — the same
+#      price v2 charged when it landed, and the last one this mechanism needs.
+INDEX_VERSION = 3
 _BATCH = 128            # texts per embedding request
 
 #: Rows between commits while storing vectors. Small enough that an interrupted
@@ -112,8 +118,14 @@ def quantized(con: sqlite3.Connection) -> bool:
 def _ensure_schema(con: sqlite3.Connection, dim: int) -> None:
     con.execute(
         "CREATE TABLE IF NOT EXISTS chunks (id INTEGER PRIMARY KEY AUTOINCREMENT,"
-        " name TEXT, tap TEXT, path TEXT, kind TEXT, cix INTEGER, snip TEXT)")
+        " name TEXT, tap TEXT, path TEXT, kind TEXT, cix INTEGER, snip TEXT,"
+        " digest TEXT)")
     con.execute("CREATE INDEX IF NOT EXISTS chunks_tap ON chunks(tap)")
+    # Keyed by (tap, path) because that is `rag.entry_key` — row identity, the
+    # thing an entry's chunks belong to. Keying it on `digest` alone would be
+    # the content question, which is a different one: two taps shipping the
+    # same file share a digest and must still be deleted independently.
+    con.execute("CREATE INDEX IF NOT EXISTS chunks_entry ON chunks(tap, path)")
     con.execute("CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT)")
     if _quantizable(dim):
         # Two relations, because one cannot do both jobs. `vec0` is the only
@@ -518,17 +530,38 @@ def build(entries: list[dict] | None = None, force: bool = False,
                     if commit and old_commits.get(safe) == commit:
                         reused.add(safe)
 
-        fresh = [e for e in entries
-                 if e["tap"].replace("/", "__") not in reused]
-        changed_taps = sorted({e["tap"] for e in fresh})
+        candidates = [e for e in entries
+                      if e["tap"].replace("/", "__") not in reused]
+        changed_taps = sorted({e["tap"] for e in candidates})
         # Prune taps that are gone entirely, not only the ones that changed.
         # `boost untap` removes a tap's entries, so it can never appear in
-        # `fresh` and its vectors survived every later incremental build —
+        # `candidates` and its vectors survived every later incremental build —
         # crowding the KNN pool on every query with rows `retrieve` then
         # discards for not being live, which is how a dense search quietly
         # returns fewer hits the longer an index has been in use.
         removed_taps = sorted(_indexed_taps(con) - {e["tap"] for e in entries})
-        _delete_taps(con, changed_taps + removed_taps)
+        _delete_taps(con, removed_taps)
+
+        # Second-level reuse, and the reason it is worth the column. A tap's
+        # commit moving says *something* in that clone changed — not that
+        # anything boost indexes did. Measured on a real 464-tap install: 19
+        # taps drifted in 6.85 h, and 10 of them changed nothing indexed at all
+        # (badge JSON, star-history SVGs, CI YAML, e2e TypeScript), yet cost
+        # 44,866 chunks — 40.0% of the incremental bill. Across all 19, 967
+        # changed files mapped to 39 indexed entries owning 659 chunks, against
+        # 112,081 re-embedded: 170x the actual change.
+        #
+        # The digest is not new identity machinery. `catalog._content_digest`
+        # already stamps every entry with a hash of exactly what `read_body`
+        # assembles, and `tests/unit/test_content_identity.py` pins that parity
+        # — so an entry whose digest matches the one stored beside its chunks
+        # is one whose embedded text is byte-identical.
+        fresh, kept = _split_by_digest(con, candidates)
+        _delete_entries(con, [entry_key(e) for e in fresh])
+        # An entry deleted upstream keeps answering queries otherwise: whole-tap
+        # deletion used to sweep it for free, and reuse is exactly what stops
+        # that happening.
+        _prune_stale_entries(con, changed_taps, {entry_key(e) for e in entries})
 
         added, failed_taps = _embed_and_store(con, fresh, tap_paths,
                                              on_progress=on_progress)
@@ -555,6 +588,12 @@ def build(entries: list[dict] | None = None, force: bool = False,
             "reindexed": changed_taps,
             "pruned": removed_taps,
             "reused": sorted(reused),
+            # Entries inside a moved tap whose content had not changed. This is
+            # the number the tap-level `reused` cannot show — a tap appears in
+            # `reindexed` the moment its commit moves, and on real data most of
+            # its entries are still untouched.
+            "reused_entries": len(kept),
+            "embedded_entries": len(fresh),
             "failed": sorted(failed_taps),
             "provider": prov,
             "model": mdl,
@@ -614,7 +653,8 @@ def export_shard(tap: str) -> dict:
             vt = "vec_raw v ON v.id = c.id" if quantized(con) \
                 else "vec_chunks v ON v.rowid = c.id"
             rows = con.execute(
-                "SELECT c.name, c.tap, c.path, c.kind, c.cix, c.snip, v.embedding "  # noqa: S608  relation name from a literal pair
+                "SELECT c.name, c.tap, c.path, c.kind, c.cix, c.snip, "  # noqa: S608  relation name from a literal pair
+                "c.digest, v.embedding "
                 "FROM chunks c JOIN %s WHERE c.tap = ? ORDER BY c.id" % vt,
                 (tap,)).fetchall()
         except sqlite3.Error as exc:
@@ -630,10 +670,16 @@ def export_shard(tap: str) -> dict:
             # the vector rows are gone or unlinked. Still not "never built".
             raise _unreadable_vectors(tap, expected, None)
         for row in rows:
-            name, ctap, path, kind, cix, snip, emb = row
+            name, ctap, path, kind, cix, snip, digest, emb = row
             chunks.append({
                 "name": name, "tap": ctap, "path": path, "kind": kind,
                 "cix": cix, "snip": snip,
+                # The digest travels with the vector or the shard defeats its
+                # own purpose: `_split_by_digest` reuses an entry only when the
+                # stored digest matches the catalog's, so an imported chunk
+                # without one is re-embedded on the very next build — paying
+                # in CPU exactly what downloading the shard was meant to save.
+                "digest": digest,
                 # base64 so the shard is plain JSON and can be published as a
                 # release artifact without a binary format of its own.
                 "embedding": base64.b64encode(bytes(emb)).decode("ascii"),
@@ -714,10 +760,15 @@ def import_shard(shard: dict, commit: str) -> tuple[bool, str]:
         mod = _load()
         for c in shard.get("chunks") or []:
             cur = con.execute(
-                "INSERT INTO chunks (name, tap, path, kind, cix, snip) "
-                "VALUES (?, ?, ?, ?, ?, ?)",
+                "INSERT INTO chunks (name, tap, path, kind, cix, snip, digest) "
+                "VALUES (?, ?, ?, ?, ?, ?, ?)",
                 (c.get("name"), c.get("tap"), c.get("path"), c.get("kind"),
-                 c.get("cix"), c.get("snip")))
+                 c.get("cix"), c.get("snip"),
+                 # A shard published before the digest travelled has none, and
+                 # None is the honest value: that tap re-embeds once, which is
+                 # what it did before this existed. Never a placeholder — a
+                 # wrong digest would suppress a real re-embed forever.
+                 c.get("digest")))
             blob = base64.b64decode(c["embedding"])
             _store_vector(con, cur.lastrowid, blob)
         commits = meta.get("commits")
@@ -748,12 +799,109 @@ def _indexed_taps(con: sqlite3.Connection) -> set:
         return set()          # no chunks table yet — nothing indexed, nothing stale
 
 
+def _delete_matching(con: sqlite3.Connection, where: str,
+                     params: tuple) -> int:
+    """Drop the chunk rows matching ``where``, and their vectors. Rows hit.
+
+    Deleting a chunk is two statements that must not drift apart: the vectors
+    go through :func:`_drop_vectors` (which knows both store layouts) and only
+    then does the row go. Tap-level and entry-level deletion had that pair
+    written out twice, which is one copy too many for a rule this easy to half-
+    apply — an orphan vector outlives every later deletion, because those are
+    all scoped through ``chunks``.
+
+    ``where`` is a literal from this module, never caller data; the parameters
+    are bound.
+    """
+    # `%` rather than `+`, to match every other interpolated statement in this
+    # module: those are the shapes ruff's S608 is known to flag here, so the
+    # suppression is one ruff actually consumes rather than an unused `noqa`
+    # that RUF100 would then reject.
+    ids = [r[0] for r in con.execute(
+        "SELECT id FROM chunks WHERE %s" % where, params)]  # noqa: S608  literal clause
+    if not ids:
+        return 0
+    _drop_vectors(con, ids)
+    con.execute("DELETE FROM chunks WHERE %s" % where, params)  # noqa: S608  same literal
+    return len(ids)
+
+
+def _delete_entries(con: sqlite3.Connection,
+                    keys: list[tuple[str, str]]) -> int:
+    """Drop every chunk row for these ``(tap, path)`` entries. Returns rows hit.
+
+    The entry-level counterpart to :func:`_delete_taps`, and it must stay
+    row-scoped rather than content-scoped: two taps can ship a byte-identical
+    file, so deleting by digest would take another tap's rows with it.
+    """
+    return sum(_delete_matching(con, "tap = ? AND path = ?", (tap, path))
+               for tap, path in keys)
+
+
+def _prune_stale_entries(con: sqlite3.Connection, taps: list[str],
+                         live: set[tuple[str, str]]) -> int:
+    """Drop rows in ``taps`` for entries the catalog no longer has.
+
+    Whole-tap deletion used to do this for free: a changed tap was emptied and
+    rebuilt, so an entry deleted upstream simply never came back. Reuse keeps
+    the rows instead, which means a file removed from a registry would answer
+    queries forever — the same silent-stale-row failure `_delete_taps`'s
+    `removed_taps` sweep exists to prevent, one level down.
+    """
+    stale = [(tap, path) for tap in taps
+             for (path,) in con.execute(
+                 "SELECT DISTINCT path FROM chunks WHERE tap = ?", (tap,))
+             if (tap, path) not in live]
+    return _delete_entries(con, stale)
+
+
+def _split_by_digest(con: sqlite3.Connection,
+                     entries: list[dict]) -> tuple[list[dict], list[dict]]:
+    """Partition ``entries`` into ``(to_embed, already_stored)``.
+
+    An entry is already stored when the store holds chunks for its
+    ``(tap, path)`` **and every one of them** carries its current content
+    digest. Every one, not any: a partially-written entry — an interrupted
+    build, or one whose digest changed mid-run — would otherwise be reused
+    with a mixture of two versions' vectors, which is worse than re-embedding
+    because nothing later notices.
+
+    An entry with no digest is never reused. `catalog._content_digest` stamps
+    them at scan time, but a cache written before ``CACHE_FORMAT`` has none,
+    and CLAUDE.md's rule is explicit: consumers degrade cleanly when ``content``
+    is absent and **must never treat two absences as a match**. Two entries
+    with no digest are not the same entry; they are two unknowns.
+    """
+    # Nothing to decide, and the early return is the common case rather than a
+    # guard: on a build where no tap moved, `candidates` is empty, and without
+    # this the store gets scanned end to end to answer a question nobody asked.
+    if not entries:
+        return [], []
+    # Scoped to the taps actually in play, via the `chunks_tap` index. Reading
+    # the whole table instead measured 3.75 s cold (0.17 s warm) against 0.08 s
+    # for the nineteen largest taps and under 0.01 s for nineteen typical ones,
+    # on a real 657,587-chunk store. The full read was also the wrong shape: it
+    # grows with the store while the work grows with the drift.
+    stored: dict[tuple[str, str], set] = {}
+    for tap in sorted({e["tap"] for e in entries}):
+        for path, digest in con.execute(
+                "SELECT path, digest FROM chunks WHERE tap = ?", (tap,)):
+            stored.setdefault((tap, path), set()).add(digest)
+    to_embed, kept = [], []
+    for e in entries:
+        digest = e.get("content")
+        have = stored.get(entry_key(e))
+        if digest and have == {digest}:
+            kept.append(e)
+        else:
+            to_embed.append(e)
+    return to_embed, kept
+
+
 def _delete_taps(con: sqlite3.Connection, taps: list[str]) -> None:
+    """Drop every chunk row belonging to these taps, and their vectors."""
     for tap in taps:
-        ids = [r[0] for r in
-               con.execute("SELECT id FROM chunks WHERE tap = ?", (tap,))]
-        _drop_vectors(con, ids)
-        con.execute("DELETE FROM chunks WHERE tap = ?", (tap,))
+        _delete_matching(con, "tap = ?", (tap,))
 
 
 def _embed_and_store(con: sqlite3.Connection, entries: list[dict],
@@ -825,10 +973,14 @@ def _embed_and_store(con: sqlite3.Connection, entries: list[dict],
             failed_taps.add(e["tap"])
             continue
         cur = con.execute(
-            "INSERT INTO chunks (name, tap, path, kind, cix, snip) "
-            "VALUES (?, ?, ?, ?, ?, ?)",
+            "INSERT INTO chunks (name, tap, path, kind, cix, snip, digest) "
+            "VALUES (?, ?, ?, ?, ?, ?, ?)",
             (e["name"], e["tap"], e["skill_md"],
-             e.get("kind", "skill"), ci, text[:200].strip()))
+             e.get("kind", "skill"), ci, text[:200].strip(),
+             # None where the catalog has no digest, which `_split_by_digest`
+             # then refuses to reuse — the honest answer for a cache written
+             # before CACHE_FORMAT stamped one.
+             e.get("content")))
         _store_vector(con, cur.lastrowid, mod.serialize_float32(vec))
         added += 1
         since_commit += 1

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55/boost_skill_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.54
+Version: 1.2.55
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/boost_skill_cli.egg-info/SOURCES.txt

@@ -684,6 +684,7 @@ tests/unit/test_dedup_trust.py
 tests/unit/test_demo_index.py
 tests/unit/test_dense.py
 tests/unit/test_dense_embed_dedupe.py
+tests/unit/test_dense_entry_reuse.py
 tests/unit/test_dense_fallback.py
 tests/unit/test_dense_fix_hint.py
 tests/unit/test_dense_progress.py

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/docs/roadmap/items/dense-reindex-reuses-whole-taps.md

@@ -2,14 +2,14 @@
 id: dense-reindex-reuses-whole-taps
 board: code
 section: internals
-status: next
+status: shipped
 category: Search · Performance
 complexity: M
 impact: High
 wow: 3
 note: one changed file re-embeds 29 min for 1.9 s of new text
 order: 126
-owner:
+owner: feat/dense-entry-digest-reuse
 pr:
 title: "Dense reuse is per <em>tap</em>, so one changed file re-embeds the whole registry"
 ---
@@ -91,3 +91,34 @@ today's incremental cost, and <code>davila7</code> changed nothing boost indexes
 <b>Shards do not cover this case.</b> Of the 19 moved taps exactly one appears in the published
 manifest, and its shard is pinned at the commit the tap just left, so <code>import_shard</code>
 refuses it. Net importable for this reindex: <b>0 of 112,081 chunks</b>.
+
+<b>Shipped.</b> <code>chunks</code> gained a <code>digest</code> column and
+<code>_split_by_digest</code> partitions a moved tap's entries on it, so only entries whose content
+really differs are deleted and re-embedded. The card's own conclusion held: no new identity
+machinery was needed &mdash; <code>catalog._content_digest</code> already hashes exactly what
+<code>rag.read_body</code> assembles, and <code>test_content_identity.py</code> pins that parity.
+
+<b>Three things would each have been a silent regression, and each has a test.</b> An entry deleted
+upstream has to stop answering &mdash; whole-tap deletion swept it for free, and reuse is precisely
+what stops that, so <code>_prune_stale_entries</code> sweeps it explicitly. An entry is reused only
+when <b>every</b> stored chunk carries the current digest, never any: a half-written entry would
+otherwise be reused as a mixture of two versions' vectors, which is worse than re-embedding because
+nothing later notices. And a <b>missing digest is never a match</b> &mdash; two absences are two
+unknowns, which is CLAUDE.md's rule and this is where it bites.
+
+<b>A fourth check exists because entry-level deletion is a shape the store had never seen.</b>
+Removing <em>some</em> of a tap's rows can leave a vector whose chunk is gone &mdash; an orphan
+<code>retrieve</code> still ranks, invisible to any chunk total because the chunk side is correct
+&mdash; or a chunk with no vector, which the KNN can never return. Verified on a real store: after a
+change and after a delete, the id sets of <code>chunks</code> and <code>vec_raw</code> match in both
+directions.
+
+<b>The digest travels in a shard too.</b> Without that, an imported shard's entries carry none and
+the next <code>reindex --dense</code> re-embeds them &mdash; spending in CPU exactly what
+downloading the shard existed to save. A shard published before this has no digest, and
+<code>None</code> is the honest value: that tap re-embeds once. Never a placeholder, which would
+suppress a real re-embed forever.
+
+<code>INDEX_VERSION</code> 2&nbsp;&rarr;&nbsp;3 <b>is</b> the migration &mdash;
+<code>_ensure_schema</code> is <code>CREATE TABLE IF NOT EXISTS</code> and cannot add a column to an
+existing table, and <code>build</code> already wipes on a version change. One full re-embed, once.

{boost_skill_cli-1.2.54 → boost_skill_cli-1.2.55}/docs/roadmap.html

@@ -4742,8 +4742,9 @@ bug report, and <code>spin.progress</code> is deliberately silent off a TTY.</p>
       </article>
 
       <article class="cap rcard" id="dense-reindex-reuses-whole-taps">
-        <div class="head"><span class="pill next">Next</span><span class="cat">Search · Performance</span></div>
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Search · Performance</span></div>
         <h3>Dense reuse is per <em>tap</em>, so one changed file re-embeds the whole registry</h3>
+        <details class="cardbody"><summary>Write-up</summary>
         <p><code>dense.build</code> decides reuse by comparing each tap's recorded commit against its current
 one, and everything below that granularity is re-done from scratch:
 <code>_delete_taps</code> is <code>DELETE FROM chunks WHERE tap = ?</code>, and <code>fresh</code>
@@ -4821,7 +4822,38 @@ today's incremental cost, and <code>davila7</code> changed nothing boost indexes
 
 <b>Shards do not cover this case.</b> Of the 19 moved taps exactly one appears in the published
 manifest, and its shard is pinned at the commit the tap just left, so <code>import_shard</code>
-refuses it. Net importable for this reindex: <b>0 of 112,081 chunks</b>.</p>
+refuses it. Net importable for this reindex: <b>0 of 112,081 chunks</b>.
+
+<b>Shipped.</b> <code>chunks</code> gained a <code>digest</code> column and
+<code>_split_by_digest</code> partitions a moved tap's entries on it, so only entries whose content
+really differs are deleted and re-embedded. The card's own conclusion held: no new identity
+machinery was needed &mdash; <code>catalog._content_digest</code> already hashes exactly what
+<code>rag.read_body</code> assembles, and <code>test_content_identity.py</code> pins that parity.
+
+<b>Three things would each have been a silent regression, and each has a test.</b> An entry deleted
+upstream has to stop answering &mdash; whole-tap deletion swept it for free, and reuse is precisely
+what stops that, so <code>_prune_stale_entries</code> sweeps it explicitly. An entry is reused only
+when <b>every</b> stored chunk carries the current digest, never any: a half-written entry would
+otherwise be reused as a mixture of two versions' vectors, which is worse than re-embedding because
+nothing later notices. And a <b>missing digest is never a match</b> &mdash; two absences are two
+unknowns, which is CLAUDE.md's rule and this is where it bites.
+
+<b>A fourth check exists because entry-level deletion is a shape the store had never seen.</b>
+Removing <em>some</em> of a tap's rows can leave a vector whose chunk is gone &mdash; an orphan
+<code>retrieve</code> still ranks, invisible to any chunk total because the chunk side is correct
+&mdash; or a chunk with no vector, which the KNN can never return. Verified on a real store: after a
+change and after a delete, the id sets of <code>chunks</code> and <code>vec_raw</code> match in both
+directions.
+
+<b>The digest travels in a shard too.</b> Without that, an imported shard's entries carry none and
+the next <code>reindex --dense</code> re-embeds them &mdash; spending in CPU exactly what
+downloading the shard existed to save. A shard published before this has no digest, and
+<code>None</code> is the honest value: that tap re-embeds once. Never a placeholder, which would
+suppress a real re-embed forever.
+
+<code>INDEX_VERSION</code> 2&nbsp;&rarr;&nbsp;3 <b>is</b> the migration &mdash;
+<code>_ensure_schema</code> is <code>CREATE TABLE IF NOT EXISTS</code> and cannot add a column to an
+existing table, and <code>build</code> already wipes on a version change. One full re-embed, once.</p></details>
         <div class="meta">
           <span class="m">Complexity <b>M</b></span>
           <span class="m hi">Impact <b>High</b></span>