boost-skill-cli 1.2.2__tar.gz → 1.2.4__tar.gz

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/.vale/styles/config/vocabularies/boost/accept.txt

@@ -67,6 +67,9 @@ reindex
 repo
 repo's
 repos
+rescan
+rescanned
+rescans
 Rerank
 rerank
 reranking

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/CLAUDE.md

@@ -182,6 +182,25 @@ line coverage. Target `boost_cli/core` behavior with assertions, not just import
   still ignored, and deliberately: it is an unsafe fix over ~1,000 call sites in
   `boost_cli` alone. Write new code in the modern style; don't bulk-rewrite old
   code in an unrelated PR.
+- **A catalog entry has two identities, and they are not interchangeable.**
+  *Row* identity is `(tap, skill_md)` (`rag.entry_key`) — which file, in which
+  tap. *Content* identity is `entry["content"]`, a truncated sha256 of
+  `name + "\n" + description + "\n" + body` stamped by `catalog._content_digest`
+  at scan time. Use the row key to address a row and the digest to ask whether
+  two rows are the same thing. The digest must stay byte-identical to what
+  `rag.read_body` assembles — `tests/unit/test_content_identity.py` pins the
+  parity, because a digest that drifts keeps *looking* right while clustering
+  nothing. Name-based keys are wrong at both layers: over 60,047 real entries,
+  hashing the body alone merges 259 clusters spanning different names, and
+  name+description over-collapses 3,383 clusters of genuinely different prose.
+  Consumers must degrade cleanly when `content` is absent (a cache written
+  before `CACHE_FORMAT`, or a synthesised entry) — and must never treat two
+  absences as a match.
+- **The tap cache is versioned; bump `catalog.CACHE_FORMAT` to backfill.** It is
+  how a new entry field reaches 460 caches on a real machine without a re-tap:
+  `load_tap` rescans a stale cache *when the clone is still there*, and serves
+  the stale entries as-is when it is not. Never make a stale cache an error —
+  that trades a missing field for a missing catalogue.
 - **Three item kinds, one scanner.** `core/catalog.scan_dir` indexes `skill`
   (SKILL.md), `rule` (.mdc/.cursorrules/.windsurfrules/.clinerules), and
   `workflow` (commands/agents/workflows Markdown). **All three install**, and

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.2
+Version: 1.2.4
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.2'
-__version_tuple__ = version_tuple = (1, 2, 2)
+__version__ = version = '1.2.4'
+__version_tuple__ = version_tuple = (1, 2, 4)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_cli/core/browse.py

@@ -163,16 +163,35 @@ def highlight_positions(query: str, text: str) -> set[int]:
 #: (`rule`, 47 distinct) is already the pathological case — a bigger one is a
 #: registry doing something strange, not a user waiting to be helped.
 _FUZZY_BUCKET_CAP = 80
+# Bucket name reserved for rows keyed by content digest. A NUL can't occur in a
+# slugified skill name, so a digested row can never share a fuzzy bucket with a
+# real one — and the fuzzy pass skips this bucket outright, because "95% similar"
+# is a question about prose and asking it of two hex digests is meaningless.
+_DIGEST_SIG = "\x00digest"
 
 
 def _sig(entry: dict) -> tuple[str, str]:
-    """Identity of a row for dedupe: name plus description, normalised.
-
-    The description is what makes this safe. Keying on the name alone would
-    collapse `code-reviewer` — 75 rows with **42 distinct descriptions** on a
-    real catalogue — into one, hiding 41 genuinely different skills. Keying on
-    the pair collapses only rows that say the same thing.
+    """Identity of a row for dedupe: the scanner's content digest when it has
+    one, else name plus description, normalised.
+
+    The digest is the right key and the pair is a fallback, not a peer. Keying
+    on name+description alone over-collapses: on a real 60,047-entry catalogue
+    it merges **3,383 clusters** the digest keeps apart — items that share a
+    name and a description while carrying genuinely different prose, one of
+    which then becomes unreachable in the browser. It is still the only key
+    available for a row from a cache written before `catalog.CACHE_FORMAT`, so
+    it stays.
+
+    Digested and undigested rows can never merge into each other: the two keys
+    live in different halves of the tuple, so an unknown is only ever equal to
+    another unknown with the same visible metadata.
+
+    Keying on the name alone was never an option — `code-reviewer` is 75 rows
+    with **42 distinct descriptions** on a real catalogue.
     """
+    content = str(entry.get("content", "")).strip()
+    if content:
+        return (_DIGEST_SIG, content)
     return (str(entry.get("name", "")).strip().lower(),
             str(entry.get("description", "")).strip().lower())
 
@@ -208,6 +227,8 @@ def dedupe(entries, threshold: float = 0.95) -> list[tuple[dict, int]]:
     # with different names are different skills however similar the prose.
     by_name: dict[str, list] = {}
     for sig in order:
+        if sig[0] == _DIGEST_SIG:
+            continue        # exact content already decided; see _DIGEST_SIG
         by_name.setdefault(sig[0], []).append(sig)
     absorbed: set[tuple[str, str]] = set()
     for sigs in by_name.values():

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_cli/core/catalog.py

@@ -17,6 +17,7 @@ that ship in the same GitHub registries:
 """
 from __future__ import annotations
 
+import hashlib
 import json
 import operator
 import os
@@ -26,6 +27,14 @@ from pathlib import Path
 from ..errors import BoostError
 from . import frontmatter, gitutil, paths, registry, util
 
+# Bumped whenever a scan starts recording something the previous scan did not,
+# so 460 caches on a real machine invalidate on read instead of needing a
+# re-tap. `catalogbundle.BUNDLE_FORMAT` and `rag.INDEX_VERSION` self-version for
+# the same reason; the tap cache was the one derived artifact that did not, and
+# without it there was no way to ship a new entry field at all.
+#   1 -- entries carry `content` (see `_content_digest`)
+CACHE_FORMAT = 1
+
 KIND_SKILL = "skill"
 KIND_RULE = "rule"
 KIND_WORKFLOW = "workflow"
@@ -61,6 +70,36 @@ def _description(meta: dict, body: str) -> str:
     return ""
 
 
+def _content_digest(name: str, description: str, body: str) -> str:
+    """Content identity for an item: what it *is*, independent of where it sits.
+
+    Deliberately not the file bytes and not the body alone. The text hashed here
+    is exactly what :func:`rag.read_body` assembles — ``name``, ``description``,
+    then the frontmatter-stripped body — because that is the string boost
+    already treats as an item's content everywhere it clusters copies, and two
+    identity keys that *nearly* agree are worse than one.
+
+    Measured over a real 460-tap install (60,047 entries), against the two
+    alternatives that look reasonable:
+
+        name+description+body   41,051 clusters   0 span >1 name   <- this
+        body only               40,372 clusters   259 span >1 name
+        name+description        37,668 clusters   2 span >1 name
+
+    Body-only merges 259 clusters that span different names — the
+    ``admin-interface-rule`` collision #366 fixed, where one registry ships two
+    unrelated rules under one name. Metadata-only over-collapses in the other
+    direction: 3,383 clusters of items that share a name and description while
+    carrying genuinely different prose. Including all three separates both.
+
+    Truncated to 16 hex chars to match the index, which stores one of these per
+    document; the full digest doubles the index for collision odds that are
+    already negligible at 60k items.
+    """
+    text = "%s\n%s\n%s" % (name, description, body)
+    return hashlib.sha256(text.strip().encode("utf-8", "replace")).hexdigest()[:16]
+
+
 def _make_entry(root: Path, defining_file: Path, kind: str, default_name: str,
                 tap_name: str, curated: bool, meta: dict, body: str) -> dict:
     name = str(meta.get("name") or "").strip() or default_name
@@ -80,6 +119,11 @@ def _make_entry(root: Path, defining_file: Path, kind: str, default_name: str,
         "rel_dir": item_dir.relative_to(root).as_posix() if item_dir != root else ".",
         "skill_md": defining_file.relative_to(root).as_posix(),
         "meta": meta,
+        # Free here and nowhere else: the body is already read, parsed and about
+        # to be discarded. `rag` used to re-open all 60k files at index time to
+        # recompute exactly this (~14 s on a real install) because the entry did
+        # not carry it.
+        "content": _content_digest(slug, description, body),
         # Lowercased substring-search blob, flattened once at index time so
         # search() never re-walks the frontmatter per query (see _search_blob).
         "search_blob": _search_blob(slug, description, meta),
@@ -176,6 +220,7 @@ def rebuild_tap(tap: registry.Tap) -> list[dict]:
     tap.cache_file.write_text(json.dumps({
         "tap": tap.name,
         "url": tap.url,
+        "format": CACHE_FORMAT,
         "generated": util.now_iso(),
         "commit": gitutil.head_commit(tap.path),
         "skills": entries,
@@ -192,12 +237,21 @@ def rebuild_tap(tap: registry.Tap) -> list[dict]:
 # the cached list is shared by reference (matching the index cache). Keying on
 # nanosecond mtime + size catches rewrites a bare second-resolution mtime would
 # miss.
-_ENTRY_CACHE: dict[str, tuple[tuple[int, int], list[dict]]] = {}
+_ENTRY_CACHE: dict[str, tuple[tuple[int, int], list[dict], bool]] = {}
+
 
+def _cached_tap(tap: registry.Tap) -> tuple[list[dict], bool] | None:
+    """``(skills, is_current)`` for a tap's cache, or ``None`` to force a
+    rebuild (file missing, unreadable, or corrupt JSON).
 
-def _cached_tap(tap: registry.Tap) -> list[dict] | None:
-    """Memoized ``skills`` for a tap's cache file, or ``None`` to force a
-    rebuild (file missing, unreadable, or corrupt JSON)."""
+    ``is_current`` says whether the cache was written by this scanner. It is
+    reported rather than acted on here because the right response depends on
+    something this function cannot see: whether the clone is still on disk to
+    rescan. A stale cache is not *wrong* — it is missing fields its consumers
+    have since learned to want, and every one of them degrades cleanly without
+    them — so discarding it unconditionally would trade a missing digest for
+    missing entries, which is the worse failure.
+    """
     p = tap.cache_file
     key = str(p)
     try:
@@ -208,14 +262,16 @@ def _cached_tap(tap: registry.Tap) -> list[dict] | None:
     stamp = (st.st_mtime_ns, st.st_size)
     cached = _ENTRY_CACHE.get(key)
     if cached is not None and cached[0] == stamp:
-        return cached[1]
+        return cached[1], cached[2]
     try:
-        skills = json.loads(p.read_text(encoding="utf-8")).get("skills", [])
+        data = json.loads(p.read_text(encoding="utf-8"))
     except (json.JSONDecodeError, OSError):
         _ENTRY_CACHE.pop(key, None)
         return None
-    _ENTRY_CACHE[key] = (stamp, skills)
-    return skills
+    skills = data.get("skills", [])
+    current = data.get("format") == CACHE_FORMAT
+    _ENTRY_CACHE[key] = (stamp, skills, current)
+    return skills, current
 
 
 def load_tap(tap: registry.Tap, rebuild: bool = False) -> list[dict]:
@@ -223,11 +279,20 @@ def load_tap(tap: registry.Tap, rebuild: bool = False) -> list[dict]:
 
     `rebuild=True` or a missing/corrupt cache forces a rescan; an
     uncloned tap yields `[]`.
+
+    A cache written by an older scanner is rescanned *when the clone is still
+    there to rescan* — that is what backfills a new entry field across every tap
+    on a machine (460 of them on a real install) without asking anyone to re-tap.
+    When the clone is gone the stale entries are served as they are: consumers
+    all degrade cleanly on a missing field, so refusing to answer would cost the
+    user their catalogue to buy a digest.
     """
     if not rebuild:
         cached = _cached_tap(tap)
         if cached is not None:
-            return cached
+            skills, current = cached
+            if current or not tap.is_cloned:
+                return skills
     if tap.is_cloned:
         return rebuild_tap(tap)
     return []
@@ -487,12 +552,48 @@ def resolve_one(name: str, path: str | None = None) -> dict:
                 % (bare, len(matches), taps[0], paths),
                 hint="that registry ships one name twice — pick one with "
                      "`--path <one of the above>`, and raise it with the tap")
+        # Several taps, one thing. Registries mirror each other constantly, so
+        # the common cross-tap collision is N byte-identical copies of one
+        # skill — and the within-tap branch above already decided that
+        # indistinguishable candidates are boost's problem to resolve, not the
+        # user's. Applying the same judgement here removes a dead end (a
+        # mirrored skill could not be installed unqualified at all) without
+        # touching the genuine-ambiguity error, which still fires whenever the
+        # copies actually differ.
+        best = _same_thing(matches)
+        if best is not None:
+            return best
         raise BoostError(
             "%r exists in multiple taps: %s" % (name, ", ".join(taps)),
             hint="qualify it, e.g. `%s:%s`" % (taps[0], bare))
     return matches[0]
 
 
+def _same_thing(matches: list[dict]) -> dict | None:
+    """The copy to install when every candidate is the same content, else None.
+
+    Returns None unless *every* candidate carries a digest and they all agree.
+    A missing digest is an unknown, never a match: assuming two unknowns equal
+    would silently install one skill when the user asked about another, which is
+    strictly worse than the error it would be replacing.
+
+    Ranked by :func:`rag.source_rank` — the user's own ``curated`` flag first,
+    the shipped registry confidence second — which exists for exactly this
+    decision and is what `dedupe_by_content` already uses to choose between
+    identical bodies in search results. Ties break on ``(tap, rel_dir)`` so the
+    pick cannot vary between runs.
+    """
+    digests = {e.get("content") for e in matches}
+    if len(digests) != 1 or not next(iter(digests)):
+        return None
+    # Local import: `rag` imports this module, so the edge only exists at call
+    # time. Same shape as `store.install`'s deferred `gitutil` import.
+    from . import rag
+    return min(matches, key=lambda e: (rag.source_rank(e),
+                                       str(e.get("tap", "")),
+                                       str(e.get("rel_dir", ""))))
+
+
 def _meta_text(meta) -> str:
     """Lowercased, space-joined flatten of a frontmatter dict for substring
     search. Cheaper than ``json.dumps(meta).lower()`` — which built and threw

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_cli/core/dense.py

@@ -580,23 +580,47 @@ def _embed_and_store(con: sqlite3.Connection, entries: list[dict],
     for e in entries:
         for ci, text in enumerate(_chunk_texts(e, tap_paths)):
             rows.append((e, ci, text))
+
+    # Embed each DISTINCT text once. Registries mirror each other, so one text
+    # arrives many times in a single build: 42.9% of 750,416 chunks on a real
+    # 460-tap install are repeats, worst case 1,464 identical copies. The
+    # provider is deterministic, so the copies were being bought at full price
+    # for a byte-identical vector — and `retrieve_any` runs `dedupe_by_content`
+    # on every retrieval path, so they were discarded before reaching a result
+    # slot. Insertion below is still per row, because `chunks.tap` is what
+    # scopes tap deletion and collapsing rows would strand a tap's vectors.
+    order: list[str] = []
+    seen: set[str] = set()
+    for _e, _ci, text in rows:
+        if text not in seen:
+            seen.add(text)
+            order.append(text)
+
+    vec_of: dict[str, object] = {}
+    for start in range(0, len(order), _BATCH):
+        batch = order[start:start + _BATCH]
+        vecs = embed.embed(batch, input_type="document")
+        if not vecs or len(vecs) != len(batch):
+            continue     # taps are attributed per row below, not per batch
+        vec_of.update(zip(batch, vecs, strict=True))
+
     added = 0
     failed_taps: set = set()
-    for start in range(0, len(rows), _BATCH):
-        batch = rows[start:start + _BATCH]
-        vecs = embed.embed([t for _e, _c, t in batch], input_type="document")
-        if not vecs or len(vecs) != len(batch):
-            failed_taps.update(e["tap"] for e, _c, _t in batch)
+    for e, ci, text in rows:
+        vec = vec_of.get(text)
+        if vec is None:
+            # Every tap owning a copy of a text the provider rejected, not just
+            # whichever copy happened to sit in the failed batch.
+            failed_taps.add(e["tap"])
             continue
-        for (e, ci, text), vec in zip(batch, vecs, strict=True):
-            cur = con.execute(
-                "INSERT INTO chunks (name, tap, path, kind, cix, snip) "
-                "VALUES (?, ?, ?, ?, ?, ?)",
-                (e["name"], e["tap"], e["skill_md"],
-                 e.get("kind", "skill"), ci, text[:200].strip()))
-            con.execute("INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
-                        (cur.lastrowid, mod.serialize_float32(vec)))
-            added += 1
+        cur = con.execute(
+            "INSERT INTO chunks (name, tap, path, kind, cix, snip) "
+            "VALUES (?, ?, ?, ?, ?, ?)",
+            (e["name"], e["tap"], e["skill_md"],
+             e.get("kind", "skill"), ci, text[:200].strip()))
+        con.execute("INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
+                    (cur.lastrowid, mod.serialize_float32(vec)))
+        added += 1
     return added, failed_taps
 
 

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_cli/core/rag.py

@@ -222,9 +222,13 @@ def _make_docs(entries: list[dict], tap_paths: dict[str, Path]) -> list[dict]:
     docs: list[dict] = []
     for e in entries:
         body = read_body(e, tap_paths)
-        # Hashed here because the body is already in hand; doing it at query
-        # time would mean re-reading ~30k files to answer one search.
-        digest = hashlib.sha256(body.strip().encode("utf-8", "replace")).hexdigest()[:16]
+        # The scanner already hashed this exact text (`catalog._content_digest`
+        # assembles name + description + body, the same string `read_body`
+        # returns), so an entry from a current cache hands the value over. The
+        # fallback is not dead code: a cache written before CACHE_FORMAT, or an
+        # entry synthesised by a caller rather than scanned, still has to index.
+        digest = e.get("content") or hashlib.sha256(
+            body.strip().encode("utf-8", "replace")).hexdigest()[:16]
         tf: dict[str, int] = defaultdict(int)
         for tok in tokenize(surface(e) + "\n" + body):
             tf[tok] += 1

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.2
+Version: 1.2.4
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/boost_skill_cli.egg-info/SOURCES.txt

@@ -204,6 +204,7 @@ docs/roadmap/items/browse-crashes-selecting-a-rule-or-workflow.md
 docs/roadmap/items/build-provenance-slsa-attestations.md
 docs/roadmap/items/cache-the-catalog-entry-set-across-rag-queries.md
 docs/roadmap/items/capability-manifest-and-least-privilege-policy.md
+docs/roadmap/items/catalogue-entries-carry-no-content-identity.md
 docs/roadmap/items/chat-cites-sources-by-a-number-it-never-prints.md
 docs/roadmap/items/chat-command-grounded-conversational-search.md
 docs/roadmap/items/ci-job-timeout-minutes.md
@@ -434,6 +435,7 @@ docs/roadmap/items/workspace-scope-install.md
 docs/roadmap/items/zip-strict-audit.md
 docs/roadmap/items/zizmor-ignores-drift-and-make-lint-never-ran-it.md
 docs/roadmap/items/zizmor-pin-is-a-yanked-release.md
+docs/superpowers/specs/2026-08-16-catalogue-content-identity-design.md
 evals/README.md
 evals/__init__.py
 evals/baseline.json
@@ -596,10 +598,12 @@ tests/unit/test_complete.py
 tests/unit/test_complete_rc_block.py
 tests/unit/test_config.py
 tests/unit/test_content_dedupe.py
+tests/unit/test_content_identity.py
 tests/unit/test_crash_redaction.py
 tests/unit/test_dedup_trust.py
 tests/unit/test_demo_index.py
 tests/unit/test_dense.py
+tests/unit/test_dense_embed_dedupe.py
 tests/unit/test_dense_fallback.py
 tests/unit/test_dense_fix_hint.py
 tests/unit/test_dense_shard_real_schema.py

boost_skill_cli-1.2.4/docs/roadmap/items/catalogue-entries-carry-no-content-identity.md

@@ -0,0 +1,56 @@
+---
+id: catalogue-entries-carry-no-content-identity
+board: code
+section: internals
+status: shipped
+category: Catalog · Correctness
+complexity: M
+impact: High
+wow: 5
+note: 42.9% of 750,416 embedded chunks are duplicates — one was embedded 1,464 times
+order: 116
+owner: feat/catalogue-content-identity
+pr: 526
+title: A catalog entry knew where it came from, never what it was
+---
+<b>boost collapses duplicate copies at both places a user sees them</b> &mdash;
+<code>rag.dedupe_by_content</code> for search results, <code>resolve_one</code> for install
+resolution &mdash; and at neither place it <em>pays</em> for them. A catalog entry carried
+<code>(tap, skill_md)</code>, which is row identity and deliberately strict, but nothing that said
+<em>this is the same thing as that</em>. Every consumer needing content identity therefore derived
+its own, and they disagreed.
+
+<b>The bill was measurable.</b> On a real 460-tap install <code>rag_vectors.sqlite</code> is
+<b>3.2&nbsp;GB</b>, and <b>42.9% of its 750,416 chunks are duplicates</b> &mdash; one chunk was
+embedded <b>1,464 times</b>. Those embeddings cost API credits and buy nothing:
+<code>retrieve_any</code> runs <code>dedupe_by_content</code> on every retrieval path, so each
+redundant vector is computed, stored, billed, then discarded before it can reach a result slot.
+
+<b>Three keys were in play, and the comment was wrong about which one shipped.</b>
+<code>dedupe_by_content</code>'s docstring says clustering is &ldquo;on the body, never the name&rdquo;
+and offers as proof that no cluster spans more than one name. Both are artefacts: the code hashes
+<code>read_body()</code>, which <em>prepends</em> name and description, so zero name-spanning clusters
+is guaranteed rather than measured. Over all 60,047 entries:
+
+<b>name + description + body</b> keeps 41,051 clusters and spans more than one name <b>zero</b> times
+&middot; <em>body alone</em> keeps 40,372 but spans more than one name <b>259</b> times, which is the
+<code>admin-interface-rule</code> collision again &middot; <em>name + description</em> keeps 37,668,
+over-collapsing <b>3,383</b> clusters of items that share metadata while carrying different prose.
+
+<b>So the entry now carries the digest the code already believed in.</b>
+<code>_make_entry</code> hashes name + description + body at scan time, which is free &mdash; the
+body is already read, parsed and about to be discarded, measured at <b>2.04&nbsp;&micro;s per entry,
+122&nbsp;ms for the whole corpus</b>. It also <em>removes</em> work: <code>rag</code> was re-opening
+all 60k files at index-build time purely to recompute this. Verified against real data by rescanning
+40 tapped repos and checking all <b>8,564</b> entries against what <code>read_body</code> hashes:
+zero mismatches.
+
+<b>Nothing is dropped.</b> Duplicate rows stay in the catalogue, because removing them breaks
+<code>--path</code> disambiguation, typosquat detection, tap-scoped repair in <code>sync_apply</code>,
+the eval harness's exemplar resolution, and the <code>source_rank</code> quality prior. The mark is
+the digest, and it is reversible by rebuilding a cache. The tap cache gained a
+<code>format</code> version to backfill 460 machines' worth of caches on read &mdash; it was the one
+derived artifact that could not self-invalidate, which is why no entry field could ever be added
+before. A stale cache whose clone is gone is still served rather than discarded: consumers all
+degrade cleanly on a missing digest, so refusing to answer would cost a user their catalogue to buy
+a hash.

{boost_skill_cli-1.2.2 → boost_skill_cli-1.2.4}/docs/roadmap.html

@@ -133,7 +133,7 @@
       <div class="stat"><b>16</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>7</b><span>Planned</span></div>
-      <div class="stat"><b>253</b><span>Loop finds</span></div>
+      <div class="stat"><b>254</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -4023,6 +4023,59 @@ storage cleanup.</p>
           <span class="m">458 taps held 12 GB to index 1.9 GB of Markdown</span>
         </div>
       </article>
+
+      <article class="cap rcard" id="catalogue-entries-carry-no-content-identity">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Catalog · Correctness</span></div>
+        <h3>A catalog entry knew where it came from, never what it was</h3>
+        <details class="cardbody"><summary>Write-up</summary>
+        <p><b>boost collapses duplicate copies at both places a user sees them</b> &mdash;
+<code>rag.dedupe_by_content</code> for search results, <code>resolve_one</code> for install
+resolution &mdash; and at neither place it <em>pays</em> for them. A catalog entry carried
+<code>(tap, skill_md)</code>, which is row identity and deliberately strict, but nothing that said
+<em>this is the same thing as that</em>. Every consumer needing content identity therefore derived
+its own, and they disagreed.
+
+<b>The bill was measurable.</b> On a real 460-tap install <code>rag_vectors.sqlite</code> is
+<b>3.2&nbsp;GB</b>, and <b>42.9% of its 750,416 chunks are duplicates</b> &mdash; one chunk was
+embedded <b>1,464 times</b>. Those embeddings cost API credits and buy nothing:
+<code>retrieve_any</code> runs <code>dedupe_by_content</code> on every retrieval path, so each
+redundant vector is computed, stored, billed, then discarded before it can reach a result slot.
+
+<b>Three keys were in play, and the comment was wrong about which one shipped.</b>
+<code>dedupe_by_content</code>'s docstring says clustering is &ldquo;on the body, never the name&rdquo;
+and offers as proof that no cluster spans more than one name. Both are artefacts: the code hashes
+<code>read_body()</code>, which <em>prepends</em> name and description, so zero name-spanning clusters
+is guaranteed rather than measured. Over all 60,047 entries:
+
+<b>name + description + body</b> keeps 41,051 clusters and spans more than one name <b>zero</b> times
+&middot; <em>body alone</em> keeps 40,372 but spans more than one name <b>259</b> times, which is the
+<code>admin-interface-rule</code> collision again &middot; <em>name + description</em> keeps 37,668,
+over-collapsing <b>3,383</b> clusters of items that share metadata while carrying different prose.
+
+<b>So the entry now carries the digest the code already believed in.</b>
+<code>_make_entry</code> hashes name + description + body at scan time, which is free &mdash; the
+body is already read, parsed and about to be discarded, measured at <b>2.04&nbsp;&micro;s per entry,
+122&nbsp;ms for the whole corpus</b>. It also <em>removes</em> work: <code>rag</code> was re-opening
+all 60k files at index-build time purely to recompute this. Verified against real data by rescanning
+40 tapped repos and checking all <b>8,564</b> entries against what <code>read_body</code> hashes:
+zero mismatches.
+
+<b>Nothing is dropped.</b> Duplicate rows stay in the catalogue, because removing them breaks
+<code>--path</code> disambiguation, typosquat detection, tap-scoped repair in <code>sync_apply</code>,
+the eval harness's exemplar resolution, and the <code>source_rank</code> quality prior. The mark is
+the digest, and it is reversible by rebuilding a cache. The tap cache gained a
+<code>format</code> version to backfill 460 machines' worth of caches on read &mdash; it was the one
+derived artifact that could not self-invalidate, which is why no entry field could ever be added
+before. A stale cache whose clone is gone is still served rather than discarded: consumers all
+degrade cleanly on a missing digest, so refusing to answer would cost a user their catalogue to buy
+a hash.</p></details>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★★</b></span>
+          <span class="m">42.9% of 750,416 embedded chunks are duplicates — one was embedded 1,464 times</span>
+        </div>
+      </article>
       <!-- /ROADMAP:cards -->
 
     </div>