boost-skill-cli 1.0.358__tar.gz → 1.0.360__tar.gz

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/CLAUDE.md

@@ -63,12 +63,22 @@ corpus: `scripts/ensure_eval_corpus.sh` first taps the pinned repo list in
 --defaults` is NOT enough, it omits every rule/workflow repo). The list is
 **twenty** repos: the first six cover every golden target, the rest exist so the
 corpus is a realistic size. That matters more than it sounds — over the six
-alone BM25 scored 1.000 / 0.791 / 0.854 / 0.882, and over the twenty it scores
-**0.863 / 0.473 / 0.607 / 0.662** on the same golden set, so three of the four
-old floors fail once the corpus stops being tiny. Each floor now sits ~10% under
-its measured value — loose enough that upstream drift can't flake the build,
-tight enough to catch a collapse. Regression-vs-baseline stays relaxed
-(`--regression-eps 1`), so the absolute floors are the real gate.
+alone (743 entries) BM25 scores 0.978 / 0.791 / 0.854 / 0.882, and over the
+twenty (10,152) it scores **0.852 / 0.473 / 0.605 / 0.657** on the same golden
+set, so three of the four old floors fail once the corpus stops being tiny.
+
+**The ranked list de-duplicates on the content hash, not the name.** A grade key
+decides both relevance and identity, and keying identity on the name collapsed
+13 different skills called `code-reviewer` into one rank slot — crediting the
+ranker with a compression that existed only in the scoring code, and worth about
+one query of recall@10. That is where the old "recall is 1.000" folklore came
+from; the six-repo corpus measures 0.978 once mirrors collapse and homonyms do
+not. Relevance is still decided by name (or by content class when a golden row
+pins an `exemplar`), so the sets can migrate a row at a time.
+
+Each floor sits ~10% under its measured value — loose enough that upstream drift
+can't flake the build, tight enough to catch a collapse. Regression-vs-baseline
+stays relaxed (`--regression-eps 1`), so the absolute floors are the real gate.
 
 **Every row of `taps.txt` pins a commit SHA**, and `scripts/eval_corpus.py`
 checks each clone out at it — the corpus is 10,152 entries, of which one

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.358
+Version: 1.0.360
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.0.358'
-__version_tuple__ = version_tuple = (1, 0, 358)
+__version__ = version = '1.0.360'
+__version_tuple__ = version_tuple = (1, 0, 360)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/boost_skill_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.0.358
+Version: 1.0.360
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License: GNU GENERAL PUBLIC LICENSE

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/boost_skill_cli.egg-info/SOURCES.txt

@@ -214,6 +214,7 @@ docs/roadmap/items/end-of-options-guard-on-git-commands.md
 docs/roadmap/items/eval-corpus-cannot-see-real-retrieval.md
 docs/roadmap/items/eval-corpus-is-96x-smaller-than-a-real-install.md
 docs/roadmap/items/eval-corpus-was-not-actually-pinned.md
+docs/roadmap/items/eval-deduped-ranked-lists-by-name.md
 docs/roadmap/items/extension-free-tests-core-dense-py.md
 docs/roadmap/items/extract-mcp-http-servers-out-of-configuration-py.md
 docs/roadmap/items/finish-mutation-hardening-across-core.md

boost_skill_cli-1.0.360/docs/roadmap/items/eval-deduped-ranked-lists-by-name.md

@@ -0,0 +1,50 @@
+---
+id: eval-deduped-ranked-lists-by-name
+board: code
+section: internals
+status: shipped
+category: Eval · Correctness
+complexity: M
+impact: High
+wow: 4
+note: 13 different skills named code-reviewer shared one rank slot — where "recall is 1.000" came from
+order: 83
+owner: loop/eval-dedup-by-body
+pr: 411
+title: The eval de-duplicated its ranked list by name, so homonyms shared a rank
+---
+<b>A grade key does two jobs, and they needed different answers.</b> It decides whether an entry is
+<em>relevant</em>, and it is the <em>identity</em> the ranked list is de-duplicated on. Both were the
+entry's name &mdash; so the thirteen genuinely different skills called <code>code-reviewer</code> in
+the pinned corpus collapsed into <b>one rank slot</b>, and every metric was computed over a list
+about a third shorter than the one a user would scroll.
+
+<b>This is not a rounding error dressed up.</b> It credited the ranker with a compression that exists
+only in the scoring code. Measured over the pinned corpus, de-duplicating on the content hash instead
+moves BM25 recall@10 from <b>0.863 to 0.852</b> &mdash; roughly one golden query &mdash; and over the
+six-repo minimal set from <b>1.000 to 0.978</b>. That second number is the interesting one: it is
+where the project's &ldquo;retrieval recall is 1.000&rdquo; folklore came from. The corpus never had
+perfect recall; it had a scoring key that merged wrong answers into right ones.
+
+<b>Exemplar rows had the mirror-image bug.</b> When a row pins an exemplar, its <em>distractors</em>
+were keyed on <code>tap::skill_md</code> &mdash; so two byte-identical mirrors of a distractor each
+took a slot and pushed the target later. The two bugs pointed opposite ways, which meant an
+exemplar-graded row and a name-graded row could not honestly be averaged into a single number. That
+blocked finishing [[golden-set-grades-by-name-not-by-skill]], whose whole design is that rows migrate
+one at a time.
+
+<b>The fix separates the two jobs.</b> Relevance is decided by name, or by content class when the row
+pins an exemplar &mdash; unchanged, so a multi-name <code>relevant</code> list still needs each
+distinct name found. Identity is always the entry's content hash: mirrors of one skill collapse
+(counting them twice rewards nothing), distinct bodies do not (a user really does see thirteen
+entries). One convention for every row.
+
+<b>What this does not fix.</b> The corpus is still 10,152 entries against a real install's far
+larger one &mdash; see [[eval-corpus-is-96x-smaller-than-a-real-install]] &mdash; and the golden set
+still grades 22 of its 50 natural-language rows against a name that resolves to several different
+skills. This makes finishing that migration possible; it does not finish it. The corrected floors are
+comfortably clear of the gate (recall 0.852 against 0.78), so no threshold moved.
+
+<b>Provenance.</b> Found while working out how to add the remaining exemplars: the blocker turned out
+not to be the 22 judgment calls but the fact that a half-migrated set would average two different
+rank conventions. Measuring that is what exposed the name-collapse underneath it.

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/docs/roadmap/items/golden-set-grades-by-name-not-by-skill.md

@@ -7,10 +7,10 @@ category: Eval · Correctness
 complexity: M
 impact: High
 wow: 4
-note: the mechanism ships; which of 59 code-reviewers each query means is still a human call
+note: 28 of 50 rows pinned mechanically with zero change to any number; 22 are genuine judgment calls
 order: 81
 owner: loop/golden-name-grading
-pr:
+pr: 412
 title: The golden set grades by name, and 35 of 53 names are ambiguous
 ---
 <b>The eval scores a hit when the top result carries the right <em>name</em>. Most of those names do
@@ -84,3 +84,24 @@ statement about intent. Guessing it would bake one opinion into the number the p
 invisibly. The harness is ready for those decisions one row at a time; each added exemplar tightens
 the metric and none of them destabilise it.
 
+<b>Progress: 28 of the 50 rows are now pinned, and pinning them changed nothing.</b> Measured over
+the SHA-pinned corpus, 28 rows have the property that every name in their <code>relevant</code> list
+resolves to exactly <b>one body</b> &mdash; so the exemplar is a lookup, not a judgment, and grading
+by content class must return the same verdict as grading by name. It does: the natural-language set
+scores <b>0.350 / 0.160 / 0.245 / 0.259</b> before and after, identical to three decimal places.
+That equality is the point of shipping them &mdash; the rows are now explicit about which skill they
+mean, at zero cost to comparability.
+
+<b>The remaining 22 are the real content of this card.</b> <code>code-reviewer</code> is 13 distinct
+skills in this corpus, <code>update-docs</code> 10, <code>commit</code> 4. There is no shortcut
+available: their descriptions share a median similarity of about <b>0.15</b>, so these are genuine
+forks rather than one skill re-published, and no rule separates them without someone saying what the
+question meant. The menu is generated rather than written down, because the candidate set is a fact
+about the corpus that is tapped:
+
+<code>python3 scripts/eval_retrieval.py --golden tests/eval/golden-natural.jsonl --worksheet</code>
+
+<b>What unblocked this.</b> Not the judgment calls &mdash; [[eval-deduped-ranked-lists-by-name]].
+A half-migrated set was averaging two different rank conventions, because name-graded rows collapsed
+homonyms into one rank slot while exemplar-graded rows gave every distractor mirror its own. Until
+both used one convention, migrating rows one at a time produced a number that meant nothing.

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/docs/roadmap.html

@@ -125,7 +125,7 @@
       <div class="stat"><b>16</b><span>Shipped</span></div>
       <div class="stat"><b>1</b><span>Next up</span></div>
       <div class="stat"><b>4</b><span>Planned</span></div>
-      <div class="stat"><b>197</b><span>Loop finds</span></div>
+      <div class="stat"><b>198</b><span>Loop finds</span></div>
       <!-- /ROADMAP:stats -->
     </div>
   </section>
@@ -2412,12 +2412,34 @@ weaker gate that still reports a number, which is the failure this card exists t
 <code>code-reviewer</code>s a query about reviewing a diff for security problems refers to is a
 statement about intent. Guessing it would bake one opinion into the number the project publishes,
 invisibly. The harness is ready for those decisions one row at a time; each added exemplar tightens
-the metric and none of them destabilise it.</p>
+the metric and none of them destabilise it.
+
+<b>Progress: 28 of the 50 rows are now pinned, and pinning them changed nothing.</b> Measured over
+the SHA-pinned corpus, 28 rows have the property that every name in their <code>relevant</code> list
+resolves to exactly <b>one body</b> &mdash; so the exemplar is a lookup, not a judgment, and grading
+by content class must return the same verdict as grading by name. It does: the natural-language set
+scores <b>0.350 / 0.160 / 0.245 / 0.259</b> before and after, identical to three decimal places.
+That equality is the point of shipping them &mdash; the rows are now explicit about which skill they
+mean, at zero cost to comparability.
+
+<b>The remaining 22 are the real content of this card.</b> <code>code-reviewer</code> is 13 distinct
+skills in this corpus, <code>update-docs</code> 10, <code>commit</code> 4. There is no shortcut
+available: their descriptions share a median similarity of about <b>0.15</b>, so these are genuine
+forks rather than one skill re-published, and no rule separates them without someone saying what the
+question meant. The menu is generated rather than written down, because the candidate set is a fact
+about the corpus that is tapped:
+
+<code>python3 scripts/eval_retrieval.py --golden tests/eval/golden-natural.jsonl --worksheet</code>
+
+<b>What unblocked this.</b> Not the judgment calls &mdash; [[eval-deduped-ranked-lists-by-name]].
+A half-migrated set was averaging two different rank conventions, because name-graded rows collapsed
+homonyms into one rank slot while exemplar-graded rows gave every distractor mirror its own. Until
+both used one convention, migrating rows one at a time produced a number that meant nothing.</p>
         <div class="meta">
           <span class="m">Complexity <b>M</b></span>
           <span class="m hi">Impact <b>High</b></span>
           <span class="m">Wow <b class="wow">★★★★</b></span>
-          <span class="m">the mechanism ships; which of 59 code-reviewers each query means is still a human call</span>
+          <span class="m">28 of 50 rows pinned mechanically with zero change to any number; 22 are genuine judgment calls</span>
         </div>
       </article>
 
@@ -2485,6 +2507,53 @@ measured on an install that was missing the repo holding 62% of it.</p></details
           <span class="m">62% of the required gate's corpus was one unpinned third-party repo, against a 1.15-query margin</span>
         </div>
       </article>
+
+      <article class="cap rcard" id="eval-deduped-ranked-lists-by-name">
+        <div class="head"><span class="pill shipped">Shipped</span><span class="cat">Eval · Correctness</span></div>
+        <h3>The eval de-duplicated its ranked list by name, so homonyms shared a rank</h3>
+        <details class="cardbody"><summary>Write-up</summary>
+        <p><b>A grade key does two jobs, and they needed different answers.</b> It decides whether an entry is
+<em>relevant</em>, and it is the <em>identity</em> the ranked list is de-duplicated on. Both were the
+entry's name &mdash; so the thirteen genuinely different skills called <code>code-reviewer</code> in
+the pinned corpus collapsed into <b>one rank slot</b>, and every metric was computed over a list
+about a third shorter than the one a user would scroll.
+
+<b>This is not a rounding error dressed up.</b> It credited the ranker with a compression that exists
+only in the scoring code. Measured over the pinned corpus, de-duplicating on the content hash instead
+moves BM25 recall@10 from <b>0.863 to 0.852</b> &mdash; roughly one golden query &mdash; and over the
+six-repo minimal set from <b>1.000 to 0.978</b>. That second number is the interesting one: it is
+where the project's &ldquo;retrieval recall is 1.000&rdquo; folklore came from. The corpus never had
+perfect recall; it had a scoring key that merged wrong answers into right ones.
+
+<b>Exemplar rows had the mirror-image bug.</b> When a row pins an exemplar, its <em>distractors</em>
+were keyed on <code>tap::skill_md</code> &mdash; so two byte-identical mirrors of a distractor each
+took a slot and pushed the target later. The two bugs pointed opposite ways, which meant an
+exemplar-graded row and a name-graded row could not honestly be averaged into a single number. That
+blocked finishing [[golden-set-grades-by-name-not-by-skill]], whose whole design is that rows migrate
+one at a time.
+
+<b>The fix separates the two jobs.</b> Relevance is decided by name, or by content class when the row
+pins an exemplar &mdash; unchanged, so a multi-name <code>relevant</code> list still needs each
+distinct name found. Identity is always the entry's content hash: mirrors of one skill collapse
+(counting them twice rewards nothing), distinct bodies do not (a user really does see thirteen
+entries). One convention for every row.
+
+<b>What this does not fix.</b> The corpus is still 10,152 entries against a real install's far
+larger one &mdash; see [[eval-corpus-is-96x-smaller-than-a-real-install]] &mdash; and the golden set
+still grades 22 of its 50 natural-language rows against a name that resolves to several different
+skills. This makes finishing that migration possible; it does not finish it. The corrected floors are
+comfortably clear of the gate (recall 0.852 against 0.78), so no threshold moved.
+
+<b>Provenance.</b> Found while working out how to add the remaining exemplars: the blocker turned out
+not to be the 22 judgment calls but the fact that a half-migrated set would average two different
+rank conventions. Measuring that is what exposed the name-collapse underneath it.</p></details>
+        <div class="meta">
+          <span class="m">Complexity <b>M</b></span>
+          <span class="m hi">Impact <b>High</b></span>
+          <span class="m">Wow <b class="wow">★★★★</b></span>
+          <span class="m">13 different skills named code-reviewer shared one rank slot — where "recall is 1.000" came from</span>
+        </div>
+      </article>
       <!-- /ROADMAP:cards -->
 
     </div>

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/scripts/eval_retrieval.py

@@ -126,8 +126,14 @@ METRICS: Dict[str, Callable[[Sequence[str], set, int], float]] = {
 # punish a correct answer for arriving from a mirror), while a different skill
 # sharing the name does not.
 #
-# Rows without an exemplar keep name grading byte-for-byte, so published
-# baselines stay comparable and the two styles can coexist during a migration.
+# Rows without an exemplar still decide RELEVANCE by name, so the sets can
+# migrate a row at a time. What is no longer name-keyed is IDENTITY: the ranked
+# list de-duplicates on the content hash for every row, exemplar or not. Keying
+# both on the name collapsed 13 different `code-reviewer`s into one rank slot
+# and inflated recall@10 by about one query (0.863 -> 0.852 over the pinned
+# corpus); keying an exemplar row's distractors on tap::skill_md did the
+# opposite, giving byte-identical mirrors a slot each. Mixed sets could not be
+# averaged into one number until both used the same convention.
 
 _EXEMPLAR_SEP = "::"
 
@@ -165,19 +171,39 @@ def prepare_row(row: dict, hashes: dict) -> dict:
 def grade_key(row: dict, entry: dict, hashes: dict) -> str:
     """The token this entry contributes to a ranked list, for scoring.
 
+    The key does two jobs, and they need different answers. It decides whether
+    an entry is RELEVANT — by name, or by content class when the row pins an
+    exemplar — and it is the IDENTITY the ranked list is de-duplicated on.
+
+    Keying both on the name conflated them: 13 genuinely different skills named
+    `code-reviewer` collapsed into one rank slot, so the eval credited the
+    ranker with a compression that exists only here. Measured over the pinned
+    corpus that was worth about one query of recall@10 (0.863 against 0.852).
+    Exemplar rows had the inverse bug — their distractors keyed on
+    ``tap::skill_md``, so byte-identical mirrors each took a slot and pushed the
+    target later.
+
+    So: relevance by name or class, identity always by content hash. One
+    convention for every row, which is what allows an exemplar-graded row and a
+    name-graded row to be averaged into the same number.
+
     ``hashes`` is passed rather than read from module state: the map is the
     thing that decides whether two entries are the same skill, so a caller must
     not be able to grade against a different one by accident.
     """
+    digest = hashes.get((entry.get("tap", ""), entry.get("skill_md", "")))
     classes = row.get("class_hashes")
-    if not classes:
+    if classes:
+        if digest and digest in classes:
+            return "cls:%s" % sorted(classes)[0]
+    elif str(entry.get("name", "")) in row["relevant_set"]:
         return str(entry.get("name", ""))
-    digest = hashes.get((entry.get("tap", ""), entry.get("skill_md", "")))
-    if digest and digest in classes:
-        return "cls:%s" % sorted(classes)[0]
-    # Distinct, so two different homonyms never collapse into one another and
-    # inflate recall.
-    return "not:%s::%s" % (entry.get("tap", ""), entry.get("skill_md", ""))
+    if digest:
+        return "body:%s" % digest
+    # No hash (an entry the index never saw): fall back to a key that is unique
+    # per entry. Colliding here would silently shorten the ranked list and
+    # flatter every metric computed from it.
+    return "nohash:%s::%s" % (entry.get("tap", ""), entry.get("skill_md", ""))
 
 
 def relevant_keys(row: dict) -> set:
@@ -187,6 +213,46 @@ def relevant_keys(row: dict) -> set:
     return {"cls:%s" % sorted(classes)[0]}
 
 
+def exemplar_worksheet(rows: List[dict], entries: List[dict],
+                       hashes: dict) -> List[dict]:
+    """Rows still graded by name whose name resolves to several bodies.
+
+    Pinning those is a judgment about what the question meant, not a lookup, so
+    this hands over the menu rather than guessing: for each undecided row, every
+    distinct body a relevant name resolves to, with its description.
+
+    Generated rather than committed as a comment in the golden file, because the
+    candidate list is a fact about the corpus that is currently tapped. Written
+    down, it would be wrong the first time a pin moves.
+    """
+    by_name: Dict[str, List[dict]] = {}
+    for entry in entries:
+        by_name.setdefault(str(entry.get("name", "")), []).append(entry)
+    sheet: List[dict] = []
+    for row in rows:
+        if row.get("class_hashes"):
+            continue                       # already decided
+        seen: Dict[str, dict] = {}
+        for name in row["relevant_set"]:
+            for entry in by_name.get(name, []):
+                digest = hashes.get((entry.get("tap", ""), entry.get("skill_md", "")))
+                if digest and digest not in seen:
+                    seen[digest] = entry
+        if len(seen) < 2:
+            continue                       # determined, or absent entirely
+        sheet.append({
+            "query": row["query"],
+            "candidates": [
+                {"spec": "%s%s%s" % (e.get("tap", ""), _EXEMPLAR_SEP,
+                                     e.get("skill_md", "")),
+                 "description": (e.get("description") or "").strip().split("\n")[0]}
+                for e in sorted(seen.values(),
+                                key=lambda x: (x.get("tap", ""), x.get("skill_md", "")))
+            ],
+        })
+    return sheet
+
+
 def dedupe_keys(keys):
     """Collapse to the first (best-ranked) occurrence of each key."""
     seen: set = set()
@@ -577,6 +643,9 @@ def main(argv: Optional[List[str]] = None) -> int:
                     help="Tier 1b: ranx significance test between engines "
                          "(opt-in [eval] extra; degrades if ranx absent)")
     ap.add_argument("--json", action="store_true", help="machine-readable JSON")
+    ap.add_argument("--worksheet", action="store_true",
+                    help="list the golden rows still graded by name whose name "
+                         "resolves to several bodies, with the candidates")
     args = ap.parse_args(argv)
 
     floors = parse_floors(args.floor)          # fail fast on a bad --floor
@@ -590,6 +659,22 @@ def main(argv: Optional[List[str]] = None) -> int:
             print("  indexed %d entries -> %d chunks across %d taps"
                   % (stats["entries"], stats["docs"], stats["taps"]))
 
+    if args.worksheet:
+        sheet = exemplar_worksheet(rows, catalog.all_entries(), rag.content_hashes())
+        if args.json:
+            print(json.dumps(sheet, indent=2))
+            return 0
+        print("%d of %d rows still graded by name resolve to several bodies.\n"
+              % (len(sheet), len(rows)))
+        for case in sheet:
+            print("  %s" % case["query"])
+            for cand in case["candidates"]:
+                print("      %s" % cand["spec"])
+                if cand["description"]:
+                    print("          %s" % cand["description"][:96])
+            print()
+        return 0
+
     if args.rerank:
         return run_rerank_lift(rows, args.k, args.json)
 

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/tests/eval/baseline.json

@@ -5,16 +5,16 @@
       "golden": "golden.jsonl",
       "engines": {
         "catalog.search": {
-          "recall@k": 0.7527472527472527,
+          "recall@k": 0.7417582417582418,
           "hit@1": 0.46153846153846156,
-          "MRR": 0.5660896594773488,
-          "nDCG@k": 0.6022495608262091
+          "MRR": 0.5622940186865697,
+          "nDCG@k": 0.5966760830419942
         },
         "BM25 full-content": {
-          "recall@k": 0.8626373626373627,
+          "recall@k": 0.8516483516483516,
           "hit@1": 0.4725274725274725,
-          "MRR": 0.607356395043468,
-          "nDCG@k": 0.6624090789490626
+          "MRR": 0.6047880662244283,
+          "nDCG@k": 0.6574937765278362
         }
       }
     },
@@ -25,14 +25,14 @@
         "catalog.search": {
           "recall@k": 0.08,
           "hit@1": 0.02,
-          "MRR": 0.04203489933881709,
-          "nDCG@k": 0.04333333333333333
+          "MRR": 0.040721853968010216,
+          "nDCG@k": 0.04244796319302442
         },
         "BM25 full-content": {
-          "recall@k": 0.37,
+          "recall@k": 0.35,
           "hit@1": 0.16,
-          "MRR": 0.24542437429912048,
-          "nDCG@k": 0.26491049774398634
+          "MRR": 0.2447192741990893,
+          "nDCG@k": 0.2591292012176286
         }
       }
     }

{boost_skill_cli-1.0.358 → boost_skill_cli-1.0.360}/tests/eval/golden-natural.jsonl

@@ -31,25 +31,45 @@
 # than flooring one. Adding it to the gate would also floor a number nobody has
 # argued for yet.
 #
-# One JSON object per line: {query, relevant[], kind, note}
-{"query": "my writing is too wordy and people skim past the important parts", "relevant": ["write-concisely"], "kind": "skill", "note": "clarity, no shared vocabulary with the name"}
-{"query": "an error surfaces deep in the call stack and I cannot tell what originally triggered it", "relevant": ["root-cause-tracing"], "kind": "skill", "note": "tracing back from a symptom"}
-{"query": "I keep patching symptoms instead of whatever is actually wrong underneath", "relevant": ["why", "root-cause-tracing"], "kind": "skill", "note": "five-whys intent"}
+# GRADING. A row may pin an `exemplar` — "tap::skill_md", the entry the query was
+# written about — and is then graded on that entry's CONTENT CLASS: byte-identical
+# mirrors from other registries still count, a different skill sharing the name
+# does not. 28 of these 50 rows carry one. They were not chosen: over the pinned
+# corpus each of their relevant names resolves to exactly ONE body, so the pin is
+# a lookup rather than a judgment, and the pinned taps.txt is what makes that
+# reproducible.
+#
+# The other 22 are genuinely undecided. `code-reviewer` is 13 different skills
+# here, `update-docs` 10, `commit` 4, and their descriptions share a median
+# similarity of ~0.15 — there is no "same skill re-published" shortcut. Deciding
+# which one a question meant is a statement about intent, and guessing it would
+# bake one opinion into a published number. To see the menu:
+#
+#   python3 scripts/eval_retrieval.py --golden tests/eval/golden-natural.jsonl --worksheet
+#
+# which prints every candidate body and its description. It is generated rather
+# than listed here on purpose: the candidate set is a fact about the corpus that
+# is tapped, so written down it would be wrong the first time a pin moves.
+#
+# One JSON object per line: {query, relevant[], exemplar?, kind, note}
+{"query": "my writing is too wordy and people skim past the important parts", "relevant": ["write-concisely"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/docs/skills/write-concisely/SKILL.md", "kind": "skill", "note": "clarity, no shared vocabulary with the name"}
+{"query": "an error surfaces deep in the call stack and I cannot tell what originally triggered it", "relevant": ["root-cause-tracing"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/root-cause-tracing/SKILL.md", "kind": "skill", "note": "tracing back from a symptom"}
+{"query": "I keep patching symptoms instead of whatever is actually wrong underneath", "relevant": ["why", "root-cause-tracing"], "exemplar": ["NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/why/SKILL.md", "NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/root-cause-tracing/SKILL.md"], "kind": "skill", "note": "five-whys intent"}
 {"query": "what convention should I follow for the messages I write when recording changes in git", "relevant": ["commit"], "kind": "skill", "note": "conventional commits"}
-{"query": "I keep stashing half-finished work every time I have to switch branches", "relevant": ["git-worktrees"], "kind": "skill", "note": "worktrees without naming them"}
-{"query": "I want to attach review status to a commit without rewriting any history", "relevant": ["git-notes"], "kind": "skill", "note": "metadata on commits"}
+{"query": "I keep stashing half-finished work every time I have to switch branches", "relevant": ["git-worktrees"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/git/skills/git-worktrees/SKILL.md", "kind": "skill", "note": "worktrees without naming them"}
+{"query": "I want to attach review status to a commit without rewriting any history", "relevant": ["git-notes"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/git/skills/git-notes/SKILL.md", "kind": "skill", "note": "metadata on commits"}
 {"query": "check what I changed for injection flaws and leaked credentials before it goes in", "relevant": ["security-auditor"], "kind": "workflow", "note": "vulnerability review"}
-{"query": "am I actually exercising the risky paths or only the easy ones", "relevant": ["test-coverage-reviewer"], "kind": "workflow", "note": "coverage quality"}
+{"query": "am I actually exercising the risky paths or only the easy ones", "relevant": ["test-coverage-reviewer"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/review/agents/test-coverage-reviewer.md", "kind": "workflow", "note": "coverage quality"}
 {"query": "read through my diff and tell me what is broken in it", "relevant": ["bug-hunter"], "kind": "workflow", "note": "defect finding"}
 {"query": "does this change follow the conventions the rest of the project uses", "relevant": ["code-reviewer"], "kind": "workflow", "note": "guideline adherence"}
-{"query": "I need a single page covering what went wrong, why it happened and what we will do", "relevant": ["analyse-problem"], "kind": "skill", "note": "A3 report"}
-{"query": "help me lay out everything that might be contributing, grouped into categories", "relevant": ["cause-and-effect"], "kind": "skill", "note": "fishbone"}
-{"query": "I want to run small controlled experiments and keep iterating on the result", "relevant": ["plan-do-check-act"], "kind": "skill", "note": "PDCA"}
+{"query": "I need a single page covering what went wrong, why it happened and what we will do", "relevant": ["analyse-problem"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/analyse-problem/SKILL.md", "kind": "skill", "note": "A3 report"}
+{"query": "help me lay out everything that might be contributing, grouped into categories", "relevant": ["cause-and-effect"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/cause-and-effect/SKILL.md", "kind": "skill", "note": "fishbone"}
+{"query": "I want to run small controlled experiments and keep iterating on the result", "relevant": ["plan-do-check-act"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/kaizen/skills/plan-do-check-act/SKILL.md", "kind": "skill", "note": "PDCA"}
 {"query": "drive my locally running site in a real browser and confirm it behaves", "relevant": ["webapp-testing"], "kind": "skill", "note": "playwright without the word"}
 {"query": "get the text out of a scanned document so I can search it", "relevant": ["pdf"], "kind": "skill", "note": "extraction"}
-{"query": "I need to edit a Word file programmatically", "relevant": ["docx"], "kind": "skill", "note": "office format"}
-{"query": "read numbers out of a spreadsheet and write some back", "relevant": ["xlsx"], "kind": "skill", "note": "office format"}
-{"query": "put together a slide deck for a presentation", "relevant": ["pptx"], "kind": "skill", "note": "office format"}
+{"query": "I need to edit a Word file programmatically", "relevant": ["docx"], "exemplar": "anthropics/skills::skills/docx/SKILL.md", "kind": "skill", "note": "office format"}
+{"query": "read numbers out of a spreadsheet and write some back", "relevant": ["xlsx"], "exemplar": "anthropics/skills::skills/xlsx/SKILL.md", "kind": "skill", "note": "office format"}
+{"query": "put together a slide deck for a presentation", "relevant": ["pptx"], "exemplar": "anthropics/skills::skills/pptx/SKILL.md", "kind": "skill", "note": "office format"}
 {"query": "make a short looping animation small enough to post in chat", "relevant": ["slack-gif-creator"], "kind": "skill", "note": "gif constraints"}
 {"query": "make this match our company colours and typefaces", "relevant": ["brand-guidelines"], "kind": "skill", "note": "brand application"}
 {"query": "the interface looks generic and I want it to feel deliberate", "relevant": ["frontend-design"], "kind": "skill", "note": "visual design intent"}
@@ -58,27 +78,27 @@
 {"query": "I am running out of room in the window and important details get dropped", "relevant": ["context-engineering"], "kind": "skill", "note": "context limits"}
 {"query": "I want to package a repeatable procedure so it can be reused later", "relevant": ["skill-creator", "create-skill"], "kind": "skill", "note": "duplicate intent in corpus"}
 {"query": "how do I tell whether my changes to the instructions actually helped", "relevant": ["agent-evaluation"], "kind": "skill", "note": "measuring prompt changes"}
-{"query": "run something automatically every time before code is committed", "relevant": ["create-hook"], "kind": "skill", "note": "git hooks"}
-{"query": "turn this ticket into a proper technical specification", "relevant": ["analyze-issue"], "kind": "skill", "note": "issue to spec"}
+{"query": "run something automatically every time before code is committed", "relevant": ["create-hook"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/customaize-agent/skills/create-hook/SKILL.md", "kind": "skill", "note": "git hooks"}
+{"query": "turn this ticket into a proper technical specification", "relevant": ["analyze-issue"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/git/skills/analyze-issue/SKILL.md", "kind": "skill", "note": "issue to spec"}
 {"query": "open a change request with the right template filled in", "relevant": ["create-pr"], "kind": "skill", "note": "PR creation"}
-{"query": "gather all the unresolved feedback on my change into a task list", "relevant": ["load-pr-comments"], "kind": "skill", "note": "review comments"}
+{"query": "gather all the unresolved feedback on my change into a task list", "relevant": ["load-pr-comments"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/git/skills/load-pr-comments/SKILL.md", "kind": "skill", "note": "review comments"}
 {"query": "the written guides no longer match what the code does", "relevant": ["update-docs"], "kind": "skill", "note": "stale documentation"}
 {"query": "walk me through writing a guide together step by step", "relevant": ["doc-coauthoring"], "kind": "skill", "note": "structured authoring"}
-{"query": "this needs careful multi-step logic rather than a quick guess", "relevant": ["thought-based-reasoning"], "kind": "skill", "note": "reasoning depth"}
-{"query": "I want several different perspectives to argue about my design before I commit to it", "relevant": ["critique"], "kind": "skill", "note": "multi-judge debate"}
-{"query": "look back over what you just produced and improve on it", "relevant": ["reflect"], "kind": "skill", "note": "self-refinement"}
-{"query": "keep this lesson around so the same mistake is not repeated", "relevant": ["memorize"], "kind": "skill", "note": "persisting insights"}
-{"query": "some of the decisions we recorded are probably out of date now", "relevant": ["decay"], "kind": "skill", "note": "evidence freshness"}
+{"query": "this needs careful multi-step logic rather than a quick guess", "relevant": ["thought-based-reasoning"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/customaize-agent/skills/thought-based-reasoning/SKILL.md", "kind": "skill", "note": "reasoning depth"}
+{"query": "I want several different perspectives to argue about my design before I commit to it", "relevant": ["critique"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/reflexion/skills/critique/SKILL.md", "kind": "skill", "note": "multi-judge debate"}
+{"query": "look back over what you just produced and improve on it", "relevant": ["reflect"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/reflexion/skills/reflect/SKILL.md", "kind": "skill", "note": "self-refinement"}
+{"query": "keep this lesson around so the same mistake is not repeated", "relevant": ["memorize"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/reflexion/skills/memorize/SKILL.md", "kind": "skill", "note": "persisting insights"}
+{"query": "some of the decisions we recorded are probably out of date now", "relevant": ["decay"], "exemplar": "NeoLabHQ/context-engineering-kit::plugins/fpf/skills/decay/SKILL.md", "kind": "skill", "note": "evidence freshness"}
 {"query": "generate visuals from code with a bit of controlled randomness", "relevant": ["algorithmic-art"], "kind": "skill", "note": "creative coding"}
 {"query": "produce a good looking printable document rather than a web page", "relevant": ["canvas-design"], "kind": "skill", "note": "print output"}
 {"query": "apply one consistent look across slides, documents and pages", "relevant": ["theme-factory"], "kind": "skill", "note": "theming"}
 {"query": "what does calling the model actually cost per token", "relevant": ["claude-api"], "kind": "skill", "note": "pricing reference"}
 {"query": "my containers keep restarting and I cannot work out why", "relevant": ["docker-expert"], "kind": "workflow", "note": "avoids the word docker"}
-{"query": "the layout falls apart on a narrow screen", "relevant": ["css-expert"], "kind": "workflow", "note": "responsive layout"}
-{"query": "I want automated tests that click through the app like a person would", "relevant": ["cypress-expert"], "kind": "workflow", "note": "e2e testing"}
-{"query": "our full-text queries have got slow as the index grew", "relevant": ["elasticsearch-expert"], "kind": "workflow", "note": "search performance"}
-{"query": "how should I choose partition and sort keys for this access pattern", "relevant": ["dynamodb-expert"], "kind": "workflow", "note": "key design"}
-{"query": "my shell script fails silently in the pipeline and I never notice", "relevant": ["bash-expert"], "kind": "workflow", "note": "defensive scripting"}
-{"query": "I need to configure a fleet of servers repeatably without doing it by hand", "relevant": ["ansible-expert"], "kind": "workflow", "note": "config management"}
-{"query": "keep database schema versions in step across environments", "relevant": ["flyway-expert"], "kind": "workflow", "note": "migrations"}
-{"query": "background jobs are piling up and not getting processed", "relevant": ["celery-expert", "bullmq-expert"], "kind": "workflow", "note": "two queue libraries, both defensible"}
+{"query": "the layout falls apart on a narrow screen", "relevant": ["css-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/css-expert.md", "kind": "workflow", "note": "responsive layout"}
+{"query": "I want automated tests that click through the app like a person would", "relevant": ["cypress-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/cypress-expert.md", "kind": "workflow", "note": "e2e testing"}
+{"query": "our full-text queries have got slow as the index grew", "relevant": ["elasticsearch-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/elasticsearch-expert.md", "kind": "workflow", "note": "search performance"}
+{"query": "how should I choose partition and sort keys for this access pattern", "relevant": ["dynamodb-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/dynamodb-expert.md", "kind": "workflow", "note": "key design"}
+{"query": "my shell script fails silently in the pipeline and I never notice", "relevant": ["bash-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/bash-expert.md", "kind": "workflow", "note": "defensive scripting"}
+{"query": "I need to configure a fleet of servers repeatably without doing it by hand", "relevant": ["ansible-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/ansible-expert.md", "kind": "workflow", "note": "config management"}
+{"query": "keep database schema versions in step across environments", "relevant": ["flyway-expert"], "exemplar": "0xfurai/claude-code-subagents::agents/flyway-expert.md", "kind": "workflow", "note": "migrations"}
+{"query": "background jobs are piling up and not getting processed", "relevant": ["celery-expert", "bullmq-expert"], "exemplar": ["0xfurai/claude-code-subagents::agents/celery-expert.md", "0xfurai/claude-code-subagents::agents/bullmq-expert.md"], "kind": "workflow", "note": "two queue libraries, both defensible"}