claude-mem-lite 5.4.0 → 5.5.1

package/.claude-plugin/marketplace.json

@@ -10,7 +10,7 @@
   "plugins": [
     {
       "name": "claude-mem-lite",
-      "version": "5.4.0",
+      "version": "5.5.1",
       "source": "./",
       "description": "Persistent long-term memory for Claude Code via MCP — captures coding decisions, bugfixes, and context across sessions. Hybrid FTS5 + TF-IDF search with episode batching. Single SQLite DB, no external services. A lighter, lower-cost alternative to claude-mem (episode batching + a smaller model; cost savings are an internal estimate, not a measured benchmark)."
     }

package/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-mem-lite",
-  "version": "5.4.0",
+  "version": "5.5.1",
   "description": "Persistent long-term memory for Claude Code via MCP — captures coding decisions, bugfixes, and context across sessions. Hybrid FTS5 + TF-IDF search with episode batching. Single SQLite DB, no external services. A lighter, lower-cost alternative to claude-mem (episode batching + a smaller model; cost savings are an internal estimate, not a measured benchmark).",
   "author": {
     "name": "sdsrss"

package/README.md

@@ -802,6 +802,7 @@ benchmark and A/B harness are calibrated against — changing them invalidates t
 | `CLAUDE_MEM_CJK_PREC_MIN` | Precision floor for CJK segmentation candidates. | `0.2` |
 | `CLAUDE_MEM_AUTO_DEEP` | `0` disables automatic deep-search escalation (one Haiku call rewriting a weak query into keyword/concept/HyDE variants). Explicit `deep: true` still works. | _(auto)_ |
 | `CLAUDE_MEM_DEEP_DISCLOSURE` | `off` suppresses the one-line caveat appended to a multi-variant deep result. The caveat exists because deep search fills the page even when the corpus cannot answer — measured at 10 of 10 slots on queries whose answers had been removed (`benchmark/deep-search-holdout.mjs`) — and `deep` is AUTO by default on the MCP surface, i.e. it escalates precisely when the honest answer is "nothing". It does not change retrieval, ranking, or which rows are returned. | _(on)_ |
+| `CLAUDE_MEM_REACH_DISCLOSURE` | `off` suppresses the one-line note that fires when a search's reported `total` exceeds what its pagination can hand back. The candidate pool is sized from `limit` alone and deliberately does not grow with `offset` (D#30 — an offset-scaled pool re-ranks its own prefix under RRF, so pages overlapped and gapped), while `total` is the full match count. Measured on a 128-row corpus: at the default limit of 20 the last non-empty offset is 59, so 60 of 128 rows are unreachable at any offset. The note reports that; it does not change retrieval, ranking, or which rows are returned. | _(on)_ |
 | `CLAUDE_MEM_AUTO_DEEP_CLI` | `0` disables the same auto-escalation on the CLI path only. | _(auto)_ |
 | `CLAUDE_MEM_VECTORS` | `1` re-enables the persisted TF-IDF vector arm (off by default; also needs a vector rebuild via `maintain`). | _(off)_ |
 | `CLAUDE_MEM_SCOPE_FILTER` | `1` stops environment-scoped observations from firing on file-triggered recall. They stay reachable via search. **Leave it off**: on the face it gates, `environment` is not the low-relevance class its premise assumes — it cites at least as well as `project` (47.5% vs 44.3%, intervals overlapping), and an earlier measurement left 173 recall groups empty with it on. | _(off)_ |

package/deep-search.mjs

@@ -191,12 +191,18 @@ export function resolveDeepMode(explicitDeep, { surface, env = process.env } = {
  * D#3. benchmark/deep-search-holdout.mjs asks the suite's own queries of a corpus with
  * their relevant_ids deleted, so the correct answer is zero rows and every returned row is
  * a false positive by construction. It reads mean FP@10 = 10.00 across 12/12 queries: deep
- * fills every slot, every time. The single-query baseline returns 1-2 rows on the same
- * negatives — the flood is the UNION across paraphrase variants, which is also where deep's
- * recall win comes from, so this is not a bug to be thresholded away. Three gates were
- * tested against both arms and rejected; suppressing OR-fallback on rewrites takes deep
- * R@10 from 0.7383 to 0.3962, because the vocab-mismatch win IS that fallback. rrfFuseN
+ * fills every slot, every time. THE FLOOD IS NOT THE PARAPHRASE UNION, and an earlier
+ * version of this paragraph said it was ("the single-query baseline returns 1-2 rows on the
+ * same negatives"). Measured 2026-09-07, same fixture: the single-variant baseline already
+ * returns mean 9.42 of 10 (min 5, max 10, n=12), so fusion adds about half a slot to a page
+ * that was already full. A counterfactual names the real source — disabling the AND->OR
+ * fallback in search-engine.mjs takes mean FP@10 from 10.00 to 0.08, with 0/12 queries
+ * flooded instead of 12/12. Read that as a MECHANISM PROBE, not a candidate fix: the same
+ * fallback IS the vocab-mismatch recall win, and suppressing it on rewrites takes deep R@10
+ * from 0.7383 to 0.3962. Three gates were tested against both arms and rejected. rrfFuseN
  * fuses by RANK, so no magnitude signal survives the merge for a downstream floor to read.
+ * The same counterfactual explains why auto-escalation never fires here (D#8): with the
+ * fallback off, plain hits drop to min 0 and the escalation reach goes 0/12 -> 12/12.
  *
  * The discrimination is not available at this layer, so the honest move is to hand the
  * caller what the caller cannot otherwise see. Two things were missing:

package/hook-optimize.mjs

@@ -102,7 +102,7 @@ export function rebuildVector(db, obsId, textPartsOrRow) {
  *
  * @param {object} db better-sqlite3 database handle
  * @param {number} limit max candidates to return
- * @param {{ scope?: 'narrow' | 'wide' | 'aliases' | 'scopes', project?: string }} [opts] Optional project filter (e.g. inferProject()-resolved name) narrows candidates to a single project — opt-in to preserve prior cross-project default.
+ * @param {{ scope?: 'narrow' | 'wide' | 'aliases' | 'scopes' | 'concepts', project?: string }} [opts] Optional project filter (e.g. inferProject()-resolved name) narrows candidates to a single project — opt-in to preserve prior cross-project default.
  */
 export function findReenrichCandidates(db, limit = 10, { scope = 'narrow', project } = {}) {
   const projectClause = project ? 'AND project = ?' : '';
@@ -129,7 +129,8 @@ export function findReenrichCandidates(db, limit = 10, { scope = 'narrow', proje
         ${projectClause}
       ORDER BY
         CASE WHEN lesson_learned IS NOT NULL AND lesson_learned != '' THEN 0 ELSE 1 END,
-        created_at_epoch DESC
+        created_at_epoch DESC,
+        id DESC
       LIMIT ?
     `);
     return project ? stmt.all(project, limit) : stmt.all(limit);
@@ -149,12 +150,48 @@ export function findReenrichCandidates(db, limit = 10, { scope = 'narrow', proje
         AND LENGTH(COALESCE(narrative, '')) > 100
         AND ${notLowSignalTitleClause('')}
         ${projectClause}
-      ORDER BY created_at_epoch DESC
+      ORDER BY created_at_epoch DESC, id DESC
+      LIMIT ?
+    `);
+    return project ? stmt.all(project, limit) : stmt.all(limit);
+  }
+  if (scope === 'concepts') {
+    // D#6 concepts backfill: substantive rows with no concepts, REGARDLESS of lesson,
+    // aliases or scope. Same shape and same reason as the two pools above, one column
+    // over — and this one exists because the P1-2 fix created it. save-enrich fires on
+    // every successful manual save and writes search_aliases (always) + lesson_learned
+    // (bugfix/decision) + scope, which are precisely narrow's, wide's, aliases' and
+    // scopes' predicates, so a save-enriched row matches NONE of the four and never
+    // receives concepts. Measured on the real DB 2026-09-07: 14/14 live observations
+    // conceptless, 14/14 with aliases, 0/14 with optimized_at, all four pools empty.
+    //
+    // Keyed on `concepts` ALONE, not on concepts+facts: idempotency here is "the column
+    // this pass fills becomes non-empty", the same contract aliases and scopes carry. A
+    // facts term in the predicate would re-select forever every row whose narrative
+    // yields no extractable fact.
+    //
+    // Deliberately NOT gated on optimized_at, for the reason the alias branch gives and
+    // one more: the general pass preserves-on-empty, so a re-enrich whose model returned
+    // no concepts leaves the row stamped AND conceptless. Gating on the stamp would
+    // strand exactly those rows — the R10 P2-2 shape, where one pass's bookkeeping
+    // evicts a row from a backfill it never visited.
+    const stmt = db.prepare(`
+      SELECT id, title, narrative, type, subtitle, concepts, facts, text, importance, project
+      FROM observations
+      WHERE ${liveObsFilterSql('')}
+        AND (concepts IS NULL OR concepts = '')
+        AND LENGTH(COALESCE(narrative, '')) > 100
+        AND ${notLowSignalTitleClause('')}
+        ${projectClause}
+      ORDER BY created_at_epoch DESC, id DESC
       LIMIT ?
     `);
     return project ? stmt.all(project, limit) : stmt.all(limit);
   }
   if (scope === 'wide') {
+    // This pool's ORDER BY leads with a CASE term and spans lines, which is exactly why the
+    // first pass of the D#9 tiebreaker missed it. The full note lives in the default pool at
+    // the bottom of this function -- read it before touching any ORDER BY here.
     const stmt = db.prepare(`
       SELECT id, title, narrative, type, subtitle, concepts, facts, search_aliases, importance, project
       FROM observations
@@ -167,7 +204,8 @@ export function findReenrichCandidates(db, limit = 10, { scope = 'narrow', proje
         ${projectClause}
       ORDER BY
         CASE type WHEN 'decision' THEN 0 WHEN 'bugfix' THEN 1 WHEN 'refactor' THEN 2 ELSE 3 END,
-        created_at_epoch DESC
+        created_at_epoch DESC,
+        id DESC
       LIMIT ?
     `);
     return project ? stmt.all(project, limit) : stmt.all(limit);
@@ -182,7 +220,36 @@ export function findReenrichCandidates(db, limit = 10, { scope = 'narrow', proje
       AND search_aliases IS NULL
       AND optimized_at IS NULL
       ${projectClause}
-    ORDER BY created_at_epoch DESC
+    -- D#9: the id term is a REACHABILITY guard, not cosmetics. Every pool in this file is
+    -- ORDER BY created_at_epoch DESC LIMIT n feeding JS-side work, so a tie AT THE
+    -- BOUNDARY decides pool MEMBERSHIP. Measured 2026-09-07: two inserts land in the same
+    -- millisecond 272/300 times, and on a tie SQLite returns ASCENDING rowid -- the exact
+    -- opposite of the "newest first" this clause states -- so the newest rows fell out of
+    -- the pool whenever the clock had not ticked. SQLite's tie order is deterministic here
+    -- (8 rows on one epoch, 200 queries, one returned order), so this is not defending
+    -- against a varying plan; it is making the stated order total.
+    --
+    -- TWO DIFFERENT COUNTS, AND AN EARLIER DRAFT OF THIS COMMENT CONFLATED THEM. This
+    -- function, findReenrichCandidates, holds FIVE pools -- five db.prepare blocks:
+    -- 'scopes', 'aliases', 'concepts', 'wide', and this default 'narrow' one. The FILE
+    -- holds SEVEN "ORDER BY ... created_at_epoch DESC" sites: those five plus
+    -- extractUniqueConcepts and findMergeCandidates. All seven now carry the id term.
+    -- DO NOT GREP FOR THE ONE-LINE FORM: two of the seven ('scopes' and 'wide') lead with
+    -- a CASE ... term and span several lines, so grepping the joined
+    -- "created_at_epoch DESC, id DESC" spelling sees only five. That is how the first pass
+    -- read six and shipped 'wide' untiebroken -- the pool the DAILY unattended path passes
+    -- explicitly, on a budget of 6, where a boundary tie decides which rows reach the LLM
+    -- on a given run. Caught later by a test driving scope 'wide'; the original boundary
+    -- case drove only 'narrow', so nothing went red.
+    -- NOT FIXED, AND NAMED SO THE COMPLETENESS CLAIM IS TRUE: findSmartCompressCandidates
+    -- carries an eighth ordering, "ORDER BY project, created_at_epoch" -- ASCENDING, no id
+    -- term, no LIMIT. It is outside the seven by construction and is left alone under Iron
+    -- Law #1: it feeds clusterForCompression, whose vector branch seeds clusters in SQL
+    -- order, so a tie could move cluster membership -- but that branch needs
+    -- CLAUDE_MEM_VECTORS=1 and is off by default, and no failing case has been built.
+    -- Unjudged, not cleared.
+    -- This comment is INSIDE a template literal, so it must never contain a backtick.
+    ORDER BY created_at_epoch DESC, id DESC
     LIMIT ?
   `);
   return project ? stmt.all(project, limit) : stmt.all(limit);
@@ -298,9 +365,23 @@ scope: ${SCOPE_PROMPT_LEGEND}`;
         // scope rides this call for free (D#135 P3). COALESCE, not a plain set:
         // an omitted or off-enum value normalizes to null and must not erase a
         // classification an earlier face already wrote.
-        db.prepare(
-          `UPDATE observations SET search_aliases = ?, text = ?, scope = COALESCE(?, scope) WHERE id = ?`,
-        ).run(safe.search_aliases, safe.text, normalizeScope(parsed.scope), cand.id);
+        // D#12: the live-row guard, on the WHERE and not merely on the SELECT that chose
+        // the row. The Haiku call above is up to BG_LLM_TIMEOUT_MS (45 s), long enough for
+        // a concurrent hook to supersede or auto-compress this row — R10 P3-3's finding,
+        // fixed then on the general branch only and carried by the concepts branch since
+        // D#6. This was the one branch of the four without it. `changes === 0` is a SKIP,
+        // not a success: it must not count as processed and must not rebuild a vector for
+        // a row that is no longer live.
+        const res = db
+          .prepare(
+            `UPDATE observations SET search_aliases = ?, text = ?, scope = COALESCE(?, scope)
+             WHERE id = ? AND ${liveObsFilterSql('')}`,
+          )
+          .run(safe.search_aliases, safe.text, normalizeScope(parsed.scope), cand.id);
+        if (res.changes === 0) {
+          skipped++;
+          continue;
+        }
         // Refresh the TF-IDF vector from the just-updated FTS text so the new
         // aliases reach the vector arm too — the narrow/wide branch rebuilds, this
         // one must as well. No-ops when the vector arm is off / vocab unbuilt.
@@ -308,6 +389,69 @@ scope: ${SCOPE_PROMPT_LEGEND}`;
         processed++;
         continue;
       }
+      if (scope === 'concepts') {
+        // Concepts-only backfill (D#6). Writes concepts + facts and APPENDS them to the
+        // existing FTS text — never rebuilds it, for the reason the alias branch gives:
+        // a rebuild from concepts/facts drops the original narrative and alias terms and
+        // regresses recall. Never touches the user's curated title / narrative / lesson /
+        // type / importance, and never stamps optimized_at, so the wide pass keeps its
+        // own candidates exactly as the alias and scopes passes leave them.
+        const conceptsPrompt = `Extract search concepts and concrete facts from this coding memory. Return ONLY valid JSON, no markdown fences.
+
+Title: ${truncate(cand.title || '(untitled)', 200)}
+Narrative: ${truncate(cand.narrative || '(no narrative)', 500)}
+
+JSON: {"concepts":["kw1","kw2"],"facts":["specific fact 1","specific fact 2"]}
+concepts: 3-8 short keyword phrases naming what this memory is ABOUT (systems, components, error classes, techniques).
+facts: 1-4 specific, checkable statements the narrative actually asserts. Omit rather than invent.`;
+        const parsed = await callModelJSONAsync(conceptsPrompt, 'haiku', {
+          timeout: BG_LLM_TIMEOUT_MS,
+          maxTokens: 300,
+        });
+        const pickStrings = (v) =>
+          Array.isArray(v) ? v.filter((s) => typeof s === 'string' && s.trim().length > 0) : [];
+        const conceptArr = pickStrings(parsed && parsed.concepts);
+        // No concepts is a SKIP, not an empty write: writing '' would leave the row in
+        // this pool forever, and the pass would burn one Haiku call per cycle on it.
+        if (!conceptArr.length) {
+          skipped++;
+          continue;
+        }
+        const factArr = pickStrings(parsed && parsed.facts);
+        const conceptsOnly = conceptArr.slice(0, 10).join(' ');
+        const factsOnly = factArr.slice(0, 10).join(' ');
+        const appendedText = [
+          cand.text || '',
+          conceptsOnly,
+          factsOnly,
+          cjkBigrams(`${conceptsOnly} ${factsOnly}`),
+        ]
+          .filter(Boolean)
+          .join(' ');
+        const safe = scrubRecord('observations', {
+          concepts: conceptsOnly,
+          facts: factsOnly,
+          text: appendedText,
+        });
+        // Fill-only-empty on `concepts` plus the live-row guard, both on the WHERE and
+        // not merely on the SELECT: the round-trip above is up to 45 s, long enough for a
+        // concurrent hook to supersede or compress the row (R10 P3-3) or for save-enrich
+        // to fill it. `facts` rides along with preserve-on-empty for the same reason the
+        // general pass preserves it — a partial answer must not wipe a filled column.
+        const res = db
+          .prepare(
+            `UPDATE observations SET concepts = ?, facts = COALESCE(NULLIF(?, ''), facts), text = ?
+             WHERE id = ? AND (concepts IS NULL OR concepts = '') AND ${liveObsFilterSql('')}`,
+          )
+          .run(safe.concepts, safe.facts, safe.text, cand.id);
+        if (res.changes === 0) {
+          skipped++;
+          continue;
+        }
+        rebuildVector(db, cand.id, [safe.text]);
+        processed++;
+        continue;
+      }
       const prompt = `Re-enrich this observation with structured metadata. Return ONLY valid JSON, no markdown fences.
 
 Title: ${truncate(cand.title || '(untitled)', 200)}
@@ -336,9 +480,24 @@ scope: ${SCOPE_PROMPT_LEGEND}`;
       // hide a real observation until manual surgery. In wide scope, fall through and let
       // clampImportance floor it to 1 (kept visible, low-ranked) instead of hiding.
       if ((parsed.importance === 0 || parsed.importance === '0') && scope !== 'wide') {
-        db.prepare(
-          `UPDATE observations SET compressed_into = ${COMPRESSED_AUTO}, optimized_at = ? WHERE id = ?`,
-        ).run(Date.now(), cand.id);
+        // D#12, and this one is not a stale-write guard — it is a POINTER guard.
+        // `compressed_into` is the child -> keeper link, and COMPRESSED_AUTO is -1. If a
+        // concurrent cluster-merge or smart-compress adopts this row during the 45 s Haiku
+        // call, it holds a POSITIVE keeper id; overwriting that with -1 does not merely
+        // stamp a dead row, it destroys the link — lib/maintain-core.mjs:316 recovers
+        // orphans with `compressed_into > 0`, and recoverChildrenOf follows the same id.
+        // The sibling write in lib/maintain-core.mjs:631 already carries this predicate,
+        // so the codebase had decided the question and this site had not been updated.
+        const res = db
+          .prepare(
+            `UPDATE observations SET compressed_into = ${COMPRESSED_AUTO}, optimized_at = ?
+             WHERE id = ? AND ${liveObsFilterSql('')}`,
+          )
+          .run(Date.now(), cand.id);
+        if (res.changes === 0) {
+          skipped++;
+          continue;
+        }
         processed++;
         continue;
       }
@@ -507,7 +666,7 @@ export function extractUniqueConcepts(db, limit = 500, { project } = {}) {
     WHERE ${liveObsFilterSql('')}
       AND concepts IS NOT NULL AND concepts != ''
       ${projectClause}
-    ORDER BY created_at_epoch DESC
+    ORDER BY created_at_epoch DESC, id DESC -- D#9: total order, see findReenrichCandidates
     LIMIT 2000
   `);
   const rows = project ? stmt.all(project) : stmt.all();
@@ -682,7 +841,10 @@ export function findMergeCandidates(db, maxClusters = 5, { project } = {}) {
       AND title IS NOT NULL AND title != ''
       AND created_at_epoch > ?
       ${projectClause}
-    ORDER BY created_at_epoch DESC
+    -- D#9: this pool's head is what the keeper reduce falls back to on a full tie, so an
+    -- arbitrary tie order decides WHICH DUPLICATE SURVIVES a merge. Same-episode rows are
+    -- exactly that tie (same project, same importance, access_count 0, same millisecond).
+    ORDER BY created_at_epoch DESC, id DESC
     LIMIT 200
   `);
   const rows = project ? stmt.all(cutoff, project) : stmt.all(cutoff);
@@ -751,14 +913,27 @@ Return ONLY valid JSON:
     });
     if (!parsed || !parsed.should_merge) return { merged: false };
 
-    // Keeper = highest importance, then highest access_count. Previously access_count
-    // alone, so a critical (importance=3) but never-accessed observation lost the keeper
-    // role to a trivial (importance=1) accessed one and was compressed away.
+    // Keeper = highest importance, then highest access_count, then highest id. Previously
+    // access_count alone, so a critical (importance=3) but never-accessed observation lost
+    // the keeper role to a trivial (importance=1) accessed one and was compressed away.
+    //
+    // D#9: the third term is the one that makes this TOTAL. Without it a full tie fell
+    // through to `cluster[0]` — the SQL head — and same-episode duplicates are exactly a
+    // full tie: same project, same importance, access_count 0, and a created_at_epoch in
+    // the same millisecond 272 times out of 300 (measured 2026-09-07). On a tie SQLite
+    // returns ASCENDING rowid while an untied pool returns the newest first, so which
+    // duplicate survived flipped on whether two writes straddled a millisecond. Ordering
+    // the pool alone would not have been enough: this reduce is exported to callers that
+    // build their own cluster, so it has to be total on its own. Highest id = written last
+    // = the version whose content the merged summary should be anchored on.
     const keeper = cluster.reduce((best, o) => {
       const oi = o.importance || 1,
         bi = best.importance || 1;
       if (oi !== bi) return oi > bi ? o : best;
-      return (o.access_count || 0) > (best.access_count || 0) ? o : best;
+      const oa = o.access_count || 0,
+        ba = best.access_count || 0;
+      if (oa !== ba) return oa > ba ? o : best;
+      return (o.id || 0) > (best.id || 0) ? o : best;
     }, cluster[0]);
     const others = cluster.filter((o) => o.id !== keeper.id);
     // Floor the merged importance at the cluster max — merging must never silently
@@ -1016,6 +1191,52 @@ export function clusterForCompression(candidates, db) {
   return clusters;
 }
 
+/**
+ * The smart-compress prompt. Exported so a RULER can measure the shipped text.
+ *
+ * Extracted for benchmark/compress-veto-rate.mjs (D#10). It has to be one string in one
+ * place: a ruler that retypes the prompt measures its own copy, which is exactly how
+ * tests/handoff-simulation.test.mjs came to assert on a re-implementation while the real
+ * hook emitted a block no user had ever seen.
+ *
+ * D#10. This prompt used to OPEN with "Summarize these related code memory observations",
+ * asserting the premise it should have been testing, and the only bail was a missing title
+ * — so the model had no way to refuse. The sibling executeMergeCluster has had
+ * `should_merge` since it was written; these two LLM cluster paths disagreed about whether
+ * the model may say no, and this is the one that HIDES its inputs (compressed_into removes
+ * them from every injection and search surface and puts them out of recoverBuriedLessons'
+ * reach).
+ *
+ * It matters because the upstream relatedness check is not always on:
+ * clusterForCompression only computes cosine similarity when getVocabulary returns a
+ * vocabulary, and that is null whenever the vector arm is off — which is the default
+ * (CLAUDE_MEM_VECTORS !== '1'). The else branch groups by a 14-day window ALONE. Measured
+ * with a control arm 2026-09-07: three unrelated observations over 12 days form 1 cluster
+ * with the arm off and 0 with it on. Until that branch is decided (D#10 option a), this
+ * veto is the only thing standing between the heuristic and an unattended write that hides
+ * real rows.
+ *
+ * @param {Array<object>} observations cluster members
+ * @returns {string}
+ */
+export function buildCompressPrompt(observations) {
+  const obsDescriptions = observations
+    .map(
+      (o, i) =>
+        `${i + 1}. [${o.type || 'change'}] "${truncate(o.title || '(untitled)', 200)}" — ${truncate(o.narrative || '(no narrative)', 500)}${o.lesson_learned ? ` | Lesson: ${truncate(o.lesson_learned, 200)}` : ''}`,
+    )
+    .join('\n');
+
+  return `These code memory observations were grouped by a heuristic that may be wrong. FIRST decide whether they are one story worth collapsing into a single memory. Return ONLY valid JSON.
+
+Observations:
+${obsDescriptions}
+
+JSON: {"should_compress":true,"title":"descriptive summary ≤120 chars","narrative":"comprehensive summary ≤800 chars preserving key decisions and lessons","concepts":["kw1","kw2"],"facts":["all specific facts preserved"],"lesson_learned":"most important synthesized lesson or 'none'","search_aliases":["alt search 1","alt search 2"]}
+should_compress: false when these are about unrelated systems, files or problems, or when a merged summary would lose more than it saves. Compressing HIDES the originals from search, so refuse when in doubt. When false, the other fields are ignored.
+When true: preserve all important decisions, lessons, and specific facts.`;
+}
+
 export async function executeSmartCompressCluster(db, observations, project) {
   if (observations.length < 3) return { compressed: false };
 
@@ -1023,25 +1244,18 @@ export async function executeSmartCompressCluster(db, observations, project) {
   if (!gotSlot) return { compressed: false };
 
   try {
-    const obsDescriptions = observations
-      .map(
-        (o, i) =>
-          `${i + 1}. [${o.type || 'change'}] "${truncate(o.title || '(untitled)', 200)}" — ${truncate(o.narrative || '(no narrative)', 500)}${o.lesson_learned ? ` | Lesson: ${truncate(o.lesson_learned, 200)}` : ''}`,
-      )
-      .join('\n');
-
-    const prompt = `Summarize these related code memory observations into ONE comprehensive summary. Preserve all important decisions, lessons, and specific facts. Return ONLY valid JSON.
-
-Observations:
-${obsDescriptions}
-
-JSON: {"title":"descriptive summary ≤120 chars","narrative":"comprehensive summary ≤800 chars preserving key decisions and lessons","concepts":["kw1","kw2"],"facts":["all specific facts preserved"],"lesson_learned":"most important synthesized lesson or 'none'","search_aliases":["alt search 1","alt search 2"]}`;
+    const prompt = buildCompressPrompt(observations);
 
     const parsed = await callModelJSONAsync(prompt, 'sonnet', {
       timeout: BG_LLM_TIMEOUT_MS,
       maxTokens: 1000,
     });
-    if (!parsed || !parsed.title) return { compressed: false };
+    // Fail CLOSED, exactly as `should_merge` does: an omitted verdict refuses. The two
+    // failure directions are not symmetric — refusing wrongly means a compression did not
+    // happen, proceeding wrongly means unrelated observations were hidden from every
+    // surface. On a path that hides its inputs, silence is not consent.
+    if (!parsed || !parsed.should_compress) return { compressed: false };
+    if (!parsed.title) return { compressed: false };
 
     // Scrub BEFORE truncate (see re-enrich note): boundary cut on scrubbed text.
     const title = truncate(scrubSecrets(parsed.title || ''), 120);
@@ -1193,6 +1407,11 @@ export function optimizePreview(db, { project, detail = false } = {}) {
   // lesson_learned per row, so counting by materialising was the one place this
   // round pulled megabytes to print an integer. (pre-tag review NOTE 11)
   const reenrichScopes = countReenrichCandidates(db, 'scopes', project);
+  // D#6: the concepts-backfill backlog. Reported for the same reason as the three
+  // above — a pool whose size is invisible cannot be sized for a one-shot drain
+  // (`optimize --run --task re-enrich --scope concepts --max N`), and this pool is
+  // the one that holds every save-enriched manual save.
+  const reenrichConcepts = findReenrichCandidates(db, 5000, { scope: 'concepts', project }).length;
 
   const concepts = extractUniqueConcepts(db, 500, { project });
   const normalizeReady = shouldRunNormalize(project) && concepts.length >= 5;
@@ -1209,6 +1428,7 @@ export function optimizePreview(db, { project, detail = false } = {}) {
     reenrichWide,
     reenrichAliases,
     reenrichScopes,
+    reenrichConcepts,
     normalize: normalizeReady ? concepts.length : 0,
     normalizeGateOpen: shouldRunNormalize(project),
     clusterMerge,
@@ -1240,7 +1460,7 @@ export function optimizePreview(db, { project, detail = false } = {}) {
  *   is budgeted separately, up to the re-enrich slice again — see the rationale at
  *   the call site. Its calls are enum-classification only (maxTokens 60).
  * @param {boolean} [opts.force=false] Bypass time-based gates (e.g. normalize interval).
- * @param {'narrow'|'wide'|'aliases'} [opts.reenrichScope='narrow'] Scope for the re-enrich task.
+ * @param {'narrow'|'wide'|'aliases'|'concepts'} [opts.reenrichScope='narrow'] Scope for the re-enrich task.
  *   'wide' targets bugfix/refactor/feature/decision with narrative but no lesson (R-7).
  *   'aliases' (P1) backfills search_aliases on substantive alias-less rows regardless
  *   of lesson (lesson-bearing manual saves) — adds ONLY aliases, never rewrites content.
@@ -1295,16 +1515,32 @@ export async function optimizeRun(
             //     mis-prices it by an order of magnitude.
             // Cap is budget.reenrich, so the daily pass adds at most that many cheap
             // classification calls and an empty pool still costs nothing.
+            //
+            // D#6 adds a FOURTH claimant, 'concepts', and it SHARES the aliases half
+            // rather than taking one of its own. Sharing keeps the boundary this comment
+            // already describes: the main scope still gets at least half the budget, so
+            // adding a pool cannot starve the lesson enrichment that is the point of the
+            // pass. Aliases is served FIRST out of that shared half, on a stated
+            // ordering: an alias-less row is paraphrase-UNFINDABLE (a recall zero),
+            // while a conceptless row is findable and merely ranks worse — measured at
+            // +0.0846 R@10 on the benchmark fixture, which is real but is not a zero.
+            // Both pools drain (each is idempotent via the column it fills), so the
+            // ordering decides which drains first, not which gets served at all.
             const half = Math.max(1, Math.floor(budget.reenrich / 2));
             const aliasBudget = Math.min(
               half,
               findReenrichCandidates(db, half, { scope: 'aliases', project }).length,
             );
+            const conceptsBudget = Math.min(
+              half - aliasBudget,
+              findReenrichCandidates(db, Math.max(0, half - aliasBudget), { scope: 'concepts', project })
+                .length,
+            );
             const scopesBudget = Math.min(
               budget.reenrich,
               findReenrichCandidates(db, budget.reenrich, { scope: 'scopes', project }).length,
             );
-            const mainRes = await executeReenrich(db, budget.reenrich - aliasBudget, {
+            const mainRes = await executeReenrich(db, budget.reenrich - aliasBudget - conceptsBudget, {
               scope: reenrichScope,
               project,
             });
@@ -1312,14 +1548,31 @@ export async function optimizeRun(
               aliasBudget > 0
                 ? await executeReenrich(db, aliasBudget, { scope: 'aliases', project })
                 : { processed: 0, skipped: 0 };
+            const conceptsRes =
+              conceptsBudget > 0
+                ? await executeReenrich(db, conceptsBudget, { scope: 'concepts', project })
+                : { processed: 0, skipped: 0 };
             const scopesRes =
               scopesBudget > 0
                 ? await executeReenrich(db, scopesBudget, { scope: 'scopes', project })
                 : { processed: 0, skipped: 0 };
             results.reenrich = {
-              processed: (mainRes.processed || 0) + (aliasRes.processed || 0) + (scopesRes.processed || 0),
-              skipped: (mainRes.skipped || 0) + (aliasRes.skipped || 0) + (scopesRes.skipped || 0),
-              byScope: { [reenrichScope]: mainRes, aliases: aliasRes, scopes: scopesRes },
+              processed:
+                (mainRes.processed || 0) +
+                (aliasRes.processed || 0) +
+                (conceptsRes.processed || 0) +
+                (scopesRes.processed || 0),
+              skipped:
+                (mainRes.skipped || 0) +
+                (aliasRes.skipped || 0) +
+                (conceptsRes.skipped || 0) +
+                (scopesRes.skipped || 0),
+              byScope: {
+                [reenrichScope]: mainRes,
+                aliases: aliasRes,
+                concepts: conceptsRes,
+                scopes: scopesRes,
+              },
             };
           } else {
             results.reenrich = await executeReenrich(db, budget.reenrich, { scope: reenrichScope, project });

package/lib/search-core.mjs

@@ -465,6 +465,67 @@ export function applyTierFilter(db, results, { tier, sourceKey, currentProject }
   });
 }
 
+/**
+ * Tell the caller when `total` promises rows this pagination can never hand back.
+ *
+ * D#5. `computePerSourceWindow` is offset-INDEPENDENT by design (D#30: an
+ * offset-scaled pool re-ranks its own prefix under RRF, so pages overlapped and
+ * gapped on a vector-populated DB). The bound is right and stays. What was never
+ * adjusted is the REPORTED NUMBER: `countSearchTotal` re-derives the full
+ * MATCH+filter population, so a search over 128 matching rows prints
+ * "Found 10 of 128" at offset 50 and "No results at offset 60" — with nothing
+ * saying that offsets past the candidate pool are empty by construction.
+ * Measured 2026-09-07 on a 128-row sandbox corpus: the last non-empty offset is
+ * 59 / 59 / 89 for limits 10 / 20 / 30, i.e. at the default `mem_search` limit of
+ * 20, 60 of 128 rows (46.9%) are unreachable at ANY offset.
+ *
+ * `reachable` is the pre-slice candidate count (`preFinalizeCount` =
+ * `results.length`), NOT a re-derived `max(limit*3, 60)`. That matters and is not
+ * a style choice: `perSourceLimit` is PER SOURCE, so a cross-source search fuses
+ * up to four such pools and its real ceiling is several times the formula. A note
+ * quoting the formula would understate the reach of every cross-source query.
+ *
+ * Silent for deep (explicit or auto-escalated): there `total` IS the fused variant
+ * set already in `results`, so `total > reachable` cannot hold and a note would be
+ * describing a bound that is not the one in force.
+ *
+ * Off switch: CLAUDE_MEM_REACH_DISCLOSURE=off (mirrors CLAUDE_MEM_DEEP_DISCLOSURE).
+ *
+ * @param {object} [opts]
+ * @param {number} [opts.total]      the reported population (countSearchTotal)
+ * @param {number} [opts.reachable]  pre-slice candidate count (preFinalizeCount)
+ * @param {number} [opts.offset]     the offset this page asked for
+ * @param {boolean} [opts.isDeep]
+ * @param {object} [opts.env]
+ * @returns {string} the note, or '' when it should not be shown
+ */
+export function reachabilityNote({
+  total = 0,
+  reachable = 0,
+  offset = 0,
+  isDeep = false,
+  env = process.env,
+} = {}) {
+  if (String(env.CLAUDE_MEM_REACH_DISCLOSURE || '').toLowerCase() === 'off') return '';
+  if (isDeep) return '';
+  if (!Number.isFinite(total) || !Number.isFinite(reachable)) return '';
+  // Nothing came back AT ALL is a different question from "this page is past the
+  // pool" — a tier filter that dropped every candidate leaves total > 0 with
+  // reachable 0, and answering that with a pagination note would misattribute it.
+  // The CLI's own zero-result branch owns that case; D#5 is about pagination reach.
+  if (!(reachable > 0)) return '';
+  if (!(total > reachable)) return '';
+  const tail =
+    'The candidate pool is sized from `limit` alone and deliberately does not grow with ' +
+    '`offset`, so same-limit pages stay stable and disjoint (D#30). Raise the limit to widen ' +
+    'the pool, or narrow the query.';
+  return offset >= reachable
+    ? `[search: offset ${offset} is past this query's reach — ${total} rows match but only the ` +
+        `first ${reachable} are pageable. ${tail}]`
+    : `[search: ${total} rows match but only the first ${reachable} are pageable; offsets at or ` +
+        `past ${reachable} return empty. ${tail}]`;
+}
+
 /**
  * Finalize a merged, scored result set into one page: compute the TRUE
  * (limit/offset-invariant) population, slice the requested page, and attach the
@@ -485,6 +546,9 @@ export function applyTierFilter(db, results, { tier, sourceKey, currentProject }
  * expansion (concept co-occurrence / PRF / vector), and `offset` is applied
  * exactly ONCE here (the per-source SQL always saw offset 0).
  *
+ * `total` therefore reports a population LARGER than this call can ever hand back —
+ * see reachabilityNote, which is what tells the caller so.
+ *
  * @returns {{ total: number, page: object[] }}
  */
 export function finalizeSearchPage(

package/mem-cli.mjs

@@ -120,6 +120,7 @@ import {
   parseDateBounds,
   parseDuration,
   coreRunSearchPipeline,
+  reachabilityNote,
 } from './lib/search-core.mjs';
 import { AUTO_MERGE_THRESHOLD } from './lib/dedup-constants.mjs';
 import { countRecentHookErrors } from './lib/hook-telemetry.mjs';
@@ -479,6 +480,19 @@ async function cmdSearch(db, args, { llm } = {}) {
     return;
   }
 
+  // D#5. Same channel and same reasoning as the deep disclosure above: `total` is the
+  // real population, the candidate pool is offset-independent by design (D#30), and
+  // nothing else tells the caller that offsets past the pool are empty by construction.
+  // Emitted BEFORE the two return paths below so it covers both the past-the-pool page
+  // and a normal page whose total is unreachable — one call, not two wordings.
+  const reachNote = reachabilityNote({
+    total,
+    reachable: res.preFinalizeCount,
+    offset,
+    isDeep,
+  });
+  if (reachNote) process.stderr.write(`${reachNote}\n`);
+
   if (paged.length === 0) {
     if (jsonOutput) {
       out(
@@ -3202,7 +3216,7 @@ Commands:
     --run-all           Execute bypassing gates
     --task T            Comma-separated: re-enrich,normalize,cluster-merge,smart-compress
     --max N             Max items per task (1-100, default 15)
-    --scope S           re-enrich scope: narrow (default) | wide | aliases | scopes
+    --scope S           re-enrich scope: narrow (default) | wide | aliases | scopes | concepts
                         (aliases: backfill search_aliases on substantive rows that
                          lack them — incl. lesson-bearing manual saves — adds ONLY
                          aliases, never rewrites title/narrative/lesson)
@@ -3477,8 +3491,8 @@ async function cmdOptimize(db, args) {
   let reenrichScope = 'narrow';
   if (scopeIdx >= 0 && args[scopeIdx + 1] !== undefined) {
     const raw = args[scopeIdx + 1];
-    if (raw !== 'narrow' && raw !== 'wide' && raw !== 'aliases' && raw !== 'scopes') {
-      fail(`[mem] Invalid --scope "${raw}". Use: narrow, wide, aliases, scopes`);
+    if (raw !== 'narrow' && raw !== 'wide' && raw !== 'aliases' && raw !== 'scopes' && raw !== 'concepts') {
+      fail(`[mem] Invalid --scope "${raw}". Use: narrow, wide, aliases, scopes, concepts`);
       return;
     }
     reenrichScope = raw;
@@ -3502,7 +3516,7 @@ async function cmdOptimize(db, args) {
     out('[mem] 🔍 LLM Optimization Preview:');
     if (project) out(`  Project filter: ${project}`);
     out(
-      `  Re-enrich candidates: ${preview.reenrich}${preview.reenrichWide !== undefined && preview.reenrichWide !== null ? `  (wide scope: ${preview.reenrichWide})` : ''}${preview.reenrichAliases ? `  (aliases scope: ${preview.reenrichAliases})` : ''}${preview.reenrichScopes ? `  (scopes scope: ${preview.reenrichScopes})` : ''}`,
+      `  Re-enrich candidates: ${preview.reenrich}${preview.reenrichWide !== undefined && preview.reenrichWide !== null ? `  (wide scope: ${preview.reenrichWide})` : ''}${preview.reenrichAliases ? `  (aliases scope: ${preview.reenrichAliases})` : ''}${preview.reenrichScopes ? `  (scopes scope: ${preview.reenrichScopes})` : ''}${preview.reenrichConcepts ? `  (concepts scope: ${preview.reenrichConcepts})` : ''}`,
     );
     out(
       `  Normalize: ${preview.normalizeGateOpen ? `${preview.normalize} unique concepts` : 'gate closed (7-day interval)'}`,

package/npm-shrinkwrap.json

@@ -1,12 +1,12 @@
 {
   "name": "claude-mem-lite",
-  "version": "5.4.0",
+  "version": "5.5.1",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "claude-mem-lite",
-      "version": "5.4.0",
+      "version": "5.5.1",
       "os": [
         "darwin",
         "linux"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-mem-lite",
-  "version": "5.4.0",
+  "version": "5.5.1",
   "description": "Persistent long-term memory for Claude Code via MCP — captures coding decisions, bugfixes, and context across sessions. Hybrid FTS5 + TF-IDF search with episode batching. Single SQLite DB, no external services. A lighter, lower-cost alternative to claude-mem (episode batching + a smaller model; cost savings are an internal estimate, not a measured benchmark).",
   "type": "module",
   "packageManager": "npm@10.9.2",
@@ -30,6 +30,8 @@
     "test:coverage": "vitest run --coverage",
     "benchmark": "node benchmark/benchmark.mjs",
     "benchmark:gate": "node benchmark/ci-gate.mjs",
+    "benchmark:multipliers": "node benchmark/multiplier-discrimination.mjs",
+    "benchmark:multipliers:gate": "node benchmark/multiplier-discrimination.mjs --self-check",
     "audit:inventory": "node scripts/audit-metrics.mjs --inventory",
     "audit:deps": "node scripts/audit-metrics.mjs --deps",
     "audit:selfcheck": "node scripts/audit-metrics.mjs --self-check",

package/scoring-sql.mjs

@@ -10,25 +10,55 @@ import { DAY_MS } from './lib/time-constants.mjs';
 // multipliers below encode PRODUCT PRIORS: recent / same-project / important /
 // high-signal-type / frequently-cited memories are more relevant to the CURRENT
 // dev session. A periodic audit tends to flag them as "0-lift dead weight" —
-// resist that on benchmark evidence alone. Measured (audit ②, obs #8773):
-//   * benchmark.mjs --matrix (micro-fixture, now models the full FULL_SCORE
-//     chain): type-quality is the TOP contributor (drop-type ΔnDCG=0.0082,
-//     ΔMRR=0.0166), decay +0.0043 nDCG, importance +0.0012; the chain lifts
-//     hybrid over bm25_only by +0.0093 nDCG / +0.0166 MRR (net 0 queries hurt).
-//     project, access and lesson read exactly 0 — but that is STRUCTURAL: the
-//     fixture is single-project, access_count=0, and has 0 lesson_learned rows,
-//     so it cannot vary those three axes.
+// resist that, and reach for the right instrument instead of the aggregate one.
+//
+// ALL EIGHT ARE ALIVE AND CARRY THE MAGNITUDE DECLARED HERE. Measured
+// 2026-09-07 at `main` @ f25e8ae with benchmark/multiplier-discrimination.mjs,
+// which ranks pairs of rows with byte-identical indexed text differing in one
+// column, so BM25 ties and the score quotient IS the multiplier:
+//     decay 1.9770  type 1.8333  project 2.0000  importance 2.0000
+//     access 1.5004  lesson 1.3000  noise 5.0000  cite 2.0000
+// Each matches its declared ratio to 4 decimals, hybrid ranks the preferred row
+// 12/12, and removing the term drops that to 6/12 — a coin flip.
+//
+// THE AGGREGATE MATRIX CANNOT SEE THAT, and its zeros must not be read as death.
+// Same tree, `benchmark.mjs --matrix`: bm25_only ALONE reads R@10 0.8996 /
+// P@10 0.9731 / nDCG 0.9728, so the fixture is saturated and all eight
+// multipliers together buy +0.0002 R@10. Five ablation arms (project, access,
+// lesson, noise, cite) read 0 on all four metrics, and dropping importance reads
+// BETTER (ΔnDCG -0.0019). Why each zero, corrected 2026-09-07 — an earlier
+// version of this note said "the fixture is single-project", which is false
+// (seed-data.json is 5 projects x 40 rows):
+//   * project — 29 of 30 queries set no project, and the one that does also
+//     FILTERS on it, which makes the boost a constant over the survivors and
+//     therefore rank-invariant. The harness used to pass the filter value as the
+//     boost; it now mirrors search-engine.mjs:606 and disables the boost under a
+//     filter, which is rank-invariant on the matrix (verified: all 11 delta
+//     blocks byte-identical across the change).
+//   * access / lesson / noise / cite — seed-data.json carries no access_count,
+//     no lesson_learned and no injection/cite counters at all, so those four
+//     columns are constant and the terms are 1.0x on every row.
+// A multiplier reading 0 there is a benchmark-MISMATCH artifact, NOT dead weight.
 //   * longmemeval.mjs --temporal (n=500, real dates): bit-identical to uniform —
 //     LongMemEval-S windows (mean 27.9d, 74% <30d) are far shorter than these
 //     half-lives, so decay moves no rank there either.
-// Where a multiplier reads 0 it is a benchmark-MISMATCH artifact (the instrument
-// can't vary that axis), NOT proven dead weight. Decision: KEEP them; do NOT
-// delete on "0 lift". Guardrail: the ci-gate `hybrid_over_bm25 >= -0.05` floor
-// (benchmark/ci-gate.mjs) covers the full modelled chain — D#121: cite + noise
-// joined the matrix MULT_EXPR after M-3 put them in FULL_SCORE (fixture carries
-// zero cite/noise state, so both read 0 by construction, same caveat as lesson;
-// their real-SQL direction pins live in benchmark/events-pipeline-probes.mjs).
-// Genuine validation of the prior-encoding axes needs a labeled real-dev-memory eval.
+//
+// DO NOT TRUST THE CI GATE TO CATCH A CHANGE HERE. The `hybrid_over_bm25 >= -0.05`
+// floor does NOT cover the chain, measured 2026-09-07 by mutating the real tree
+// and reverting it: neutering the importance multiplier left the gate at exit 0
+// with all four checks PASS and `hybrid_over_bm25` going UP (R 0.0002 -> 0.0019),
+// because importance is a negative contributor on that fixture; changing lesson's
+// 0.3 to 0.5 left the gate's output byte-identical. The eight per-term ablation
+// deltas the matrix prints are gated by nothing. Retune a constant in this file
+// and re-run benchmark/multiplier-discrimination.mjs, which reports MISMATCH on
+// exactly that shape — the aggregate gate will not.
+// D#121: cite + noise joined the matrix MULT_EXPR after M-3 put them in
+// FULL_SCORE; their real-SQL direction pins live in
+// benchmark/events-pipeline-probes.mjs.
+// Still open, and NOT what the ruler above answers: whether these priors help a
+// REAL user. "Wired up with the declared magnitude" is a different question from
+// "correctly calibrated", and the second one still needs a labeled
+// real-dev-memory eval.
 
 // ─── Type-Differentiated Recency Decay ──────────────────────────────────────
 

package/server.mjs

@@ -30,6 +30,7 @@ import {
   parseDateBounds,
   parseDuration,
   coreRunSearchPipeline,
+  reachabilityNote,
 } from './lib/search-core.mjs';
 import {
   runMaintainOps,
@@ -595,6 +596,20 @@ async function runSearchPipeline(db, args, { llm, rerankLlm } = {}) {
     });
     if (disclosure) output.content[0].text += `\n\n${disclosure}`;
   }
+  // D#5. Same split as the deep note: an MCP client reads the tool RESULT, not stderr,
+  // so a caller that pages past the candidate pool has to be told inside the payload or
+  // it reads "0 results" against a total it was just handed and concludes the corpus is
+  // empty. `reachable` is r.preFinalizeCount — the pre-slice candidate count — because
+  // perSourceLimit is PER SOURCE and a cross-source query fuses several of those pools.
+  if (output.content?.[0]?.type === 'text') {
+    const reachNote = reachabilityNote({
+      total: r.total,
+      reachable: r.preFinalizeCount,
+      offset,
+      isDeep: r.isDeep,
+    });
+    if (reachNote) output.content[0].text += `\n\n${reachNote}`;
+  }
   appendDeferredTrailer(output);
 
   // Expose structured fields for tests + the MCP content blob.