dsh-loci 2.7.1 → 2.8.0

package/CHANGELOG.md

@@ -4,6 +4,43 @@ All notable changes to this project are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project follows
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html) from 1.0.0.
 
+## [2.8.0] — 2026-11-15
+
+Two real defects, both found by measuring rather than reading: search returned nothing for
+normal questions, and the one conditional injection path recorded nothing at all.
+
+### Fixed
+
+- **`loci_search` returned nothing for sentences.** A CJK run is matched as an FTS5 phrase,
+  which requires it to appear contiguously — so one word the store had never seen zeroed the
+  whole query. Measured: `回滚窗口` found the entry, `回滚窗口多久？` found nothing. People
+  type sentences, so this made the tool useless for its main use.
+  Two changes: the run's bigrams are also offered as individual OR terms, and admission now
+  accepts a **contiguous** stretch of ≥3 characters instead of requiring the whole run.
+  A 2-character stretch needs to be a non-hub token *in the title*, which keeps the original
+  false positive out — `完全无关的问题` used to be admitted by an entry whose body happened to
+  contain `问题`.
+
+  Effect on `scripts/acceptance-v2.mjs`, same 30 facts, same questions, no corpus change:
+  **lexical top-1 went 0/30 → 8/30, top-3 0/30 → 9/30.** The remaining 21 misses share no
+  bigram at all with their target — a genuine vocabulary gap that no lexical method closes,
+  which is why the index is the primary channel.
+
+- **`recallAssist` recorded nothing.** The only conditional injection in the product wrote no
+  `request_stat` row, bumped no per-entry counter, put no ids in the audit line, and did not
+  even pass a session id to its own search — so the panel could not answer "why did this
+  entry appear in that session", which is the project's own R4 rule. `renderAssistBlock` now
+  returns `{text, ids, bytes}`, and the injection is recorded with `source: 'assist'`, its
+  ids, and the usual per-entry counters. Recording failures are caught: a logging problem
+  must never break the turn it is recording.
+
+### Schema
+
+- **v4** adds `request_stat.source` (`section` | `assist`), with the migration marking
+  existing rows `section` — which is what they were, since nothing but the index wrote to
+  that table before. `ALTER TABLE … DROP COLUMN` re-parses the stored statement, so the
+  column's comment lives outside the `CREATE TABLE` body.
+
 ## [2.7.1] — 2026-11-15
 
 A correction, and a decision not to ship something.

package/docs/ACCEPTANCE.md

@@ -9,7 +9,7 @@
 
 # A. v1.0.0 (historical)
 
-Everything below was measured on **v1.0.0**. The tree is 2.6.x: schema v3, auto-verified
+Everything below was measured on **v1.0.0**. The tree is 2.8.x: schema v4, auto-verified
 capture, the pending-review notice, lesson fields, the review-kind filter and the two-part
 quote gate all landed afterwards. Treat this as a record of what was true then.
 

package/docs/DESIGN.md

@@ -165,6 +165,10 @@ Query construction:
 
 - CJK run of 3+ chars → FTS5 **phrase** of its consecutive bigrams
   (`"路径 径断 断言"`), which behaves like a substring match;
+- **and** each of its bigrams as an individual OR term. The phrase is precise but requires
+  the run to appear contiguously, so a single word the store has never seen used to zero
+  the whole query: `回滚窗口` found the entry, `回滚窗口多久？` found nothing. People type
+  sentences, so both forms are offered — the phrase for precision, the bigrams for recall;
 - 1–2 char CJK → the single bigram;
 - Latin → prefix query (`pnpm*`);
 - terms are OR-ed; tokens whose document frequency exceeds 15% are dropped as hub
@@ -174,6 +178,30 @@ A one-character CJK query has no index token; it falls back to a bounded `LIKE`
 inside the scope. That is the only non-indexed path, and it is bounded by
 `CHANNEL_LIMIT`.
 
+### 4.1.1 Admission
+
+Candidate generation is deliberately loose; admission is where precision is decided, in
+process, so the FTS expression never has to be fragile.
+
+When the query contains a CJK run, an entry is admitted on `lexical` terms only if a
+**contiguous stretch of the run** appears in it:
+
+| Longest contiguous stretch | Admitted? |
+|---|---|
+| the whole run | yes |
+| ≥3 characters | yes |
+| 2 characters, a non-hub token, **in the title** | yes |
+| 2 characters anywhere else | no |
+| words present but not contiguously | no |
+
+Two failure modes bracket the rule. All-or-nothing rejected `回滚窗口多久？` for a
+6-character entry match, because `多久` is in no document — so any sentence with one
+unfamiliar word returned nothing, which is what `scripts/acceptance-v2.mjs` measured as
+0/30 before the change and 8/30 after. Plain token coverage admits a note holding two of
+the words in different clauses, which is the false positive the run rule existed to stop.
+The title requirement handles the weak case: `完全无关的问题` was admitted by an entry whose
+*body* happened to contain `问题`, and no title did.
+
 ### 4.2 Channels and fusion
 
 | Channel | Candidate source | Weight | Notes |

package/docs/PRIOR-ART.md

@@ -137,7 +137,7 @@ R-b 只在**正文参与判定**时成立。由人工核实过的结构化字段
 1. **R1 正文不参与注入选取**（限定为索引段）：**索引段**的选取不读正文 —— SELECT 里没有 body，只有 id/kind/date/title。`recallAssist` 是**明确的例外**：按正文检索、每会话一次、注入为 user message、有字节上限、默认关闭。例外不算违规，前提是它被写下来且被单独约束（R3′/R4/R5）。将来任何按正文选取注入集的提案，仍然按拒绝处理。
 2. **R2 提前一个边界，而非禁止改变**：条件求值不得发生在**步内**；允许发生在会话/日边界。已发货的待审计数是**边界例外**：它随写入变化，代价是前缀可能失效，这是知情选择（§2.1 R-a 记为未测量的经济赌注），不是疏忽。
 3. **R3 只允许在会话边界求值**，且输入必须是可复现快照。**当前 `recallAssist` 只满足前半句**：它读实时库并用墙钟做时效加权，同一库重放可能得到不同排序。所以 R3 目前是一条**待实现**的规则，不是已满足的事实。
-4. **R4 可解释**：每条被条件选中的条目，必须在面板或审计里回答"为什么这次出现"。**索引段满足；`recallAssist` 不满足** —— 它不写 `request_stat`、不写 `memory_usage`，审计行里没有 id 也没有 session。这是待补的缺口。
+4. **R4 可解释**：每条被条件选中的条目，必须在面板或审计里回答"为什么这次出现"。**索引段一直满足；`recallAssist` 在 2.8.0 起满足** —— 它此前不写 `request_stat`、不写 `memory_usage`，审计行里既没有 id 也没有 session，等于这个产品唯一的条件注入路径无法自证。现在 `request_stat.source` 区分 `section`/`assist`，审计行带 id，per-entry 计数照常累加，`renderAssistBlock` 返回 `{text, ids, bytes}` 而不是一段裸文本。
 5. **R5 可选且有界**：默认关闭、有字节上限、可随时关掉。**已满足**（`recallAssist` 默认 `off`，`recallAssistBytes` 默认 512、上限 2048，每会话一次）。
 
 > 修正原则：一条规则如果代码违反它、且没有任何测试或检查发现，那么真正生效的不是规则，是代码。把规则改成代码实际的样子（并标明待补的部分），比保留一句漂亮话有用。

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-loci",
-  "version": "2.7.1",
+  "version": "2.8.0",
   "description": "Layered long-term memory for DeepSeek Harness: the prompt sees only a stable index, full text is one lookup away. Bi-temporal facts, strict scope isolation, verified-only injection.",
   "type": "module",
   "license": "MIT",

package/src/host/index.js

@@ -23,7 +23,7 @@ import { registerCommands } from './commands.js';
 import { runDoctor } from './doctor.js';
 
 /** Version stamped into the database and diagnostics. */
-export const VERSION = '2.7.1';
+export const VERSION = '2.8.0';
 
 /** Optional collaborators; without them the plugin still serves tools. */
 export const inject = ['systemPrompt', 'tools'];
@@ -230,15 +230,35 @@ export function apply(ctx, rawConfig = {}) {
             const query = firstUserText(decision.messages);
             if (query.length < 2) return decision;
             const scopes = resolveScopes({ sessionId: session.id, cwd: session.header?.cwd });
-            const text = await renderAssistBlock({
+            const assistScopeKeys = readScopeKeys(scopes, { includeGlobal: read('includeGlobal') });
+            const assist = await renderAssistBlock({
               store,
-              scopeKeys: readScopeKeys(scopes, { includeGlobal: read('includeGlobal') }),
+              scopeKeys: assistScopeKeys,
               query,
               maxBytes: read('recallAssistBytes'),
+              sessionId: session.id,
+              scopeKey: scopes.workspace,
             });
             assisted.add(session.id);
             if (assisted.size > ASSIST_MEMORY) assisted.clear();
-            if (text.length === 0) return decision;
+            if (assist.text.length === 0) return decision;
+            // Record it. This path injected entries into the conversation and wrote
+            // nothing anywhere, so the panel could not answer "why did this appear in
+            // that session" — the project's own R4 rule, broken by its own feature.
+            try {
+              store.recordInjection({
+                sessionId: session.id,
+                scopeKey: scopes.workspace,
+                bytes: assist.bytes,
+                listed: assist.ids.length,
+                hidden: 0,
+                ids: assist.ids,
+                source: 'assist',
+              });
+            } catch (error) {
+              // Recording must never break the turn it is recording.
+              log.warn(`loci: could not record the recall injection: ${String(error?.message ?? error)}`);
+            }
             return {
               ...decision,
               messages: [
@@ -246,7 +266,7 @@ export function apply(ctx, rawConfig = {}) {
                 {
                   id: `loci-recall-${randomUUID()}`,
                   role: 'user',
-                  content: [{ type: 'text', text }],
+                  content: [{ type: 'text', text: assist.text }],
                   source: { kind: 'plugin:loci', form: 'block', summary: 'memory recall' },
                 },
               ],

package/src/host/inject/index-section.js

@@ -275,23 +275,31 @@ export function createIndexSection(options) {
  * turn's matches. Off by default; when on it is injected once per session as a
  * user message, never per turn.
  *
- * @param options - `store`, `scopeKeys`, `query`, `maxBytes`, `limit`.
- * @returns the message text, or `''`.
+ * @param options - `store`, `scopeKeys`, `query`, `maxBytes`, `limit`, `sessionId`, `scopeKey`.
+ * @returns `{ text, ids, bytes }`; `text` is `''` when nothing was rendered.
  */
 export async function renderAssistBlock(options) {
-  const { store, scopeKeys, query, maxBytes = 512, limit = 3 } = options;
-  if (typeof query !== 'string' || query.trim().length < 2) return '';
-  const result = await search(store, { query, scopeKeys, statuses: ['verified'], limit, detail: 'snippet' });
-  if (result.hits.length === 0) return '';
+  const { store, scopeKeys, query, maxBytes = 512, limit = 3, sessionId, scopeKey } = options;
+  const empty = { text: '', ids: [], bytes: 0 };
+  if (typeof query !== 'string' || query.trim().length < 2) return empty;
+  // `sessionId`/`scopeKey` go into the search call so its own audit row can say which
+  // session and scope asked. They used to be omitted, which made the recall audit line
+  // unattributable even when it was written.
+  const result = await search(store, { query, scopeKeys, statuses: ['verified'], limit, detail: 'snippet', sessionId, scopeKey });
+  if (result.hits.length === 0) return empty;
   const lines = ['<loci-recall note="possibly relevant; confirm with loci_get before relying on it">'];
+  const ids = [];
   let used = bytes(lines[0]);
   for (const hit of result.hits) {
     const line = `- ${hit.id} · ${renderSafe(indexTitle(hit.title, 60))}`;
     const lineBytes = bytes(line) + 1;
     if (used + lineBytes > maxBytes) break;
     lines.push(line);
+    ids.push(hit.id);
     used += lineBytes;
   }
   lines.push('</loci-recall>');
-  return lines.length > 2 ? lines.join('\n') : '';
+  if (ids.length === 0) return empty;
+  const text = lines.join('\n');
+  return { text, ids, bytes: bytes(text) };
 }

package/src/host/store/migrate.js

@@ -49,6 +49,18 @@ export const SHIPPED_MIGRATIONS = [
       db.exec(MEMORY_LESSON_DDL);
     },
   },
+  {
+    from: 3,
+    to: 4,
+    // One nullable column recording which surface injected a row. Existing rows get
+    // NULL and are read as `section`, which is what they were: nothing but the index
+    // wrote to this table before the assist started recording itself.
+    up(db) {
+      db.exec('ALTER TABLE request_stat ADD COLUMN source TEXT');
+      db.exec('CREATE INDEX IF NOT EXISTS ix_request_stat_source ON request_stat (source)');
+      db.exec("UPDATE request_stat SET source = 'section' WHERE source IS NULL");
+    },
+  },
 ];
 
 /**

package/src/host/store/schema.js

@@ -8,7 +8,7 @@
  */
 
 /** Schema version this build writes and expects. */
-export const SCHEMA_VERSION = 3;
+export const SCHEMA_VERSION = 4;
 
 /**
  * DDL for the v2 per-entry usage table.
@@ -156,6 +156,12 @@ CREATE TABLE IF NOT EXISTS token_stat (
 ${MEMORY_USAGE_DDL}
 
 
+-- request_stat.source (v4) records which surface produced an injection: 'section' is the
+-- per-request system-prompt index, 'assist' is the optional once-per-session recall block.
+-- Without it the two were indistinguishable, so recallAssist injections left no trace
+-- anywhere and "why did this entry appear" could not be answered for them.
+-- Kept outside the CREATE TABLE body on purpose: SQLite's ALTER TABLE ... DROP COLUMN
+-- re-parses the stored statement and fails with "incomplete input" when it holds comments.
 CREATE TABLE IF NOT EXISTS request_stat (
   id             INTEGER PRIMARY KEY AUTOINCREMENT,
   ts             INTEGER NOT NULL,
@@ -163,9 +169,11 @@ CREATE TABLE IF NOT EXISTS request_stat (
   scope_key      TEXT,
   section_bytes  INTEGER NOT NULL,
   entries_listed INTEGER NOT NULL,
-  entries_hidden INTEGER NOT NULL
+  entries_hidden INTEGER NOT NULL,
+  source         TEXT
 );
 CREATE INDEX IF NOT EXISTS ix_request_stat_ts ON request_stat (ts DESC);
+CREATE INDEX IF NOT EXISTS ix_request_stat_source ON request_stat (source);
 
 CREATE VIRTUAL TABLE IF NOT EXISTS memory_fts USING fts5 (
   tokens,

package/src/host/store/search.js

@@ -241,22 +241,69 @@ export function lexicalDetail(row, plan) {
     }
   }
   let runHit = false;
+  let runSpan = 0;
+  let runSpanHub = true;
+  let runSpanInTitle = false;
+  const isHub = typeof plan.hub === 'function' ? plan.hub : () => false;
   for (const run of runs) {
-    let all = true;
+    // Longest contiguous stretch of the run whose bigrams are all present. All-or-nothing
+    // was the old rule, and it made a sentence with one unseen word return nothing:
+    // "回滚窗口" found the entry, "回滚窗口多久？" did not, because 多久 appears in no
+    // document. Requiring a *contiguous* stretch still rejects the case the old rule was
+    // written for — a note that happens to contain 路径 and 断言 in unrelated clauses has
+    // a longest span of 2 characters, not 4.
+    let span = 0;
+    let current = 0;
+    let spanHub = true;
+    let currentHub = true;
+    let spanEnd = 0;
     for (let i = 0; i + 1 < run.length; i += 1) {
       const bigram = run.slice(i, i + 2);
-      if (!title.includes(bigram) && !body.includes(bigram)) {
-        all = false;
-        break;
+      if (title.includes(bigram) || body.includes(bigram)) {
+        if (!isHub(bigram)) currentHub = false;
+        current += 1;
+        if (current > span) {
+          span = current;
+          spanHub = currentHub;
+          spanEnd = i + 2;
+        }
+      } else {
+        current = 0;
+        currentHub = true;
       }
     }
-    if (all) {
+    // `span` counts bigrams; a stretch of n bigrams covers n+1 characters.
+    const spanChars = span === 0 ? 0 : span + 1;
+    const spanText = span === 0 ? '' : run.slice(spanEnd - spanChars, spanEnd);
+    // A two-character span only counts as evidence when it is a real content word *and*
+    // it appears in the title. Without the title requirement, "完全无关的问题？" was
+    // admitted by an entry whose body happened to contain 问题 — the query had no topic
+    // overlap at all. The title is the entry's subject, so that is where weak evidence
+    // has to be.
+    const inTitle = spanText.length > 0 && title.includes(spanText);
+    if (spanChars > runSpan) {
+      runSpan = spanChars;
+      runSpanHub = spanHub;
+      runSpanInTitle = inTitle;
+    } else if (spanChars === runSpan) {
+      if (spanHub === false) runSpanHub = false;
+      if (inTitle) runSpanInTitle = true;
+    }
+    if (spanChars === run.length) {
       runHit = true;
       score += 3;
     }
   }
   const coverage = tokens.length === 0 ? 0 : matched / tokens.length;
-  return { score: tokens.length === 0 ? 0 : score / (tokens.length + 1), runHit, coverage, hasRuns: runs.length > 0 };
+  return {
+    score: tokens.length === 0 ? 0 : score / (tokens.length + 1),
+    runHit,
+    runSpan,
+    runSpanHub,
+    runSpanInTitle,
+    coverage,
+    hasRuns: runs.length > 0,
+  };
 }
 
 /**
@@ -335,7 +382,23 @@ export async function search(store, request) {
     // Admission: a CJK run must be fully evidenced; word-only queries need at
     // least half their tokens. Recency is a tie-breaker, never an admission
     // ticket, so a row no other signal matched does not appear just by being new.
-    const lexicalAdmitted = lexical.hasRuns ? lexical.runHit : lexical.coverage >= 0.5;
+    // Admission. When the query contains a CJK run, a contiguous stretch of at least three
+  // characters in the *same* run has to appear in the entry. Three was chosen against the
+  // two failure modes:
+  //   - all-or-nothing (the old rule) rejected "回滚窗口多久？" for a 6-character entry
+  //     match, because 多久 is in no document — so any sentence with one unfamiliar word
+  //     returned nothing at all;
+  //   - plain token coverage admitted notes that merely contain two of the words in
+  //     different clauses, which is the false positive the run rule existed to stop.
+  // A contiguous span is the middle ground: scattered words do not form one, and a real
+  // entry match always does.
+  // A 2-character span is admitted only when it is a real content word rather than a term
+  // that appears in a sixth of the store — `hub` is the existing frequency predicate, so
+  // "回滚" passes and "我们" does not. Below 20 entries every token is a non-hub, which is
+  // the right answer for a small store: missing a recall is worse than one extra candidate.
+  const lexicalAdmitted = lexical.hasRuns
+    ? lexical.runHit || lexical.runSpan >= 3 || (lexical.runSpan === 2 && lexical.runSpanHub === false && lexical.runSpanInTitle)
+    : lexical.coverage >= 0.5;
     if (!lexicalAdmitted && entityHits === 0 && vectorScore === 0) continue;
     const age = Math.max(0, now - Number(row.updated_at ?? now));
     const recency = Math.exp(-age / (30 * DAY));

package/src/host/store/store.js

@@ -1087,7 +1087,7 @@ export class Store {
         .get(...keys)?.n ?? 0,
     );
     const lastInjection = this.db
-      .prepare('SELECT ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden FROM request_stat ORDER BY id DESC LIMIT 1')
+      .prepare('SELECT ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden, source FROM request_stat ORDER BY id DESC LIMIT 1')
       .get();
     const revisions = Number(this.db.prepare('SELECT COUNT(*) AS n FROM memory').get()?.n ?? 0);
     return {
@@ -1113,6 +1113,7 @@ export class Store {
               bytes: Number(lastInjection.section_bytes),
               listed: Number(lastInjection.entries_listed),
               hidden: Number(lastInjection.entries_hidden),
+              source: lastInjection.source ?? 'section',
             },
       dbBytes: existsSync(this.paths.db) ? statSync(this.paths.db).size : 0,
       vector: { ...this.vector, ...this.embedding.health() },
@@ -1125,11 +1126,16 @@ export class Store {
    * @param row - `sessionId`, `scopeKey`, `bytes`, `listed`, `hidden`.
    */
   recordInjection(row) {
+    // `source` separates the two injection surfaces. The index runs every request; the
+    // recall assist runs once per session and, before this column existed, recorded
+    // nothing at all — so the one conditional path in the product was also the one with
+    // no answer to "why did this appear".
+    const source = typeof row.source === 'string' && row.source.length > 0 ? row.source : 'section';
     this.db
       .prepare(
-        'INSERT INTO request_stat (ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden) VALUES (?, ?, ?, ?, ?, ?)',
+        'INSERT INTO request_stat (ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden, source) VALUES (?, ?, ?, ?, ?, ?, ?)',
       )
-      .run(this.now(), row.sessionId ?? null, row.scopeKey ?? null, row.bytes, row.listed, row.hidden);
+      .run(this.now(), row.sessionId ?? null, row.scopeKey ?? null, row.bytes, row.listed, row.hidden, source);
     // Per-entry counters, so "which line earns its bytes" stays answerable after the
     // aggregate rows rotate out. A listed entry never fetched by id is the dead
     // weight this product exists to find.
@@ -1151,6 +1157,10 @@ export class Store {
       bytes: row.bytes,
       listed: row.listed,
       hidden: row.hidden,
+      source,
+      // The ids are what makes the line answerable. Without them an audit record says
+      // "3 entries were injected" and nothing about which three.
+      ids: Array.isArray(row.ids) ? row.ids : undefined,
     });
   }
 

package/src/host/store/tokenize.js

@@ -136,6 +136,19 @@ export function buildQueryPlan(query, options = {}) {
 
   const terms = [];
   for (const phrase of phrases) terms.push(phrase);
+  // Each CJK run is matched as a phrase, which is precise — but a phrase has to appear
+  // contiguously and in order, so **one word the store has never seen zeroes the whole
+  // query**. Measured before this change: "回滚窗口" found the entry, "回滚窗口多久？"
+  // found nothing, because 多久 is not in any document and the run is a single phrase
+  // with no alternative to fall back on. People type sentences, so that made search
+  // return nothing for most real questions.
+  //
+  // The bigrams are therefore also offered individually. Recall widens; the in-process
+  // scorer still rewards a contiguous run (`runs`), so precision is decided there rather
+  // than by making the FTS expression fragile.
+  for (const phrase of phrases) {
+    for (const bigram of phrase.replace(/"/g, '').split(' ')) terms.push(bigram);
+  }
   for (const token of singles) terms.push(token);
   // Exact terms, not prefix queries: FTS5 has no prefix index here, so `word*`
   // scans the whole term dictionary (measured: 1.4 s on a 10k-entry store).

package/src/mcp/store-facade.js

@@ -36,7 +36,7 @@ export function openFacadeStore(options = {}) {
     memoryRoot: root,
     paths,
     config,
-    pluginVersion: '2.7.1',
+    pluginVersion: '2.8.0',
     readOnly: options.readOnly === true,
   });
   return { store, paths };