dsh-session-recall 0.1.0 → 0.3.0

package/README.md

@@ -33,7 +33,7 @@ recall({ query, session_id })            → search the events of one session
 recall({ query, limit, cursor })         → page through results
 ```
 
-Each hit carries the session id, title (best-effort), date, and a match snippet; the result renders as a native search card in the Web UI (`SearchMatchesResultView`). Zero-hit CJK queries get a tokenizer hint: the FTS `unicode61` tokenizer indexes uninterrupted CJK runs as single tokens, so the tool teaches the model to retry with short space-separated keywords.
+Each hit carries the session id, title (best-effort), date, and a match snippet; the result renders as a native search card in the Web UI (`SearchMatchesResultView`). Because the FTS `unicode61` tokenizer indexes an uninterrupted CJK run as a single token, a short Chinese phrase inside a longer sentence would otherwise never match the index — so a zero-hit CJK query automatically falls back to a substring scan over session text (the `sessionQuery.filterEvents` literal text clause). Every whitespace-separated term must match, so `简历 模板` still recovers `简历模板`; the hint reports when that path matched.
 
 ## Scoping (the authorization gap)
 
@@ -77,7 +77,9 @@ Plugin row config (all optional):
     allowAllProjects: true  # honor the tool's all_projects argument
     defaultLimit: 5         # page size when the model omits limit (1..10)
     maxLimit: 10            # largest accepted page size (1..25)
-    cjkHint: true           # zero-hit CJK tokenizer workaround hint
+    cjkHint: true           # explain CJK zero-hit results
+    cjkFallback: true       # CJK zero-hit → exact substring scan over session text
+    cjkFallbackScanMax: 50  # max sessions scanned per cross-session fallback (1..500)
 ```
 
 ## Failure behavior
@@ -87,8 +89,29 @@ Every failure returns a friendly `hint` instead of a raw exception: a disabled i
 ## Known limitations
 
 - First search after startup walks the durable logs to build the index (the tool description warns the model); subsequent searches are incremental.
-- `unicode61` matches whole tokens/phrases, not substrings — `AI` does not match `BRAID`. The zero-hit CJK hint mitigates the worst case; a substring fallback via `filterEvents()` is a possible v2.
+- `unicode61` matches whole tokens/phrases, not substrings — `AI` does not match `BRAID`. CJK queries that get zero full-text hits fall back to a substring scan (`filterEvents`) whose whitespace-separated terms are ANDed, so `简历 模板` also recovers `简历模板`; the hint reports when that path matched.
 - One process must own the index file (single-writer SQLite, per the official backend).
+- Matches return transcript text verbatim — there is no credential or local-path redaction. A token or sensitive path pasted into an earlier session can be surfaced by a matching search. Default cwd scoping and `allowAllProjects: false` are the only containment; fingerprinting or redaction is future work.
+
+## Benchmark
+
+Measured on a real headless profile (Node 25, Apple Silicon, warm filesystem cache).
+
+| Corpus | |
+|---|---|
+| Sessions | 31 |
+| Events in the durable logs | 187,706 (~104 MB uncompressed, 49.7 MB zstd) |
+| Indexed text events | 8,187 |
+| FTS index on disk | 15 MB |
+
+| Query | Hits | Warm latency (FTS5 `MATCH`) |
+|---|---|---|
+| EN `font` | 100 (capped) | 1.1 ms |
+| EN `resume template` | 46 | 0.6 ms |
+| CN `字体` | 29 | 0.3 ms |
+| CN `简历 模板` | 10 | 0.1 ms |
+
+Warm searches run sub-millisecond to ~1.5 ms against the on-disk index. Cold start: scanning the 49.7 MB of session logs takes ~4.7 s (decompress + line scan) and inserting the 8,187 text events into a fresh FTS5 table takes ~190 ms; the first `recall` in a fresh profile completes within the tool's 10 s timeout. After that, restarts reuse the persisted index with incremental reconciliation.
 
 ## Development
 

package/README.zh.md

@@ -33,7 +33,7 @@ recall({ query, session_id })            → 只搜指定会话内的事件
 recall({ query, limit, cursor })         → 翻页
 ```
 
-每条命中带会话 id、标题（尽力补全）、日期、命中摘录；结果在 Web UI 里渲染成原生搜索卡片（`SearchMatchesResultView`）。CJK 查询零命中时会给出分词提示：FTS 的 `unicode61` 分词器把连续中文当成一个 token，工具会教模型改用空格分隔的短关键词重试。
+每条命中带会话 id、标题（尽力补全）、日期、命中摘录；结果在 Web UI 里渲染成原生搜索卡片（`SearchMatchesResultView`）。因为 FTS 的 `unicode61` 分词器会把连续中文当成一个 token，短中文短语一旦嵌在长句里就匹配不到索引——所以 CJK 查询零命中时会自动回退到对会话文本的子串扫描（走 `sessionQuery.filterEvents` 的字面文本子句），空格拆出的每个词都必须命中，因此 `简历 模板` 也能找回 `简历模板`；hint 会说明这条回退路径是否命中。
 
 ## 授权边界（官方明确留给工具层的责任）
 
@@ -77,7 +77,9 @@ dsh plugin --profile web add github:kittimzhe/dsh-session-recall
     allowAllProjects: true  # 是否允许工具的 all_projects 参数
     defaultLimit: 5         # 模型省略 limit 时的页大小（1..10）
     maxLimit: 10            # 最大页大小（1..25）
-    cjkHint: true           # CJK 零命中的分词提示开关
+    cjkHint: true           # CJK 零命中的提示开关
+    cjkFallback: true       # CJK 零命中 → 对会话文本做精确子串扫描
+    cjkFallbackScanMax: 50  # 跨会话回退时最多扫描的会话数（1..500）
 ```
 
 ## 失败行为
@@ -87,8 +89,29 @@ dsh plugin --profile web add github:kittimzhe/dsh-session-recall
 ## 已知限制
 
 - 启动后第一次搜索会扫全量日志建索引（工具描述里已警告模型）；之后增量更新。
-- `unicode61` 按完整 token/短语匹配，不支持子串——`AI` 匹配不到 `BRAID`。CJK 零命中提示缓解了最坏情况；基于 `filterEvents()` 的子串兜底是 v2 方向。
+- `unicode61` 按完整 token/短语匹配，不支持子串——`AI` 匹配不到 `BRAID`。CJK 查询零命中时会回退到子串扫描（`filterEvents`），空格分隔的各词按 AND 语义都必须命中，因此 `简历 模板` 也能找回 `简历模板`；hint 会说明是否命中。
 - 索引文件单进程独占（官方后端的单写者 SQLite 约束）。
+- 命中结果按原文照摘，**没有任何凭据或本地路径脱敏**——更早的会话里粘贴过的 token 或敏感路径可能被检索出来。目前只有默认 cwd 收窄与 `allowAllProjects: false` 两道闸；指纹识别/脱敏是后续增强。
+
+## 基准
+
+真机 headless profile 实测（Node 25，Apple Silicon，暖文件缓存）。
+
+| 语料 | |
+|---|---|
+| 会话数 | 31 |
+| 日志总事件 | 187,706（解压后约 104 MB，zstd 压缩后 49.7 MB） |
+| 已索引文本事件 | 8,187 |
+| FTS 索引体积 | 15 MB |
+
+| 查询 | 命中 | 暖查询耗时（FTS5 `MATCH`） |
+|---|---|---|
+| 英文 `font` | 100（上限） | 1.1 ms |
+| 英文 `resume template` | 46 | 0.6 ms |
+| 中文 `字体` | 29 | 0.3 ms |
+| 中文 `简历 模板` | 10 | 0.1 ms |
+
+暖查询（索引已在盘上）耗时 0.1～1.5 ms。冷启动首建：扫描 49.7 MB 日志约 4.7 s（解压 + 逐行扫描），把 8,187 条文本事件写入全新 FTS5 表约 190 ms；全新 profile 的首次 `recall` 在工具 10 s 超时内完成。此后重启复用持久化索引、增量 reconcile。
 
 ## 开发
 

package/lib/index.d.ts

@@ -1,6 +1,6 @@
 import { ToolDefinition } from "@deepseek-ai/dsh-tools";
-import { SessionEventSearchPage, SessionEventSearchRequest, SessionSearchExecContext, SessionSearchHit, SessionSearchPage, SessionSearchRequest, SessionTitleObservationResult } from "@deepseek-ai/dsh-session-query";
-import { JsonValue } from "@deepseek-ai/dsh-session";
+import { SessionEventResultFilter, SessionEventSearchDocument, SessionEventSearchPage, SessionEventSearchRequest, SessionRecord, SessionSearchExecContext, SessionSearchHit, SessionSearchPage, SessionSearchRequest, SessionTitleObservationResult } from "@deepseek-ai/dsh-session-query";
+import { JsonValue, SessionId } from "@deepseek-ai/dsh-session";
 import { Context } from "@deepseek-ai/cordis";
 import { ContentBlock } from "@deepseek-ai/dsh-llm";
 //#region src/config.d.ts
@@ -12,8 +12,12 @@ interface RecallConfig {
   defaultLimit?: number;
   /** Largest accepted page size. Default `10`. */
   maxLimit?: number;
-  /** Teach the model the CJK tokenizer workaround on zero hits. Default `true`. */
+  /** Explain CJK zero-hit results with a hint line. Default `true`. */
   cjkHint?: boolean;
+  /** Fall back to an exact substring scan when a CJK query gets no full-text hits. Default `true`. */
+  cjkFallback?: boolean;
+  /** Max sessions to scan on a cross-session CJK fallback. Default `50`. */
+  cjkFallbackScanMax?: number;
 }
 /** Validated, fully defaulted configuration. */
 interface NormalizedRecallConfig {
@@ -21,6 +25,8 @@ interface NormalizedRecallConfig {
   readonly defaultLimit: number;
   readonly maxLimit: number;
   readonly cjkHint: boolean;
+  readonly cjkFallback: boolean;
+  readonly cjkFallbackScanMax: number;
 }
 /** Default, clamp, and cross-check every optional field. */
 declare function normalizeRecallConfig(config?: RecallConfig): NormalizedRecallConfig;
@@ -76,6 +82,8 @@ interface RecallQueryEngine {
   searchSessions(request: SessionSearchRequest, exec?: SessionSearchExecContext): Promise<SessionSearchPage<SessionSearchHit>>;
   searchEvents(request: SessionEventSearchRequest, exec?: SessionSearchExecContext): Promise<SessionEventSearchPage>;
   readTitleSnapshots(sessionIds: readonly string[], signal?: AbortSignal): Promise<SessionTitleObservationResult[]>;
+  listSessions(signal?: AbortSignal): Promise<SessionRecord[]>;
+  filterEvents(sessionId: SessionId, filters: readonly SessionEventResultFilter[]): Promise<SessionEventSearchDocument[]>;
 }
 declare const RECALL_TOOL_DESCRIPTION: string;
 /**
@@ -94,7 +102,12 @@ declare function renderRecallText(result: RecallResult): string;
 declare function recallContentBlocks(result: RecallResult): ContentBlock[];
 /** Replayable presentation payload consumed by `presentResult` on the UI plane. */
 declare function recallPresentationMeta(result: RecallResult): JsonValue;
-/** The zero-hit CJK tokenizer hint, or `null` when not applicable. */
+/**
+ * Hint shown when the CJK substring fallback succeeded — explains why the
+ * full-text index missed the query and reports how many sessions matched.
+ */
+declare function cjkFallbackHint(matched: number, enabled: boolean): string | null;
+/** The zero-hit hint, shown only when both the full-text and substring paths miss. */
 declare function cjkZeroHitHint(query: string, zeroHits: boolean, enabled: boolean): string | null;
 //#endregion
 //#region src/util.d.ts
@@ -114,6 +127,12 @@ declare function hasCJK(text: string): boolean;
 declare function normalizeQuery(text: string): string;
 /** First line of `text` with control characters stripped, clipped to `limit` code points. */
 declare function firstLineClipped(text: string, limit: number): string;
+/**
+ * Single-line snippet clipped around the first case-insensitive occurrence of
+ * `query`, with ellipses at either end when text was cut. Falls back to a
+ * head clip when the query is empty or absent.
+ */
+declare function snippetAround(text: string, query: string, limit: number): string;
 //#endregion
 //#region src/index.d.ts
 declare const name = "session-recall";
@@ -121,4 +140,4 @@ declare const inject: string[];
 /** Plugin entry: mount the `recall` tool on the global tool registry. */
 declare function apply(ctx: Context, config?: RecallConfig): void;
 //#endregion
-export { type NormalizedRecallConfig, RECALL_TOOL_DESCRIPTION, type RecallArgs, type RecallBestMatch, type RecallConfig, type RecallItem, type RecallQueryEngine, type RecallResult, type RecallScope, apply, cjkZeroHitHint, clamp, createRecallTool, firstLineClipped, formatDate, hasCJK, id8, inject, name, normalizeQuery, normalizeRecallConfig, recallContentBlocks, recallPresentationMeta, renderRecallText };
+export { type NormalizedRecallConfig, RECALL_TOOL_DESCRIPTION, type RecallArgs, type RecallBestMatch, type RecallConfig, type RecallItem, type RecallQueryEngine, type RecallResult, type RecallScope, apply, cjkFallbackHint, cjkZeroHitHint, clamp, createRecallTool, firstLineClipped, formatDate, hasCJK, id8, inject, name, normalizeQuery, normalizeRecallConfig, recallContentBlocks, recallPresentationMeta, renderRecallText, snippetAround };

package/lib/index.js

@@ -12,7 +12,9 @@ function normalizeRecallConfig(config) {
 		allowAllProjects: config?.allowAllProjects !== false,
 		defaultLimit,
 		maxLimit: Math.max(defaultLimit, intIn(config?.maxLimit, 10, 1, 25)),
-		cjkHint: config?.cjkHint !== false
+		cjkHint: config?.cjkHint !== false,
+		cjkFallback: config?.cjkFallback !== false,
+		cjkFallbackScanMax: intIn(config?.cjkFallbackScanMax, 50, 1, 500)
 	};
 }
 //#endregion
@@ -45,6 +47,10 @@ function hasCJK(text) {
 function normalizeQuery(text) {
 	return text.trim().replaceAll(/\s+/g, " ");
 }
+/** Split into whitespace-separated terms, dropping empty pieces. */
+function splitTerms(text) {
+	return text.split(/\s+/u).filter((term) => term.length > 0);
+}
 /** First line of `text` with control characters stripped, clipped to `limit` code points. */
 function firstLineClipped(text, limit) {
 	const line = text.split("\n", 1)[0] ?? "";
@@ -56,6 +62,21 @@ function firstLineClipped(text, limit) {
 	}
 	return out;
 }
+/**
+* Single-line snippet clipped around the first case-insensitive occurrence of
+* `query`, with ellipses at either end when text was cut. Falls back to a
+* head clip when the query is empty or absent.
+*/
+function snippetAround(text, query, limit) {
+	const flat = text.replaceAll(/\s+/g, " ");
+	const q = normalizeQuery(query).toLowerCase();
+	const idx = q === "" ? -1 : flat.toLowerCase().indexOf(q);
+	if (idx < 0) return flat.slice(0, limit);
+	const pad = Math.floor(limit / 3);
+	const start = Math.max(0, idx - pad);
+	const end = Math.min(flat.length, start + limit);
+	return `${start > 0 ? "…" : ""}${flat.slice(start, end)}${end < flat.length ? "…" : ""}`;
+}
 //#endregion
 //#region src/render.ts
 const SNIPPET_CHARS = 120;
@@ -108,10 +129,18 @@ function recallPresentationMeta(result) {
 		}))
 	};
 }
-/** The zero-hit CJK tokenizer hint, or `null` when not applicable. */
+/**
+* Hint shown when the CJK substring fallback succeeded — explains why the
+* full-text index missed the query and reports how many sessions matched.
+*/
+function cjkFallbackHint(matched, enabled) {
+	if (!enabled || matched <= 0) return null;
+	return `full-text search returned no CJK hits (SQLite FTS5 tokenizer "unicode61" does not segment CJK), so an exact substring scan over session text was used instead and matched ${matched} session(s).`;
+}
+/** The zero-hit hint, shown only when both the full-text and substring paths miss. */
 function cjkZeroHitHint(query, zeroHits, enabled) {
 	if (!enabled || !zeroHits || !hasCJK(query)) return null;
-	return "no hits for a CJK query — the FTS unicode61 tokenizer indexes uninterrupted CJK runs as single tokens. Retry with short space-separated keywords (e.g. \"简历 模板\" instead of \"我的简历模板在这里\"), or an English term.";
+	return `no matches for this CJK query. Try a shorter phrase, or an English/code term.`;
 }
 //#endregion
 //#region src/tool.ts
@@ -130,7 +159,7 @@ function cjkZeroHitHint(query, zeroHits, enabled) {
 const RECALL_TOOL_DESCRIPTION = [
 	"Search the FULL TEXT of past and current session transcripts on this machine (your own conversation history with this user).",
 	"Use it when the user refers to earlier work (\"that bug we fixed last week\", \"the font we chose for my resume\") or when prior context was compacted away.",
-	"Matches whole words/phrases (English and code identifiers work best); returns the best-matching event snippet per session plus the session id.",
+	"Matches whole words/phrases for English and code identifiers; a zero-hit Chinese (CJK) query automatically falls back to a substring scan in which every whitespace-separated term must match. Returns the best-matching event snippet per session plus the session id.",
 	"Then use the read tool on files, or ask the user, to go deeper — this tool only points at history, it does not resume sessions.",
 	"Scoping: by default only sessions started in the current project directory; pass all_projects=true to search everywhere.",
 	"The first search after startup may be slow while the index builds."
@@ -213,6 +242,63 @@ function eventItems(page, sessionId) {
 		}
 	}));
 }
+/** Snippet window kept consistent with the tool's text projection. */
+const CJK_SNIPPET_CHARS = 120;
+/**
+* Decompose a query into ANDed literal-text clauses. A space-separated CJK
+* query like "简历 模板" must match each term as a substring (so it recovers
+* "简历模板"), not a whitespace-joined literal (which would require the space
+* to be present verbatim). `highlight` is the first term, guaranteed present
+* whenever the ANDed scan matches.
+*/
+function cjkTextFilters(query) {
+	const terms = splitTerms(query);
+	return {
+		filters: terms.map((term) => ({
+			kind: "text",
+			text: term
+		})),
+		highlight: terms[0] ?? query
+	};
+}
+/**
+* CJK substring-scan fallback for zero-hit full-text searches. SQLite FTS5's
+* `unicode61` tokenizer treats an uninterrupted CJK run as one token, so a
+* short Chinese phrase inside a longer sentence never matches the index. The
+* official sessionQuery service's `filterEvents` text clause is a literal
+* Unicode/case-insensitive regex scan that is deliberately independent of FTS
+* providers, so scanning each scoped session with it recovers the exact
+* substring matches the full-text index cannot see.
+*/
+async function cjkScanSessions(engine, query, agentCwd, wantAll, scanMax, limit, signal) {
+	const all = await engine.listSessions(signal);
+	const candidates = !wantAll && agentCwd != null ? all.filter((record) => record.header.cwd === agentCwd) : all;
+	const items = [];
+	for (const record of candidates.slice(0, scanMax)) {
+		if (items.length >= limit) break;
+		const { filters, highlight } = cjkTextFilters(query);
+		const docs = await engine.filterEvents(record.header.id, filters);
+		if (docs.length === 0) continue;
+		const doc = docs[0];
+		if (doc === void 0) continue;
+		items.push({
+			sessionId: record.header.id,
+			id8: id8(record.header.id),
+			title: null,
+			createdAt: record.header.createdAt,
+			cwd: record.header.cwd ?? null,
+			live: record.live,
+			persisted: record.persisted,
+			bestMatch: {
+				seq: doc.seq,
+				type: doc.type,
+				time: doc.time,
+				snippet: snippetAround(doc.text, highlight, CJK_SNIPPET_CHARS)
+			}
+		});
+	}
+	return items;
+}
 const nullableString = { oneOf: [{ type: "string" }, { type: "null" }] };
 const recallOutputSchema = {
 	type: "object",
@@ -320,7 +406,28 @@ function createRecallTool(config, engine) {
 						limit,
 						cursor: brand(args.cursor)
 					}, { signal: exec.signal });
-					const items = eventItems(page, sessionId);
+					let items = eventItems(page, sessionId);
+					let hint = null;
+					if (items.length === 0 && hasCJK(query) && cfg.cjkFallback) {
+						const { filters, highlight } = cjkTextFilters(query);
+						const docs = await engine.filterEvents(sessionId, filters);
+						if (docs.length > 0) items = docs.slice(0, limit).map((doc) => ({
+							sessionId,
+							id8: id8(sessionId),
+							title: null,
+							createdAt: page.session.createdAt,
+							cwd: page.session.cwd ?? null,
+							live: true,
+							persisted: false,
+							bestMatch: {
+								seq: doc.seq,
+								type: doc.type,
+								time: doc.time,
+								snippet: snippetAround(doc.text, highlight, CJK_SNIPPET_CHARS)
+							}
+						}));
+						hint = items.length > 0 ? cjkFallbackHint(items.length, cfg.cjkHint) : cjkZeroHitHint(query, true, cfg.cjkHint);
+					} else if (items.length === 0) hint = cjkZeroHitHint(query, true, cfg.cjkHint);
 					return {
 						query,
 						scope,
@@ -328,7 +435,7 @@ function createRecallTool(config, engine) {
 						hasMore: page.nextCursor != null,
 						items,
 						nextCursor: page.nextCursor ?? null,
-						hint: cjkZeroHitHint(query, items.length === 0, cfg.cjkHint)
+						hint
 					};
 				}
 				const request = {
@@ -342,15 +449,27 @@ function createRecallTool(config, engine) {
 				if (args.cursor != null && args.cursor !== "") request.cursor = brand(args.cursor);
 				const page = await engine.searchSessions(request, { signal: exec.signal });
 				const titles = await titlesFor(engine, page.items.map((hit) => hit.header.id), exec.signal);
-				const items = toItems(page.items, titles);
+				let items = toItems(page.items, titles);
+				let hint = null;
+				let fallbackRan = false;
+				if (items.length === 0 && hasCJK(query) && cfg.cjkFallback) {
+					fallbackRan = true;
+					const scanned = await cjkScanSessions(engine, query, agentCwd, wantAll, cfg.cjkFallbackScanMax, limit, exec.signal);
+					const scanTitles = await titlesFor(engine, scanned.map((item) => item.sessionId), exec.signal);
+					items = scanned.map((item) => ({
+						...item,
+						title: scanTitles.get(item.sessionId) ?? null
+					}));
+					hint = items.length > 0 ? cjkFallbackHint(items.length, cfg.cjkHint) : cjkZeroHitHint(query, true, cfg.cjkHint);
+				} else if (items.length === 0) hint = cjkZeroHitHint(query, true, cfg.cjkHint);
 				return {
 					query,
 					scope,
 					count: items.length,
-					hasMore: page.nextCursor != null,
+					hasMore: !fallbackRan && page.nextCursor != null,
 					items,
-					nextCursor: page.nextCursor ?? null,
-					hint: cjkZeroHitHint(query, items.length === 0, cfg.cjkHint)
+					nextCursor: !fallbackRan ? page.nextCursor ?? null : null,
+					hint
 				};
 			} catch (error) {
 				return recallError(query, error);
@@ -387,4 +506,4 @@ function apply(ctx, config) {
 	}, "session-recall lifecycle");
 }
 //#endregion
-export { RECALL_TOOL_DESCRIPTION, apply, cjkZeroHitHint, clamp, createRecallTool, firstLineClipped, formatDate, hasCJK, id8, inject, name, normalizeQuery, normalizeRecallConfig, recallContentBlocks, recallPresentationMeta, renderRecallText };
+export { RECALL_TOOL_DESCRIPTION, apply, cjkFallbackHint, cjkZeroHitHint, clamp, createRecallTool, firstLineClipped, formatDate, hasCJK, id8, inject, name, normalizeQuery, normalizeRecallConfig, recallContentBlocks, recallPresentationMeta, renderRecallText, snippetAround };

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-session-recall",
   "description": "Cross-session full-text recall for DeepSeek Harness: the model-facing `recall` tool searches past session transcripts through ctx.sessionQuery",
-  "version": "0.1.0",
+  "version": "0.3.0",
   "publishConfig": {
     "access": "public"
   },