dsh-loci 2.7.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +69 -0
- package/docs/ACCEPTANCE.md +49 -6
- package/docs/DESIGN.md +28 -0
- package/docs/PRIOR-ART.md +1 -1
- package/package.json +1 -1
- package/src/host/index.js +25 -5
- package/src/host/inject/index-section.js +15 -7
- package/src/host/store/migrate.js +12 -0
- package/src/host/store/schema.js +10 -2
- package/src/host/store/search.js +70 -7
- package/src/host/store/store.js +13 -3
- package/src/host/store/tokenize.js +13 -0
- package/src/mcp/store-facade.js +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,75 @@ All notable changes to this project are documented here. The format follows
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project follows
|
|
5
5
|
[Semantic Versioning](https://semver.org/spec/v2.0.0.html) from 1.0.0.
|
|
6
6
|
|
|
7
|
+
## [2.8.0] — 2026-11-15
|
|
8
|
+
|
|
9
|
+
Two real defects, both found by measuring rather than reading: search returned nothing for
|
|
10
|
+
normal questions, and the one conditional injection path recorded nothing at all.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- **`loci_search` returned nothing for sentences.** A CJK run is matched as an FTS5 phrase,
|
|
15
|
+
which requires it to appear contiguously — so one word the store had never seen zeroed the
|
|
16
|
+
whole query. Measured: `回滚窗口` found the entry, `回滚窗口多久?` found nothing. People
|
|
17
|
+
type sentences, so this made the tool useless for its main use.
|
|
18
|
+
Two changes: the run's bigrams are also offered as individual OR terms, and admission now
|
|
19
|
+
accepts a **contiguous** stretch of ≥3 characters instead of requiring the whole run.
|
|
20
|
+
A 2-character stretch needs to be a non-hub token *in the title*, which keeps the original
|
|
21
|
+
false positive out — `完全无关的问题` used to be admitted by an entry whose body happened to
|
|
22
|
+
contain `问题`.
|
|
23
|
+
|
|
24
|
+
Effect on `scripts/acceptance-v2.mjs`, same 30 facts, same questions, no corpus change:
|
|
25
|
+
**lexical top-1 went 0/30 → 8/30, top-3 0/30 → 9/30.** The remaining 21 misses share no
|
|
26
|
+
bigram at all with their target — a genuine vocabulary gap that no lexical method closes,
|
|
27
|
+
which is why the index is the primary channel.
|
|
28
|
+
|
|
29
|
+
- **`recallAssist` recorded nothing.** The only conditional injection in the product wrote no
|
|
30
|
+
`request_stat` row, bumped no per-entry counter, put no ids in the audit line, and did not
|
|
31
|
+
even pass a session id to its own search — so the panel could not answer "why did this
|
|
32
|
+
entry appear in that session", which is the project's own R4 rule. `renderAssistBlock` now
|
|
33
|
+
returns `{text, ids, bytes}`, and the injection is recorded with `source: 'assist'`, its
|
|
34
|
+
ids, and the usual per-entry counters. Recording failures are caught: a logging problem
|
|
35
|
+
must never break the turn it is recording.
|
|
36
|
+
|
|
37
|
+
### Schema
|
|
38
|
+
|
|
39
|
+
- **v4** adds `request_stat.source` (`section` | `assist`), with the migration marking
|
|
40
|
+
existing rows `section` — which is what they were, since nothing but the index wrote to
|
|
41
|
+
that table before. `ALTER TABLE … DROP COLUMN` re-parses the stored statement, so the
|
|
42
|
+
column's comment lives outside the `CREATE TABLE` body.
|
|
43
|
+
|
|
44
|
+
## [2.7.1] — 2026-11-15
|
|
45
|
+
|
|
46
|
+
A correction, and a decision not to ship something.
|
|
47
|
+
|
|
48
|
+
### Corrected
|
|
49
|
+
|
|
50
|
+
2.7.0's acceptance note said the index does not scale — "arm C asked for 12/12 ids, so at 200
|
|
51
|
+
entries the model pulls the whole store". **That was wrong, and the error was in the
|
|
52
|
+
experiment.** Arm C asked **12 questions about 12 different entries**; fetching 12 bodies is
|
|
53
|
+
the correct answer to 12 questions. Concluding "it over-fetches" from `questions == fetches`
|
|
54
|
+
was a mistake in the arm's design, not a property of the product.
|
|
55
|
+
|
|
56
|
+
The fair version — **one** question against many candidates:
|
|
57
|
+
|
|
58
|
+
| Arm | Candidates | Result |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| D1 | 12 titles, 1 relevant | fetched **1**, the right one |
|
|
61
|
+
| D2 | 40 titles, 1 relevant, plus a decoy sharing the query word | fetched **1**, the right one |
|
|
62
|
+
| E | 40 titles, nothing relevant | answered `NOT_IN_MEMORY`, substituted nothing |
|
|
63
|
+
|
|
64
|
+
### Not shipped, on purpose
|
|
65
|
+
|
|
66
|
+
The work item was "make the index guide the model to converge". It already converges, so no
|
|
67
|
+
steering text was added to the footer. Two arms tested a footer that explicitly said "do not
|
|
68
|
+
batch-fetch titles that merely look plausible": it changed nothing. The footer is 83 bytes on
|
|
69
|
+
every request, and this design's entire argument is that prompt bytes are the scarce
|
|
70
|
+
resource — a sentence that changes no behaviour is pure cost.
|
|
71
|
+
|
|
72
|
+
The lesson is worth more than the fix would have been: **a measurement that counts fetches
|
|
73
|
+
against questions must keep those two numbers independent.** `docs/ACCEPTANCE.md` now carries
|
|
74
|
+
the correction in place, rather than quietly deleting the wrong claim.
|
|
75
|
+
|
|
7
76
|
## [2.7.0] — 2026-11-15
|
|
8
77
|
|
|
9
78
|
The acceptance was measuring less than it claimed. This release re-runs it properly and
|
package/docs/ACCEPTANCE.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
# A. v1.0.0 (historical)
|
|
11
11
|
|
|
12
|
-
Everything below was measured on **v1.0.0**. The tree is 2.
|
|
12
|
+
Everything below was measured on **v1.0.0**. The tree is 2.8.x: schema v4, auto-verified
|
|
13
13
|
capture, the pending-review notice, lesson fields, the review-kind filter and the two-part
|
|
14
14
|
quote gate all landed afterwards. Treat this as a record of what was true then.
|
|
15
15
|
|
|
@@ -104,7 +104,51 @@ Each arm answered in one turn with no tools.
|
|
|
104
104
|
|---|---|---|---|
|
|
105
105
|
| **A — no memory** | the 20 questions only | **0/20** | answered `UNKNOWN` to all 20 — it abstained rather than hallucinating |
|
|
106
106
|
| **B — index only** | the 20 questions + 20 index lines whose titles *are* the facts | **20/20** | answered every question with the exact value |
|
|
107
|
-
| **C — index + fetch** | 12
|
|
107
|
+
| **C — index + fetch** | 12 questions, 12 summary titles, values only in bodies | asked for 12/12 ids | correct: 12 questions need 12 bodies — see the correction below |
|
|
108
|
+
| **D1 — one question, 12 candidates, one relevant** | one question + 12 summary titles | fetched **1/1**, the right one | no over-fetch |
|
|
109
|
+
| **D2 — one question, 40 candidates, one relevant** | one question + 40 summary titles (including a decoy sharing the word 发布) | fetched **1/1**, the right one | no over-fetch at 40 |
|
|
110
|
+
| **E — one question, 40 candidates, nothing relevant** | the same 40 titles, asking about something absent | answered `NOT_IN_MEMORY` | did not substitute a nearby entry |
|
|
111
|
+
|
|
112
|
+
### Correction: arm C did not show what it first appeared to show
|
|
113
|
+
|
|
114
|
+
The first write-up of this section said arm C revealed greedy fetching — "asked for 12/12
|
|
115
|
+
ids, did not discriminate, pulls the whole index at 200 entries". **That was wrong, and the
|
|
116
|
+
error was in the experiment, not the product.** Arm C asked **12 questions about 12
|
|
117
|
+
different entries**; fetching 12 bodies is the *correct* answer to 12 questions. Concluding
|
|
118
|
+
"it over-fetches" from `questions == fetches` was a design mistake in the arm.
|
|
119
|
+
|
|
120
|
+
D1 and D2 are the fair version: **one** question, many candidates, exactly one relevant.
|
|
121
|
+
Both fetched a single body, and D2 found it among 40 titles while ignoring a decoy that
|
|
122
|
+
shares the query word (发布说明 vs 发布窗口).
|
|
123
|
+
|
|
124
|
+
Two follow-up arms then tested whether the instruction text changed the behaviour, using a
|
|
125
|
+
footer that explicitly said "do not batch-fetch titles that merely look plausible":
|
|
126
|
+
|
|
127
|
+
| Variant | Result |
|
|
128
|
+
|---|---|
|
|
129
|
+
| current footer | fetched the needed ids |
|
|
130
|
+
| steering footer | fetched the same ids |
|
|
131
|
+
|
|
132
|
+
The instruction made no difference, so it was **not** added to the shipped footer. The
|
|
133
|
+
footer is 83 bytes on every request; a sentence that changes no behaviour is pure prompt
|
|
134
|
+
cost, and this repository's whole argument for the index shape is that prompt bytes are the
|
|
135
|
+
scarce resource.
|
|
136
|
+
|
|
137
|
+
### Decision: no code change, on purpose
|
|
138
|
+
|
|
139
|
+
The work item that produced arms C–E was "make the index guide the model to converge
|
|
140
|
+
instead of fetching everything". The evidence says it already converges:
|
|
141
|
+
|
|
142
|
+
| Situation | Behaviour |
|
|
143
|
+
|---|---|
|
|
144
|
+
| one question, one matching title among 12 | fetches that one |
|
|
145
|
+
| one question, one matching title among 40, with a decoy | fetches that one |
|
|
146
|
+
| one question, no matching title among 40 | `NOT_IN_MEMORY`, no substitute |
|
|
147
|
+
| N questions, N matching titles | fetches N — which is correct |
|
|
148
|
+
|
|
149
|
+
The only case that looked like over-fetching was the one where the experiment asked N
|
|
150
|
+
questions and counted N fetches. Shipping a fix for it would have been shipping a change
|
|
151
|
+
justified by a badly designed measurement.
|
|
108
152
|
|
|
109
153
|
Read honestly:
|
|
110
154
|
|
|
@@ -113,10 +157,9 @@ Read honestly:
|
|
|
113
157
|
- **The abstention in arm A is a good sign.** The model did not invent values; a
|
|
114
158
|
memory-less assistant says "I don't know" instead of confabulating. So the value here is
|
|
115
159
|
recall, not anti-hallucination.
|
|
116
|
-
- **
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
is a design problem, not a formatting problem.
|
|
160
|
+
- **The index scales better than the first draft of this page claimed.** One question
|
|
161
|
+
against 40 candidate titles resolved to a single body. The scaling worry this section
|
|
162
|
+
raised was the author's, not the design's.
|
|
120
163
|
|
|
121
164
|
## B.5 What §B still does not measure
|
|
122
165
|
|
package/docs/DESIGN.md
CHANGED
|
@@ -165,6 +165,10 @@ Query construction:
|
|
|
165
165
|
|
|
166
166
|
- CJK run of 3+ chars → FTS5 **phrase** of its consecutive bigrams
|
|
167
167
|
(`"路径 径断 断言"`), which behaves like a substring match;
|
|
168
|
+
- **and** each of its bigrams as an individual OR term. The phrase is precise but requires
|
|
169
|
+
the run to appear contiguously, so a single word the store has never seen used to zero
|
|
170
|
+
the whole query: `回滚窗口` found the entry, `回滚窗口多久?` found nothing. People type
|
|
171
|
+
sentences, so both forms are offered — the phrase for precision, the bigrams for recall;
|
|
168
172
|
- 1–2 char CJK → the single bigram;
|
|
169
173
|
- Latin → prefix query (`pnpm*`);
|
|
170
174
|
- terms are OR-ed; tokens whose document frequency exceeds 15% are dropped as hub
|
|
@@ -174,6 +178,30 @@ A one-character CJK query has no index token; it falls back to a bounded `LIKE`
|
|
|
174
178
|
inside the scope. That is the only non-indexed path, and it is bounded by
|
|
175
179
|
`CHANNEL_LIMIT`.
|
|
176
180
|
|
|
181
|
+
### 4.1.1 Admission
|
|
182
|
+
|
|
183
|
+
Candidate generation is deliberately loose; admission is where precision is decided, in
|
|
184
|
+
process, so the FTS expression never has to be fragile.
|
|
185
|
+
|
|
186
|
+
When the query contains a CJK run, an entry is admitted on `lexical` terms only if a
|
|
187
|
+
**contiguous stretch of the run** appears in it:
|
|
188
|
+
|
|
189
|
+
| Longest contiguous stretch | Admitted? |
|
|
190
|
+
|---|---|
|
|
191
|
+
| the whole run | yes |
|
|
192
|
+
| ≥3 characters | yes |
|
|
193
|
+
| 2 characters, a non-hub token, **in the title** | yes |
|
|
194
|
+
| 2 characters anywhere else | no |
|
|
195
|
+
| words present but not contiguously | no |
|
|
196
|
+
|
|
197
|
+
Two failure modes bracket the rule. All-or-nothing rejected `回滚窗口多久?` for a
|
|
198
|
+
6-character entry match, because `多久` is in no document — so any sentence with one
|
|
199
|
+
unfamiliar word returned nothing, which is what `scripts/acceptance-v2.mjs` measured as
|
|
200
|
+
0/30 before the change and 8/30 after. Plain token coverage admits a note holding two of
|
|
201
|
+
the words in different clauses, which is the false positive the run rule existed to stop.
|
|
202
|
+
The title requirement handles the weak case: `完全无关的问题` was admitted by an entry whose
|
|
203
|
+
*body* happened to contain `问题`, and no title did.
|
|
204
|
+
|
|
177
205
|
### 4.2 Channels and fusion
|
|
178
206
|
|
|
179
207
|
| Channel | Candidate source | Weight | Notes |
|
package/docs/PRIOR-ART.md
CHANGED
|
@@ -137,7 +137,7 @@ R-b 只在**正文参与判定**时成立。由人工核实过的结构化字段
|
|
|
137
137
|
1. **R1 正文不参与注入选取**(限定为索引段):**索引段**的选取不读正文 —— SELECT 里没有 body,只有 id/kind/date/title。`recallAssist` 是**明确的例外**:按正文检索、每会话一次、注入为 user message、有字节上限、默认关闭。例外不算违规,前提是它被写下来且被单独约束(R3′/R4/R5)。将来任何按正文选取注入集的提案,仍然按拒绝处理。
|
|
138
138
|
2. **R2 提前一个边界,而非禁止改变**:条件求值不得发生在**步内**;允许发生在会话/日边界。已发货的待审计数是**边界例外**:它随写入变化,代价是前缀可能失效,这是知情选择(§2.1 R-a 记为未测量的经济赌注),不是疏忽。
|
|
139
139
|
3. **R3 只允许在会话边界求值**,且输入必须是可复现快照。**当前 `recallAssist` 只满足前半句**:它读实时库并用墙钟做时效加权,同一库重放可能得到不同排序。所以 R3 目前是一条**待实现**的规则,不是已满足的事实。
|
|
140
|
-
4. **R4 可解释**:每条被条件选中的条目,必须在面板或审计里回答"为什么这次出现"
|
|
140
|
+
4. **R4 可解释**:每条被条件选中的条目,必须在面板或审计里回答"为什么这次出现"。**索引段一直满足;`recallAssist` 在 2.8.0 起满足** —— 它此前不写 `request_stat`、不写 `memory_usage`,审计行里既没有 id 也没有 session,等于这个产品唯一的条件注入路径无法自证。现在 `request_stat.source` 区分 `section`/`assist`,审计行带 id,per-entry 计数照常累加,`renderAssistBlock` 返回 `{text, ids, bytes}` 而不是一段裸文本。
|
|
141
141
|
5. **R5 可选且有界**:默认关闭、有字节上限、可随时关掉。**已满足**(`recallAssist` 默认 `off`,`recallAssistBytes` 默认 512、上限 2048,每会话一次)。
|
|
142
142
|
|
|
143
143
|
> 修正原则:一条规则如果代码违反它、且没有任何测试或检查发现,那么真正生效的不是规则,是代码。把规则改成代码实际的样子(并标明待补的部分),比保留一句漂亮话有用。
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-loci",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.8.0",
|
|
4
4
|
"description": "Layered long-term memory for DeepSeek Harness: the prompt sees only a stable index, full text is one lookup away. Bi-temporal facts, strict scope isolation, verified-only injection.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/src/host/index.js
CHANGED
|
@@ -23,7 +23,7 @@ import { registerCommands } from './commands.js';
|
|
|
23
23
|
import { runDoctor } from './doctor.js';
|
|
24
24
|
|
|
25
25
|
/** Version stamped into the database and diagnostics. */
|
|
26
|
-
export const VERSION = '2.
|
|
26
|
+
export const VERSION = '2.8.0';
|
|
27
27
|
|
|
28
28
|
/** Optional collaborators; without them the plugin still serves tools. */
|
|
29
29
|
export const inject = ['systemPrompt', 'tools'];
|
|
@@ -230,15 +230,35 @@ export function apply(ctx, rawConfig = {}) {
|
|
|
230
230
|
const query = firstUserText(decision.messages);
|
|
231
231
|
if (query.length < 2) return decision;
|
|
232
232
|
const scopes = resolveScopes({ sessionId: session.id, cwd: session.header?.cwd });
|
|
233
|
-
const
|
|
233
|
+
const assistScopeKeys = readScopeKeys(scopes, { includeGlobal: read('includeGlobal') });
|
|
234
|
+
const assist = await renderAssistBlock({
|
|
234
235
|
store,
|
|
235
|
-
scopeKeys:
|
|
236
|
+
scopeKeys: assistScopeKeys,
|
|
236
237
|
query,
|
|
237
238
|
maxBytes: read('recallAssistBytes'),
|
|
239
|
+
sessionId: session.id,
|
|
240
|
+
scopeKey: scopes.workspace,
|
|
238
241
|
});
|
|
239
242
|
assisted.add(session.id);
|
|
240
243
|
if (assisted.size > ASSIST_MEMORY) assisted.clear();
|
|
241
|
-
if (text.length === 0) return decision;
|
|
244
|
+
if (assist.text.length === 0) return decision;
|
|
245
|
+
// Record it. This path injected entries into the conversation and wrote
|
|
246
|
+
// nothing anywhere, so the panel could not answer "why did this appear in
|
|
247
|
+
// that session" — the project's own R4 rule, broken by its own feature.
|
|
248
|
+
try {
|
|
249
|
+
store.recordInjection({
|
|
250
|
+
sessionId: session.id,
|
|
251
|
+
scopeKey: scopes.workspace,
|
|
252
|
+
bytes: assist.bytes,
|
|
253
|
+
listed: assist.ids.length,
|
|
254
|
+
hidden: 0,
|
|
255
|
+
ids: assist.ids,
|
|
256
|
+
source: 'assist',
|
|
257
|
+
});
|
|
258
|
+
} catch (error) {
|
|
259
|
+
// Recording must never break the turn it is recording.
|
|
260
|
+
log.warn(`loci: could not record the recall injection: ${String(error?.message ?? error)}`);
|
|
261
|
+
}
|
|
242
262
|
return {
|
|
243
263
|
...decision,
|
|
244
264
|
messages: [
|
|
@@ -246,7 +266,7 @@ export function apply(ctx, rawConfig = {}) {
|
|
|
246
266
|
{
|
|
247
267
|
id: `loci-recall-${randomUUID()}`,
|
|
248
268
|
role: 'user',
|
|
249
|
-
content: [{ type: 'text', text }],
|
|
269
|
+
content: [{ type: 'text', text: assist.text }],
|
|
250
270
|
source: { kind: 'plugin:loci', form: 'block', summary: 'memory recall' },
|
|
251
271
|
},
|
|
252
272
|
],
|
|
@@ -275,23 +275,31 @@ export function createIndexSection(options) {
|
|
|
275
275
|
* turn's matches. Off by default; when on it is injected once per session as a
|
|
276
276
|
* user message, never per turn.
|
|
277
277
|
*
|
|
278
|
-
* @param options - `store`, `scopeKeys`, `query`, `maxBytes`, `limit`.
|
|
279
|
-
* @returns
|
|
278
|
+
* @param options - `store`, `scopeKeys`, `query`, `maxBytes`, `limit`, `sessionId`, `scopeKey`.
|
|
279
|
+
* @returns `{ text, ids, bytes }`; `text` is `''` when nothing was rendered.
|
|
280
280
|
*/
|
|
281
281
|
export async function renderAssistBlock(options) {
|
|
282
|
-
const { store, scopeKeys, query, maxBytes = 512, limit = 3 } = options;
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
282
|
+
const { store, scopeKeys, query, maxBytes = 512, limit = 3, sessionId, scopeKey } = options;
|
|
283
|
+
const empty = { text: '', ids: [], bytes: 0 };
|
|
284
|
+
if (typeof query !== 'string' || query.trim().length < 2) return empty;
|
|
285
|
+
// `sessionId`/`scopeKey` go into the search call so its own audit row can say which
|
|
286
|
+
// session and scope asked. They used to be omitted, which made the recall audit line
|
|
287
|
+
// unattributable even when it was written.
|
|
288
|
+
const result = await search(store, { query, scopeKeys, statuses: ['verified'], limit, detail: 'snippet', sessionId, scopeKey });
|
|
289
|
+
if (result.hits.length === 0) return empty;
|
|
286
290
|
const lines = ['<loci-recall note="possibly relevant; confirm with loci_get before relying on it">'];
|
|
291
|
+
const ids = [];
|
|
287
292
|
let used = bytes(lines[0]);
|
|
288
293
|
for (const hit of result.hits) {
|
|
289
294
|
const line = `- ${hit.id} · ${renderSafe(indexTitle(hit.title, 60))}`;
|
|
290
295
|
const lineBytes = bytes(line) + 1;
|
|
291
296
|
if (used + lineBytes > maxBytes) break;
|
|
292
297
|
lines.push(line);
|
|
298
|
+
ids.push(hit.id);
|
|
293
299
|
used += lineBytes;
|
|
294
300
|
}
|
|
295
301
|
lines.push('</loci-recall>');
|
|
296
|
-
|
|
302
|
+
if (ids.length === 0) return empty;
|
|
303
|
+
const text = lines.join('\n');
|
|
304
|
+
return { text, ids, bytes: bytes(text) };
|
|
297
305
|
}
|
|
@@ -49,6 +49,18 @@ export const SHIPPED_MIGRATIONS = [
|
|
|
49
49
|
db.exec(MEMORY_LESSON_DDL);
|
|
50
50
|
},
|
|
51
51
|
},
|
|
52
|
+
{
|
|
53
|
+
from: 3,
|
|
54
|
+
to: 4,
|
|
55
|
+
// One nullable column recording which surface injected a row. Existing rows get
|
|
56
|
+
// NULL and are read as `section`, which is what they were: nothing but the index
|
|
57
|
+
// wrote to this table before the assist started recording itself.
|
|
58
|
+
up(db) {
|
|
59
|
+
db.exec('ALTER TABLE request_stat ADD COLUMN source TEXT');
|
|
60
|
+
db.exec('CREATE INDEX IF NOT EXISTS ix_request_stat_source ON request_stat (source)');
|
|
61
|
+
db.exec("UPDATE request_stat SET source = 'section' WHERE source IS NULL");
|
|
62
|
+
},
|
|
63
|
+
},
|
|
52
64
|
];
|
|
53
65
|
|
|
54
66
|
/**
|
package/src/host/store/schema.js
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
*/
|
|
9
9
|
|
|
10
10
|
/** Schema version this build writes and expects. */
|
|
11
|
-
export const SCHEMA_VERSION =
|
|
11
|
+
export const SCHEMA_VERSION = 4;
|
|
12
12
|
|
|
13
13
|
/**
|
|
14
14
|
* DDL for the v2 per-entry usage table.
|
|
@@ -156,6 +156,12 @@ CREATE TABLE IF NOT EXISTS token_stat (
|
|
|
156
156
|
${MEMORY_USAGE_DDL}
|
|
157
157
|
|
|
158
158
|
|
|
159
|
+
-- request_stat.source (v4) records which surface produced an injection: 'section' is the
|
|
160
|
+
-- per-request system-prompt index, 'assist' is the optional once-per-session recall block.
|
|
161
|
+
-- Without it the two were indistinguishable, so recallAssist injections left no trace
|
|
162
|
+
-- anywhere and "why did this entry appear" could not be answered for them.
|
|
163
|
+
-- Kept outside the CREATE TABLE body on purpose: SQLite's ALTER TABLE ... DROP COLUMN
|
|
164
|
+
-- re-parses the stored statement and fails with "incomplete input" when it holds comments.
|
|
159
165
|
CREATE TABLE IF NOT EXISTS request_stat (
|
|
160
166
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
161
167
|
ts INTEGER NOT NULL,
|
|
@@ -163,9 +169,11 @@ CREATE TABLE IF NOT EXISTS request_stat (
|
|
|
163
169
|
scope_key TEXT,
|
|
164
170
|
section_bytes INTEGER NOT NULL,
|
|
165
171
|
entries_listed INTEGER NOT NULL,
|
|
166
|
-
entries_hidden INTEGER NOT NULL
|
|
172
|
+
entries_hidden INTEGER NOT NULL,
|
|
173
|
+
source TEXT
|
|
167
174
|
);
|
|
168
175
|
CREATE INDEX IF NOT EXISTS ix_request_stat_ts ON request_stat (ts DESC);
|
|
176
|
+
CREATE INDEX IF NOT EXISTS ix_request_stat_source ON request_stat (source);
|
|
169
177
|
|
|
170
178
|
CREATE VIRTUAL TABLE IF NOT EXISTS memory_fts USING fts5 (
|
|
171
179
|
tokens,
|
package/src/host/store/search.js
CHANGED
|
@@ -241,22 +241,69 @@ export function lexicalDetail(row, plan) {
|
|
|
241
241
|
}
|
|
242
242
|
}
|
|
243
243
|
let runHit = false;
|
|
244
|
+
let runSpan = 0;
|
|
245
|
+
let runSpanHub = true;
|
|
246
|
+
let runSpanInTitle = false;
|
|
247
|
+
const isHub = typeof plan.hub === 'function' ? plan.hub : () => false;
|
|
244
248
|
for (const run of runs) {
|
|
245
|
-
|
|
249
|
+
// Longest contiguous stretch of the run whose bigrams are all present. All-or-nothing
|
|
250
|
+
// was the old rule, and it made a sentence with one unseen word return nothing:
|
|
251
|
+
// "回滚窗口" found the entry, "回滚窗口多久?" did not, because 多久 appears in no
|
|
252
|
+
// document. Requiring a *contiguous* stretch still rejects the case the old rule was
|
|
253
|
+
// written for — a note that happens to contain 路径 and 断言 in unrelated clauses has
|
|
254
|
+
// a longest span of 2 characters, not 4.
|
|
255
|
+
let span = 0;
|
|
256
|
+
let current = 0;
|
|
257
|
+
let spanHub = true;
|
|
258
|
+
let currentHub = true;
|
|
259
|
+
let spanEnd = 0;
|
|
246
260
|
for (let i = 0; i + 1 < run.length; i += 1) {
|
|
247
261
|
const bigram = run.slice(i, i + 2);
|
|
248
|
-
if (
|
|
249
|
-
|
|
250
|
-
|
|
262
|
+
if (title.includes(bigram) || body.includes(bigram)) {
|
|
263
|
+
if (!isHub(bigram)) currentHub = false;
|
|
264
|
+
current += 1;
|
|
265
|
+
if (current > span) {
|
|
266
|
+
span = current;
|
|
267
|
+
spanHub = currentHub;
|
|
268
|
+
spanEnd = i + 2;
|
|
269
|
+
}
|
|
270
|
+
} else {
|
|
271
|
+
current = 0;
|
|
272
|
+
currentHub = true;
|
|
251
273
|
}
|
|
252
274
|
}
|
|
253
|
-
|
|
275
|
+
// `span` counts bigrams; a stretch of n bigrams covers n+1 characters.
|
|
276
|
+
const spanChars = span === 0 ? 0 : span + 1;
|
|
277
|
+
const spanText = span === 0 ? '' : run.slice(spanEnd - spanChars, spanEnd);
|
|
278
|
+
// A two-character span only counts as evidence when it is a real content word *and*
|
|
279
|
+
// it appears in the title. Without the title requirement, "完全无关的问题?" was
|
|
280
|
+
// admitted by an entry whose body happened to contain 问题 — the query had no topic
|
|
281
|
+
// overlap at all. The title is the entry's subject, so that is where weak evidence
|
|
282
|
+
// has to be.
|
|
283
|
+
const inTitle = spanText.length > 0 && title.includes(spanText);
|
|
284
|
+
if (spanChars > runSpan) {
|
|
285
|
+
runSpan = spanChars;
|
|
286
|
+
runSpanHub = spanHub;
|
|
287
|
+
runSpanInTitle = inTitle;
|
|
288
|
+
} else if (spanChars === runSpan) {
|
|
289
|
+
if (spanHub === false) runSpanHub = false;
|
|
290
|
+
if (inTitle) runSpanInTitle = true;
|
|
291
|
+
}
|
|
292
|
+
if (spanChars === run.length) {
|
|
254
293
|
runHit = true;
|
|
255
294
|
score += 3;
|
|
256
295
|
}
|
|
257
296
|
}
|
|
258
297
|
const coverage = tokens.length === 0 ? 0 : matched / tokens.length;
|
|
259
|
-
return {
|
|
298
|
+
return {
|
|
299
|
+
score: tokens.length === 0 ? 0 : score / (tokens.length + 1),
|
|
300
|
+
runHit,
|
|
301
|
+
runSpan,
|
|
302
|
+
runSpanHub,
|
|
303
|
+
runSpanInTitle,
|
|
304
|
+
coverage,
|
|
305
|
+
hasRuns: runs.length > 0,
|
|
306
|
+
};
|
|
260
307
|
}
|
|
261
308
|
|
|
262
309
|
/**
|
|
@@ -335,7 +382,23 @@ export async function search(store, request) {
|
|
|
335
382
|
// Admission: a CJK run must be fully evidenced; word-only queries need at
|
|
336
383
|
// least half their tokens. Recency is a tie-breaker, never an admission
|
|
337
384
|
// ticket, so a row no other signal matched does not appear just by being new.
|
|
338
|
-
|
|
385
|
+
// Admission. When the query contains a CJK run, a contiguous stretch of at least three
|
|
386
|
+
// characters in the *same* run has to appear in the entry. Three was chosen against the
|
|
387
|
+
// two failure modes:
|
|
388
|
+
// - all-or-nothing (the old rule) rejected "回滚窗口多久?" for a 6-character entry
|
|
389
|
+
// match, because 多久 is in no document — so any sentence with one unfamiliar word
|
|
390
|
+
// returned nothing at all;
|
|
391
|
+
// - plain token coverage admitted notes that merely contain two of the words in
|
|
392
|
+
// different clauses, which is the false positive the run rule existed to stop.
|
|
393
|
+
// A contiguous span is the middle ground: scattered words do not form one, and a real
|
|
394
|
+
// entry match always does.
|
|
395
|
+
// A 2-character span is admitted only when it is a real content word rather than a term
|
|
396
|
+
// that appears in a sixth of the store — `hub` is the existing frequency predicate, so
|
|
397
|
+
// "回滚" passes and "我们" does not. Below 20 entries every token is a non-hub, which is
|
|
398
|
+
// the right answer for a small store: missing a recall is worse than one extra candidate.
|
|
399
|
+
const lexicalAdmitted = lexical.hasRuns
|
|
400
|
+
? lexical.runHit || lexical.runSpan >= 3 || (lexical.runSpan === 2 && lexical.runSpanHub === false && lexical.runSpanInTitle)
|
|
401
|
+
: lexical.coverage >= 0.5;
|
|
339
402
|
if (!lexicalAdmitted && entityHits === 0 && vectorScore === 0) continue;
|
|
340
403
|
const age = Math.max(0, now - Number(row.updated_at ?? now));
|
|
341
404
|
const recency = Math.exp(-age / (30 * DAY));
|
package/src/host/store/store.js
CHANGED
|
@@ -1087,7 +1087,7 @@ export class Store {
|
|
|
1087
1087
|
.get(...keys)?.n ?? 0,
|
|
1088
1088
|
);
|
|
1089
1089
|
const lastInjection = this.db
|
|
1090
|
-
.prepare('SELECT ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden FROM request_stat ORDER BY id DESC LIMIT 1')
|
|
1090
|
+
.prepare('SELECT ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden, source FROM request_stat ORDER BY id DESC LIMIT 1')
|
|
1091
1091
|
.get();
|
|
1092
1092
|
const revisions = Number(this.db.prepare('SELECT COUNT(*) AS n FROM memory').get()?.n ?? 0);
|
|
1093
1093
|
return {
|
|
@@ -1113,6 +1113,7 @@ export class Store {
|
|
|
1113
1113
|
bytes: Number(lastInjection.section_bytes),
|
|
1114
1114
|
listed: Number(lastInjection.entries_listed),
|
|
1115
1115
|
hidden: Number(lastInjection.entries_hidden),
|
|
1116
|
+
source: lastInjection.source ?? 'section',
|
|
1116
1117
|
},
|
|
1117
1118
|
dbBytes: existsSync(this.paths.db) ? statSync(this.paths.db).size : 0,
|
|
1118
1119
|
vector: { ...this.vector, ...this.embedding.health() },
|
|
@@ -1125,11 +1126,16 @@ export class Store {
|
|
|
1125
1126
|
* @param row - `sessionId`, `scopeKey`, `bytes`, `listed`, `hidden`.
|
|
1126
1127
|
*/
|
|
1127
1128
|
recordInjection(row) {
|
|
1129
|
+
// `source` separates the two injection surfaces. The index runs every request; the
|
|
1130
|
+
// recall assist runs once per session and, before this column existed, recorded
|
|
1131
|
+
// nothing at all — so the one conditional path in the product was also the one with
|
|
1132
|
+
// no answer to "why did this appear".
|
|
1133
|
+
const source = typeof row.source === 'string' && row.source.length > 0 ? row.source : 'section';
|
|
1128
1134
|
this.db
|
|
1129
1135
|
.prepare(
|
|
1130
|
-
'INSERT INTO request_stat (ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden) VALUES (?, ?, ?, ?, ?, ?)',
|
|
1136
|
+
'INSERT INTO request_stat (ts, session_id, scope_key, section_bytes, entries_listed, entries_hidden, source) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
|
1131
1137
|
)
|
|
1132
|
-
.run(this.now(), row.sessionId ?? null, row.scopeKey ?? null, row.bytes, row.listed, row.hidden);
|
|
1138
|
+
.run(this.now(), row.sessionId ?? null, row.scopeKey ?? null, row.bytes, row.listed, row.hidden, source);
|
|
1133
1139
|
// Per-entry counters, so "which line earns its bytes" stays answerable after the
|
|
1134
1140
|
// aggregate rows rotate out. A listed entry never fetched by id is the dead
|
|
1135
1141
|
// weight this product exists to find.
|
|
@@ -1151,6 +1157,10 @@ export class Store {
|
|
|
1151
1157
|
bytes: row.bytes,
|
|
1152
1158
|
listed: row.listed,
|
|
1153
1159
|
hidden: row.hidden,
|
|
1160
|
+
source,
|
|
1161
|
+
// The ids are what makes the line answerable. Without them an audit record says
|
|
1162
|
+
// "3 entries were injected" and nothing about which three.
|
|
1163
|
+
ids: Array.isArray(row.ids) ? row.ids : undefined,
|
|
1154
1164
|
});
|
|
1155
1165
|
}
|
|
1156
1166
|
|
|
@@ -136,6 +136,19 @@ export function buildQueryPlan(query, options = {}) {
|
|
|
136
136
|
|
|
137
137
|
const terms = [];
|
|
138
138
|
for (const phrase of phrases) terms.push(phrase);
|
|
139
|
+
// Each CJK run is matched as a phrase, which is precise — but a phrase has to appear
|
|
140
|
+
// contiguously and in order, so **one word the store has never seen zeroes the whole
|
|
141
|
+
// query**. Measured before this change: "回滚窗口" found the entry, "回滚窗口多久?"
|
|
142
|
+
// found nothing, because 多久 is not in any document and the run is a single phrase
|
|
143
|
+
// with no alternative to fall back on. People type sentences, so that made search
|
|
144
|
+
// return nothing for most real questions.
|
|
145
|
+
//
|
|
146
|
+
// The bigrams are therefore also offered individually. Recall widens; the in-process
|
|
147
|
+
// scorer still rewards a contiguous run (`runs`), so precision is decided there rather
|
|
148
|
+
// than by making the FTS expression fragile.
|
|
149
|
+
for (const phrase of phrases) {
|
|
150
|
+
for (const bigram of phrase.replace(/"/g, '').split(' ')) terms.push(bigram);
|
|
151
|
+
}
|
|
139
152
|
for (const token of singles) terms.push(token);
|
|
140
153
|
// Exact terms, not prefix queries: FTS5 has no prefix index here, so `word*`
|
|
141
154
|
// scans the whole term dictionary (measured: 1.4 s on a 10k-entry store).
|