dsh-limit-break 1.4.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,26 @@ All notable changes to `dsh-limit-break`. Format follows
5
5
 
6
6
  ## Unreleased
7
7
 
8
+ ## 1.5.0 — 2026-10-22
9
+
10
+ ### Changed
11
+ - **`memory_recall` looks where you point it.** The scorer used raw substring counts over five
12
+ lowercased blobs, which let common words outvote the one term that identifies a fact ("the",
13
+ "how", "long"). It now drops English stopwords, stems (`slides`/`slide`, `rendering`/`render`),
14
+ matches whole words with a prefix fallback (`ray` → `raytracer`), splits dotted ids into segments
15
+ (`bench.gpu_monte_carlo` → `bench`/`gpu`/`monte`/`carlo`), weights each query token by idf
16
+ against the live store, and adds a bonus for a whole-phrase hit in the statement.
17
+ - **`tools/recall-bench.mjs`** (new): 20 hand-labelled questions → hit@1, hit@3, MRR, misses
18
+ printed. Same 20-fact store, before and after: **hit@1 0.75 → 0.95**, MRR **0.867 → 0.975**,
19
+ hit@3 1.0 → 1.0. The remaining miss ("monte carlo pi benchmark on the CPU") is ambiguous
20
+ between `bench.cpu_suite` and `bench.gpu_monte_carlo` and ranks second.
21
+
22
+ ### Notes
23
+ - No tool added, removed or renamed: the four-tool ABI is untouched (see 1.4.0). `tokenize`,
24
+ `scoreFact` and `rankFacts` are newly exported from `lib/index.js` only so the benchmark measures
25
+ the shipped scorer instead of a copy of it.
26
+ - Store format unchanged; 1.2.x–1.4.0 stores load as-is.
27
+
8
28
  ## 1.4.0 — 2026-10-22
9
29
 
10
30
  ### Removed
package/MAINTAIN.md CHANGED
@@ -108,7 +108,16 @@ git add -A && git commit -m "..." && git push
108
108
 
109
109
  ---
110
110
 
111
- ## Backlog(按 2026-10-20 20:06 的真实状态)
111
+ ## Backlog(按 2026-10-22 20:10 的真实状态)
112
+
113
+ - [x] **1.5.0 召回排序改进**(2026-10-22):`memory_recall` 由“子串计数”改为 stopword 过滤 +
114
+ 轻量词干 + 整词/前缀匹配 + dotted id 分段 + idf 加权 + 短语加分;新增
115
+ `tools/recall-bench.mjs`(20 题),实测 hit@1 **0.75 → 0.95**、MRR **0.867 → 0.975**、hit@3 1.0。
116
+ 四工具 ABI 未变,1.4.0 的“工具集冻结”规则仍然成立。
117
+ - [ ] 基准里剩下的 miss(“monte carlo pi benchmark on the CPU”)是 `bench.cpu_suite` 与
118
+ `bench.gpu_monte_carlo` 的真实歧义;再往上做要引入上下文/近义词表,性价比低,当作已知上限。
119
+ - [ ] desktop profile 可能钉在旧版本——每次发布后用
120
+ `dsh.cmd plugin --profile desktop add dsh-limit-break@<version>` 消漂移。
112
121
 
113
122
  - [x] **1.2.0 已上线**(2026-10-20T12:02:52Z):npm `latest` = `1.2.0`,GitHub `origin/main` =
114
123
  `fd844a4`,`verify` PASS,`test_plugin.mjs` 6/6 PASS。用户 20:04 完成了 OTP 发布和推送。
@@ -120,7 +129,8 @@ git add -A && git commit -m "..." && git push
120
129
  - [ ] npm 上还挂着占位包 `0.0.0-stage`(`stub: true`)。`npm deprecate` 也要 OTP(本轮 EOTP 失败);
121
130
  用户一条命令即可:`npm deprecate dsh-limit-break@0.0.0-stage "placeholder - use 1.2.0"`。
122
131
  - [ ] `1.0.0` 是无元数据的旧包体(author/repository 都不在)。同样一条 `npm deprecate` 可清掉噪音(需 OTP)。
123
- - [ ] `facts.json` 里 `env.gpu` 仍写着"no GPU compute benchmark was run"——已过时(GPU 实测 54.3×)。
132
+ - [x] `facts.json` 的 `env.gpu` 已改写成“GPU compute throughput is recorded separately in
133
+ bench.gpu_monte_carlo and bench.render”(2026-10-21 的回写),陈旧描述不再存在。
124
134
  - [x] 插件工具对**已经打开的会话**不可见:已在 README「Known limits」第 1 条写明
125
135
  (tool list 是 per-session 快照,插件侧修不了)。事实 `harness.plugin-tool-visibility` 有实测。
126
136
  - [ ] `resilient_fetch` 目前按"主机名别名"降级,可以继续加:`--resolve` 到已知 IP、
package/README.md CHANGED
@@ -13,6 +13,14 @@ self-benchmark: two things an agent is normally told it cannot do, made cheap en
13
13
  Zero npm dependencies. No import-time side effects. Everything fails soft — a broken store or a dead
14
14
  network returns an error object, never an exception that could take the host down.
15
15
 
16
+ **Recall quality.** `memory_recall` is measured, not vibed: `tools/recall-bench.mjs` runs 20
17
+ hand-labelled questions against the live store and prints hit@1 / hit@3 / MRR. At 1.5.0 the ranking
18
+ skips English stopwords, stems, matches whole words (with a prefix fallback, so `ray` finds
19
+ `raytracer`), splits dotted ids into segments, weights each query token by inverse document
20
+ frequency, and rewards a whole-phrase hit: **hit@1 0.75 → 0.95, MRR 0.867 → 0.975** on the same 20
21
+ questions and the same 20 facts. The one remaining miss is a genuinely ambiguous question that
22
+ matches two benchmark facts.
23
+
16
24
  **Runtime requirement: `curl`.** Rung 1 of the fetch ladder is plain `fetch` and needs nothing; rungs
17
25
  2–4 (IPv6 → IPv4 → sibling hostname) shell out to `curl`, so it has to be on `PATH`. Windows 10+,
18
26
  macOS and mainstream Linux all ship it. Without it the ladder degrades to rung 1 and says so in the
@@ -87,9 +95,10 @@ Measured, not guessed — each line is something a session actually hit.
87
95
  machine is the intended use.
88
96
  * **The fetch ladder is hostname-level.** It does not defeat IP-level blocks, TLS interception,
89
97
  or a proxy that terminates the connection; it does not skip certificate verification.
90
- * **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement), no embeddings, no
91
- stemming above the tokenisation in `tokenize()`. A query that shares no token with a fact will
92
- miss it.
98
+ * **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement/details), no
99
+ embeddings: stopword-filtered, crudely stemmed word matching weighted by idf, so it is good at
100
+ "which fact did I mean" and bad at paraphrase. `tools/recall-bench.mjs` states the current
101
+ numbers (hit@1 0.95 on 20 questions); a query that shares no word with a fact still misses it.
93
102
  * **`resilient_fetch` reads at most 2 MiB per response** and returns at most 20 000 characters;
94
103
  bigger payloads are truncated, not streamed.
95
104
 
package/lib/index.js CHANGED
@@ -182,31 +182,105 @@ async function saveStore(dir, store) {
182
182
  return { file, bytes: Buffer.byteLength(body) }
183
183
  }
184
184
 
185
+ // Ranking inputs. The old scorer used raw substring counts, which let a common word
186
+ // ("the", "how", "long") outvote the one rare word that identifies the fact. Words are
187
+ // matched against word sets instead, suffixed words share a stem, and each query token is
188
+ // weighted by how rare it is in the store (idf), so a distinctive term wins a tie.
189
+ const STOPWORDS = new Set([
190
+ 'a', 'about', 'all', 'also', 'an', 'and', 'any', 'are', 'as', 'at', 'be', 'been', 'but', 'by',
191
+ 'can', 'could', 'did', 'do', 'does', 'each', 'few', 'for', 'from', 'had', 'has', 'have', 'here',
192
+ 'how', 'if', 'in', 'into', 'is', 'it', 'its', 'just', 'many', 'me', 'more', 'most', 'much', 'my',
193
+ 'no', 'not', 'of', 'on', 'once', 'only', 'or', 'other', 'our', 'out', 'over', 'should', 'some',
194
+ 'such', 'than', 'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'those', 'to',
195
+ 'too', 'under', 'up', 'very', 'was', 'we', 'were', 'what', 'when', 'where', 'which', 'who', 'why',
196
+ 'will', 'with', 'would', 'you', 'your'
197
+ ])
198
+
199
+ function stem(word) {
200
+ let w = word
201
+ if (w.length > 5 && w.endsWith('ies')) return `${w.slice(0, -3)}y`
202
+ if (w.length > 4 && w.endsWith('es')) w = w.slice(0, -2)
203
+ else if (w.length > 3 && w.endsWith('s') && !w.endsWith('ss')) w = w.slice(0, -1)
204
+ if (w.length > 5 && w.endsWith('ing')) w = w.slice(0, -3)
205
+ else if (w.length > 4 && w.endsWith('ed')) w = w.slice(0, -2)
206
+ if (w.length > 5 && w.endsWith('ly')) w = w.slice(0, -2)
207
+ return w
208
+ }
209
+
185
210
  function tokenize(query) {
186
211
  return String(query || '')
212
+ .toLowerCase()
213
+ // "memory_prune" should match the id segment "memory_prune" as well as "memory"/"prune".
214
+ .replace(/_+/g, ' ')
215
+ .split(/[^\p{L}\p{N}_.+-]+/u)
216
+ .filter((part) => part.length > 1 && !STOPWORDS.has(part))
217
+ .map(stem)
218
+ }
219
+
220
+ function words(text) {
221
+ return String(text || '')
187
222
  .toLowerCase()
188
223
  .split(/[^\p{L}\p{N}_.+-]+/u)
189
- .filter((part) => part.length > 1)
224
+ .filter(Boolean)
225
+ .map(stem)
226
+ }
227
+
228
+ function wordSet(text) {
229
+ return new Set(words(text))
190
230
  }
191
231
 
192
- function scoreFact(fact, tokens) {
232
+ /** 1 for the exact word, 0.5 when the query token is a prefix of a stored word ("ray" ~ "raytracer"). */
233
+ function wordMatch(set, token) {
234
+ if (set.has(token)) return 1
235
+ for (const word of set) {
236
+ if (word.length > token.length && word.startsWith(token)) return 0.5
237
+ }
238
+ return 0
239
+ }
240
+
241
+ const FIELD_WEIGHTS = { id: 7, topic: 2, tags: 3, statement: 3, details: 1 }
242
+
243
+ function scoreFact(fact, tokens, idf) {
193
244
  if (!tokens.length) return 0
194
- const id = String(fact.id || '').toLowerCase()
195
- const statement = String(fact.statement || '').toLowerCase()
196
- const topic = String(fact.topic || '').toLowerCase()
197
- const tags = (Array.isArray(fact.tags) ? fact.tags : []).join(' ').toLowerCase()
198
- const details = fact.details ? JSON.stringify(fact.details).toLowerCase() : ''
245
+ // Ids are dotted paths ("bench.gpu_monte_carlo"); split them so a token can match a segment.
246
+ const id = new Set(String(fact.id || '').toLowerCase().split(/[^\p{L}\p{N}]+/u).filter(Boolean).map(stem))
247
+ const topic = wordSet(fact.topic)
248
+ const tags = wordSet((Array.isArray(fact.tags) ? fact.tags : []).join(' '))
249
+ const statement = wordSet(fact.statement)
250
+ const details = wordSet(fact.details ? JSON.stringify(fact.details) : '')
199
251
  let score = 0
200
252
  for (const token of tokens) {
201
- if (id.includes(token)) score += 5
202
- if (topic.includes(token)) score += 2
203
- if (tags.includes(token)) score += 2
204
- if (statement.includes(token)) score += 3
205
- if (details.includes(token)) score += 1
253
+ const boost = idf instanceof Map ? idf.get(token) || 1 : 1
254
+ score += boost * (
255
+ FIELD_WEIGHTS.id * wordMatch(id, token) +
256
+ FIELD_WEIGHTS.topic * wordMatch(topic, token) +
257
+ FIELD_WEIGHTS.tags * wordMatch(tags, token) +
258
+ FIELD_WEIGHTS.statement * wordMatch(statement, token) +
259
+ FIELD_WEIGHTS.details * wordMatch(details, token)
260
+ )
206
261
  }
262
+ const phrase = tokens.join(' ')
263
+ if (tokens.length > 1 && String(fact.statement || '').toLowerCase().includes(phrase)) score += 6
207
264
  return score
208
265
  }
209
266
 
267
+ /** Scored ordering used by memory_recall; exported so the recall benchmark can measure it. */
268
+ function rankFacts(facts, query) {
269
+ const tokens = tokenize(query)
270
+ const living = (facts || []).filter(Boolean)
271
+ if (!tokens.length) return living.map((fact) => ({ fact, score: 0 }))
272
+ const idf = new Map()
273
+ for (const token of tokens) {
274
+ let df = 0
275
+ for (const fact of living) if (scoreFact(fact, [token]) > 0) df += 1
276
+ idf.set(token, +(1 + Math.log((living.length + 1) / (df + 1))).toFixed(4))
277
+ }
278
+ return living
279
+ .map((fact) => ({ fact, score: scoreFact(fact, tokens, idf) }))
280
+ .filter((row) => row.score > 0)
281
+ .sort((a, b) => b.score - a.score)
282
+ }
283
+
210
284
  function shapeFact(fact) {
211
285
  return {
212
286
  id: fact.id,
@@ -426,11 +500,7 @@ export function apply(ctx, config) {
426
500
  const tokens = tokenize(input.query)
427
501
  let picked = store.facts.filter(Boolean)
428
502
  if (!input.all && tokens.length) {
429
- picked = picked
430
- .map((fact) => ({ fact, score: scoreFact(fact, tokens) }))
431
- .filter((row) => row.score > 0)
432
- .sort((a, b) => b.score - a.score)
433
- .map((row) => row.fact)
503
+ picked = rankFacts(picked, input.query).map((row) => row.fact)
434
504
  }
435
505
  // Number(input.limit) || 8 turned 0/NaN into 8 but let Infinity through to slice()
436
506
  // (which returns everything) and gave no upper bound below the cap.
@@ -636,4 +706,4 @@ export function apply(ctx, config) {
636
706
  }
637
707
 
638
708
  export default { name, inject, apply }
639
- export { resolveStoreDir, loadStore, saveStore, alternatesFor, clip }
709
+ export { resolveStoreDir, loadStore, saveStore, alternatesFor, clip, tokenize, scoreFact, rankFacts }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-limit-break",
3
- "version": "1.4.0",
3
+ "version": "1.5.0",
4
4
  "description": "Durable cross-session memory and a fetch ladder that survives hostname-keyed blocking — four small tools for DeepSeek Harness.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
@@ -0,0 +1,72 @@
1
+ /**
2
+ * Recall benchmark: how often does memory_recall's ranking put the intended fact first?
3
+ *
4
+ * It reads the live store (read-only) and runs a fixed query set whose expected ids were
5
+ * written by hand from the fact statements. Hit@1 is the number that matters: an agent asks
6
+ * one question and reads the top answer. Run: node tools/recall-bench.mjs [--json]
7
+ */
8
+ import { readFile } from 'node:fs/promises'
9
+ import { join } from 'node:path'
10
+ import { homedir } from 'node:os'
11
+ import { rankFacts } from '../lib/index.js'
12
+
13
+ const STORE = join(homedir(), '.dsh', 'memory', 'facts.json')
14
+
15
+ /** query -> expected fact id. Phrased the way a later session actually asks. */
16
+ const CASES = [
17
+ ['which CPU is in this machine', 'env.cpu'],
18
+ ['how many cores does the processor have', 'env.cpu'],
19
+ ['how much VRAM does the GPU have', 'env.gpu'],
20
+ ['what python version is bundled', 'env.python'],
21
+ ['where is cupy installed', 'gpu.cupy_installed'],
22
+ ['what is the DSH home directory', 'env.dsh_home'],
23
+ ['which node and npm versions are installed', 'env.node_npm'],
24
+ ['monte carlo pi benchmark on the CPU', 'bench.cpu_suite'],
25
+ ['how long did the ray tracer take at 1080p', 'bench.render'],
26
+ ['GPU monte carlo speedup over CPU', 'bench.gpu_monte_carlo'],
27
+ ['how long did the batch of four pipelines run', 'batch.summary'],
28
+ ['why is wikipedia blocked by SNI', 'net.wikipedia_access'],
29
+ ['is github push failing', 'network.github-git'],
30
+ ['deck rating for physical world execution', 'deck.ratings'],
31
+ ['how many slides in the round 2 deck', 'deck.round2'],
32
+ ['which tools does the memory plugin provide', 'tool.limit_break_plugin'],
33
+ ['weekly maintenance objective for the plugin', 'task.limit-break.maintenance'],
34
+ ['what is the latest npm release', 'release.dsh-limit-break'],
35
+ ['why was memory_prune removed', 'tool.memory_prune'],
36
+ ['what renders the film with numpy', 'film.wotaiyang']
37
+ ]
38
+
39
+ const raw = JSON.parse(await readFile(STORE, 'utf8'))
40
+ const facts = (raw.facts || []).filter(Boolean)
41
+
42
+ let hit1 = 0
43
+ let hit3 = 0
44
+ let rr = 0
45
+ const misses = []
46
+ for (const [query, want] of CASES) {
47
+ const ranked = rankFacts(facts, query).map((row) => row.fact.id)
48
+ const at = ranked.indexOf(want)
49
+ if (at === 0) hit1 += 1
50
+ if (at >= 0 && at < 3) hit3 += 1
51
+ if (at >= 0) rr += 1 / (at + 1)
52
+ if (at !== 0) misses.push({ query, want, at: at < 0 ? 'miss' : at + 1, top: ranked.slice(0, 3) })
53
+ }
54
+
55
+ const report = {
56
+ store: STORE,
57
+ facts: facts.length,
58
+ cases: CASES.length,
59
+ hit_at_1: +(hit1 / CASES.length).toFixed(3),
60
+ hit_at_3: +(hit3 / CASES.length).toFixed(3),
61
+ mrr: +(rr / CASES.length).toFixed(3),
62
+ misses
63
+ }
64
+
65
+ if (process.argv.includes('--json')) {
66
+ console.log(JSON.stringify(report, null, 2))
67
+ } else {
68
+ console.log(`facts=${report.facts} cases=${report.cases} hit@1=${report.hit_at_1} hit@3=${report.hit_at_3} mrr=${report.mrr}`)
69
+ for (const miss of report.misses) {
70
+ console.log(` miss: "${miss.query}" wanted ${miss.want} got rank ${miss.at} -> ${miss.top.join(', ')}`)
71
+ }
72
+ }