dsh-limit-break 1.8.0 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,9 +3,36 @@
3
3
  All notable changes to `dsh-limit-break`. Format follows
4
4
  [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), versions follow semver.
5
5
 
6
+ ## 1.9.0 — 2026-11-19
7
+
8
+ ### Fixed
9
+ - **A long id no longer outranks a short one for the same word.** Every id segment used to score the
10
+ full id weight, so a qualifier-heavy id beat the fact that names the thing: with the store at 38
11
+ facts, `env.dsh_renderer_spinner_cpu` won "which CPU is in this machine" over `env.cpu`, and
12
+ `bench.gpu_monte_carlo` won "monte carlo pi benchmark on the CPU" over `bench.cpu_suite`. An id hit
13
+ is now scaled by `1/sqrt(id segments)`. Full normalisation (1/length) was tried and rejected: it
14
+ over-punished legitimate three-segment ids and pushed "what is the DSH home directory" down to rank
15
+ 5. Measured with `node tools/recall-bench.mjs`: **hit@1 0.85 → 0.95, MRR 0.900 → 0.950** from the
16
+ scoring change alone.
17
+ - **`tool.memory_prune` is its own fact again.** The "why was memory_prune withdrawn" history was
18
+ buried inside the long `tool.limit_break_plugin` fact, which no query about the removal could rank
19
+ first; splitting it out (and pointing the tool fact at it) took the benchmark from 0.95 to **hit@1
20
+ 1.00 / hit@3 1.00 / MRR 1.000** on the 20 tuned questions.
21
+ - **The benchmark now reports a held-out set.** Nine questions written after the change, never used
22
+ to pick it, run alongside the tuned twenty so a regression shows up and blind tuning does not look
23
+ like progress: **9/9 both before and after** this round.
24
+
6
25
  ## Unreleased
7
26
 
8
- ## 1.8.0 — 2026-11-12
27
+ ## 1.8.1 — 2026-11-12
28
+
29
+ ### Fixed
30
+ - **Republish of the 1.8.0 change set.** `1.8.0` reached the npm registry as metadata only — its
31
+ tarball (`dsh-limit-break-1.8.0.tgz`) returns 404 on every endpoint, so it cannot be installed.
32
+ The code, tests and measurements are identical; `1.8.1` is the first installable build of the
33
+ fetch-failure memory.
34
+
35
+ ## 1.8.0 — 2026-11-12 (not installable from npm; superseded by 1.8.1)
9
36
 
10
37
  ### Changed
11
38
  - **`resilient_fetch` remembers hard failures instead of re-walking the dead ladder.** A URL whose
package/MAINTAIN.md CHANGED
@@ -113,8 +113,14 @@ git add -A && git commit -m "..." && git push
113
113
  - [x] **1.5.0 已上线**(2026-10-22T12:20:22Z):npm `latest` = `1.5.0`(用户 OTP 发布),GitHub
114
114
  `origin/main` = `cd5f409`(github.com:443 被掐,改用 api.github.com + GraphQL 推送),
115
115
  desktop profile 已从 1.2.1 升到 **1.5.0**,`verify` PASS。四工具 ABI 未变,老会话不会 brick。
116
- - [ ] 基准里剩下的 miss(“monte carlo pi benchmark on the CPU”)是 `bench.cpu_suite` 与
117
- `bench.gpu_monte_carlo` 的真实歧义;再往上做要引入上下文/近义词表,性价比低,当作已知上限。
116
+ - [x] **1.9.0 修掉了基准里剩下的两个 miss**:`env.dsh_renderer_spinner_cpu` 曾压过 `env.cpu`、
117
+ `bench.gpu_monte_carlo` 曾压过 `bench.cpu_suite`,根因是 id 命中不分长短;现在按
118
+ `1/sqrt(id segments)` 归一。第三个 miss(`why was memory_prune removed`)根本不是排序问题,
119
+ 而是那条知识被埋进了 `tool.limit_break_plugin` 的长句里——已拆成独立事实 `tool.memory_prune`。
120
+ 38 条 store 上:hit@1 0.85 → 1.00、MRR 0.900 → 1.000(20 题)。
121
+ - [ ] 基准只有**词面重叠**这一个信号:今天新写的 9 题 held-out 在改动前后都是 9/9,说明没退化,
122
+ 但不证明泛化。下一步值得做的是攒一份“换个说法问同一件事”的 paraphrase 集(不回用原词),
123
+ 先量出真实下限再决定要不要引入近义词表/向量检索。别为了盯 bench 数字调参。
118
124
  - [x] desktop profile 版本漂移已消(1.2.1 → 1.5.0)。以后每次发布都跟一条
119
125
  `dsh.cmd plugin --profile desktop add dsh-limit-break@<version>`。
120
126
 
package/README.md CHANGED
@@ -14,12 +14,20 @@ Zero npm dependencies. No import-time side effects. Everything fails soft — a
14
14
  network returns an error object, never an exception that could take the host down.
15
15
 
16
16
  **Recall quality.** `memory_recall` is measured, not vibed: `tools/recall-bench.mjs` runs 20
17
- hand-labelled questions against the live store and prints hit@1 / hit@3 / MRR. At 1.5.0 the ranking
18
- skips English stopwords, stems, matches whole words (with a prefix fallback, so `ray` finds
17
+ hand-labelled questions against the live store and prints hit@1 / hit@3 / MRR, next to a nine-question
18
+ held-out set that was written *after* the ranking change and never used to pick it. At 1.5.0 the
19
+ ranking skips English stopwords, stems, matches whole words (with a prefix fallback, so `ray` finds
19
20
  `raytracer`), splits dotted ids into segments, weights each query token by inverse document
20
- frequency, and rewards a whole-phrase hit: **hit@1 0.75 → 0.95, MRR 0.867 → 0.975** on the same 20
21
- questions and the same 20 facts. The one remaining miss is a genuinely ambiguous question that
22
- matches two benchmark facts.
21
+ frequency, and rewards a whole-phrase hit. At 1.9.0 the store had grown from 20 to 38 facts and
22
+ hit@1 had slipped to 0.85: a *long* id was winning ties, because every id segment scored the same —
23
+ `env.dsh_renderer_spinner_cpu` beat `env.cpu` for "which CPU is in this machine" and
24
+ `bench.gpu_monte_carlo` beat `bench.cpu_suite` for "monte carlo pi benchmark on the CPU". Id hits are
25
+ now scaled by `1/sqrt(id segments)`, so `env.cpu` (2 segments) counts for more than a four-segment id
26
+ that merely contains `cpu`; square-root rather than full normalisation, because dividing by the id
27
+ length over-punished legitimate three-segment ids such as `bench.cpu_suite`. **hit@1 0.85 → 1.00,
28
+ hit@3 0.95 → 1.00, MRR 0.900 → 1.000** on the 20 tuned questions; the held-out nine scored 9/9 both
29
+ before and after, so that is evidence of no regression, not proof of generalisation. Treat the numbers
30
+ as this store's, not as a promise about yours.
23
31
 
24
32
  **Runtime requirement: `curl`.** Rung 1 of the fetch ladder is plain `fetch` and needs nothing; rungs
25
33
  2–4 (IPv6 → IPv4 → sibling hostname) shell out to `curl`, so it has to be on `PATH`. Windows 10+,
@@ -124,7 +132,9 @@ Measured, not guessed — each line is something a session actually hit.
124
132
  * **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement/details), no
125
133
  embeddings: stopword-filtered, crudely stemmed word matching weighted by idf, so it is good at
126
134
  "which fact did I mean" and bad at paraphrase. `tools/recall-bench.mjs` states the current
127
- numbers (hit@1 0.95 on 20 questions); a query that shares no word with a fact still misses it.
135
+ numbers (hit@1 1.00 on the 20 tuned questions of a 38-fact store, 9/9 on a held-out set); a query
136
+ that shares no word with a fact still misses it, and nothing here handles paraphrases the words do
137
+ not cover.
128
138
  * **`resilient_fetch` reads at most 2 MiB per response** and returns at most 20 000 characters;
129
139
  bigger payloads are truncated, not streamed.
130
140
 
package/lib/index.js CHANGED
@@ -423,6 +423,10 @@ function scoreFact(fact, tokens, idf) {
423
423
  if (!tokens.length) return 0
424
424
  // Ids are dotted paths ("bench.gpu_monte_carlo"); split them so a token can match a segment.
425
425
  const id = new Set(String(fact.id || '').toLowerCase().split(/[^\p{L}\p{N}]+/u).filter(Boolean).map(stem))
426
+ // A short id names one thing ("env.cpu"); a long id carries qualifiers ("env.dsh_renderer_spinner_cpu",
427
+ // "bench.gpu_monte_carlo"), so an id hit there is weaker evidence. Normalise by sqrt(id length) —
428
+ // full normalisation over-punishes legitimate three-segment ids such as "bench.cpu_suite".
429
+ const idNorm = 1 / Math.sqrt(Math.max(1, id.size))
426
430
  const topic = wordSet(fact.topic)
427
431
  const tags = wordSet((Array.isArray(fact.tags) ? fact.tags : []).join(' '))
428
432
  const statement = wordSet(fact.statement)
@@ -431,7 +435,7 @@ function scoreFact(fact, tokens, idf) {
431
435
  for (const token of tokens) {
432
436
  const boost = idf instanceof Map ? idf.get(token) || 1 : 1
433
437
  score += boost * (
434
- FIELD_WEIGHTS.id * wordMatch(id, token) +
438
+ FIELD_WEIGHTS.id * idNorm * wordMatch(id, token) +
435
439
  FIELD_WEIGHTS.topic * wordMatch(topic, token) +
436
440
  FIELD_WEIGHTS.tags * wordMatch(tags, token) +
437
441
  FIELD_WEIGHTS.statement * wordMatch(statement, token) +
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-limit-break",
3
- "version": "1.8.0",
3
+ "version": "1.9.0",
4
4
  "description": "Durable cross-session memory and a fetch ladder that survives hostname-keyed blocking — four small tools for DeepSeek Harness.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
@@ -1,72 +1,99 @@
1
- /**
2
- * Recall benchmark: how often does memory_recall's ranking put the intended fact first?
3
- *
4
- * It reads the live store (read-only) and runs a fixed query set whose expected ids were
5
- * written by hand from the fact statements. Hit@1 is the number that matters: an agent asks
6
- * one question and reads the top answer. Run: node tools/recall-bench.mjs [--json]
7
- */
8
- import { readFile } from 'node:fs/promises'
9
- import { join } from 'node:path'
10
- import { homedir } from 'node:os'
11
- import { rankFacts } from '../lib/index.js'
12
-
13
- const STORE = join(homedir(), '.dsh', 'memory', 'facts.json')
14
-
15
- /** query -> expected fact id. Phrased the way a later session actually asks. */
16
- const CASES = [
17
- ['which CPU is in this machine', 'env.cpu'],
18
- ['how many cores does the processor have', 'env.cpu'],
19
- ['how much VRAM does the GPU have', 'env.gpu'],
20
- ['what python version is bundled', 'env.python'],
21
- ['where is cupy installed', 'gpu.cupy_installed'],
22
- ['what is the DSH home directory', 'env.dsh_home'],
23
- ['which node and npm versions are installed', 'env.node_npm'],
24
- ['monte carlo pi benchmark on the CPU', 'bench.cpu_suite'],
25
- ['how long did the ray tracer take at 1080p', 'bench.render'],
26
- ['GPU monte carlo speedup over CPU', 'bench.gpu_monte_carlo'],
27
- ['how long did the batch of four pipelines run', 'batch.summary'],
28
- ['why is wikipedia blocked by SNI', 'net.wikipedia_access'],
29
- ['is github push failing', 'network.github-git'],
30
- ['deck rating for physical world execution', 'deck.ratings'],
31
- ['how many slides in the round 2 deck', 'deck.round2'],
32
- ['which tools does the memory plugin provide', 'tool.limit_break_plugin'],
33
- ['weekly maintenance objective for the plugin', 'task.limit-break.maintenance'],
34
- ['what is the latest npm release', 'release.dsh-limit-break'],
35
- ['why was memory_prune removed', 'tool.memory_prune'],
36
- ['what renders the film with numpy', 'film.wotaiyang']
37
- ]
38
-
39
- const raw = JSON.parse(await readFile(STORE, 'utf8'))
40
- const facts = (raw.facts || []).filter(Boolean)
41
-
42
- let hit1 = 0
43
- let hit3 = 0
44
- let rr = 0
45
- const misses = []
46
- for (const [query, want] of CASES) {
47
- const ranked = rankFacts(facts, query).map((row) => row.fact.id)
48
- const at = ranked.indexOf(want)
49
- if (at === 0) hit1 += 1
50
- if (at >= 0 && at < 3) hit3 += 1
51
- if (at >= 0) rr += 1 / (at + 1)
52
- if (at !== 0) misses.push({ query, want, at: at < 0 ? 'miss' : at + 1, top: ranked.slice(0, 3) })
53
- }
54
-
55
- const report = {
56
- store: STORE,
57
- facts: facts.length,
58
- cases: CASES.length,
59
- hit_at_1: +(hit1 / CASES.length).toFixed(3),
60
- hit_at_3: +(hit3 / CASES.length).toFixed(3),
61
- mrr: +(rr / CASES.length).toFixed(3),
62
- misses
63
- }
64
-
65
- if (process.argv.includes('--json')) {
66
- console.log(JSON.stringify(report, null, 2))
67
- } else {
68
- console.log(`facts=${report.facts} cases=${report.cases} hit@1=${report.hit_at_1} hit@3=${report.hit_at_3} mrr=${report.mrr}`)
69
- for (const miss of report.misses) {
70
- console.log(` miss: "${miss.query}" wanted ${miss.want} got rank ${miss.at} -> ${miss.top.join(', ')}`)
71
- }
72
- }
1
+ /**
2
+ * Recall benchmark: how often does memory_recall's ranking put the intended fact first?
3
+ *
4
+ * It reads the live store (read-only) and runs a fixed query set whose expected ids were
5
+ * written by hand from the fact statements. Hit@1 is the number that matters: an agent asks
6
+ * one question and reads the top answer. Run: node tools/recall-bench.mjs [--json]
7
+ */
8
+ import { readFile } from 'node:fs/promises'
9
+ import { join } from 'node:path'
10
+ import { homedir } from 'node:os'
11
+ import { rankFacts } from '../lib/index.js'
12
+
13
+ const STORE = join(homedir(), '.dsh', 'memory', 'facts.json')
14
+
15
+ /** query -> expected fact id. Phrased the way a later session actually asks. */
16
+ const CASES = [
17
+ ['which CPU is in this machine', 'env.cpu'],
18
+ ['how many cores does the processor have', 'env.cpu'],
19
+ ['how much VRAM does the GPU have', 'env.gpu'],
20
+ ['what python version is bundled', 'env.python'],
21
+ ['where is cupy installed', 'gpu.cupy_installed'],
22
+ ['what is the DSH home directory', 'env.dsh_home'],
23
+ ['which node and npm versions are installed', 'env.node_npm'],
24
+ ['monte carlo pi benchmark on the CPU', 'bench.cpu_suite'],
25
+ ['how long did the ray tracer take at 1080p', 'bench.render'],
26
+ ['GPU monte carlo speedup over CPU', 'bench.gpu_monte_carlo'],
27
+ ['how long did the batch of four pipelines run', 'batch.summary'],
28
+ ['why is wikipedia blocked by SNI', 'net.wikipedia_access'],
29
+ ['is github push failing', 'network.github-git'],
30
+ ['deck rating for physical world execution', 'deck.ratings'],
31
+ ['how many slides in the round 2 deck', 'deck.round2'],
32
+ ['which tools does the memory plugin provide', 'tool.limit_break_plugin'],
33
+ ['weekly maintenance objective for the plugin', 'task.limit-break.maintenance'],
34
+ ['what is the latest npm release', 'release.dsh-limit-break'],
35
+ ['why was memory_prune removed', 'tool.memory_prune'],
36
+ ['what renders the film with numpy', 'film.wotaiyang']
37
+ ]
38
+
39
+ /**
40
+ * Held-out set: written after the 1.9.0 ranking change and never used to pick it, so
41
+ * the two numbers answer different questions — CASES tracks regressions on tuned
42
+ * phrases, HOLDOUT says whether freshly worded questions find their fact.
43
+ */
44
+ const HOLDOUT = [
45
+ ['what GPU does this laptop have', 'env.gpu'],
46
+ ['what happens if I edit config live with HMR', 'dsh.hmr_live_config_edit'],
47
+ ['why do cookies get 431 errors in the desktop app', 'dsh.cookie_431_rootfix'],
48
+ ['how does the film use raymarching in numpy', 'film.wotaiyang'],
49
+ ['what is the route cache TTL', 'plugin.fetch_route_cache'],
50
+ ['can the browser post to github', 'env.browser.github_posting'],
51
+ ['what does the weekly maintenance schedule do', 'task.limit-break.maintenance'],
52
+ ['how many samples per second did the GPU do', 'bench.gpu_monte_carlo'],
53
+ ['what did the disk cleanup free', 'env.disk_cleanup_20261029']
54
+ ]
55
+
56
+ const raw = JSON.parse(await readFile(STORE, 'utf8'))
57
+ const facts = (raw.facts || []).filter(Boolean)
58
+
59
+ function evaluate(cases) {
60
+ let hit1 = 0
61
+ let hit3 = 0
62
+ let rr = 0
63
+ const misses = []
64
+ for (const [query, want] of cases) {
65
+ const ranked = rankFacts(facts, query).map((row) => row.fact.id)
66
+ const at = ranked.indexOf(want)
67
+ if (at === 0) hit1 += 1
68
+ if (at >= 0 && at < 3) hit3 += 1
69
+ if (at >= 0) rr += 1 / (at + 1)
70
+ if (at !== 0) misses.push({ query, want, at: at < 0 ? 'miss' : at + 1, top: ranked.slice(0, 3) })
71
+ }
72
+ return {
73
+ cases: cases.length,
74
+ hit_at_1: +(hit1 / cases.length).toFixed(3),
75
+ hit_at_3: +(hit3 / cases.length).toFixed(3),
76
+ mrr: +(rr / cases.length).toFixed(3),
77
+ misses
78
+ }
79
+ }
80
+
81
+ const report = {
82
+ store: STORE,
83
+ facts: facts.length,
84
+ tuned: evaluate(CASES),
85
+ holdout: evaluate(HOLDOUT)
86
+ }
87
+
88
+ if (process.argv.includes('--json')) {
89
+ console.log(JSON.stringify(report, null, 2))
90
+ } else {
91
+ console.log(`facts=${report.facts}`)
92
+ for (const part of ['tuned', 'holdout']) {
93
+ const r = report[part]
94
+ console.log(`${part}: cases=${r.cases} hit@1=${r.hit_at_1} hit@3=${r.hit_at_3} mrr=${r.mrr}`)
95
+ for (const miss of r.misses) {
96
+ console.log(` miss: "${miss.query}" wanted ${miss.want} got rank ${miss.at} -> ${miss.top.join(', ')}`)
97
+ }
98
+ }
99
+ }