dsh-limit-break 1.8.1 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/MAINTAIN.md +8 -2
- package/README.md +16 -6
- package/lib/index.js +5 -1
- package/package.json +1 -1
- package/tools/recall-bench.mjs +99 -72
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,25 @@
|
|
|
3
3
|
All notable changes to `dsh-limit-break`. Format follows
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), versions follow semver.
|
|
5
5
|
|
|
6
|
+
## 1.9.0 — 2026-11-19
|
|
7
|
+
|
|
8
|
+
### Fixed
|
|
9
|
+
- **A long id no longer outranks a short one for the same word.** Every id segment used to score the
|
|
10
|
+
full id weight, so a qualifier-heavy id beat the fact that names the thing: with the store at 38
|
|
11
|
+
facts, `env.dsh_renderer_spinner_cpu` won "which CPU is in this machine" over `env.cpu`, and
|
|
12
|
+
`bench.gpu_monte_carlo` won "monte carlo pi benchmark on the CPU" over `bench.cpu_suite`. An id hit
|
|
13
|
+
is now scaled by `1/sqrt(id segments)`. Full normalisation (1/length) was tried and rejected: it
|
|
14
|
+
over-punished legitimate three-segment ids and pushed "what is the DSH home directory" down to rank
|
|
15
|
+
5. Measured with `node tools/recall-bench.mjs`: **hit@1 0.85 → 0.95, MRR 0.900 → 0.950** from the
|
|
16
|
+
scoring change alone.
|
|
17
|
+
- **`tool.memory_prune` is its own fact again.** The "why was memory_prune withdrawn" history was
|
|
18
|
+
buried inside the long `tool.limit_break_plugin` fact, which no query about the removal could rank
|
|
19
|
+
first; splitting it out (and pointing the tool fact at it) took the benchmark from 0.95 to **hit@1
|
|
20
|
+
1.00 / hit@3 1.00 / MRR 1.000** on the 20 tuned questions.
|
|
21
|
+
- **The benchmark now reports a held-out set.** Nine questions written after the change, never used
|
|
22
|
+
to pick it, run alongside the tuned twenty so a regression shows up and blind tuning does not look
|
|
23
|
+
like progress: **9/9 both before and after** this round.
|
|
24
|
+
|
|
6
25
|
## Unreleased
|
|
7
26
|
|
|
8
27
|
## 1.8.1 — 2026-11-12
|
package/MAINTAIN.md
CHANGED
|
@@ -113,8 +113,14 @@ git add -A && git commit -m "..." && git push
|
|
|
113
113
|
- [x] **1.5.0 已上线**(2026-10-22T12:20:22Z):npm `latest` = `1.5.0`(用户 OTP 发布),GitHub
|
|
114
114
|
`origin/main` = `cd5f409`(github.com:443 被掐,改用 api.github.com + GraphQL 推送),
|
|
115
115
|
desktop profile 已从 1.2.1 升到 **1.5.0**,`verify` PASS。四工具 ABI 未变,老会话不会 brick。
|
|
116
|
-
- [
|
|
117
|
-
`bench.gpu_monte_carlo`
|
|
116
|
+
- [x] **1.9.0 修掉了基准里剩下的两个 miss**:`env.dsh_renderer_spinner_cpu` 曾压过 `env.cpu`、
|
|
117
|
+
`bench.gpu_monte_carlo` 曾压过 `bench.cpu_suite`,根因是 id 命中不分长短;现在按
|
|
118
|
+
`1/sqrt(id segments)` 归一。第三个 miss(`why was memory_prune removed`)根本不是排序问题,
|
|
119
|
+
而是那条知识被埋进了 `tool.limit_break_plugin` 的长句里——已拆成独立事实 `tool.memory_prune`。
|
|
120
|
+
38 条 store 上:hit@1 0.85 → 1.00、MRR 0.900 → 1.000(20 题)。
|
|
121
|
+
- [ ] 基准只有**词面重叠**这一个信号:今天新写的 9 题 held-out 在改动前后都是 9/9,说明没退化,
|
|
122
|
+
但不证明泛化。下一步值得做的是攒一份“换个说法问同一件事”的 paraphrase 集(不回用原词),
|
|
123
|
+
先量出真实下限再决定要不要引入近义词表/向量检索。别为了盯 bench 数字调参。
|
|
118
124
|
- [x] desktop profile 版本漂移已消(1.2.1 → 1.5.0)。以后每次发布都跟一条
|
|
119
125
|
`dsh.cmd plugin --profile desktop add dsh-limit-break@<version>`。
|
|
120
126
|
|
package/README.md
CHANGED
|
@@ -14,12 +14,20 @@ Zero npm dependencies. No import-time side effects. Everything fails soft — a
|
|
|
14
14
|
network returns an error object, never an exception that could take the host down.
|
|
15
15
|
|
|
16
16
|
**Recall quality.** `memory_recall` is measured, not vibed: `tools/recall-bench.mjs` runs 20
|
|
17
|
-
hand-labelled questions against the live store and prints hit@1 / hit@3 / MRR
|
|
18
|
-
|
|
17
|
+
hand-labelled questions against the live store and prints hit@1 / hit@3 / MRR, next to a nine-question
|
|
18
|
+
held-out set that was written *after* the ranking change and never used to pick it. At 1.5.0 the
|
|
19
|
+
ranking skips English stopwords, stems, matches whole words (with a prefix fallback, so `ray` finds
|
|
19
20
|
`raytracer`), splits dotted ids into segments, weights each query token by inverse document
|
|
20
|
-
frequency, and rewards a whole-phrase hit
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
frequency, and rewards a whole-phrase hit. At 1.9.0 the store had grown from 20 to 38 facts and
|
|
22
|
+
hit@1 had slipped to 0.85: a *long* id was winning ties, because every id segment scored the same —
|
|
23
|
+
`env.dsh_renderer_spinner_cpu` beat `env.cpu` for "which CPU is in this machine" and
|
|
24
|
+
`bench.gpu_monte_carlo` beat `bench.cpu_suite` for "monte carlo pi benchmark on the CPU". Id hits are
|
|
25
|
+
now scaled by `1/sqrt(id segments)`, so `env.cpu` (2 segments) counts for more than a four-segment id
|
|
26
|
+
that merely contains `cpu`; square-root rather than full normalisation, because dividing by the id
|
|
27
|
+
length over-punished legitimate three-segment ids such as `bench.cpu_suite`. **hit@1 0.85 → 1.00,
|
|
28
|
+
hit@3 0.95 → 1.00, MRR 0.900 → 1.000** on the 20 tuned questions; the held-out nine scored 9/9 both
|
|
29
|
+
before and after, so that is evidence of no regression, not proof of generalisation. Treat the numbers
|
|
30
|
+
as this store's, not as a promise about yours.
|
|
23
31
|
|
|
24
32
|
**Runtime requirement: `curl`.** Rung 1 of the fetch ladder is plain `fetch` and needs nothing; rungs
|
|
25
33
|
2–4 (IPv6 → IPv4 → sibling hostname) shell out to `curl`, so it has to be on `PATH`. Windows 10+,
|
|
@@ -124,7 +132,9 @@ Measured, not guessed — each line is something a session actually hit.
|
|
|
124
132
|
* **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement/details), no
|
|
125
133
|
embeddings: stopword-filtered, crudely stemmed word matching weighted by idf, so it is good at
|
|
126
134
|
"which fact did I mean" and bad at paraphrase. `tools/recall-bench.mjs` states the current
|
|
127
|
-
numbers (hit@1
|
|
135
|
+
numbers (hit@1 1.00 on the 20 tuned questions of a 38-fact store, 9/9 on a held-out set); a query
|
|
136
|
+
that shares no word with a fact still misses it, and nothing here handles paraphrases the words do
|
|
137
|
+
not cover.
|
|
128
138
|
* **`resilient_fetch` reads at most 2 MiB per response** and returns at most 20 000 characters;
|
|
129
139
|
bigger payloads are truncated, not streamed.
|
|
130
140
|
|
package/lib/index.js
CHANGED
|
@@ -423,6 +423,10 @@ function scoreFact(fact, tokens, idf) {
|
|
|
423
423
|
if (!tokens.length) return 0
|
|
424
424
|
// Ids are dotted paths ("bench.gpu_monte_carlo"); split them so a token can match a segment.
|
|
425
425
|
const id = new Set(String(fact.id || '').toLowerCase().split(/[^\p{L}\p{N}]+/u).filter(Boolean).map(stem))
|
|
426
|
+
// A short id names one thing ("env.cpu"); a long id carries qualifiers ("env.dsh_renderer_spinner_cpu",
|
|
427
|
+
// "bench.gpu_monte_carlo"), so an id hit there is weaker evidence. Normalise by sqrt(id length) —
|
|
428
|
+
// full normalisation over-punishes legitimate three-segment ids such as "bench.cpu_suite".
|
|
429
|
+
const idNorm = 1 / Math.sqrt(Math.max(1, id.size))
|
|
426
430
|
const topic = wordSet(fact.topic)
|
|
427
431
|
const tags = wordSet((Array.isArray(fact.tags) ? fact.tags : []).join(' '))
|
|
428
432
|
const statement = wordSet(fact.statement)
|
|
@@ -431,7 +435,7 @@ function scoreFact(fact, tokens, idf) {
|
|
|
431
435
|
for (const token of tokens) {
|
|
432
436
|
const boost = idf instanceof Map ? idf.get(token) || 1 : 1
|
|
433
437
|
score += boost * (
|
|
434
|
-
FIELD_WEIGHTS.id * wordMatch(id, token) +
|
|
438
|
+
FIELD_WEIGHTS.id * idNorm * wordMatch(id, token) +
|
|
435
439
|
FIELD_WEIGHTS.topic * wordMatch(topic, token) +
|
|
436
440
|
FIELD_WEIGHTS.tags * wordMatch(tags, token) +
|
|
437
441
|
FIELD_WEIGHTS.statement * wordMatch(statement, token) +
|
package/package.json
CHANGED
package/tools/recall-bench.mjs
CHANGED
|
@@ -1,72 +1,99 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Recall benchmark: how often does memory_recall's ranking put the intended fact first?
|
|
3
|
-
*
|
|
4
|
-
* It reads the live store (read-only) and runs a fixed query set whose expected ids were
|
|
5
|
-
* written by hand from the fact statements. Hit@1 is the number that matters: an agent asks
|
|
6
|
-
* one question and reads the top answer. Run: node tools/recall-bench.mjs [--json]
|
|
7
|
-
*/
|
|
8
|
-
import { readFile } from 'node:fs/promises'
|
|
9
|
-
import { join } from 'node:path'
|
|
10
|
-
import { homedir } from 'node:os'
|
|
11
|
-
import { rankFacts } from '../lib/index.js'
|
|
12
|
-
|
|
13
|
-
const STORE = join(homedir(), '.dsh', 'memory', 'facts.json')
|
|
14
|
-
|
|
15
|
-
/** query -> expected fact id. Phrased the way a later session actually asks. */
|
|
16
|
-
const CASES = [
|
|
17
|
-
['which CPU is in this machine', 'env.cpu'],
|
|
18
|
-
['how many cores does the processor have', 'env.cpu'],
|
|
19
|
-
['how much VRAM does the GPU have', 'env.gpu'],
|
|
20
|
-
['what python version is bundled', 'env.python'],
|
|
21
|
-
['where is cupy installed', 'gpu.cupy_installed'],
|
|
22
|
-
['what is the DSH home directory', 'env.dsh_home'],
|
|
23
|
-
['which node and npm versions are installed', 'env.node_npm'],
|
|
24
|
-
['monte carlo pi benchmark on the CPU', 'bench.cpu_suite'],
|
|
25
|
-
['how long did the ray tracer take at 1080p', 'bench.render'],
|
|
26
|
-
['GPU monte carlo speedup over CPU', 'bench.gpu_monte_carlo'],
|
|
27
|
-
['how long did the batch of four pipelines run', 'batch.summary'],
|
|
28
|
-
['why is wikipedia blocked by SNI', 'net.wikipedia_access'],
|
|
29
|
-
['is github push failing', 'network.github-git'],
|
|
30
|
-
['deck rating for physical world execution', 'deck.ratings'],
|
|
31
|
-
['how many slides in the round 2 deck', 'deck.round2'],
|
|
32
|
-
['which tools does the memory plugin provide', 'tool.limit_break_plugin'],
|
|
33
|
-
['weekly maintenance objective for the plugin', 'task.limit-break.maintenance'],
|
|
34
|
-
['what is the latest npm release', 'release.dsh-limit-break'],
|
|
35
|
-
['why was memory_prune removed', 'tool.memory_prune'],
|
|
36
|
-
['what renders the film with numpy', 'film.wotaiyang']
|
|
37
|
-
]
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
}
|
|
72
|
-
|
|
1
|
+
/**
|
|
2
|
+
* Recall benchmark: how often does memory_recall's ranking put the intended fact first?
|
|
3
|
+
*
|
|
4
|
+
* It reads the live store (read-only) and runs a fixed query set whose expected ids were
|
|
5
|
+
* written by hand from the fact statements. Hit@1 is the number that matters: an agent asks
|
|
6
|
+
* one question and reads the top answer. Run: node tools/recall-bench.mjs [--json]
|
|
7
|
+
*/
|
|
8
|
+
import { readFile } from 'node:fs/promises'
|
|
9
|
+
import { join } from 'node:path'
|
|
10
|
+
import { homedir } from 'node:os'
|
|
11
|
+
import { rankFacts } from '../lib/index.js'
|
|
12
|
+
|
|
13
|
+
const STORE = join(homedir(), '.dsh', 'memory', 'facts.json')
|
|
14
|
+
|
|
15
|
+
/** query -> expected fact id. Phrased the way a later session actually asks. */
|
|
16
|
+
const CASES = [
|
|
17
|
+
['which CPU is in this machine', 'env.cpu'],
|
|
18
|
+
['how many cores does the processor have', 'env.cpu'],
|
|
19
|
+
['how much VRAM does the GPU have', 'env.gpu'],
|
|
20
|
+
['what python version is bundled', 'env.python'],
|
|
21
|
+
['where is cupy installed', 'gpu.cupy_installed'],
|
|
22
|
+
['what is the DSH home directory', 'env.dsh_home'],
|
|
23
|
+
['which node and npm versions are installed', 'env.node_npm'],
|
|
24
|
+
['monte carlo pi benchmark on the CPU', 'bench.cpu_suite'],
|
|
25
|
+
['how long did the ray tracer take at 1080p', 'bench.render'],
|
|
26
|
+
['GPU monte carlo speedup over CPU', 'bench.gpu_monte_carlo'],
|
|
27
|
+
['how long did the batch of four pipelines run', 'batch.summary'],
|
|
28
|
+
['why is wikipedia blocked by SNI', 'net.wikipedia_access'],
|
|
29
|
+
['is github push failing', 'network.github-git'],
|
|
30
|
+
['deck rating for physical world execution', 'deck.ratings'],
|
|
31
|
+
['how many slides in the round 2 deck', 'deck.round2'],
|
|
32
|
+
['which tools does the memory plugin provide', 'tool.limit_break_plugin'],
|
|
33
|
+
['weekly maintenance objective for the plugin', 'task.limit-break.maintenance'],
|
|
34
|
+
['what is the latest npm release', 'release.dsh-limit-break'],
|
|
35
|
+
['why was memory_prune removed', 'tool.memory_prune'],
|
|
36
|
+
['what renders the film with numpy', 'film.wotaiyang']
|
|
37
|
+
]
|
|
38
|
+
|
|
39
|
+
/**
|
|
40
|
+
* Held-out set: written after the 1.9.0 ranking change and never used to pick it, so
|
|
41
|
+
* the two numbers answer different questions — CASES tracks regressions on tuned
|
|
42
|
+
* phrases, HOLDOUT says whether freshly worded questions find their fact.
|
|
43
|
+
*/
|
|
44
|
+
const HOLDOUT = [
|
|
45
|
+
['what GPU does this laptop have', 'env.gpu'],
|
|
46
|
+
['what happens if I edit config live with HMR', 'dsh.hmr_live_config_edit'],
|
|
47
|
+
['why do cookies get 431 errors in the desktop app', 'dsh.cookie_431_rootfix'],
|
|
48
|
+
['how does the film use raymarching in numpy', 'film.wotaiyang'],
|
|
49
|
+
['what is the route cache TTL', 'plugin.fetch_route_cache'],
|
|
50
|
+
['can the browser post to github', 'env.browser.github_posting'],
|
|
51
|
+
['what does the weekly maintenance schedule do', 'task.limit-break.maintenance'],
|
|
52
|
+
['how many samples per second did the GPU do', 'bench.gpu_monte_carlo'],
|
|
53
|
+
['what did the disk cleanup free', 'env.disk_cleanup_20261029']
|
|
54
|
+
]
|
|
55
|
+
|
|
56
|
+
const raw = JSON.parse(await readFile(STORE, 'utf8'))
|
|
57
|
+
const facts = (raw.facts || []).filter(Boolean)
|
|
58
|
+
|
|
59
|
+
function evaluate(cases) {
|
|
60
|
+
let hit1 = 0
|
|
61
|
+
let hit3 = 0
|
|
62
|
+
let rr = 0
|
|
63
|
+
const misses = []
|
|
64
|
+
for (const [query, want] of cases) {
|
|
65
|
+
const ranked = rankFacts(facts, query).map((row) => row.fact.id)
|
|
66
|
+
const at = ranked.indexOf(want)
|
|
67
|
+
if (at === 0) hit1 += 1
|
|
68
|
+
if (at >= 0 && at < 3) hit3 += 1
|
|
69
|
+
if (at >= 0) rr += 1 / (at + 1)
|
|
70
|
+
if (at !== 0) misses.push({ query, want, at: at < 0 ? 'miss' : at + 1, top: ranked.slice(0, 3) })
|
|
71
|
+
}
|
|
72
|
+
return {
|
|
73
|
+
cases: cases.length,
|
|
74
|
+
hit_at_1: +(hit1 / cases.length).toFixed(3),
|
|
75
|
+
hit_at_3: +(hit3 / cases.length).toFixed(3),
|
|
76
|
+
mrr: +(rr / cases.length).toFixed(3),
|
|
77
|
+
misses
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
const report = {
|
|
82
|
+
store: STORE,
|
|
83
|
+
facts: facts.length,
|
|
84
|
+
tuned: evaluate(CASES),
|
|
85
|
+
holdout: evaluate(HOLDOUT)
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
if (process.argv.includes('--json')) {
|
|
89
|
+
console.log(JSON.stringify(report, null, 2))
|
|
90
|
+
} else {
|
|
91
|
+
console.log(`facts=${report.facts}`)
|
|
92
|
+
for (const part of ['tuned', 'holdout']) {
|
|
93
|
+
const r = report[part]
|
|
94
|
+
console.log(`${part}: cases=${r.cases} hit@1=${r.hit_at_1} hit@3=${r.hit_at_3} mrr=${r.mrr}`)
|
|
95
|
+
for (const miss of r.misses) {
|
|
96
|
+
console.log(` miss: "${miss.query}" wanted ${miss.want} got rank ${miss.at} -> ${miss.top.join(', ')}`)
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
}
|