dsh-limit-break 1.5.0 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,50 @@ All notable changes to `dsh-limit-break`. Format follows
5
5
 
6
6
  ## Unreleased
7
7
 
8
+ ## 1.7.0 — 2026-11-05
9
+
10
+ ### Changed
11
+ - **`resilient_fetch` remembers which rung worked instead of rediscovering it on every call.** A
12
+ non-direct success is written back as `network.route.<host>` (14-day TTL, written under the same
13
+ lock as `memory_save`) and the next call to that host starts at that rung; if a plain fetch starts
14
+ working again the route is deleted, so an unblocked host heals itself. Measured on
15
+ `en.wikipedia.org/wiki/Monte_Carlo_method` with the default 12 s per-rung timeout: cold **12.11 s**
16
+ with attempts `direct>alternate-host`, warm **1.08 s / 1.14 s** with a single `alternate-host`
17
+ attempt — **11.2× faster**; the tool now also reports `millis`, `route`
18
+ (`learned|reused|dropped|none`) and `cached_via`.
19
+ - **`tools/fetch-route-bench.mjs`** (new): cold-vs-warm measurement for one blocked host plus a
20
+ self-heal check (a seeded stale route must be dropped once direct works). 6/6 checks PASS.
21
+ - Route facts are ordinary facts: visible to `memory_recall`/`memory_stats`, removable with any
22
+ store edit. Only fallback successes and self-heal events write, so a working network writes nothing.
23
+
24
+ ### Notes
25
+ - No tool added, removed or renamed; store format unchanged. The tool description text now mentions
26
+ the remembered route, which an already-open session will not see until its tool snapshot refreshes.
27
+ - `maintain.py status` still prints `market listing not live yet (PR #8296 closed)` while the same
28
+ call reports `listed_in_market: true`: the check treats a merged-and-closed PR as not live. Left
29
+ as-is, documented in MAINTAIN.md so the next round does not read it as a regression.
30
+
31
+ ## 1.6.0 — 2026-10-29
32
+
33
+ ### Fixed
34
+ - **`memory_save` no longer loses writes when two sessions save at the same time.** Saving was
35
+ load → mutate → write-temp → rename with nothing serialising it, so concurrent writers each
36
+ wrote a store missing the other's fact, and on Windows the losing `rename` often failed outright
37
+ with `EPERM`. Saves now take a lock file created with `O_EXCL` around the whole read-modify-write,
38
+ released in a `finally`; a lock left behind by a dead process is stolen after 15 s, and a writer
39
+ that cannot get in within 8 s returns `{ ok: false, code: 'memory.locked' }` instead of throwing.
40
+ A failed `stat` is never read as "stale" — that is exactly how two writers end up inside at once.
41
+ - **`tools/concurrency-selftest.mjs`** (new): spawns N worker processes that each save M facts,
42
+ reads the store back, and counts what survived; it also checks stale-lock recovery.
43
+ 8 workers × 6 saves × 3 rounds on the same machine: **16–17 of 48 saves lost** and **up to 25 of
44
+ 48 saves failed with `EPERM`** before; **0 lost, 0 failed** after (also clean at 12 × 5 × 4 = 240
45
+ saves, and a 60 s-old lock is recovered with 6/6 saves kept).
46
+
47
+ ### Notes
48
+ - No tool added, removed or renamed: the four-tool ABI is untouched. Store format unchanged, and
49
+ single-writer behaviour is identical (the lock is uncontended: one `open` + one `rm` per save).
50
+ - Read paths (`memory_recall`, `memory_stats`) do not take the lock.
51
+
8
52
  ## 1.5.0 — 2026-10-22
9
53
 
10
54
  ### Changed
package/MAINTAIN.md CHANGED
@@ -110,22 +110,27 @@ git add -A && git commit -m "..." && git push
110
110
 
111
111
  ## Backlog(按 2026-10-22 20:10 的真实状态)
112
112
 
113
- - [x] **1.5.0 召回排序改进**(2026-10-22):`memory_recall` 由“子串计数”改为 stopword 过滤 +
114
- 轻量词干 + 整词/前缀匹配 + dotted id 分段 + idf 加权 + 短语加分;新增
115
- `tools/recall-bench.mjs`(20 题),实测 hit@1 **0.75 → 0.95**、MRR **0.867 → 0.975**、hit@3 1.0。
116
- 四工具 ABI 未变,1.4.0 的“工具集冻结”规则仍然成立。
113
+ - [x] **1.5.0 已上线**(2026-10-22T12:20:22Z):npm `latest` = `1.5.0`(用户 OTP 发布),GitHub
114
+ `origin/main` = `cd5f409`(github.com:443 被掐,改用 api.github.com + GraphQL 推送),
115
+ desktop profile 已从 1.2.1 升到 **1.5.0**,`verify` PASS。四工具 ABI 未变,老会话不会 brick。
117
116
  - [ ] 基准里剩下的 miss(“monte carlo pi benchmark on the CPU”)是 `bench.cpu_suite` 与
118
117
  `bench.gpu_monte_carlo` 的真实歧义;再往上做要引入上下文/近义词表,性价比低,当作已知上限。
119
- - [ ] desktop profile 可能钉在旧版本——每次发布后用
120
- `dsh.cmd plugin --profile desktop add dsh-limit-break@<version>` 消漂移。
118
+ - [x] desktop profile 版本漂移已消(1.2.1 → 1.5.0)。以后每次发布都跟一条
119
+ `dsh.cmd plugin --profile desktop add dsh-limit-break@<version>`。
121
120
 
122
121
  - [x] **1.2.0 已上线**(2026-10-20T12:02:52Z):npm `latest` = `1.2.0`,GitHub `origin/main` =
123
122
  `fd844a4`,`verify` PASS,`test_plugin.mjs` 6/6 PASS。用户 20:04 完成了 OTP 发布和推送。
124
123
  `env.node_npm` 与 `release.dsh-limit-break` 两条事实已回写。
125
124
  - [x] 旧定时任务 `schedule-803cdbdc`(会话 5133bee1)已由用户在面板删除;`storages/schedule.json`
126
125
  里只剩 `schedule-0b6c60ea`(hello world)和新的 `schedule-b3675d28`。无需再处理。
127
- - [ ] 市场条目 PR `awesome-dsh-plugin/awesome-dsh-plugin#8296` 仍是 open;合并前 `status` 会一直报
128
- `market listing not live yet`。找合并节奏,不要重提。
126
+ - [x] 市场条目**已合并上线**(2026-10-22 之后):`awesome-dsh-plugin/awesome-dsh-plugin#8296`
127
+ merged,条目文件 `data/plugins/Ma354__dsh-limit-break.yml`(category: memory);
128
+ 2026-10-27 01:06 的 `#9282`(加 storefront icon,`assets/icon-180.png`)也已 merged。
129
+ `maintain.py status` 现在 `listed_in_market: true`、`problems: []`。不要再提新 PR。
130
+ - [ ] 本地仓库落后远端:本地 `94eafb7`,远端 `b84161a`(用户 10-25 推了 icon 等);
131
+ `github.com:443` 被掐,`git fetch` 连试 3 次全失败。等网络恢复后跑
132
+ `git fetch origin && git reset --hard origin/main`。在此之前文档改动走
133
+ `..\_maintain\ghpush.mjs`(GraphQL,按远端 head 建提交,内容全量替换,不受分叉影响)。
129
134
  - [ ] npm 上还挂着占位包 `0.0.0-stage`(`stub: true`)。`npm deprecate` 也要 OTP(本轮 EOTP 失败);
130
135
  用户一条命令即可:`npm deprecate dsh-limit-break@0.0.0-stage "placeholder - use 1.2.0"`。
131
136
  - [ ] `1.0.0` 是无元数据的旧包体(author/repository 都不在)。同样一条 `npm deprecate` 可清掉噪音(需 OTP)。
@@ -133,6 +138,15 @@ git add -A && git commit -m "..." && git push
133
138
  bench.gpu_monte_carlo and bench.render”(2026-10-21 的回写),陈旧描述不再存在。
134
139
  - [x] 插件工具对**已经打开的会话**不可见:已在 README「Known limits」第 1 条写明
135
140
  (tool list 是 per-session 快照,插件侧修不了)。事实 `harness.plugin-tool-visibility` 有实测。
136
- - [ ] `resilient_fetch` 目前按"主机名别名"降级,可以继续加:`--resolve` 到已知 IP、
137
- IPv6 优先、以及把成功路径写回记忆库(现在是只读阶梯)。
138
- - [ ] 没有任何自动化测试覆盖 `memory_save` 的并发写。单进程够用,多进程未验证。
141
+ - [x] **`resilient_fetch` 会把成功路径写回记忆库**(1.7.0,2026-11-05):非 direct 成功写成
142
+ `network.route.<host>`(14 天 TTL,走 `memory_save` 同一把锁),下次同主机直接从该级起步;
143
+ direct 恢复后删掉路由(自愈)。实测 `en.wikipedia.org/wiki/Monte_Carlo_method`、12 s 超时:
144
+ cold 12.11 s(`direct>alternate-host`)→ warm 1.08 s / 1.14 s(单次 `alternate-host`),11.2×。
145
+ 跑 `node tools/fetch-route-bench.mjs <url> <timeoutMs>` 复测(约 15 s,6 项检查)。
146
+ 仍可继续加:`--resolve` 到已知 IP、IPv6 优先(backlog)。
147
+ - [ ] `maintain.py status` 的 `problems` 里 `market listing not live yet (PR #8296 closed)` 是误报:
148
+ 同一份输出里 `listed_in_market: true`。原因未查,先记为已知噪音,别当成回归。
149
+ - [x] **`memory_save` 的并发写已有自动化覆盖**(1.6.0,2026-10-29):`tools/concurrency-selftest.mjs`
150
+ 起 N 个独立进程同时写同一个 store,再读回数存活条数,另测“陈旧锁被接管”。同机 8 进程 × 6 次 × 3 轮:
151
+ 修复前丢 16–17/48、且最多 25/48 直接 `EPERM`(rename 输家);加 `O_EXCL` 锁后 0 丢 0 失败,
152
+ 12 × 5 × 4 = 240 次同样全存活。改了 `memory_save` 的写入路径就跑一遍它(约 20 s)。
package/README.md CHANGED
@@ -43,6 +43,12 @@ IPv6 → IPv4 → a **sibling hostname of the same service** whose certificate s
43
43
  It does **not** disable certificate verification and does not route through a proxy that would
44
44
  impersonate the origin. Those are MITM, not workarounds.
45
45
 
46
+ **The ladder learns.** A fallback rung that worked is written back as `network.route.<host>` and the
47
+ next call to that host starts there; if a plain fetch starts working again the route is deleted (host
48
+ self-heals). Measured on `en.wikipedia.org/wiki/Monte_Carlo_method`, 12 s per-rung timeout: cold
49
+ **12.1 s** (rung 1 burned the timeout, rung 4 answered) → warm **1.08 s / 1.14 s**, one attempt
50
+ instead of two — **11.2× faster**. Numbers come from `tools/fetch-route-bench.mjs`.
51
+
46
52
  ## Install
47
53
 
48
54
  ```sh
@@ -75,7 +81,10 @@ package the profile does not declare is silently ignored.
75
81
  ```
76
82
 
77
83
  Plain JSON on purpose: anything that can read a file can use it, with or without this plugin.
78
- Writes are tmp-file + rename.
84
+ Writes are tmp-file + rename, and a save takes a `.facts.lock` (created `O_EXCL`, stolen after
85
+ 15 s if its writer died) around the whole read-modify-write, so two sessions saving at the same
86
+ moment both keep their facts. `tools/concurrency-selftest.mjs` states the numbers: 16–17 of 48
87
+ saves lost before the lock, 0 after.
79
88
 
80
89
  ## Disable / remove
81
90
 
@@ -90,11 +99,15 @@ Measured, not guessed — each line is something a session actually hit.
90
99
  * **Tools are visible only to sessions opened after the mount.** A session created before the
91
100
  plugin mounted keeps its tool snapshot; new tools do not appear until a new session. Not fixable
92
101
  from the plugin side (the tool list is materialised per session).
93
- * **One writer per store.** Writes are tmp-file + rename, which is crash-safe, but two processes
94
- writing `facts.json` concurrently still lose one update — there is no lock. One agent per
95
- machine is the intended use.
102
+ * **Concurrent saves serialise, and wait at most 8 s.** A second session saving at the same moment
103
+ waits for the lock instead of losing its fact; if the holder is wedged, the save returns
104
+ `{ ok: false, code: "memory.locked" }` rather than hanging. Read paths take no lock, so a save
105
+ still cannot stop a reader from seeing the previous snapshot.
96
106
  * **The fetch ladder is hostname-level.** It does not defeat IP-level blocks, TLS interception,
97
107
  or a proxy that terminates the connection; it does not skip certificate verification.
108
+ * **A remembered route is a bet with a 14-day expiry.** The ladder trusts `network.route.<host>` for
109
+ two weeks and drops it as soon as a direct request works; if the winning rung starts failing the
110
+ call still walks the rest of the ladder, it just pays that one rung first.
98
111
  * **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement/details), no
99
112
  embeddings: stopword-filtered, crudely stemmed word matching weighted by idf, so it is good at
100
113
  "which fact did I mean" and bad at paraphrase. `tools/recall-bench.mjs` states the current
package/lib/index.js CHANGED
@@ -18,7 +18,7 @@
18
18
 
19
19
  import { execFile } from 'node:child_process'
20
20
  import { appendFileSync, mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'
21
- import { mkdir, readFile, rename, stat, writeFile } from 'node:fs/promises'
21
+ import { mkdir, open, readFile, rename, rm, stat, writeFile } from 'node:fs/promises'
22
22
  import { homedir } from 'node:os'
23
23
  import { dirname, isAbsolute, join, resolve } from 'node:path'
24
24
  import { fileURLToPath } from 'node:url'
@@ -182,6 +182,115 @@ async function saveStore(dir, store) {
182
182
  return { file, bytes: Buffer.byteLength(body) }
183
183
  }
184
184
 
185
+ // memory_save is load -> mutate -> save, and every DSH window is its own process, so two sessions
186
+ // that save at the same moment both write a store that is missing the other's fact (and on Windows
187
+ // the loser's rename can fail outright with EPERM). A lock file created with O_EXCL is enough:
188
+ // writes are tiny and rare, and a crashed writer is recovered by stealing a lock older than
189
+ // LOCK_STALE_MS. A failed stat is treated as "still held", never as "stale" — stealing on a
190
+ // transient EPERM is exactly how two writers end up in the critical section at once.
191
+ const LOCK_STALE_MS = 15000
192
+ const LOCK_WAIT_MS = 8000
193
+
194
+ async function withStoreLock(dir, fn) {
195
+ const lock = join(dir, '.facts.lock')
196
+ await mkdir(dir, { recursive: true })
197
+ const started = Date.now()
198
+ for (;;) {
199
+ let handle = null
200
+ try {
201
+ handle = await open(lock, 'wx')
202
+ await handle.writeFile(`${process.pid} ${new Date().toISOString()}\n`)
203
+ await handle.close()
204
+ break
205
+ } catch (error) {
206
+ if (handle) await handle.close().catch(() => {})
207
+ // On Windows a create that races a delete comes back as EPERM, not EEXIST, so both mean
208
+ // "somebody else is inside the critical section, or just left it".
209
+ if (!error || (error.code !== 'EEXIST' && error.code !== 'EPERM')) throw error
210
+ const info = await stat(lock).catch((problem) => problem)
211
+ if (!info || info.code) {
212
+ // The lock vanished or is pending deletion: that is a released lock, retry promptly.
213
+ await new Promise((resolve) => setTimeout(resolve, 5))
214
+ continue
215
+ }
216
+ const age = typeof info.mtimeMs === 'number' ? Date.now() - info.mtimeMs : 0
217
+ if (age > LOCK_STALE_MS) {
218
+ await rm(lock, { force: true }).catch(() => {})
219
+ continue
220
+ }
221
+ if (Date.now() - started > LOCK_WAIT_MS) {
222
+ const err = new Error(`memory store is locked by another process (${lock}); gave up after ${LOCK_WAIT_MS} ms`)
223
+ err.code = 'memory.locked'
224
+ throw err
225
+ }
226
+ await new Promise((resolve) => setTimeout(resolve, 10 + Math.random() * 30))
227
+ }
228
+ }
229
+ try {
230
+ return await fn()
231
+ } finally {
232
+ await rm(lock, { force: true }).catch(() => {})
233
+ }
234
+ }
235
+
236
+ // A failed fetch tells you which rung of the ladder works for that host *today*; throwing that
237
+ // away means paying the same dead rungs on every call. A non-direct success is remembered per
238
+ // host (network.route.<host>) for two weeks and the ladder starts at that rung next time. When
239
+ // direct starts working again the route is dropped, so a host that unblocks heals itself.
240
+ const ROUTE_TTL_MS = 14 * 24 * 60 * 60 * 1000
241
+
242
+ function routeId(host) {
243
+ return `network.route.${String(host).toLowerCase().replace(/[^a-z0-9.-]+/g, '-')}`
244
+ }
245
+
246
+ async function readRoute(dir, host) {
247
+ const store = await loadStore(dir)
248
+ const fact = store.facts.find((f) => f && String(f.id) === routeId(host))
249
+ if (!fact) return null
250
+ const details = fact.details && typeof fact.details === 'object' ? fact.details : {}
251
+ const observed = Date.parse(String(details.observedAt || fact.timestamp || ''))
252
+ if (!Number.isFinite(observed) || Date.now() - observed > ROUTE_TTL_MS) return null
253
+ const via = typeof details.via === 'string' ? details.via : ''
254
+ if (!via || via === 'direct') return null
255
+ return { via, url: typeof details.url === 'string' ? details.url : '', observedAt: new Date(observed).toISOString() }
256
+ }
257
+
258
+ async function writeRoute(dir, host, info) {
259
+ const now = new Date().toISOString()
260
+ const id = routeId(host)
261
+ return withStoreLock(dir, async () => {
262
+ const store = await loadStore(dir)
263
+ const record = {
264
+ id,
265
+ topic: 'network',
266
+ statement: `${host} answers through the ${info.via} rung of the fetch ladder (${info.url}); a direct request to it fails today, so the ladder should start at ${info.via}.`,
267
+ value: null,
268
+ unit: null,
269
+ details: { host, via: info.via, url: info.url, observedAt: now, millis: info.millis || null },
270
+ source: 'resilient_fetch',
271
+ timestamp: now,
272
+ tags: ['network', 'fetch-route']
273
+ }
274
+ const at = store.facts.findIndex((f) => f && String(f.id) === id)
275
+ if (at >= 0) store.facts[at] = record
276
+ else store.facts.push(record)
277
+ await saveStore(dir, store)
278
+ return record
279
+ })
280
+ }
281
+
282
+ async function dropRoute(dir, host) {
283
+ const id = routeId(host)
284
+ return withStoreLock(dir, async () => {
285
+ const store = await loadStore(dir)
286
+ const before = store.facts.length
287
+ store.facts = store.facts.filter((f) => !(f && String(f.id) === id))
288
+ if (store.facts.length === before) return false
289
+ await saveStore(dir, store)
290
+ return true
291
+ })
292
+ }
293
+
185
294
  // Ranking inputs. The old scorer used raw substring counts, which let a common word
186
295
  // ("the", "how", "long") outvote the one rare word that identifies the fact. Words are
187
296
  // matched against word sets instead, suffixed words share a stem, and each query token is
@@ -565,25 +674,34 @@ export function apply(ctx, config) {
565
674
  if (id.length > MAX_ID_CHARS || statement.length > MAX_TEXT) {
566
675
  return { ok: false, code: 'memory.invalid', message: `id must be <= ${MAX_ID_CHARS} chars and statement <= ${MAX_TEXT} chars.` }
567
676
  }
568
- const store = await loadStore(storeDir)
569
- const now = new Date().toISOString()
570
- const record = {
571
- id,
572
- topic: String(id).includes('.') ? String(id).split('.')[0] : 'general',
573
- statement,
574
- value: input.value === undefined ? null : input.value,
575
- unit: input.unit === undefined ? null : input.unit,
576
- details: input.details && typeof input.details === 'object' ? input.details : null,
577
- source: input.source ? String(input.source) : null,
578
- timestamp: now,
579
- tags: Array.isArray(input.tags) ? input.tags.map(String) : []
677
+ try {
678
+ return await withStoreLock(storeDir, async () => {
679
+ const store = await loadStore(storeDir)
680
+ const now = new Date().toISOString()
681
+ const record = {
682
+ id,
683
+ topic: String(id).includes('.') ? String(id).split('.')[0] : 'general',
684
+ statement,
685
+ value: input.value === undefined ? null : input.value,
686
+ unit: input.unit === undefined ? null : input.unit,
687
+ details: input.details && typeof input.details === 'object' ? input.details : null,
688
+ source: input.source ? String(input.source) : null,
689
+ timestamp: now,
690
+ tags: Array.isArray(input.tags) ? input.tags.map(String) : []
691
+ }
692
+ const at = store.facts.findIndex((f) => f && String(f.id) === id)
693
+ const action = at >= 0 ? 'updated' : 'created'
694
+ if (at >= 0) store.facts[at] = record
695
+ else store.facts.push(record)
696
+ const written = await saveStore(storeDir, store)
697
+ return { ok: true, id, action, total: store.facts.length, bytes: written.bytes, file: written.file }
698
+ })
699
+ } catch (error) {
700
+ if (error && error.code === 'memory.locked') {
701
+ return { ok: false, code: 'memory.locked', message: error.message }
702
+ }
703
+ throw error
580
704
  }
581
- const at = store.facts.findIndex((f) => f && String(f.id) === id)
582
- const action = at >= 0 ? 'updated' : 'created'
583
- if (at >= 0) store.facts[at] = record
584
- else store.facts.push(record)
585
- const written = await saveStore(storeDir, store)
586
- return { ok: true, id, action, total: store.facts.length, bytes: written.bytes, file: written.file }
587
705
  }
588
706
  }),
589
707
  'limit-break: memory_save tool'
@@ -595,7 +713,7 @@ export function apply(ctx, config) {
595
713
  ctx.tools.register({
596
714
  name: 'resilient_fetch',
597
715
  description:
598
- 'Fetch a URL through a 4-step ladder for hosts whose block is keyed on the TLS hostname (SNI) rather than the IP: plain fetch -> IPv6 -> IPv4 -> a same-service alternate hostname. Use when web_fetch or a plain request fails with a timeout, a connection reset, or "resolves to a non-public IP", and you still need the content.',
716
+ 'Fetch a URL through a 4-step ladder for hosts whose block is keyed on the TLS hostname (SNI) rather than the IP: plain fetch -> IPv6 -> IPv4 -> a same-service alternate hostname. A host that only answered on a later rung is remembered in the memory store for two weeks and the next call starts at that rung. Use when web_fetch or a plain request fails with a timeout, a connection reset, or "resolves to a non-public IP", and you still need the content.',
599
717
  parameters: {
600
718
  type: 'object',
601
719
  properties: {
@@ -609,7 +727,7 @@ export function apply(ctx, config) {
609
727
  schema: { type: 'object' },
610
728
  render: (_args, value) =>
611
729
  value && value.ok
612
- ? [{ type: 'text', text: `HTTP ${value.status} via ${value.via} (${value.bytes} bytes)\n${String(value.text || '').slice(0, 400)}` }]
730
+ ? [{ type: 'text', text: `HTTP ${value.status} via ${value.via} in ${(Number(value.millis) / 1000).toFixed(1)}s [route ${value.route}] (${value.bytes} bytes)\n${String(value.text || '').slice(0, 400)}` }]
613
731
  : [{ type: 'text', text: `All attempts failed: ${(value && value.attempts ? value.attempts : []).map((a) => `${a.via}=${a.error || a.status}`).join(' | ')}` }]
614
732
  },
615
733
  presentCall: (args) => ({
@@ -622,11 +740,23 @@ export function apply(ctx, config) {
622
740
  const input = args && typeof args === 'object' ? args : {}
623
741
  const url = typeof input.url === 'string' ? input.url.trim() : ''
624
742
  if (!/^https?:\/\//i.test(url)) return { ok: false, code: 'fetch.invalid', attempts: [], message: 'url must start with http:// or https://' }
743
+ let host = ''
744
+ try {
745
+ host = new URL(url).hostname
746
+ } catch {
747
+ return { ok: false, code: 'fetch.invalid', attempts: [], message: 'url is not a valid absolute URL' }
748
+ }
625
749
  const per = Math.max(2000, Number(input.timeoutMs) || timeoutMs)
626
- const targets = [{ url, via: 'direct' }, ...alternatesFor(url).map((u) => ({ url: u, via: 'alternate-host' }))]
750
+ const cached = await readRoute(storeDir, host).catch(() => null)
751
+ let targets = [{ url, via: 'direct' }, ...alternatesFor(url).map((u) => ({ url: u, via: 'alternate-host' }))]
627
752
  if (shellFallback) {
628
753
  targets.push({ url, via: 'curl-ipv6' }, { url, via: 'curl-ipv4' })
629
754
  }
755
+ if (cached) {
756
+ const preferred = targets.filter((target) => target.via === cached.via)
757
+ if (preferred.length) targets = [...preferred, ...targets.filter((target) => target.via !== cached.via)]
758
+ }
759
+ const startedAt = Date.now()
630
760
  const attempts = []
631
761
  for (const target of targets) {
632
762
  try {
@@ -637,6 +767,18 @@ export function apply(ctx, config) {
637
767
  attempts.push({ via: target.via, url: target.url, status: result.status })
638
768
  if (result.ok && result.text) {
639
769
  const clipped = clip(result.text)
770
+ const millis = Date.now() - startedAt
771
+ let route = cached ? 'reused' : 'none'
772
+ try {
773
+ if (target.via === 'direct') {
774
+ if (cached) route = (await dropRoute(storeDir, host)) ? 'dropped' : 'none'
775
+ } else {
776
+ await writeRoute(storeDir, host, { via: target.via, url: target.url, millis })
777
+ route = cached && cached.via === target.via ? 'reused' : 'learned'
778
+ }
779
+ } catch {
780
+ route = 'unwritten'
781
+ }
640
782
  return {
641
783
  ok: true,
642
784
  status: result.status,
@@ -646,14 +788,17 @@ export function apply(ctx, config) {
646
788
  bytes: clipped.bytes,
647
789
  truncated: clipped.truncated,
648
790
  text: clipped.text,
649
- attempts
791
+ attempts,
792
+ millis,
793
+ route,
794
+ cached_via: cached ? cached.via : null
650
795
  }
651
796
  }
652
797
  } catch (error) {
653
798
  attempts.push({ via: target.via, url: target.url, error: String((error && error.message) || error).slice(0, 300) })
654
799
  }
655
800
  }
656
- return { ok: false, code: 'fetch.failed', requested: url, attempts }
801
+ return { ok: false, code: 'fetch.failed', requested: url, cached_via: cached ? cached.via : null, attempts }
657
802
  }
658
803
  }),
659
804
  'limit-break: resilient_fetch tool'
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "dsh-limit-break",
3
- "version": "1.5.0",
3
+ "version": "1.7.0",
4
4
  "description": "Durable cross-session memory and a fetch ladder that survives hostname-keyed blocking — four small tools for DeepSeek Harness.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
@@ -0,0 +1,112 @@
1
+ /**
2
+ * Concurrency probe: do simultaneous memory_save calls lose facts?
3
+ *
4
+ * memory_save is load -> mutate -> save (tmp file + rename). Without a lock, two processes that
5
+ * load the same store each write their own version back and the later rename wins, so the other
6
+ * process' fact disappears. On Windows the losing writer can even fail the rename outright
7
+ * (EPERM: the target is open in the other process).
8
+ *
9
+ * This test makes the damage visible in plain numbers: N worker processes each attempt M saves,
10
+ * then the store is read back and the surviving ids are counted.
11
+ *
12
+ * Run: node tools/concurrency-selftest.mjs [workers] [saves-per-worker] [rounds]
13
+ */
14
+ import { execFile } from 'node:child_process'
15
+ import { mkdir, mkdtemp, readdir, readFile, rm, utimes, writeFile } from 'node:fs/promises'
16
+ import { tmpdir } from 'node:os'
17
+ import { join } from 'node:path'
18
+ import { fileURLToPath } from 'node:url'
19
+ import { promisify } from 'node:util'
20
+
21
+ const exec = promisify(execFile)
22
+ const LIB = fileURLToPath(new URL('../lib/index.js', import.meta.url))
23
+ const workers = Number(process.argv[2] || 8)
24
+ const perWorker = Number(process.argv[3] || 6)
25
+ const rounds = Number(process.argv[4] || 3)
26
+
27
+ const worker = `
28
+ import(process.env.LIB).then(async (m) => {
29
+ const tools = new Map()
30
+ const ctx = {
31
+ effect(fn) { fn() },
32
+ tools: { register(d) { tools.set(d.name, d); return () => tools.delete(d.name) } }
33
+ }
34
+ m.apply(ctx, { storeDir: process.env.STORE_DIR })
35
+ const save = tools.get('memory_save')
36
+ const w = Number(process.env.WORKER)
37
+ let ok = 0
38
+ const errors = []
39
+ for (let i = 0; i < ${perWorker}; i += 1) {
40
+ try {
41
+ const res = await save.execute({ id: 'race.w' + w + '_' + i, statement: 'concurrent write w' + w + ' i' + i + ' from a separate process' })
42
+ if (res && res.ok) ok += 1
43
+ else errors.push((res && res.code ? res.code : 'not-ok') + ' ' + JSON.stringify(res))
44
+ } catch (e) {
45
+ const detail = [e && e.code, e && e.syscall, e && e.path, e && e.message].filter(Boolean).join(' | ')
46
+ errors.push(detail || String(e))
47
+ }
48
+ }
49
+ console.log('RESULT ' + JSON.stringify({ worker: w, ok, errors }))
50
+ process.exit(0)
51
+ }).catch((e) => { console.error(e); process.exit(1) })
52
+ `
53
+
54
+ const results = []
55
+ for (let round = 1; round <= rounds; round += 1) {
56
+ const dir = await mkdtemp(join(tmpdir(), 'limit-break-race-'))
57
+ const storeDir = join(dir, 'memory')
58
+ const replies = await Promise.all(
59
+ Array.from({ length: workers }, (_, w) =>
60
+ exec(process.execPath, ['--input-type=module', '-e', worker], {
61
+ env: { ...process.env, LIB: `file:///${LIB.replaceAll('\\', '/')}`, STORE_DIR: storeDir, WORKER: String(w) },
62
+ timeout: 120000
63
+ }).then((r) => r.stdout)
64
+ )
65
+ )
66
+ const reports = replies.flatMap((out) =>
67
+ out.split('\n').filter((line) => line.startsWith('RESULT ')).map((line) => JSON.parse(line.slice(7)))
68
+ )
69
+ const succeeded = reports.reduce((sum, r) => sum + r.ok, 0)
70
+ const errorCodes = reports.flatMap((r) => r.errors)
71
+ const store = JSON.parse(await readFile(join(storeDir, 'facts.json'), 'utf8'))
72
+ const kept = store.facts.filter((f) => f && /^race\.w/.test(String(f.id))).length
73
+ const leftover = (await readdir(storeDir)).filter((name) => name !== 'facts.json' && name !== 'plugin-load.log')
74
+ results.push({ round, attempted: workers * perWorker, succeeded, kept, lost: succeeded - kept, failed: workers * perWorker - succeeded, errorCodes, leftover })
75
+ await rm(dir, { recursive: true, force: true })
76
+ }
77
+
78
+ for (const r of results) {
79
+ const codes = r.errorCodes.length ? ` (errors: ${[...new Set(r.errorCodes)].join(',')})` : ''
80
+ console.log(`round ${r.round}: attempted ${r.attempted}, save-ok ${r.succeeded}, kept ${r.kept}, lost ${r.lost}${codes}${r.leftover.length ? ` leftover: ${r.leftover.join(',')}` : ''}`)
81
+ }
82
+ // A writer that dies while holding the lock must not wedge the store forever: a lock older than
83
+ // the stale window is stolen, and the save goes through.
84
+ const staleDir = await mkdtemp(join(tmpdir(), 'limit-break-stale-'))
85
+ const staleStore = join(staleDir, 'memory')
86
+ await mkdir(staleStore, { recursive: true })
87
+ await writeFile(join(staleStore, '.facts.lock'), '999999 2020-01-01T00:00:00.000Z\n')
88
+ const past = new Date(Date.now() - 60000)
89
+ await utimes(join(staleStore, '.facts.lock'), past, past)
90
+ const staleOut = await exec(process.execPath, ['--input-type=module', '-e', worker], {
91
+ env: { ...process.env, LIB: `file:///${LIB.replaceAll('\\', '/')}`, STORE_DIR: staleStore, WORKER: '99' },
92
+ timeout: 120000
93
+ })
94
+ const stale = JSON.parse(staleOut.stdout.split('\n').filter((line) => line.startsWith('RESULT ')).map((line) => line.slice(7))[0])
95
+ const staleStoreJson = JSON.parse(await readFile(join(staleStore, 'facts.json'), 'utf8'))
96
+ const staleKept = staleStoreJson.facts.filter((f) => f && /^race\.w99/.test(String(f.id))).length
97
+ const staleLeftover = (await readdir(staleStore)).filter((name) => name !== 'facts.json' && name !== 'plugin-load.log')
98
+ await rm(staleDir, { recursive: true, force: true })
99
+ console.log(`stale lock: save-ok ${stale.ok}/${perWorker}, kept ${staleKept}, leftover ${staleLeftover.length ? staleLeftover.join(',') : 'none'}`)
100
+
101
+ const lost = results.map((r) => r.lost)
102
+ const failed = results.map((r) => r.failed)
103
+ const worstLost = Math.max(...lost)
104
+ const avgLost = +(lost.reduce((a, b) => a + b, 0) / lost.length).toFixed(1)
105
+ const worstFailed = Math.max(...failed)
106
+ console.log(`${workers} workers x ${perWorker} saves x ${rounds} rounds -> lost facts: worst ${worstLost}, average ${avgLost}; failed saves: worst ${worstFailed}`)
107
+ const staleBad = stale.ok !== perWorker || staleKept !== perWorker || staleLeftover.length > 0
108
+ if (worstLost > 0 || worstFailed > 0 || staleBad) {
109
+ console.log('FAIL: concurrent saves lose facts or fail outright')
110
+ process.exit(1)
111
+ }
112
+ console.log('PASS: every concurrent save survived and none failed')
@@ -0,0 +1,98 @@
1
+ /**
2
+ * Route-cache probe: does remembering which rung of the fetch ladder works actually pay?
3
+ *
4
+ * `resilient_fetch` walks direct -> alternates -> curl IPv6 -> curl IPv4. On a host whose direct
5
+ * request is blocked (SNI-keyed), every call used to rediscover that by burning a timeout on rung 1
6
+ * before rung 4 answered. Since 1.7.0 a non-direct success is stored as `network.route.<host>` and
7
+ * the next call starts at that rung.
8
+ *
9
+ * This measures one blocked host: cold call (cache miss) vs warm calls (cache hit), printing
10
+ * attempts and seconds, then checks the store holds the route fact.
11
+ *
12
+ * Run: node tools/fetch-route-bench.mjs [url] [timeoutMs]
13
+ */
14
+ import { mkdtemp, readFile } from 'node:fs/promises'
15
+ import { tmpdir } from 'node:os'
16
+ import { join } from 'node:path'
17
+ import { fileURLToPath, pathToFileURL } from 'node:url'
18
+
19
+ const LIB = fileURLToPath(new URL('../lib/index.js', import.meta.url))
20
+ const url = process.argv[2] || 'https://en.wikipedia.org/wiki/Monte_Carlo_method'
21
+ const timeoutMs = Number(process.argv[3] || 6000)
22
+ const host = new URL(url).hostname
23
+
24
+ const { apply } = await import(pathToFileURL(LIB).href)
25
+ const mod = await import(pathToFileURL(LIB).href)
26
+ const tools = new Map()
27
+ const ctx = {
28
+ effect(fn) {
29
+ fn()
30
+ },
31
+ tools: {
32
+ register(tool) {
33
+ tools.set(tool.name, tool)
34
+ return () => tools.delete(tool.name)
35
+ }
36
+ }
37
+ }
38
+ const storeDir = join(await mkdtemp(join(tmpdir(), 'limit-break-route-')), 'memory')
39
+ apply(ctx, { storeDir })
40
+ const fetchTool = tools.get('resilient_fetch')
41
+
42
+ const call = async (label) => {
43
+ const started = Date.now()
44
+ const result = await fetchTool.execute({ url, timeoutMs })
45
+ const seconds = +((Date.now() - started) / 1000).toFixed(2)
46
+ console.log(
47
+ `${label}: ok=${result.ok} via=${result.via || '-'} route=${result.route || '-'} attempts=${(result.attempts || []).map((a) => a.via).join('>')} seconds=${seconds}`
48
+ )
49
+ return { result, seconds }
50
+ }
51
+
52
+ const cold = await call('cold')
53
+ if (!cold.result.ok || cold.result.via === 'direct') {
54
+ console.log('SKIP: direct fetch to this host works today, so there is no failed rung to remember')
55
+ process.exit(0)
56
+ }
57
+ const warm1 = await call('warm1')
58
+ const warm2 = await call('warm2')
59
+
60
+ const store = JSON.parse(await readFile(join(storeDir, 'facts.json'), 'utf8'))
61
+ const fact = store.facts.find((row) => row && row.id === `network.route.${host.toLowerCase()}`)
62
+ console.log(`route fact: ${fact ? JSON.stringify(fact.details) : 'MISSING'}`)
63
+
64
+ // Self-heal: if a host has a remembered route but a direct request now works, the stale route
65
+ // must be dropped instead of being trusted forever.
66
+ const healUrl = 'https://api.wikimedia.org/core/v1/wikipedia/en/page/Monte_Carlo_method'
67
+ const healHost = new URL(healUrl).hostname
68
+ const seeded = await mod.loadStore(storeDir)
69
+ seeded.facts.push({
70
+ id: `network.route.${healHost}`,
71
+ topic: 'network',
72
+ statement: 'seeded stale route for the self-heal check',
73
+ details: { host: healHost, via: 'alternate-host', url: healUrl, observedAt: new Date().toISOString(), millis: null },
74
+ source: 'bench',
75
+ timestamp: new Date().toISOString(),
76
+ tags: ['network']
77
+ })
78
+ await mod.saveStore(storeDir, seeded)
79
+ const heal = await fetchTool.execute({ url: healUrl, timeoutMs })
80
+ const afterHeal = await mod.loadStore(storeDir)
81
+ const healDropped = !afterHeal.facts.some((row) => row && row.id === `network.route.${healHost}`)
82
+ console.log(`self-heal: ok=${heal.ok} via=${heal.via} route=${heal.route} factDropped=${healDropped}`)
83
+
84
+ const checks = [
85
+ ['cold used a fallback rung', cold.result.via !== 'direct' && cold.result.attempts.length > 1],
86
+ ['warm call starts at the remembered rung', warm1.result.attempts.length === 1 && warm1.result.attempts[0].via === cold.result.via],
87
+ ['warm call is at least 2x faster', warm1.seconds * 2 <= cold.seconds],
88
+ ['warm call is stable', warm2.result.via === cold.result.via && warm2.result.route === 'reused'],
89
+ ['route fact written', Boolean(fact) && fact.details && fact.details.via === cold.result.via],
90
+ ['stale route dropped once direct works', heal.ok && heal.via === 'direct' && heal.route === 'dropped' && healDropped]
91
+ ]
92
+ let failed = 0
93
+ for (const [name, ok] of checks) {
94
+ console.log(`${ok ? 'PASS' : 'FAIL'} ${name}`)
95
+ if (!ok) failed += 1
96
+ }
97
+ console.log(`cold ${cold.seconds}s -> warm ${warm1.seconds}s / ${warm2.seconds}s (${(cold.seconds / Math.max(warm1.seconds, 0.01)).toFixed(1)}x faster)`)
98
+ process.exit(failed ? 1 : 0)