dsh-limit-break 1.6.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/MAINTAIN.md +8 -2
- package/README.md +9 -0
- package/lib/index.js +90 -5
- package/package.json +1 -1
- package/tools/fetch-route-bench.mjs +98 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,29 @@ All notable changes to `dsh-limit-break`. Format follows
|
|
|
5
5
|
|
|
6
6
|
## Unreleased
|
|
7
7
|
|
|
8
|
+
## 1.7.0 — 2026-11-05
|
|
9
|
+
|
|
10
|
+
### Changed
|
|
11
|
+
- **`resilient_fetch` remembers which rung worked instead of rediscovering it on every call.** A
|
|
12
|
+
non-direct success is written back as `network.route.<host>` (14-day TTL, written under the same
|
|
13
|
+
lock as `memory_save`) and the next call to that host starts at that rung; if a plain fetch starts
|
|
14
|
+
working again the route is deleted, so an unblocked host heals itself. Measured on
|
|
15
|
+
`en.wikipedia.org/wiki/Monte_Carlo_method` with the default 12 s per-rung timeout: cold **12.11 s**
|
|
16
|
+
with attempts `direct>alternate-host`, warm **1.08 s / 1.14 s** with a single `alternate-host`
|
|
17
|
+
attempt — **11.2× faster**; the tool now also reports `millis`, `route`
|
|
18
|
+
(`learned|reused|dropped|none`) and `cached_via`.
|
|
19
|
+
- **`tools/fetch-route-bench.mjs`** (new): cold-vs-warm measurement for one blocked host plus a
|
|
20
|
+
self-heal check (a seeded stale route must be dropped once direct works). 6/6 checks PASS.
|
|
21
|
+
- Route facts are ordinary facts: visible to `memory_recall`/`memory_stats`, removable with any
|
|
22
|
+
store edit. Only fallback successes and self-heal events write, so a working network writes nothing.
|
|
23
|
+
|
|
24
|
+
### Notes
|
|
25
|
+
- No tool added, removed or renamed; store format unchanged. The tool description text now mentions
|
|
26
|
+
the remembered route, which an already-open session will not see until its tool snapshot refreshes.
|
|
27
|
+
- `maintain.py status` still prints `market listing not live yet (PR #8296 closed)` while the same
|
|
28
|
+
call reports `listed_in_market: true`: the check treats a merged-and-closed PR as not live. Left
|
|
29
|
+
as-is, documented in MAINTAIN.md so the next round does not read it as a regression.
|
|
30
|
+
|
|
8
31
|
## 1.6.0 — 2026-10-29
|
|
9
32
|
|
|
10
33
|
### Fixed
|
package/MAINTAIN.md
CHANGED
|
@@ -138,8 +138,14 @@ git add -A && git commit -m "..." && git push
|
|
|
138
138
|
bench.gpu_monte_carlo and bench.render”(2026-10-21 的回写),陈旧描述不再存在。
|
|
139
139
|
- [x] 插件工具对**已经打开的会话**不可见:已在 README「Known limits」第 1 条写明
|
|
140
140
|
(tool list 是 per-session 快照,插件侧修不了)。事实 `harness.plugin-tool-visibility` 有实测。
|
|
141
|
-
- [
|
|
142
|
-
|
|
141
|
+
- [x] **`resilient_fetch` 会把成功路径写回记忆库**(1.7.0,2026-11-05):非 direct 成功写成
|
|
142
|
+
`network.route.<host>`(14 天 TTL,走 `memory_save` 同一把锁),下次同主机直接从该级起步;
|
|
143
|
+
direct 恢复后删掉路由(自愈)。实测 `en.wikipedia.org/wiki/Monte_Carlo_method`、12 s 超时:
|
|
144
|
+
cold 12.11 s(`direct>alternate-host`)→ warm 1.08 s / 1.14 s(单次 `alternate-host`),11.2×。
|
|
145
|
+
跑 `node tools/fetch-route-bench.mjs <url> <timeoutMs>` 复测(约 15 s,6 项检查)。
|
|
146
|
+
仍可继续加:`--resolve` 到已知 IP、IPv6 优先(backlog)。
|
|
147
|
+
- [ ] `maintain.py status` 的 `problems` 里 `market listing not live yet (PR #8296 closed)` 是误报:
|
|
148
|
+
同一份输出里 `listed_in_market: true`。原因未查,先记为已知噪音,别当成回归。
|
|
143
149
|
- [x] **`memory_save` 的并发写已有自动化覆盖**(1.6.0,2026-10-29):`tools/concurrency-selftest.mjs`
|
|
144
150
|
起 N 个独立进程同时写同一个 store,再读回数存活条数,另测“陈旧锁被接管”。同机 8 进程 × 6 次 × 3 轮:
|
|
145
151
|
修复前丢 16–17/48、且最多 25/48 直接 `EPERM`(rename 输家);加 `O_EXCL` 锁后 0 丢 0 失败,
|
package/README.md
CHANGED
|
@@ -43,6 +43,12 @@ IPv6 → IPv4 → a **sibling hostname of the same service** whose certificate s
|
|
|
43
43
|
It does **not** disable certificate verification and does not route through a proxy that would
|
|
44
44
|
impersonate the origin. Those are MITM, not workarounds.
|
|
45
45
|
|
|
46
|
+
**The ladder learns.** A fallback rung that worked is written back as `network.route.<host>` and the
|
|
47
|
+
next call to that host starts there; if a plain fetch starts working again the route is deleted (host
|
|
48
|
+
self-heals). Measured on `en.wikipedia.org/wiki/Monte_Carlo_method`, 12 s per-rung timeout: cold
|
|
49
|
+
**12.1 s** (rung 1 burned the timeout, rung 4 answered) → warm **1.08 s / 1.14 s**, one attempt
|
|
50
|
+
instead of two — **11.2× faster**. Numbers come from `tools/fetch-route-bench.mjs`.
|
|
51
|
+
|
|
46
52
|
## Install
|
|
47
53
|
|
|
48
54
|
```sh
|
|
@@ -99,6 +105,9 @@ Measured, not guessed — each line is something a session actually hit.
|
|
|
99
105
|
still cannot stop a reader from seeing the previous snapshot.
|
|
100
106
|
* **The fetch ladder is hostname-level.** It does not defeat IP-level blocks, TLS interception,
|
|
101
107
|
or a proxy that terminates the connection; it does not skip certificate verification.
|
|
108
|
+
* **A remembered route is a bet with a 14-day expiry.** The ladder trusts `network.route.<host>` for
|
|
109
|
+
two weeks and drops it as soon as a direct request works; if the winning rung starts failing the
|
|
110
|
+
call still walks the rest of the ladder, it just pays that one rung first.
|
|
102
111
|
* **`memory_recall` ranks by keyword overlap only** (id/topic/tags/statement/details), no
|
|
103
112
|
embeddings: stopword-filtered, crudely stemmed word matching weighted by idf, so it is good at
|
|
104
113
|
"which fact did I mean" and bad at paraphrase. `tools/recall-bench.mjs` states the current
|
package/lib/index.js
CHANGED
|
@@ -233,6 +233,64 @@ async function withStoreLock(dir, fn) {
|
|
|
233
233
|
}
|
|
234
234
|
}
|
|
235
235
|
|
|
236
|
+
// A failed fetch tells you which rung of the ladder works for that host *today*; throwing that
|
|
237
|
+
// away means paying the same dead rungs on every call. A non-direct success is remembered per
|
|
238
|
+
// host (network.route.<host>) for two weeks and the ladder starts at that rung next time. When
|
|
239
|
+
// direct starts working again the route is dropped, so a host that unblocks heals itself.
|
|
240
|
+
const ROUTE_TTL_MS = 14 * 24 * 60 * 60 * 1000
|
|
241
|
+
|
|
242
|
+
function routeId(host) {
|
|
243
|
+
return `network.route.${String(host).toLowerCase().replace(/[^a-z0-9.-]+/g, '-')}`
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
async function readRoute(dir, host) {
|
|
247
|
+
const store = await loadStore(dir)
|
|
248
|
+
const fact = store.facts.find((f) => f && String(f.id) === routeId(host))
|
|
249
|
+
if (!fact) return null
|
|
250
|
+
const details = fact.details && typeof fact.details === 'object' ? fact.details : {}
|
|
251
|
+
const observed = Date.parse(String(details.observedAt || fact.timestamp || ''))
|
|
252
|
+
if (!Number.isFinite(observed) || Date.now() - observed > ROUTE_TTL_MS) return null
|
|
253
|
+
const via = typeof details.via === 'string' ? details.via : ''
|
|
254
|
+
if (!via || via === 'direct') return null
|
|
255
|
+
return { via, url: typeof details.url === 'string' ? details.url : '', observedAt: new Date(observed).toISOString() }
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
async function writeRoute(dir, host, info) {
|
|
259
|
+
const now = new Date().toISOString()
|
|
260
|
+
const id = routeId(host)
|
|
261
|
+
return withStoreLock(dir, async () => {
|
|
262
|
+
const store = await loadStore(dir)
|
|
263
|
+
const record = {
|
|
264
|
+
id,
|
|
265
|
+
topic: 'network',
|
|
266
|
+
statement: `${host} answers through the ${info.via} rung of the fetch ladder (${info.url}); a direct request to it fails today, so the ladder should start at ${info.via}.`,
|
|
267
|
+
value: null,
|
|
268
|
+
unit: null,
|
|
269
|
+
details: { host, via: info.via, url: info.url, observedAt: now, millis: info.millis || null },
|
|
270
|
+
source: 'resilient_fetch',
|
|
271
|
+
timestamp: now,
|
|
272
|
+
tags: ['network', 'fetch-route']
|
|
273
|
+
}
|
|
274
|
+
const at = store.facts.findIndex((f) => f && String(f.id) === id)
|
|
275
|
+
if (at >= 0) store.facts[at] = record
|
|
276
|
+
else store.facts.push(record)
|
|
277
|
+
await saveStore(dir, store)
|
|
278
|
+
return record
|
|
279
|
+
})
|
|
280
|
+
}
|
|
281
|
+
|
|
282
|
+
async function dropRoute(dir, host) {
|
|
283
|
+
const id = routeId(host)
|
|
284
|
+
return withStoreLock(dir, async () => {
|
|
285
|
+
const store = await loadStore(dir)
|
|
286
|
+
const before = store.facts.length
|
|
287
|
+
store.facts = store.facts.filter((f) => !(f && String(f.id) === id))
|
|
288
|
+
if (store.facts.length === before) return false
|
|
289
|
+
await saveStore(dir, store)
|
|
290
|
+
return true
|
|
291
|
+
})
|
|
292
|
+
}
|
|
293
|
+
|
|
236
294
|
// Ranking inputs. The old scorer used raw substring counts, which let a common word
|
|
237
295
|
// ("the", "how", "long") outvote the one rare word that identifies the fact. Words are
|
|
238
296
|
// matched against word sets instead, suffixed words share a stem, and each query token is
|
|
@@ -655,7 +713,7 @@ export function apply(ctx, config) {
|
|
|
655
713
|
ctx.tools.register({
|
|
656
714
|
name: 'resilient_fetch',
|
|
657
715
|
description:
|
|
658
|
-
'Fetch a URL through a 4-step ladder for hosts whose block is keyed on the TLS hostname (SNI) rather than the IP: plain fetch -> IPv6 -> IPv4 -> a same-service alternate hostname. Use when web_fetch or a plain request fails with a timeout, a connection reset, or "resolves to a non-public IP", and you still need the content.',
|
|
716
|
+
'Fetch a URL through a 4-step ladder for hosts whose block is keyed on the TLS hostname (SNI) rather than the IP: plain fetch -> IPv6 -> IPv4 -> a same-service alternate hostname. A host that only answered on a later rung is remembered in the memory store for two weeks and the next call starts at that rung. Use when web_fetch or a plain request fails with a timeout, a connection reset, or "resolves to a non-public IP", and you still need the content.',
|
|
659
717
|
parameters: {
|
|
660
718
|
type: 'object',
|
|
661
719
|
properties: {
|
|
@@ -669,7 +727,7 @@ export function apply(ctx, config) {
|
|
|
669
727
|
schema: { type: 'object' },
|
|
670
728
|
render: (_args, value) =>
|
|
671
729
|
value && value.ok
|
|
672
|
-
? [{ type: 'text', text: `HTTP ${value.status} via ${value.via} (${value.bytes} bytes)\n${String(value.text || '').slice(0, 400)}` }]
|
|
730
|
+
? [{ type: 'text', text: `HTTP ${value.status} via ${value.via} in ${(Number(value.millis) / 1000).toFixed(1)}s [route ${value.route}] (${value.bytes} bytes)\n${String(value.text || '').slice(0, 400)}` }]
|
|
673
731
|
: [{ type: 'text', text: `All attempts failed: ${(value && value.attempts ? value.attempts : []).map((a) => `${a.via}=${a.error || a.status}`).join(' | ')}` }]
|
|
674
732
|
},
|
|
675
733
|
presentCall: (args) => ({
|
|
@@ -682,11 +740,23 @@ export function apply(ctx, config) {
|
|
|
682
740
|
const input = args && typeof args === 'object' ? args : {}
|
|
683
741
|
const url = typeof input.url === 'string' ? input.url.trim() : ''
|
|
684
742
|
if (!/^https?:\/\//i.test(url)) return { ok: false, code: 'fetch.invalid', attempts: [], message: 'url must start with http:// or https://' }
|
|
743
|
+
let host = ''
|
|
744
|
+
try {
|
|
745
|
+
host = new URL(url).hostname
|
|
746
|
+
} catch {
|
|
747
|
+
return { ok: false, code: 'fetch.invalid', attempts: [], message: 'url is not a valid absolute URL' }
|
|
748
|
+
}
|
|
685
749
|
const per = Math.max(2000, Number(input.timeoutMs) || timeoutMs)
|
|
686
|
-
const
|
|
750
|
+
const cached = await readRoute(storeDir, host).catch(() => null)
|
|
751
|
+
let targets = [{ url, via: 'direct' }, ...alternatesFor(url).map((u) => ({ url: u, via: 'alternate-host' }))]
|
|
687
752
|
if (shellFallback) {
|
|
688
753
|
targets.push({ url, via: 'curl-ipv6' }, { url, via: 'curl-ipv4' })
|
|
689
754
|
}
|
|
755
|
+
if (cached) {
|
|
756
|
+
const preferred = targets.filter((target) => target.via === cached.via)
|
|
757
|
+
if (preferred.length) targets = [...preferred, ...targets.filter((target) => target.via !== cached.via)]
|
|
758
|
+
}
|
|
759
|
+
const startedAt = Date.now()
|
|
690
760
|
const attempts = []
|
|
691
761
|
for (const target of targets) {
|
|
692
762
|
try {
|
|
@@ -697,6 +767,18 @@ export function apply(ctx, config) {
|
|
|
697
767
|
attempts.push({ via: target.via, url: target.url, status: result.status })
|
|
698
768
|
if (result.ok && result.text) {
|
|
699
769
|
const clipped = clip(result.text)
|
|
770
|
+
const millis = Date.now() - startedAt
|
|
771
|
+
let route = cached ? 'reused' : 'none'
|
|
772
|
+
try {
|
|
773
|
+
if (target.via === 'direct') {
|
|
774
|
+
if (cached) route = (await dropRoute(storeDir, host)) ? 'dropped' : 'none'
|
|
775
|
+
} else {
|
|
776
|
+
await writeRoute(storeDir, host, { via: target.via, url: target.url, millis })
|
|
777
|
+
route = cached && cached.via === target.via ? 'reused' : 'learned'
|
|
778
|
+
}
|
|
779
|
+
} catch {
|
|
780
|
+
route = 'unwritten'
|
|
781
|
+
}
|
|
700
782
|
return {
|
|
701
783
|
ok: true,
|
|
702
784
|
status: result.status,
|
|
@@ -706,14 +788,17 @@ export function apply(ctx, config) {
|
|
|
706
788
|
bytes: clipped.bytes,
|
|
707
789
|
truncated: clipped.truncated,
|
|
708
790
|
text: clipped.text,
|
|
709
|
-
attempts
|
|
791
|
+
attempts,
|
|
792
|
+
millis,
|
|
793
|
+
route,
|
|
794
|
+
cached_via: cached ? cached.via : null
|
|
710
795
|
}
|
|
711
796
|
}
|
|
712
797
|
} catch (error) {
|
|
713
798
|
attempts.push({ via: target.via, url: target.url, error: String((error && error.message) || error).slice(0, 300) })
|
|
714
799
|
}
|
|
715
800
|
}
|
|
716
|
-
return { ok: false, code: 'fetch.failed', requested: url, attempts }
|
|
801
|
+
return { ok: false, code: 'fetch.failed', requested: url, cached_via: cached ? cached.via : null, attempts }
|
|
717
802
|
}
|
|
718
803
|
}),
|
|
719
804
|
'limit-break: resilient_fetch tool'
|
package/package.json
CHANGED
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Route-cache probe: does remembering which rung of the fetch ladder works actually pay?
|
|
3
|
+
*
|
|
4
|
+
* `resilient_fetch` walks direct -> alternates -> curl IPv6 -> curl IPv4. On a host whose direct
|
|
5
|
+
* request is blocked (SNI-keyed), every call used to rediscover that by burning a timeout on rung 1
|
|
6
|
+
* before rung 4 answered. Since 1.7.0 a non-direct success is stored as `network.route.<host>` and
|
|
7
|
+
* the next call starts at that rung.
|
|
8
|
+
*
|
|
9
|
+
* This measures one blocked host: cold call (cache miss) vs warm calls (cache hit), printing
|
|
10
|
+
* attempts and seconds, then checks the store holds the route fact.
|
|
11
|
+
*
|
|
12
|
+
* Run: node tools/fetch-route-bench.mjs [url] [timeoutMs]
|
|
13
|
+
*/
|
|
14
|
+
import { mkdtemp, readFile } from 'node:fs/promises'
|
|
15
|
+
import { tmpdir } from 'node:os'
|
|
16
|
+
import { join } from 'node:path'
|
|
17
|
+
import { fileURLToPath, pathToFileURL } from 'node:url'
|
|
18
|
+
|
|
19
|
+
const LIB = fileURLToPath(new URL('../lib/index.js', import.meta.url))
|
|
20
|
+
const url = process.argv[2] || 'https://en.wikipedia.org/wiki/Monte_Carlo_method'
|
|
21
|
+
const timeoutMs = Number(process.argv[3] || 6000)
|
|
22
|
+
const host = new URL(url).hostname
|
|
23
|
+
|
|
24
|
+
const { apply } = await import(pathToFileURL(LIB).href)
|
|
25
|
+
const mod = await import(pathToFileURL(LIB).href)
|
|
26
|
+
const tools = new Map()
|
|
27
|
+
const ctx = {
|
|
28
|
+
effect(fn) {
|
|
29
|
+
fn()
|
|
30
|
+
},
|
|
31
|
+
tools: {
|
|
32
|
+
register(tool) {
|
|
33
|
+
tools.set(tool.name, tool)
|
|
34
|
+
return () => tools.delete(tool.name)
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
const storeDir = join(await mkdtemp(join(tmpdir(), 'limit-break-route-')), 'memory')
|
|
39
|
+
apply(ctx, { storeDir })
|
|
40
|
+
const fetchTool = tools.get('resilient_fetch')
|
|
41
|
+
|
|
42
|
+
const call = async (label) => {
|
|
43
|
+
const started = Date.now()
|
|
44
|
+
const result = await fetchTool.execute({ url, timeoutMs })
|
|
45
|
+
const seconds = +((Date.now() - started) / 1000).toFixed(2)
|
|
46
|
+
console.log(
|
|
47
|
+
`${label}: ok=${result.ok} via=${result.via || '-'} route=${result.route || '-'} attempts=${(result.attempts || []).map((a) => a.via).join('>')} seconds=${seconds}`
|
|
48
|
+
)
|
|
49
|
+
return { result, seconds }
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const cold = await call('cold')
|
|
53
|
+
if (!cold.result.ok || cold.result.via === 'direct') {
|
|
54
|
+
console.log('SKIP: direct fetch to this host works today, so there is no failed rung to remember')
|
|
55
|
+
process.exit(0)
|
|
56
|
+
}
|
|
57
|
+
const warm1 = await call('warm1')
|
|
58
|
+
const warm2 = await call('warm2')
|
|
59
|
+
|
|
60
|
+
const store = JSON.parse(await readFile(join(storeDir, 'facts.json'), 'utf8'))
|
|
61
|
+
const fact = store.facts.find((row) => row && row.id === `network.route.${host.toLowerCase()}`)
|
|
62
|
+
console.log(`route fact: ${fact ? JSON.stringify(fact.details) : 'MISSING'}`)
|
|
63
|
+
|
|
64
|
+
// Self-heal: if a host has a remembered route but a direct request now works, the stale route
|
|
65
|
+
// must be dropped instead of being trusted forever.
|
|
66
|
+
const healUrl = 'https://api.wikimedia.org/core/v1/wikipedia/en/page/Monte_Carlo_method'
|
|
67
|
+
const healHost = new URL(healUrl).hostname
|
|
68
|
+
const seeded = await mod.loadStore(storeDir)
|
|
69
|
+
seeded.facts.push({
|
|
70
|
+
id: `network.route.${healHost}`,
|
|
71
|
+
topic: 'network',
|
|
72
|
+
statement: 'seeded stale route for the self-heal check',
|
|
73
|
+
details: { host: healHost, via: 'alternate-host', url: healUrl, observedAt: new Date().toISOString(), millis: null },
|
|
74
|
+
source: 'bench',
|
|
75
|
+
timestamp: new Date().toISOString(),
|
|
76
|
+
tags: ['network']
|
|
77
|
+
})
|
|
78
|
+
await mod.saveStore(storeDir, seeded)
|
|
79
|
+
const heal = await fetchTool.execute({ url: healUrl, timeoutMs })
|
|
80
|
+
const afterHeal = await mod.loadStore(storeDir)
|
|
81
|
+
const healDropped = !afterHeal.facts.some((row) => row && row.id === `network.route.${healHost}`)
|
|
82
|
+
console.log(`self-heal: ok=${heal.ok} via=${heal.via} route=${heal.route} factDropped=${healDropped}`)
|
|
83
|
+
|
|
84
|
+
const checks = [
|
|
85
|
+
['cold used a fallback rung', cold.result.via !== 'direct' && cold.result.attempts.length > 1],
|
|
86
|
+
['warm call starts at the remembered rung', warm1.result.attempts.length === 1 && warm1.result.attempts[0].via === cold.result.via],
|
|
87
|
+
['warm call is at least 2x faster', warm1.seconds * 2 <= cold.seconds],
|
|
88
|
+
['warm call is stable', warm2.result.via === cold.result.via && warm2.result.route === 'reused'],
|
|
89
|
+
['route fact written', Boolean(fact) && fact.details && fact.details.via === cold.result.via],
|
|
90
|
+
['stale route dropped once direct works', heal.ok && heal.via === 'direct' && heal.route === 'dropped' && healDropped]
|
|
91
|
+
]
|
|
92
|
+
let failed = 0
|
|
93
|
+
for (const [name, ok] of checks) {
|
|
94
|
+
console.log(`${ok ? 'PASS' : 'FAIL'} ${name}`)
|
|
95
|
+
if (!ok) failed += 1
|
|
96
|
+
}
|
|
97
|
+
console.log(`cold ${cold.seconds}s -> warm ${warm1.seconds}s / ${warm2.seconds}s (${(cold.seconds / Math.max(warm1.seconds, 0.01)).toFixed(1)}x faster)`)
|
|
98
|
+
process.exit(failed ? 1 : 0)
|