@fffangchenghao/dsh-web-search 0.11.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +85 -0
- package/README.md +307 -46
- package/README.zh-CN.md +307 -46
- package/client.js +44 -3
- package/lib/cache-persist.d.ts +96 -0
- package/lib/cache-persist.js +249 -0
- package/lib/cache.d.ts +48 -4
- package/lib/cache.js +134 -9
- package/lib/config.d.ts +25 -0
- package/lib/config.js +42 -0
- package/lib/engines.d.ts +41 -1
- package/lib/engines.js +33 -2
- package/lib/fetch.d.ts +33 -0
- package/lib/fetch.js +218 -6
- package/lib/index.d.ts +26 -0
- package/lib/index.js +113 -27
- package/lib/markdown.d.ts +84 -0
- package/lib/markdown.js +847 -0
- package/lib/rank.js +4 -1
- package/lib/read-page.d.ts +136 -0
- package/lib/read-page.js +496 -0
- package/lib/render.d.ts +193 -0
- package/lib/render.js +354 -0
- package/lib/search-many.d.ts +117 -0
- package/lib/search-many.js +286 -0
- package/lib/transport.js +19 -0
- package/lib/types.d.ts +13 -4
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,90 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.13.0
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- **Optional JS rendering.** A page whose HTML is an empty shell can be rendered
|
|
7
|
+
by a real browser — but only if `playwright` or `playwright-core` is already
|
|
8
|
+
installed. Nothing is bundled, nothing is imported at load time, and a missing
|
|
9
|
+
module is a silent fallback, not an error. `renderJs` picks `auto` (render only
|
|
10
|
+
suspected shells), `always` or `off`; `renderTimeoutMs` bounds the attempt. A
|
|
11
|
+
render that fails, times out or is aborted returns the earlier rung's result.
|
|
12
|
+
- **`read_page`, for pages longer than one answer.** The host fetch request
|
|
13
|
+
carries only a url, so `web_fetch` can truncate but never continue. This tool
|
|
14
|
+
takes `offset` and returns `nextOffset`, and consecutive chunks concatenate to
|
|
15
|
+
exactly the full converted page — verified live against a long documentation
|
|
16
|
+
page. Pages are fetched once and sliced, so continuing does not download again.
|
|
17
|
+
- **A retrieval quality benchmark.** Twenty queries (ten Chinese, ten English)
|
|
18
|
+
across documentation, releases, packages and news, each with expected domains.
|
|
19
|
+
`scripts/bench.mjs` records real engine output once, then replays it offline so
|
|
20
|
+
a ranking change is a number, not an opinion. Metrics: precision@8, recall@8,
|
|
21
|
+
MRR@8, per-engine contribution and consensus. Baseline 0.3625 / 0.7417 / 0.7600.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
- An anti-bot page served with HTTP 200 was parsed as "zero results", which
|
|
25
|
+
removed an engine silently while keeping its slot reserved and never engaging
|
|
26
|
+
its cooldown. Such pages now report `SEARCH_BLOCKED`, and 360 — walled from
|
|
27
|
+
this network through both the direct and tunnel paths — moved to the fallback
|
|
28
|
+
tier so it no longer slows every search.
|
|
29
|
+
|
|
30
|
+
### Measured, not assumed
|
|
31
|
+
- `rrfConstant` 40 scores a higher MRR (0.7767) than the default 60 (0.7600) and
|
|
32
|
+
a lower precision@8 (0.3500 vs 0.3625); `rrfConstant` 120 flips that trade in
|
|
33
|
+
the other direction. Twenty queries and one-to-four-hit gaps are not enough to
|
|
34
|
+
justify changing a default, so 60 stays and the numbers are in `bench/`.
|
|
35
|
+
- Dropping authority weighting costs 0.08 precision@8 and breaks Chinese
|
|
36
|
+
results, which is why it remains on by default.
|
|
37
|
+
|
|
38
|
+
## 0.13.0
|
|
39
|
+
|
|
40
|
+
### Added
|
|
41
|
+
- **Optional JS rendering.** A page whose HTML is an empty shell can be rendered
|
|
42
|
+
by a real browser — but only if `playwright` or `playwright-core` is already
|
|
43
|
+
installed. Nothing is bundled, nothing is imported at load time, and a missing
|
|
44
|
+
module is a silent fallback, not an error. `renderJs` picks `auto` (render only
|
|
45
|
+
suspected shells), `always` or `off`; `renderTimeoutMs` bounds the attempt. A
|
|
46
|
+
render that fails, times out or is aborted returns the earlier rung's result.
|
|
47
|
+
- **A retrieval quality benchmark.** Twenty queries (ten Chinese, ten English)
|
|
48
|
+
across documentation, releases, packages and news, each with expected domains.
|
|
49
|
+
`scripts/bench.mjs` records real engine output once, then replays it offline so
|
|
50
|
+
a ranking change is a number, not an opinion. Metrics: precision@8, recall@8,
|
|
51
|
+
MRR@8, per-engine contribution and consensus. Baseline 0.3625 / 0.7417 / 0.7600.
|
|
52
|
+
|
|
53
|
+
### Fixed
|
|
54
|
+
- An anti-bot page served with HTTP 200 was parsed as "zero results", which
|
|
55
|
+
removed an engine silently while keeping its slot reserved and never engaging
|
|
56
|
+
its cooldown. Such pages now report `SEARCH_BLOCKED`, and 360 — walled from
|
|
57
|
+
this network through both the direct and tunnel paths — moved to the fallback
|
|
58
|
+
tier so it no longer slows every search.
|
|
59
|
+
|
|
60
|
+
## 0.12.0
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
- **Fetched pages come back as article Markdown**, not raw markup. Extraction
|
|
64
|
+
picks the article root, prunes page furniture, renders headings, lists, code
|
|
65
|
+
fences, tables, entities and absolute links, and cuts at `fetchMaxLength` with
|
|
66
|
+
a notice naming the original length. `fetchFormat` can return plain text.
|
|
67
|
+
- **`search_many`** answers one to five queries in a single call with a
|
|
68
|
+
concurrency cap of three and per-query error isolation.
|
|
69
|
+
- **The cache survives a restart.** A versioned snapshot keeps summaries only,
|
|
70
|
+
is written atomically, caps itself at 4 MiB with LRU eviction, and treats a
|
|
71
|
+
corrupt or unreadable file as an empty cache. `cachePersist` and `cacheFile`
|
|
72
|
+
control it.
|
|
73
|
+
|
|
74
|
+
### Fixed
|
|
75
|
+
- `cacheTtlMs`, `cacheEntries`, `cachePersist` and `cacheFile` are read live, so
|
|
76
|
+
a Settings edit takes effect without a remount, like every other volatile
|
|
77
|
+
field.
|
|
78
|
+
|
|
79
|
+
- `fetchFormat` / `fetchMaxLength` are resolved per request, so the Settings page
|
|
80
|
+
no longer needs a reload for them either; only the cache file path and the
|
|
81
|
+
persist switch are read at mount.
|
|
82
|
+
|
|
83
|
+
### Known limits
|
|
84
|
+
- The host fetch request carries only a url, so a page cannot be resumed at an
|
|
85
|
+
offset: raise `fetchMaxLength` or fetch a narrower page.
|
|
86
|
+
- Two processes writing the same cache file is last-writer-wins.
|
|
87
|
+
|
|
3
88
|
## 0.11.0
|
|
4
89
|
|
|
5
90
|
### Added
|