@fffangchenghao/dsh-web-search 0.10.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +82 -0
- package/README.md +247 -41
- package/README.zh-CN.md +247 -41
- package/client.js +66 -9
- package/lib/cache-persist.d.ts +96 -0
- package/lib/cache-persist.js +249 -0
- package/lib/cache.d.ts +56 -8
- package/lib/cache.js +141 -13
- package/lib/config.d.ts +33 -1
- package/lib/config.js +54 -2
- package/lib/domain-tool.d.ts +119 -0
- package/lib/domain-tool.js +196 -0
- package/lib/engines.js +68 -10
- package/lib/fetch.d.ts +7 -0
- package/lib/fetch.js +36 -3
- package/lib/freshness.d.ts +102 -0
- package/lib/freshness.js +359 -0
- package/lib/index.d.ts +44 -2
- package/lib/index.js +144 -23
- package/lib/markdown.d.ts +84 -0
- package/lib/markdown.js +847 -0
- package/lib/parsers.js +59 -5
- package/lib/rank.d.ts +14 -1
- package/lib/rank.js +18 -0
- package/lib/search-many.d.ts +117 -0
- package/lib/search-many.js +286 -0
- package/lib/transport.d.ts +4 -2
- package/lib/transport.js +4 -2
- package/lib/types.d.ts +22 -6
- package/lib/verticals.d.ts +164 -0
- package/lib/verticals.js +539 -0
- package/package.json +4 -4
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,87 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.12.0
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- **Fetched pages come back as article Markdown**, not raw markup. Extraction
|
|
7
|
+
picks the article root, prunes page furniture, renders headings, lists, code
|
|
8
|
+
fences, tables, entities and absolute links, and cuts at `fetchMaxLength` with
|
|
9
|
+
a notice naming the original length. `fetchFormat` can return plain text.
|
|
10
|
+
- **`search_many`** answers one to five queries in a single call with a
|
|
11
|
+
concurrency cap of three and per-query error isolation.
|
|
12
|
+
- **The cache survives a restart.** A versioned snapshot keeps summaries only,
|
|
13
|
+
is written atomically, caps itself at 4 MiB with LRU eviction, and treats a
|
|
14
|
+
corrupt or unreadable file as an empty cache. `cachePersist` and `cacheFile`
|
|
15
|
+
control it.
|
|
16
|
+
|
|
17
|
+
### Fixed
|
|
18
|
+
- `cacheTtlMs`, `cacheEntries`, `cachePersist` and `cacheFile` are read live, so
|
|
19
|
+
a Settings edit takes effect without a remount, like every other volatile
|
|
20
|
+
field.
|
|
21
|
+
|
|
22
|
+
- `fetchFormat` / `fetchMaxLength` are resolved per request, so the Settings page
|
|
23
|
+
no longer needs a reload for them either; only the cache file path and the
|
|
24
|
+
persist switch are read at mount.
|
|
25
|
+
|
|
26
|
+
### Known limits
|
|
27
|
+
- The host fetch request carries only a url, so a page cannot be resumed at an
|
|
28
|
+
offset: raise `fetchMaxLength` or fetch a narrower page.
|
|
29
|
+
- Two processes writing the same cache file is last-writer-wins.
|
|
30
|
+
|
|
31
|
+
## 0.11.0
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
- **`domain_search`, a second tool for the sources a web search cannot reach.**
|
|
35
|
+
Eight keyless adapters — GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News,
|
|
36
|
+
Stack Exchange, crates.io — each with its own timeout, each failing alone:
|
|
37
|
+
one dead source never empties the answer, and its name comes back in
|
|
38
|
+
`failures`.
|
|
39
|
+
- **`timeRange`, and publication dates behind it.** Parsers now read dates from
|
|
40
|
+
Bing, 360, the DuckDuckGo timestamp and the AnySearch content trailer;
|
|
41
|
+
DuckDuckGo receives the window as `df=`, the rest are filtered locally, and
|
|
42
|
+
`freshnessBoost` / `freshnessDays` finally score something real instead of
|
|
43
|
+
sitting on a missing field.
|
|
44
|
+
|
|
45
|
+
### Changed
|
|
46
|
+
- Undated results no longer survive a time window by default. A window keeps the
|
|
47
|
+
dated rows inside it; only when nothing dated falls inside does the undated
|
|
48
|
+
list come back, so "the last day" is not answered with pages of unknown age.
|
|
49
|
+
- The Settings card has twenty controls in four groups, including a new
|
|
50
|
+
dedicated-search group.
|
|
51
|
+
|
|
52
|
+
### Known limits
|
|
53
|
+
- PyPI publishes no search API, so its source matches the exact package name
|
|
54
|
+
taken from the first word of the query.
|
|
55
|
+
- Wikipedia answers in one language: Chinese/Japanese/Korean questions use zh,
|
|
56
|
+
everything else en.
|
|
57
|
+
- Bing and 360 accept time-filter parameters that the pages display as selected
|
|
58
|
+
while returning the same results, so those two are filtered locally.
|
|
59
|
+
|
|
60
|
+
## Unreleased
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
- **`domain_search`: key-free vertical search for packages, papers, code, Q&A
|
|
64
|
+
and reference material.** Eight credential-free adapters — GitHub
|
|
65
|
+
repositories, npm, PyPI (exact package name, since PyPI has no public search
|
|
66
|
+
API), arXiv, Wikipedia (zh for CJK queries, en otherwise), Hacker News,
|
|
67
|
+
Stack Exchange and crates.io — run concurrently, interleave by rank and merge
|
|
68
|
+
into the same `{ url, title, snippet, publishedAt }` shape `web_search`
|
|
69
|
+
returns. One source failing, timing out or answering HTTP 500 is reported as
|
|
70
|
+
a failure line and never blocks the others; every request carries a
|
|
71
|
+
User-Agent and its own deadline.
|
|
72
|
+
- Three settings fields, all `.volatile()`: `domainSources` (enabled adapters),
|
|
73
|
+
`domainLimit` (results per source, default 5) and `domainTimeoutMs` (per-source
|
|
74
|
+
deadline, default 8000 ms). The dedicated-sources group is editable from the
|
|
75
|
+
Settings page.
|
|
76
|
+
- The tool registers through the host's real `ToolRuntime.register(definition)`
|
|
77
|
+
API, read with `ctx.get('tools')` rather than `inject` so a deployment without
|
|
78
|
+
a tool registry still mounts the plugin and keeps `web_search`. If another
|
|
79
|
+
plugin owns `domain_search`, it registers as `web_domain_search`.
|
|
80
|
+
- `scripts/verify-tool-registration.mjs` proves the definition against a real
|
|
81
|
+
`ToolRuntime`; `scripts/smoke-verticals.mjs` is the live, non-CI smoke.
|
|
82
|
+
- Offline fixtures and tests for all eight parsers: normal and empty payloads,
|
|
83
|
+
HTTP 500, timeout on an injected fetch, and single-source failure isolation.
|
|
84
|
+
|
|
3
85
|
## 0.10.0
|
|
4
86
|
|
|
5
87
|
### Added
|
package/README.md
CHANGED
|
@@ -5,18 +5,19 @@
|
|
|
5
5
|
- [中文说明](README.zh-CN.md)
|
|
6
6
|
- [Changelog](CHANGELOG.md) · [Why a plugin's settings go missing: the four-layer chain](docs/plugin-settings-form.md)
|
|
7
7
|
|
|
8
|
-
> Version 0.
|
|
8
|
+
> Version 0.12.0. What follows starts with what this means for you; the technical detail is collected in the appendix.
|
|
9
9
|
|
|
10
10
|
## In one sentence
|
|
11
11
|
|
|
12
|
-
With this plugin installed, you can ask something like "how do I configure the version released last week?" and the model can call `web_search` against public search engines by itself, then work from a ranked list of titles, links and snippets. When a snippet is not enough, it can call `web_fetch` on one of those links and
|
|
12
|
+
With this plugin installed, you can ask something like "how do I configure the version released last week?" and the model can call `web_search` against public search engines by itself, then work from a ranked list of titles, links and snippets. When a snippet is not enough, it can call `web_fetch` on one of those links and get the page back as clean Markdown instead of a wall of HTML. When one question needs several separate searches, it can call `search_many` and get each query's results separately. When the question is about an npm package, a GitHub repository, a paper or a Q&A thread, it can call `domain_search` instead and ask those dedicated sources directly.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Three things change for you:
|
|
15
15
|
|
|
16
16
|
- When a question involves "search for", "latest", "official docs" or "the docs", the model has a real chance to look it up instead of answering "my knowledge has a cutoff date".
|
|
17
|
+
- When a question is about a package, a repository or a paper, the model can ask the matching source directly instead of hoping a web search understands.
|
|
17
18
|
- You do not need a search API key, and you do not pay for search separately.
|
|
18
19
|
|
|
19
|
-
Under the hood it is one sentence too: the plugin registers two providers with the harness's `web` service — `bing` for search and `http-proxy` for fetching pages.
|
|
20
|
+
Under the hood it is one sentence too: the plugin registers two providers with the harness's `web` service — `bing` for search and `http-proxy` for fetching pages — and adds two more tools: `domain_search` for packages, repositories, papers, Q&A and reference entries, and `search_many` for several searches in one call. These names come up again below.
|
|
20
21
|
|
|
21
22
|
## The problem it solves
|
|
22
23
|
|
|
@@ -34,7 +35,7 @@ You upgraded a library. A config field was renamed. You ask the model:
|
|
|
34
35
|
|
|
35
36
|
**With this plugin**:
|
|
36
37
|
|
|
37
|
-
1. The model calls `web_search`. The plugin queries several engines at once, drops duplicate URLs, re-ranks what is left by cross-engine agreement, official-documentation status and freshness, and returns roughly eight candidates;
|
|
38
|
+
1. The model calls `web_search`. The plugin queries several engines at once, drops duplicate URLs, re-ranks what is left by cross-engine agreement, official-documentation status and freshness, and returns roughly eight candidates (fewer with a time window, which keeps only results inside it);
|
|
38
39
|
2. It notices that one result is on `kubernetes.io`, calls `web_fetch`, and reads that page;
|
|
39
40
|
3. It answers from the page text and can hand you the link.
|
|
40
41
|
|
|
@@ -46,6 +47,7 @@ You upgraded a library. A config field was renamed. You ask the model:
|
|
|
46
47
|
| You paste a link and say "read this" | A direct request that times out (`github.com`, for example) is the end of it | Direct first, then the local proxy, then AnySearch extraction |
|
|
47
48
|
| Where results come from | One backend, depending on credentials | Up to six sources, cross-checked and de-duplicated |
|
|
48
49
|
| A question in Chinese | Dictionary and encyclopedia pages creep in | Chinese sources lead, without losing English documentation |
|
|
50
|
+
| A question about an npm package, a repository or a paper | A web search at best, often landing on second-hand pages | `domain_search` queries the matching source for the package, repository or paper entry |
|
|
49
51
|
| An engine starts throttling | Every call fails and retries | That engine rests for 5–30 minutes; the others carry on |
|
|
50
52
|
|
|
51
53
|
### What it does not do
|
|
@@ -73,12 +75,12 @@ A **tool** is a fixed capability handed to the agent: search, read a page, run a
|
|
|
73
75
|
This is the pair people mix up most. They solve different problems:
|
|
74
76
|
|
|
75
77
|
- **`web_search` finds URLs.** Give it keywords; it returns a list with titles, links and a line or two of snippet.
|
|
76
|
-
- **`web_fetch` reads a page.** Give it one URL; it brings back that page's content.
|
|
78
|
+
- **`web_fetch` reads a page.** Give it one URL; it brings back that page's content, cleaned into readable Markdown.
|
|
77
79
|
|
|
78
80
|
| | `web_search` | `web_fetch` |
|
|
79
81
|
| --- | --- | --- |
|
|
80
82
|
| You give it | A few keywords | One URL (`http://` or `https://`) |
|
|
81
|
-
| It gives you |
|
|
83
|
+
| It gives you | Eight candidates by default: title, link, snippet (fewer with a time window) | The page body, cleaned into Markdown (or plain text) |
|
|
82
84
|
| Can it answer the question | Only from snippets, often not enough | Yes — the text is right there |
|
|
83
85
|
| When to use it | You don't know where to look, or want several sources | You know which page to read |
|
|
84
86
|
| Which provider | `bing` (this plugin) | `http-proxy` (this plugin) |
|
|
@@ -92,6 +94,28 @@ In everyday terms:
|
|
|
92
94
|
|
|
93
95
|
You need both. Search without fetch leaves the model reading blurbs forever. Fetch without search requires a URL — and the model usually does not have one.
|
|
94
96
|
|
|
97
|
+
What fetch hands back is the **article**, not the page's furniture: navigation, ads, scripts, footers and forms are stripped, and headings, lists, code and tables survive as Markdown.
|
|
98
|
+
|
|
99
|
+
### Dedicated search (domain_search): sources, not pages
|
|
100
|
+
|
|
101
|
+
`web_search` asks "what is on the open web". `domain_search` asks something narrower: "does this package exist, is this repository maintained, is there a paper on this, has someone answered this?"
|
|
102
|
+
|
|
103
|
+
It queries eight public sources directly: GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News, Stack Exchange and crates.io. Each offers a keyless public endpoint and returns structured records — title, link, snippet — rather than a page of search results.
|
|
104
|
+
|
|
105
|
+
Analogy: `web_search` is asking passers-by where the hardware shop is and getting directions; `domain_search` is walking to the counter and quoting a part number. Ask the right shop and the answer is short.
|
|
106
|
+
|
|
107
|
+
| | `web_search` | `domain_search` |
|
|
108
|
+
| --- | --- | --- |
|
|
109
|
+
| Good for | Concepts, news, official sites, tutorials | Packages, repositories, papers, Q&A, reference entries |
|
|
110
|
+
| Where results come from | Public search-engine result pages | Each source's own public endpoint |
|
|
111
|
+
| How many at once | Eight by default, de-duplicated and ranked | Five per source by default, interleaved by source |
|
|
112
|
+
| Needs a key | No | No |
|
|
113
|
+
| When one fails | The error names each engine's reason | The dead source goes into `failures`; the rest answer |
|
|
114
|
+
|
|
115
|
+
They team up well: use `domain_search` to find the package or paper, then `web_fetch` to open its homepage or README and read the detail.
|
|
116
|
+
|
|
117
|
+
One more helper sits beside them: **`search_many`**. When one question needs several distinct searches — one per candidate library, say — it runs one to five queries in a single call and keeps each query's results apart, so one failed query cannot empty the batch.
|
|
118
|
+
|
|
95
119
|
### DeepSeek messages and sessions
|
|
96
120
|
|
|
97
121
|
The model has no memory of its own. On every request, the harness resends the **whole sequence of messages**: the system prompt, what you said earlier, what the model answered, and the results of tools. That sequence is the messages array; one continuous conversation is stored in a **session**.
|
|
@@ -151,7 +175,7 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
151
175
|
|
|
152
176
|
4. **Ranking looks beyond position — at official status and freshness.**
|
|
153
177
|
Domains such as `kubernetes.io`, `developer.mozilla.org` and `rfc-editor.org` get a boost, as does anything published within 30 days.
|
|
154
|
-
*Technical: `score = votes / (rrfConstant + position)`, then `× (1 + authorityBoost)` and a freshness factor; defaults `rrfConstant: 60`, `authorityBoost: 0.25`, `freshnessBoost: 0.2`, `freshnessDays: 30`.
|
|
178
|
+
*Technical: `score = votes / (rrfConstant + position)`, then `× (1 + authorityBoost)` and a freshness factor; defaults `rrfConstant: 60`, `authorityBoost: 0.25`, `freshnessBoost: 0.2`, `freshnessDays: 30`. Publication dates are now parsed from each engine's results, so the freshness term scores a real date instead of sitting idle.*
|
|
155
179
|
|
|
156
180
|
5. **Chinese and English are handled separately.**
|
|
157
181
|
English queries lead with global sources, Chinese queries with Chinese sources, so "how to …" is not answered with a dictionary page for the word `how`.
|
|
@@ -163,28 +187,48 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
163
187
|
|
|
164
188
|
7. **Repeating a question does not repeat the work.**
|
|
165
189
|
The same query within ten minutes is answered from the previous run, almost instantly.
|
|
166
|
-
*Technical: TTL + LRU cache, `cacheTtlMs: 600000` (ten minutes) and `cacheEntries: 100`; the key is `lead|wait|limit|sorted engine ids|normalised query
|
|
190
|
+
*Technical: TTL + LRU cache, `cacheTtlMs: 600000` (ten minutes) and `cacheEntries: 100`; the key is `lead|wait|limit|sorted engine ids|time window|normalised query`, where the window only appears when it is not `off`, so an unfiltered search keeps its old cache behaviour.*
|
|
167
191
|
|
|
168
192
|
8. **A throttled engine takes a rest.**
|
|
169
193
|
If one engine serves a verification page, it is skipped for a few minutes while the rest keep working.
|
|
170
194
|
*Technical: `baseCooldownMs: 300000`, doubling per consecutive failure up to `maxCooldownMs: 1800000`; the service's own `Retry-After` wins; parse failures and cancellations never cool an engine down.*
|
|
171
195
|
|
|
172
196
|
9. **It can read the page, not just the snippet.**
|
|
173
|
-
Search returns links; `web_fetch` returns text, and reroutes when a direct request fails.
|
|
174
|
-
*Technical: a three-rung ladder — direct (5 s cap) → local tunnel → AnySearch extraction (needs a key);
|
|
197
|
+
Search returns links; `web_fetch` returns readable text, and reroutes when a direct request fails.
|
|
198
|
+
*Technical: a three-rung ladder — direct (5 s cap) → local tunnel → AnySearch extraction (needs a key); the raw body keeps at most 1 MiB (`1048576` bytes), which is then converted (item 10) and cut to `fetchMaxLength`.*
|
|
199
|
+
|
|
200
|
+
10. **A fetched page comes back as readable Markdown, not raw HTML.**
|
|
201
|
+
Navigation, scripts, footers and forms are dropped; headings, paragraphs, lists, code blocks, quotes, links, images and tables stay — tables become Markdown tables. A page longer than the default 20,000 characters ends with one line naming the original length.
|
|
202
|
+
*Technical: `fetchFormat` (`markdown` by default, or `text`) and `fetchMaxLength` (default `20000`, range 1000–200000). Measured on `docs.python.org/3/library/urllib.parse.html`: 119,614 characters of raw HTML became 40,891 characters of body Markdown, and the tool returned 20,080 characters plus one truncation line. The body is chosen from `<article>`, then `<main>`/`role=main`, then the densest text container; relative links become absolute. **No resume**: the host's `WebFetchRequest` carries only `url` — no offset, no maxLength, no format — so a page cannot be continued from the middle; raise `fetchMaxLength` or fetch a narrower page.*
|
|
175
203
|
|
|
176
|
-
|
|
204
|
+
11. **It works on a restricted network.**
|
|
177
205
|
With a tunnel running and set as the system proxy, nothing needs configuring. With the tunnel off, it falls back to direct requests instead of failing.
|
|
178
206
|
*Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then macOS `scutil --proxy`; HTTP proxies only (`CONNECT`), not `socks5://`.*
|
|
179
207
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
*Technical: all
|
|
208
|
+
12. **Every setting is editable in the UI.**
|
|
209
|
+
Twenty-four controls in five groups; a save takes effect on the next search, apart from the few the plugin reads once at mount.
|
|
210
|
+
*Technical: all twenty-four fields carry `.volatile()`, so the Host commits them in place instead of remounting the plugin; `cacheTtlMs` and `cacheEntries` are re-read per use, while the fetch provider takes `proxy`, `timeoutMs`, `fetchFormat` and `fetchMaxLength` at mount and the cache wiring takes `cachePersist` and `cacheFile` there too.*
|
|
183
211
|
|
|
184
|
-
|
|
212
|
+
13. **Failures are explained, not silently empty.**
|
|
185
213
|
When nothing succeeds, the error lists which engines were tried and why each one failed.
|
|
186
214
|
*Technical: failures are classified `SEARCH_THROTTLED`, `SEARCH_BLOCKED`, `SEARCH_TIMEOUT`, `SEARCH_NETWORK`, `SEARCH_PARSE`, `SEARCH_ABORTED`; they are return values, not exceptions — only caller cancellation throws.*
|
|
187
215
|
|
|
216
|
+
14. **Packages, repositories, papers and Q&A are one hop away.**
|
|
217
|
+
For "is this npm package still maintained?" or "is there a paper on RAG?", the model can call `domain_search` against eight dedicated sources instead of hoping a web search finds them.
|
|
218
|
+
*Technical: eight keyless sources — `github`, `npm`, `pypi`, `arxiv`, `wikipedia`, `hn` (Hacker News), `stackexchange`, `crates` (crates.io); arguments `query` (required), `sources` (optional array, default = every enabled source), `limit` (optional, default 5, capped at 20). Each source has its own deadline (8000 ms by default), one failure never affects the others, and the failed names come back in `failures`. Measured with `react` at limit 3: github 3, npm 3, pypi 1, arxiv 3, wikipedia 3, hn 3, stackexchange 3, crates 3. Known limits: PyPI has no public search API, so it matches the **exact package name** taken from the first word of the query; Wikipedia searches one language only (`zh` for CJK, otherwise `en`). A host without `ctx.tools` does not register the tool and the rest of the plugin works as usual.*
|
|
219
|
+
|
|
220
|
+
15. **You can look only at recently published results.**
|
|
221
|
+
Choose "last 24 hours / 7 days / 30 days / 1 year" in Settings and only results inside that window stay — useful for news and releases.
|
|
222
|
+
*Technical: `timeRange` takes `off` (default) / `day` / `week` / `month` / `year` — 24 hours, 7 days, 30 days, 1 year. Publication dates are parsed from each engine's results (Bing's snippet date and `news_dt`, 360's `发贴时间`, DuckDuckGo's timestamp and snippet, AnySearch's `date:` trailer). DuckDuckGo receives the window as `df=d/w/m/y`; Bing's and 360's parameters were measured to have no effect (the page shows the option selected while the results stay the same), so those two and AnySearch filter locally. With a window on, only **dated** results inside it are kept; only when not one dated result falls inside does the undated list come back, so the answer is never empty for that reason alone.*
|
|
223
|
+
|
|
224
|
+
16. **Several questions, one call.**
|
|
225
|
+
For "compare these three libraries", the model can hand `search_many` one to five queries and get each query's results back separately; up to three run at once, and one failed query does not take the others down.
|
|
226
|
+
*Technical: `queries` (1–5, required), `limit` (default 5, 1–20), `withSites` (default false; true attaches each result's hostname). `SEARCH_MANY_CONCURRENCY = 3`. Each query goes through the same engine layer, cache and cooldown tracker as `web_search`; a thrown search becomes that query's `failures` entry, never the batch's. Use `search_many` for several questions at once, `web_search` for one, `domain_search` for packages, papers or Q&A. The tool's deadline is 30 s and it is concurrency-safe. It registers as `web_search_many` if the name is taken, and is not registered at all when the host has no `ctx.tools`.*
|
|
227
|
+
|
|
228
|
+
17. **The cache survives a restart.**
|
|
229
|
+
Ask the same question again after restarting DSH within ten minutes and the previous result is still there; nothing is sent to the engines.
|
|
230
|
+
*Technical: `cachePersist` (default `true`) and `cacheFile` (default empty = `~/.dsh/cache/dsh-web-search/cache.json`). Only the summary — `url` / `title` / `snippet` / `publishedAt` — is written, never page bodies. 4 MiB default budget, LRU eviction, atomic write (sibling temporary file + `rename`); a corrupt, foreign or unreadable file loads as an empty cache and never stops the plugin. Two processes writing the same file is last-writer-wins, and a file is never left half-written. `cacheTtlMs` and `cacheEntries` apply immediately, no restart.*
|
|
231
|
+
|
|
188
232
|
## Installing and enabling it
|
|
189
233
|
|
|
190
234
|
### What you need
|
|
@@ -260,6 +304,9 @@ Optionally, add a config row:
|
|
|
260
304
|
proxy: "" # empty means auto-detect
|
|
261
305
|
anysearchApiKey: "" # empty means anonymous access
|
|
262
306
|
anysearchDomain: "" # or finance / academic / security / legal / code
|
|
307
|
+
timeRange: off # off / day / week / month / year
|
|
308
|
+
domainLimit: 5 # results per domain_search source
|
|
309
|
+
domainTimeoutMs: 8000
|
|
263
310
|
```
|
|
264
311
|
|
|
265
312
|
**Where the key goes**: into the profile's `cordis.patch.yml`, never into the repository. The bundle's default config is empty so that publishing it cannot carry your credential along.
|
|
@@ -268,13 +315,13 @@ Optionally, add a config row:
|
|
|
268
315
|
|
|
269
316
|
After changing code or profile config, **quit DSH entirely and open it again** (on macOS, quit from the menu or press `Cmd+Q`; closing the window is not enough). Modules are evaluated once per process, and a running process never sees a file you edited underneath it.
|
|
270
317
|
|
|
271
|
-
Values saved from the Settings page
|
|
318
|
+
Values saved from the Settings page apply immediately; only `cachePersist` and `cacheFile` need the plugin to reload.
|
|
272
319
|
|
|
273
320
|
### Step 5: confirm it works
|
|
274
321
|
|
|
275
322
|
Any one of these:
|
|
276
323
|
|
|
277
|
-
1. **Look at Settings**: Settings → "Web search (Bing)" should show
|
|
324
|
+
1. **Look at Settings**: Settings → "Web search (Bing)" should show five groups of controls, 24 in all. That means the plugin loaded and the Host accepted its schema.
|
|
278
325
|
2. **Look at the log**: no warning such as `1 entry did not activate`.
|
|
279
326
|
3. **Just try it**: ask something that obviously needs the network, such as "search for a DSH plugin tutorial and give me the links". Watch whether `web_search` is called and whether the links are real.
|
|
280
327
|
|
|
@@ -282,13 +329,15 @@ If the Settings card appears but search still fails, start with [docs/troublesho
|
|
|
282
329
|
|
|
283
330
|
## Using the Settings page
|
|
284
331
|
|
|
285
|
-
Open Settings → "Web search (Bing)".
|
|
332
|
+
Open Settings → "Web search (Bing)". Twenty-four controls, arranged in five groups. Each group owns one kind of decision:
|
|
286
333
|
|
|
287
334
|
| Group | Controls | What it decides |
|
|
288
335
|
| --- | --- | --- |
|
|
289
336
|
| Engines and transport | 6 | Which engines run, how long to wait, whether to use a proxy, AnySearch credentials and vertical domain |
|
|
290
|
-
| Results and ranking |
|
|
291
|
-
|
|
|
337
|
+
| Results and ranking | 7 | How many results, how far back to look, how they merge and sort, how much official docs and fresh content count |
|
|
338
|
+
| Dedicated search | 3 | Which sources `domain_search` queries, results per source, deadline per source |
|
|
339
|
+
| Fetching | 2 | What `web_fetch` returns — Markdown or plain text — and how many characters |
|
|
340
|
+
| Cache and cooldowns | 6 | How long results are remembered, how many, whether the cache is saved to disk, and how long a throttled engine rests |
|
|
292
341
|
|
|
293
342
|
### Engines and transport (6)
|
|
294
343
|
|
|
@@ -301,39 +350,61 @@ Open Settings → "Web search (Bing)". Sixteen controls, arranged in three group
|
|
|
301
350
|
| AnySearch key | `anysearchApiKey` | empty (anonymous) | Optional. Sent only to `api.anysearch.com`; raises limits and unlocks page extraction |
|
|
302
351
|
| Vertical domain | `anysearchDomain` | empty (general) | Typical values: `finance` / `academic` / `security` / `legal` / `code`; a dotted value such as `finance.us_stock` selects a sub-domain |
|
|
303
352
|
|
|
304
|
-
### Results and ranking (
|
|
353
|
+
### Results and ranking (7)
|
|
305
354
|
|
|
306
355
|
| Control | Key | Default | Plain meaning |
|
|
307
356
|
| --- | --- | --- | --- |
|
|
308
357
|
| Results per search | `limit` | `8` | How many results at most, 1–20; a smaller caller budget still wins |
|
|
358
|
+
| Date window | `timeRange` | `off` | `off` disables it; `day` 24 hours, `week` 7 days, `month` 30 days, `year` 1 year. A window keeps only results inside it |
|
|
309
359
|
| Result blending | `rrfConstant` | `60` | Larger flattens the curve; smaller favours each engine's own top results |
|
|
310
360
|
| Trusted sites | `authorityDomains` | empty (built-in list) | Extra domains to boost, on top of the built-in official-docs list |
|
|
311
361
|
| Trusted-site boost | `authorityBoost` | `0.25` | Multiplies an authoritative result by `1 + this`, 0–10 |
|
|
312
362
|
| Freshness boost | `freshnessBoost` | `0.2` | Boost for newly published content, 0–10, fading linearly to zero |
|
|
313
363
|
| Freshness window (days) | `freshnessDays` | `30` | Age beyond which the freshness boost is gone |
|
|
314
364
|
|
|
315
|
-
###
|
|
365
|
+
### Dedicated search (3)
|
|
316
366
|
|
|
317
367
|
| Control | Key | Default | Plain meaning |
|
|
318
368
|
| --- | --- | --- | --- |
|
|
319
|
-
|
|
|
320
|
-
|
|
|
369
|
+
| Sources | `domainSources` | all eight | Keep only some of `github`, `npm`, `pypi`, `arxiv`, `wikipedia`, `hn`, `stackexchange`, `crates`, comma-separated; empty means all |
|
|
370
|
+
| Results per source | `domainLimit` | `5` | How many results `domain_search` takes from each source, 1–20 |
|
|
371
|
+
| Deadline per source | `domainTimeoutMs` | `8000` | Per-source deadline in ms, at least 500; raise it to 15000 on a slow network |
|
|
372
|
+
|
|
373
|
+
### Fetching (2)
|
|
374
|
+
|
|
375
|
+
| Control | Key | Default | Plain meaning |
|
|
376
|
+
| --- | --- | --- | --- |
|
|
377
|
+
| Output format | `fetchFormat` | `markdown` | `markdown` keeps headings, lists, tables, links and code; `text` keeps only the words. Leave it on `markdown` for reading; choose `text` for a smaller, plainer result. Applies immediately |
|
|
378
|
+
| Characters kept | `fetchMaxLength` | `20000` | How much converted text one fetch returns, 1000–200000; the cut is announced with the original length. Raise it to `50000` for long documentation. Applies immediately |
|
|
379
|
+
|
|
380
|
+
### Cache and cooldowns (6)
|
|
381
|
+
|
|
382
|
+
| Control | Key | Default | Plain meaning |
|
|
383
|
+
| --- | --- | --- | --- |
|
|
384
|
+
| Cache lifetime | `cacheTtlMs` | `600000` | Ten minutes; `0` disables the cache. A change applies to the next search |
|
|
385
|
+
| Cache size | `cacheEntries` | `100` | How many queries to remember; the least recently used goes first. A change applies to the next search |
|
|
386
|
+
| Save the cache to disk | `cachePersist` | `true` | Keep the cache in a file so a restart can still hit it; turn it off to leave no trace on disk or when the disk is read-only. Applies immediately |
|
|
387
|
+
| Cache file | `cacheFile` | empty (default path) | Where the snapshot goes; empty means `~/.dsh/cache/dsh-web-search/cache.json`. Read when the plugin mounts |
|
|
321
388
|
| Cooldown (ms) | `baseCooldownMs` | `300000` | First rest after throttling, five minutes |
|
|
322
389
|
| Max cooldown (ms) | `maxCooldownMs` | `1800000` | Ceiling for the doubling backoff, thirty minutes |
|
|
323
390
|
|
|
324
391
|
### My advice
|
|
325
392
|
|
|
326
|
-
**For normal use: leave all
|
|
393
|
+
**For normal use: leave all twenty-four alone. The defaults are the tuned values.**
|
|
327
394
|
|
|
328
|
-
Only
|
|
395
|
+
Only these situations are worth changing:
|
|
329
396
|
|
|
330
397
|
- **Your proxy is not detected**: put the local proxy address in `proxy`, for example `http://127.0.0.1:7890`. HTTP proxies only — swap `socks5://` for the client's HTTP/mixed port.
|
|
331
398
|
- **You want AnySearch's higher limits or page extraction**: fill in `anysearchApiKey`, and add `anysearchDomain` if you want one vertical index.
|
|
399
|
+
- **You only want `domain_search` to hit a few sources**: narrow `domainSources` to a set such as `github,npm`; empty means all eight.
|
|
400
|
+
- **A fetched page is cut too early, or you want plain text**: raise `fetchMaxLength` to `50000`, or set `fetchFormat` to `text`.
|
|
401
|
+
- **You do not want a cache file on disk**: turn `cachePersist` off, or point `cacheFile` somewhere else.
|
|
332
402
|
|
|
333
|
-
|
|
403
|
+
Three things to keep in mind:
|
|
334
404
|
|
|
335
|
-
- A save is immediate and applies to the next search, no restart.
|
|
336
|
-
- **Ranking changes can be hidden by the cache**: the cache key holds `lead|wait|limit|engine set|query`, not the ranking knobs. Ask the same question within ten minutes and you get the old order. Rephrase it, or wait ten minutes.
|
|
405
|
+
- A save is immediate and applies to the next search, no restart — except the controls the plugin reads at mount (`fetchFormat`, `fetchMaxLength`, `cachePersist`, `cacheFile`), which need a restart.
|
|
406
|
+
- **Ranking changes can be hidden by the cache**: the cache key holds `lead|wait|limit|engine set|time window|query`, not the ranking knobs. Ask the same question within ten minutes and you get the old order — and with persistence on, restarting DSH does not clear it. Rephrase it, or wait ten minutes. Changing the window changes the key, so an old entry cannot interfere.
|
|
407
|
+
- **A page cannot be continued from the middle**: `web_fetch` makes one request per URL, so if the tail is missing, raise `fetchMaxLength` or fetch a narrower page.
|
|
337
408
|
|
|
338
409
|
## FAQ
|
|
339
410
|
|
|
@@ -361,6 +432,16 @@ Four usual causes:
|
|
|
361
432
|
3. **It needs a login or a subscription**: fetching carries no session of yours.
|
|
362
433
|
4. **Direct requests time out**: 5 seconds direct, then the tunnel; if neither works and no key is configured you get an error, and with a key there is one more rung, server-side extraction. The message names each rung's failure.
|
|
363
434
|
|
|
435
|
+
### Why does fetched content differ from what the browser shows?
|
|
436
|
+
|
|
437
|
+
Because `web_fetch` is not a browser:
|
|
438
|
+
|
|
439
|
+
- It keeps the **article**, not the page: navigation, ads, scripts, footers, forms and hidden nodes are dropped before conversion. Headings, paragraphs, lists, code, quotes, links, images and tables survive.
|
|
440
|
+
- It does **not run JavaScript**: a page that renders its body client-side comes back nearly empty, while the browser fills it in.
|
|
441
|
+
- It **cuts the tail**: converted text beyond `fetchMaxLength` (20000 characters by default) is replaced by one line naming the original length, so a long page stops early.
|
|
442
|
+
|
|
443
|
+
If the text stops too soon, raise `fetchMaxLength` (`50000` is a reasonable value). If what comes back is an empty shell, the page probably needs a browser — try its print view, a static mirror or the site's API instead.
|
|
444
|
+
|
|
364
445
|
### Why do the results differ from what I get searching by hand?
|
|
365
446
|
|
|
366
447
|
Because it is not the same operation:
|
|
@@ -371,11 +452,38 @@ Because it is not the same operation:
|
|
|
371
452
|
- It may serve a ten-minute-old cache, and an engine may be sitting out a cooldown;
|
|
372
453
|
- It knows nothing about your account, location or browsing history, so there is no personalisation.
|
|
373
454
|
|
|
455
|
+
### Why did turning on the time filter return fewer results?
|
|
456
|
+
|
|
457
|
+
Because a window removes results whose date falls outside it. With a window on, only **dated** results inside it are kept; undated results do not count, because nothing proves they are recent — a deliberate trade: better a few results than a page of unknown age posing as "the last 24 hours".
|
|
458
|
+
|
|
459
|
+
The one exception is the fallback: if not a single **dated** result falls inside the window, the undated results come back so you are not left empty-handed. So a windowed search returns somewhere between zero and eight results, not a fixed eight.
|
|
460
|
+
|
|
461
|
+
### Will the cache give me the same old answer forever?
|
|
462
|
+
|
|
463
|
+
No. Each entry expires `cacheTtlMs` after it was written — ten minutes by default — so a repeated question inside that window gets the old answer, and the next one goes back to the engines. Three things keep it bounded:
|
|
464
|
+
|
|
465
|
+
- `cacheTtlMs: 0` turns the cache off, and `cacheEntries` (default 100) caps how many answers are kept, evicting the least recently used;
|
|
466
|
+
- the disk snapshot stores **summaries only** (`url`, `title`, `snippet`, `publishedAt`), never page text;
|
|
467
|
+
- expiry is absolute: an entry reloaded from disk keeps its original deadline, and an expired one is dropped at load, so restarting DSH cannot extend an entry's life.
|
|
468
|
+
|
|
469
|
+
### `domain_search` or `web_search` — which one?
|
|
470
|
+
|
|
471
|
+
Depends on whether you want a **page** or an **entry**:
|
|
472
|
+
|
|
473
|
+
- Concepts, news, official sites, tutorials, error messages → `web_search`, which reads the open web;
|
|
474
|
+
- Packages, repositories, papers, Q&A, reference entries → `domain_search`, which asks GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News, Stack Exchange and crates.io directly.
|
|
475
|
+
|
|
476
|
+
They chain: `domain_search` finds the package or paper entry, then `web_fetch` opens its homepage or README for the detail.
|
|
477
|
+
|
|
478
|
+
### What is the fastest way to ask several questions?
|
|
479
|
+
|
|
480
|
+
Use `search_many`. One call carries one to five `queries`, runs three at a time, and returns each query's results separately; `limit` (default 5) caps results per query, and `withSites: true` adds each result's hostname. A failed query becomes its own `failures` line instead of ending the batch. Use `web_search` for a single question, and `domain_search` when the target is a package, a repository, a paper or a Q&A thread.
|
|
481
|
+
|
|
374
482
|
### I changed a setting, why did nothing happen?
|
|
375
483
|
|
|
376
484
|
Three cases:
|
|
377
485
|
|
|
378
|
-
- **Changed on the Settings page**: saved immediately and read on the next search. But if you ask the same question again within ten minutes you may hit the cache and see no difference — try rephrasing.
|
|
486
|
+
- **Changed on the Settings page**: saved immediately and read on the next search, except `fetchFormat`, `fetchMaxLength`, `cachePersist` and `cacheFile`, which are read when the plugin mounts. But if you ask the same question again within ten minutes you may hit the cache and see no difference — try rephrasing; with persistence on, a restart does not clear the cache either.
|
|
379
487
|
- **Changed the profile's `cordis.patch.yml`**: requires a full restart of the harness.
|
|
380
488
|
- **Changed `src/*.ts`**: requires `npm run build` and then a restart. Source edits without a build still run the old code in `lib/`.
|
|
381
489
|
|
|
@@ -385,7 +493,7 @@ A little, and it is usually worth it:
|
|
|
385
493
|
|
|
386
494
|
- A measured search takes 1.4–3.8 s (English waits a bit longer, Chinese is faster); a cache hit is nearly instant.
|
|
387
495
|
- The model is waiting for a tool result during that time, not hanging; if the deadline arrives, it gets whatever has already come in.
|
|
388
|
-
- The real cost is context: search results and page text both consume the model's input, so longer pages make the turn more expensive and leave less room for everything else. That is why search returns eight results by default and a page is capped at 1 MiB.
|
|
496
|
+
- The real cost is context: search results and page text both consume the model's input, so longer pages make the turn more expensive and leave less room for everything else. That is why search returns eight results by default and a fetched page is converted and then cut at 20,000 characters (the raw body is capped at 1 MiB first).
|
|
389
497
|
|
|
390
498
|
### Why does the model sometimes not search at all?
|
|
391
499
|
|
|
@@ -404,11 +512,13 @@ web_search call
|
|
|
404
512
|
→ filter by the hosts allow-list; drop viaProxy engines when no proxy exists
|
|
405
513
|
→ skip engines in cooldown (but never end up with none)
|
|
406
514
|
→ decide the leading region from the query script (ASCII → global, CJK → cn)
|
|
407
|
-
→ look up the cache (key: lead|wait|limit|sorted engine ids|normalised query)
|
|
515
|
+
→ look up the cache in memory, then the on-disk snapshot (key: lead|wait|limit|sorted engine ids|time window|normalised query)
|
|
408
516
|
→ run the primary tier concurrently, release on the first satisfied condition
|
|
409
517
|
→ if nothing came back, run the fallback tier
|
|
518
|
+
→ each engine filters locally by timeRange (DuckDuckGo also gets df pushdown)
|
|
410
519
|
→ fuse per region, rank with RRF, slice to limit
|
|
411
|
-
→
|
|
520
|
+
→ filter once more after the merge (undated rows return only when nothing dated is inside)
|
|
521
|
+
→ store in the cache, mark the snapshot dirty, return { sources, failures, proxy }
|
|
412
522
|
```
|
|
413
523
|
|
|
414
524
|
A tier releases on whichever comes first: every engine settled; every engine of the leading region settled (success or failure); enough sources from enough contributors; the soft deadline. Engines still running are aborted and reported as `still running at deadline` or `aborted once enough results had arrived`.
|
|
@@ -429,7 +539,29 @@ score(source) = votes / (rrfConstant + position)
|
|
|
429
539
|
- Freshness factor: `1 + freshnessBoost × (1 − ageDays / freshnessDays)`, and exactly 1 outside the window. An unparsable date also counts as 1 — it is not penalised.
|
|
430
540
|
- Ties break on engine order → rank within that engine → URL string, so identical input always produces an identical order.
|
|
431
541
|
|
|
432
|
-
**
|
|
542
|
+
**Freshness now scores real dates**: `publishedAt` is parsed from each engine's results, so `freshnessBoost` / `freshnessDays` are no longer idle. An unparsable or missing date counts as 1 — neither punished nor rewarded. Time-window filtering happens in the engine layer and again after the merge; ranking only scores.
|
|
543
|
+
|
|
544
|
+
### Publication dates and the time window
|
|
545
|
+
|
|
546
|
+
Each engine writes its date somewhere different; the parsers normalise all of them to ISO 8601 UTC:
|
|
547
|
+
|
|
548
|
+
| Engine | Where the date comes from |
|
|
549
|
+
| --- | --- |
|
|
550
|
+
| Bing | A date at the head of the snippet (`2025年9月25日`, `2 天之前`), or a `news_dt` timestamp element |
|
|
551
|
+
| 360 | Its own `发贴时间:2026年1月22日` stamp |
|
|
552
|
+
| DuckDuckGo | A `result-timestamp` / `result__timestamp` element, or a date in the snippet |
|
|
553
|
+
| AnySearch | The `date: 22 hours ago` trailer appended to the content |
|
|
554
|
+
|
|
555
|
+
Details:
|
|
556
|
+
|
|
557
|
+
- Every absolute date is built with `Date.UTC` and anchored at **noon UTC**, so one date is the same instant in Shanghai, Berlin and Los Angeles and cannot drift into the previous day through the machine's timezone.
|
|
558
|
+
- Relative stamps (`3 days ago`, `3 天前`, `yesterday`) are rolling rather than calendar subtraction, which avoids timezone and daylight-saving questions.
|
|
559
|
+
- The five `timeRange` values map to 24 hours, 7 days, 30 days and 365 days: `day` / `week` / `month` / `year`; `off` disables filtering.
|
|
560
|
+
- **Pushdown versus local filtering**: DuckDuckGo's `df=d/w/m/y` is a real parameter — measured to change the result set. Bing's and 360's time parameters were measured to have no effect (the page shows the option selected while the results stay the same), so those two, like AnySearch, filter locally on the parsed date.
|
|
561
|
+
- **What happens to undated results**: with a window on, only **dated** results inside it are kept; only when not one dated result falls inside does the undated list come back, so the answer is never empty for that reason alone. An unparsable date is treated exactly like a missing one, because "could not parse" is not evidence of age.
|
|
562
|
+
- A future date is not out of bounds: it is still inside the window, and ranking clamps its age to zero.
|
|
563
|
+
- The cache key carries an extra `timeRange` segment when the window is not `off`, so switching windows cannot hit an entry from another window; the `off` key is identical to the previous release.
|
|
564
|
+
- Measured: `latest AI news` returns 8 results with `off` (5 of them dated), and 5 with `day`, all dated.
|
|
433
565
|
|
|
434
566
|
### Cache and cooldowns
|
|
435
567
|
|
|
@@ -437,6 +569,17 @@ score(source) = votes / (rrfConstant + position)
|
|
|
437
569
|
|
|
438
570
|
**Cooldowns** exist because scraping public result pages is not a supported interface. `SEARCH_THROTTLED` (429), `SEARCH_BLOCKED` (403, other 4xx, a failed echoed-query check), `SEARCH_TIMEOUT` and `SEARCH_NETWORK` put an engine to rest. `SEARCH_PARSE` (upstream markup drift — retrying cannot help) and `SEARCH_ABORTED` (the caller cancelled) never do. The duration prefers the service's own `Retry-After` (delta seconds or an HTTP date, read from the final redirect hop); otherwise `baseCooldownMs × 2^(consecutive failures − 1)`, capped at `maxCooldownMs`. One success clears the streak. If every engine is cooling, the run still uses them, so the engine set is never empty.
|
|
439
571
|
|
|
572
|
+
### The cache on disk
|
|
573
|
+
|
|
574
|
+
`cachePersist` (default `true`) turns the file on; `cacheFile` defaults to empty, which resolves to `~/.dsh/cache/dsh-web-search/cache.json` (a leading `~` expands). The document is `{ version: 1, savedAt, entries: [{ key, expiresAt, sources }] }`, and each entry carries only the summary — `url`, `title`, `snippet`, `publishedAt`; any other property a source picked up is dropped at the serialisation boundary, and page bodies never reach the file.
|
|
575
|
+
|
|
576
|
+
- **Write**: the snapshot goes to a sibling temporary file (`<file>.<pid>.<n>.tmp`) and is then `rename`d over the target, which is atomic on one filesystem; a failure before the rename leaves the previous document intact and removes the temporary file. Writes are throttled (2 s) and also flushed on dispose and on process exit.
|
|
577
|
+
- **Read**: an absent, unreadable, malformed or foreign-`version` file, and any individually invalid entry, all collapse to an empty cache; loading never throws and never stops the plugin from mounting.
|
|
578
|
+
- **Bound**: the serialised document is trimmed in LRU order until it fits 4 MiB (`DEFAULT_CACHE_FILE_MAX_BYTES`), and `cacheEntries` still caps the in-memory map.
|
|
579
|
+
- **Expiry**: `expiresAt` is absolute, so an entry reloaded from disk keeps its original deadline; expired entries are dropped at load, and a restart cannot extend a cache entry's life.
|
|
580
|
+
- **Two processes**: last writer wins. One process may replace another's newer snapshot with an older one, but a half-written file is impossible.
|
|
581
|
+
- **Live reading**: `cacheTtlMs` and `cacheEntries` are functions read on every use, so a Settings edit applies to the next search; `cachePersist` and `cacheFile` decide the wiring at mount, so changing those two needs a restart.
|
|
582
|
+
|
|
440
583
|
### The three-rung web_fetch ladder
|
|
441
584
|
|
|
442
585
|
The `http-proxy` fetch provider tries three routes in order, each only after the previous one failed:
|
|
@@ -449,7 +592,53 @@ The `http-proxy` fetch provider tries three routes in order, each only after the
|
|
|
449
592
|
|
|
450
593
|
The direct deadline is deliberately shorter than the tunnel's: a blocked host usually fails by timing out, and spending the whole budget on a route that cannot work leaves the route that can with nothing.
|
|
451
594
|
|
|
452
|
-
Other details: only `http(s)` URLs are accepted;
|
|
595
|
+
Other details: only `http(s)` URLs are accepted; the raw body keeps at most `1 MiB` (`1048576` bytes) before conversion, the rest truncated and flagged `truncated`; a non-2xx response is a **result**, not a failure, and comes back with its status code and body; only when all three rungs fail does the provider throw, and the message names each rung's reason.
|
|
596
|
+
|
|
597
|
+
### From HTML to Markdown
|
|
598
|
+
|
|
599
|
+
The conversion is dependency-free (`src/markdown.ts`): a tolerant tokenizer and tree builder, because real pages are not XML; a pruning pass that drops chrome tags, comments and hidden nodes; a body-selection pass that prefers `<article>`, then `<main>` / `role=main`, then the densest text container; and a renderer for headings, paragraphs, lists, code, quotes, rules, links, images and tables.
|
|
600
|
+
|
|
601
|
+
- `markdown` keeps the syntax; `text` keeps only the words. Entities are decoded and relative URLs become absolute.
|
|
602
|
+
- Tables become Markdown tables; a table outside the chosen article root is dropped with the rest of the furniture.
|
|
603
|
+
- The character budget applies **after** conversion: `truncateFetchText` cuts at `fetchMaxLength`, trims trailing whitespace, avoids splitting a surrogate pair, and appends `[truncated at 20000 chars; original length 40891 chars; the rest is not shown]`.
|
|
604
|
+
- Measured on `docs.python.org/3/library/urllib.parse.html`: 119,614 characters of HTML → 40,891 characters of Markdown → 20,080 characters returned.
|
|
605
|
+
- The provider is built with `fetchFormat` and `fetchMaxLength` at mount, so changing either on the Settings page needs a restart; search-side settings are re-read per call.
|
|
606
|
+
- **No resume**: the host's `WebFetchRequest` has exactly one field, `url` — no offset, no maxLength, no format. A fetch therefore always starts at the top; the only remedies are raising `fetchMaxLength` or fetching a narrower page.
|
|
607
|
+
|
|
608
|
+
### The domain_search pipeline
|
|
609
|
+
|
|
610
|
+
```
|
|
611
|
+
domain_search call
|
|
612
|
+
→ read config (domainSources / domainLimit / domainTimeoutMs, re-read per call)
|
|
613
|
+
→ resolve the proxy (config → environment → system proxy)
|
|
614
|
+
→ sources default to the enabled set; limit clamped to 1–20, default domainLimit
|
|
615
|
+
→ eight sources in parallel GET, each with its own deadline (domainTimeoutMs)
|
|
616
|
+
→ interleave by source, slice to limit
|
|
617
|
+
→ return { sources, failures, truncated }
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
All eight are public, keyless endpoints: GitHub repository search, the npm registry, PyPI, arXiv Atom, the Wikipedia API, Hacker News (Algolia), the Stack Exchange API and crates.io. Each has its own parser; when upstream markup drifts, that source degrades to an empty answer instead of failing the run. A timeout, a 5xx or an unreachable host on one source adds one `failures` line and leaves the others answering. GitHub and crates.io require a `User-Agent`, which every request carries.
|
|
621
|
+
|
|
622
|
+
Two sources need a word of their own:
|
|
623
|
+
|
|
624
|
+
- **PyPI has no public search API**, so this is not full-text search: it takes the first word of the query as an **exact package name**, and a missing package is an empty answer, not a failure.
|
|
625
|
+
- **Wikipedia searches one language only**: a query containing CJK characters uses `zh`, everything else `en`.
|
|
626
|
+
|
|
627
|
+
Registration goes through the host's real tool registry, read with `ctx.get('tools')` rather than `inject` — an inject entry is a hard dependency, so a deployment without a tool service would fail to mount the plugin and `web_search` would disappear with it. When the registry is absent the tool is not registered and the rest of the plugin works as usual. If another plugin already owns the name, it registers as `web_domain_search`.
|
|
628
|
+
|
|
629
|
+
### The search_many pipeline
|
|
630
|
+
|
|
631
|
+
```
|
|
632
|
+
search_many call
|
|
633
|
+
→ queries 1–5 (required), limit default 5 (valid 1–20), withSites default false
|
|
634
|
+
→ mapLimit over the queries with SEARCH_MANY_CONCURRENCY = 3
|
|
635
|
+
→ each query: the same engine layer, cache and cooldown tracker as web_search
|
|
636
|
+
→ a thrown query becomes { sources: [], failures: [message] }; the batch continues
|
|
637
|
+
→ results keep input order; withSites adds the lower-cased hostname as `site`
|
|
638
|
+
→ return { results, truncated }
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
Each query sees the configured `limit` as its ceiling: a caller may ask for fewer results, never for more. The tool's own deadline is 30 s, it declares itself concurrency-safe, and it registers as `web_search_many` when another plugin owns `search_many`. Like `domain_search`, it is read from `ctx.get('tools')`: a host without a registry does not register it, and the rest of the plugin keeps working.
|
|
453
642
|
|
|
454
643
|
### Proxy
|
|
455
644
|
|
|
@@ -463,21 +652,38 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
|
|
|
463
652
|
- **Google is unreachable from this network.** A tunnelled `www.google.com/search` returns a JavaScript shell (94 KB, 1–3 links); headless Chrome is redirected to `google.com.hk` with a reCAPTCHA. `gbv=1`, a consent cookie and a mobile user agent all failed. Eight public SearXNG instances were unusable too (429/403/not JSON).
|
|
464
653
|
- **It depends on upstream HTML structure.** A redesign breaks a parser; that engine records a reason and yields to the next one rather than silently returning nothing.
|
|
465
654
|
- **Titles, links and snippets only** — no page text. That is `web_fetch`'s job.
|
|
466
|
-
- **
|
|
655
|
+
- **A fetched page cannot be resumed from an offset**: the host's `WebFetchRequest` carries only `url`, so text beyond `fetchMaxLength` needs a higher limit or a narrower page.
|
|
656
|
+
- **Extraction keeps only what looks like the article**: a page built entirely by JavaScript comes back nearly empty, and content outside the chosen article root — a sidebar table, say — is dropped with the navigation.
|
|
657
|
+
- **`search_many` is capped at five queries and three in flight.**
|
|
658
|
+
- **Two processes sharing one cache file write last-writer-wins**: neither can corrupt the file, but one process's newer snapshot can be replaced by another's older one.
|
|
659
|
+
- **A time window can only filter results that carry a date**: dates are parsed from result pages, so an engine that prints none leaves that row without an age; when nothing dated falls inside the window the undated list comes back. Bing's and 360's time parameters were measured to have no effect; only DuckDuckGo's `df=` is a real pushdown.
|
|
660
|
+
- **PyPI matches exact package names only**: it has no public search API, so the plugin uses the first word of the query; a typo yields an empty answer.
|
|
661
|
+
- **Wikipedia searches one language only**: `zh` for CJK queries, `en` otherwise.
|
|
662
|
+
- **`domain_search` needs a host tool registry**: without `ctx.tools` it is not registered, and `web_search` is unaffected.
|
|
467
663
|
- **Anonymous AnySearch caps at 10 results per call.**
|
|
468
664
|
- **HTTP proxies only** — no `socks5://`.
|
|
469
|
-
- **`limit` is capped at 20
|
|
665
|
+
- **`limit` is capped at 20, and so is `domainLimit`.**
|
|
470
666
|
|
|
471
667
|
### Defaults at a glance
|
|
472
668
|
|
|
473
669
|
| Item | Value |
|
|
474
670
|
| --- | --- |
|
|
475
671
|
| Result budget `limit` | `8` (1–20) |
|
|
672
|
+
| Time window `timeRange` | `off` (no limit; `day` / `week` / `month` / `year`) |
|
|
673
|
+
| Results per source `domainLimit` | `5` (1–20) |
|
|
674
|
+
| Per-source deadline `domainTimeoutMs` | `8000` ms |
|
|
476
675
|
| Cache TTL `cacheTtlMs` | `600000` ms = 600 s |
|
|
477
676
|
| Cache entries `cacheEntries` | `100` |
|
|
677
|
+
| Cache persisted `cachePersist` | `true` |
|
|
678
|
+
| Cache file `cacheFile` | empty = `~/.dsh/cache/dsh-web-search/cache.json` |
|
|
679
|
+
| Persisted cache budget | `4194304` bytes = 4 MiB |
|
|
478
680
|
| Per-engine timeout `timeoutMs` | `15000` ms |
|
|
479
681
|
| `web_fetch` direct deadline | `5000` ms |
|
|
480
|
-
|
|
|
682
|
+
| Raw bytes kept per page | `1048576` (1 MiB) |
|
|
683
|
+
| Fetch format `fetchFormat` | `markdown` (or `text`) |
|
|
684
|
+
| Characters kept per fetch `fetchMaxLength` | `20000` (1000–200000) |
|
|
685
|
+
| `search_many` queries / concurrency | 1–5 / 3 |
|
|
686
|
+
| `search_many` limit | `5` (1–20) |
|
|
481
687
|
| RRF constant `rrfConstant` | `60` |
|
|
482
688
|
| Base cooldown `baseCooldownMs` | `300000` ms = 5 min |
|
|
483
689
|
| Max cooldown `maxCooldownMs` | `1800000` ms = 30 min |
|
|
@@ -485,14 +691,14 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
|
|
|
485
691
|
### Development and tests
|
|
486
692
|
|
|
487
693
|
```sh
|
|
488
|
-
node --test test/*.test.ts # the whole suite
|
|
694
|
+
node --test test/*.test.ts # the whole suite (283 tests)
|
|
489
695
|
node --test test/proxy.test.ts # one file
|
|
490
696
|
npm run typecheck # tsc --noEmit
|
|
491
697
|
npm run build # compile src/ to lib/
|
|
492
698
|
npm run check # typecheck + tests
|
|
493
699
|
```
|
|
494
700
|
|
|
495
|
-
The suite runs offline: parser tests replay captured fixtures from `test/fixtures
|
|
701
|
+
The suite runs offline — 283 tests: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for all eight dedicated sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, time-window tests pin the date with an injected `now`, and the newer suites cover Markdown extraction (`markdown.test.ts`), batch search (`search-many.test.ts`) and the disk snapshot (`cache-persist.test.ts`).
|
|
496
702
|
|
|
497
703
|
CI (`.github/workflows/ci.yml`) runs `npm ci`, `npm run typecheck`, `npm test` and `npm run build` on Node 24 for Ubuntu and macOS, then checks that the build emitted `lib/index.js` and `lib/index.d.ts` and that both `apply` and `Config` import.
|
|
498
704
|
|