@fffangchenghao/dsh-web-search 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,59 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.11.0
4
+
5
+ ### Added
6
+ - **`domain_search`, a second tool for the sources a web search cannot reach.**
7
+ Eight keyless adapters — GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News,
8
+ Stack Exchange, crates.io — each with its own timeout, each failing alone:
9
+ one dead source never empties the answer, and its name comes back in
10
+ `failures`.
11
+ - **`timeRange`, and publication dates behind it.** Parsers now read dates from
12
+ Bing, 360, the DuckDuckGo timestamp and the AnySearch content trailer;
13
+ DuckDuckGo receives the window as `df=`, the rest are filtered locally, and
14
+ `freshnessBoost` / `freshnessDays` finally score something real instead of
15
+ sitting on a missing field.
16
+
17
+ ### Changed
18
+ - Undated results no longer survive a time window by default. A window keeps the
19
+ dated rows inside it; only when nothing dated falls inside does the undated
20
+ list come back, so "the last day" is not answered with pages of unknown age.
21
+ - The Settings card has twenty controls in four groups, including a new
22
+ dedicated-search group.
23
+
24
+ ### Known limits
25
+ - PyPI publishes no search API, so its source matches the exact package name
26
+ taken from the first word of the query.
27
+ - Wikipedia answers in one language: Chinese/Japanese/Korean questions use zh,
28
+ everything else en.
29
+ - Bing and 360 accept time-filter parameters that the pages display as selected
30
+ while returning the same results, so those two are filtered locally.
31
+
32
+ ## Unreleased
33
+
34
+ ### Added
35
+ - **`domain_search`: key-free vertical search for packages, papers, code, Q&A
36
+ and reference material.** Eight credential-free adapters — GitHub
37
+ repositories, npm, PyPI (exact package name, since PyPI has no public search
38
+ API), arXiv, Wikipedia (zh for CJK queries, en otherwise), Hacker News,
39
+ Stack Exchange and crates.io — run concurrently, interleave by rank and merge
40
+ into the same `{ url, title, snippet, publishedAt }` shape `web_search`
41
+ returns. One source failing, timing out or answering HTTP 500 is reported as
42
+ a failure line and never blocks the others; every request carries a
43
+ User-Agent and its own deadline.
44
+ - Three settings fields, all `.volatile()`: `domainSources` (enabled adapters),
45
+ `domainLimit` (results per source, default 5) and `domainTimeoutMs` (per-source
46
+ deadline, default 8000 ms). The dedicated-sources group is editable from the
47
+ Settings page.
48
+ - The tool registers through the host's real `ToolRuntime.register(definition)`
49
+ API, read with `ctx.get('tools')` rather than `inject` so a deployment without
50
+ a tool registry still mounts the plugin and keeps `web_search`. If another
51
+ plugin owns `domain_search`, it registers as `web_domain_search`.
52
+ - `scripts/verify-tool-registration.mjs` proves the definition against a real
53
+ `ToolRuntime`; `scripts/smoke-verticals.mjs` is the live, non-CI smoke.
54
+ - Offline fixtures and tests for all eight parsers: normal and empty payloads,
55
+ HTTP 500, timeout on an injected fetch, and single-source failure isolation.
56
+
3
57
  ## 0.10.0
4
58
 
5
59
  ### Added
package/README.md CHANGED
@@ -5,18 +5,19 @@
5
5
  - [中文说明](README.zh-CN.md)
6
6
  - [Changelog](CHANGELOG.md) · [Why a plugin's settings go missing: the four-layer chain](docs/plugin-settings-form.md)
7
7
 
8
- > Version 0.10.0. What follows starts with what this means for you; the technical detail is collected in the appendix.
8
+ > Version 0.11.0. What follows starts with what this means for you; the technical detail is collected in the appendix.
9
9
 
10
10
  ## In one sentence
11
11
 
12
- With this plugin installed, you can ask something like "how do I configure the version released last week?" and the model can call `web_search` against public search engines by itself, then work from a ranked list of titles, links and snippets. When a snippet is not enough, it can call `web_fetch` on one of those links and pull the page text into the conversation.
12
+ With this plugin installed, you can ask something like "how do I configure the version released last week?" and the model can call `web_search` against public search engines by itself, then work from a ranked list of titles, links and snippets. When a snippet is not enough, it can call `web_fetch` on one of those links and pull the page text into the conversation. When the question is about an npm package, a GitHub repository, a paper or a Q&A thread, it can call `domain_search` instead and ask those dedicated sources directly.
13
13
 
14
- Two things change for you:
14
+ Three things change for you:
15
15
 
16
16
  - When a question involves "search for", "latest", "official docs" or "the docs", the model has a real chance to look it up instead of answering "my knowledge has a cutoff date".
17
+ - When a question is about a package, a repository or a paper, the model can ask the matching source directly instead of hoping a web search understands.
17
18
  - You do not need a search API key, and you do not pay for search separately.
18
19
 
19
- Under the hood it is one sentence too: the plugin registers two providers with the harness's `web` service — `bing` for search and `http-proxy` for fetching pages. Both names come up again below.
20
+ Under the hood it is one sentence too: the plugin registers two providers with the harness's `web` service — `bing` for search and `http-proxy` for fetching pages — and adds one more tool, `domain_search`, for packages, repositories, papers, Q&A and reference entries. These names come up again below.
20
21
 
21
22
  ## The problem it solves
22
23
 
@@ -34,7 +35,7 @@ You upgraded a library. A config field was renamed. You ask the model:
34
35
 
35
36
  **With this plugin**:
36
37
 
37
- 1. The model calls `web_search`. The plugin queries several engines at once, drops duplicate URLs, re-ranks what is left by cross-engine agreement, official-documentation status and freshness, and returns roughly eight candidates;
38
+ 1. The model calls `web_search`. The plugin queries several engines at once, drops duplicate URLs, re-ranks what is left by cross-engine agreement, official-documentation status and freshness, and returns roughly eight candidates (fewer with a time window, which keeps only results inside it);
38
39
  2. It notices that one result is on `kubernetes.io`, calls `web_fetch`, and reads that page;
39
40
  3. It answers from the page text and can hand you the link.
40
41
 
@@ -46,6 +47,7 @@ You upgraded a library. A config field was renamed. You ask the model:
46
47
  | You paste a link and say "read this" | A direct request that times out (`github.com`, for example) is the end of it | Direct first, then the local proxy, then AnySearch extraction |
47
48
  | Where results come from | One backend, depending on credentials | Up to six sources, cross-checked and de-duplicated |
48
49
  | A question in Chinese | Dictionary and encyclopedia pages creep in | Chinese sources lead, without losing English documentation |
50
+ | A question about an npm package, a repository or a paper | A web search at best, often landing on second-hand pages | `domain_search` queries the matching source for the package, repository or paper entry |
49
51
  | An engine starts throttling | Every call fails and retries | That engine rests for 5–30 minutes; the others carry on |
50
52
 
51
53
  ### What it does not do
@@ -78,7 +80,7 @@ This is the pair people mix up most. They solve different problems:
78
80
  | | `web_search` | `web_fetch` |
79
81
  | --- | --- | --- |
80
82
  | You give it | A few keywords | One URL (`http://` or `https://`) |
81
- | It gives you | Around eight candidates: title, link, snippet | The page body (HTML or plain text) |
83
+ | It gives you | Eight candidates by default: title, link, snippet (fewer with a time window) | The page body (HTML or plain text) |
82
84
  | Can it answer the question | Only from snippets, often not enough | Yes — the text is right there |
83
85
  | When to use it | You don't know where to look, or want several sources | You know which page to read |
84
86
  | Which provider | `bing` (this plugin) | `http-proxy` (this plugin) |
@@ -92,6 +94,24 @@ In everyday terms:
92
94
 
93
95
  You need both. Search without fetch leaves the model reading blurbs forever. Fetch without search requires a URL — and the model usually does not have one.
94
96
 
97
+ ### Dedicated search (domain_search): sources, not pages
98
+
99
+ `web_search` asks "what is on the open web". `domain_search` asks something narrower: "does this package exist, is this repository maintained, is there a paper on this, has someone answered this?"
100
+
101
+ It queries eight public sources directly: GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News, Stack Exchange and crates.io. Each offers a keyless public endpoint and returns structured records — title, link, snippet — rather than a page of search results.
102
+
103
+ Analogy: `web_search` is asking passers-by where the hardware shop is and getting directions; `domain_search` is walking to the counter and quoting a part number. Ask the right shop and the answer is short.
104
+
105
+ | | `web_search` | `domain_search` |
106
+ | --- | --- | --- |
107
+ | Good for | Concepts, news, official sites, tutorials | Packages, repositories, papers, Q&A, reference entries |
108
+ | Where results come from | Public search-engine result pages | Each source's own public endpoint |
109
+ | How many at once | Eight by default, de-duplicated and ranked | Five per source by default, interleaved by source |
110
+ | Needs a key | No | No |
111
+ | When one fails | The error names each engine's reason | The dead source goes into `failures`; the rest answer |
112
+
113
+ They team up well: use `domain_search` to find the package or paper, then `web_fetch` to open its homepage or README and read the detail.
114
+
95
115
  ### DeepSeek messages and sessions
96
116
 
97
117
  The model has no memory of its own. On every request, the harness resends the **whole sequence of messages**: the system prompt, what you said earlier, what the model answered, and the results of tools. That sequence is the messages array; one continuous conversation is stored in a **session**.
@@ -151,7 +171,7 @@ Each line is the plain version first, the technical note second. The technical n
151
171
 
152
172
  4. **Ranking looks beyond position — at official status and freshness.**
153
173
  Domains such as `kubernetes.io`, `developer.mozilla.org` and `rfc-editor.org` get a boost, as does anything published within 30 days.
154
- *Technical: `score = votes / (rrfConstant + position)`, then `× (1 + authorityBoost)` and a freshness factor; defaults `rrfConstant: 60`, `authorityBoost: 0.25`, `freshnessBoost: 0.2`, `freshnessDays: 30`. No engine supplies a publication date yet, so the freshness term is currently idle.*
174
+ *Technical: `score = votes / (rrfConstant + position)`, then `× (1 + authorityBoost)` and a freshness factor; defaults `rrfConstant: 60`, `authorityBoost: 0.25`, `freshnessBoost: 0.2`, `freshnessDays: 30`. Publication dates are now parsed from each engine's results, so the freshness term scores a real date instead of sitting idle.*
155
175
 
156
176
  5. **Chinese and English are handled separately.**
157
177
  English queries lead with global sources, Chinese queries with Chinese sources, so "how to …" is not answered with a dictionary page for the word `how`.
@@ -163,7 +183,7 @@ Each line is the plain version first, the technical note second. The technical n
163
183
 
164
184
  7. **Repeating a question does not repeat the work.**
165
185
  The same query within ten minutes is answered from the previous run, almost instantly.
166
- *Technical: TTL + LRU cache, `cacheTtlMs: 600000` (ten minutes) and `cacheEntries: 100`; the key is `lead|wait|limit|sorted engine ids|normalised query`.*
186
+ *Technical: TTL + LRU cache, `cacheTtlMs: 600000` (ten minutes) and `cacheEntries: 100`; the key is `lead|wait|limit|sorted engine ids|time window|normalised query`, where the window only appears when it is not `off`, so an unfiltered search keeps its old cache behaviour.*
167
187
 
168
188
  8. **A throttled engine takes a rest.**
169
189
  If one engine serves a verification page, it is skipped for a few minutes while the rest keep working.
@@ -178,13 +198,21 @@ Each line is the plain version first, the technical note second. The technical n
178
198
  *Technical: proxy sources in order — the `proxy` setting, then `DSH_WEB_SEARCH_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` / `HTTP_PROXY`, then macOS `scutil --proxy`; HTTP proxies only (`CONNECT`), not `socks5://`.*
179
199
 
180
200
  11. **Every setting is editable in the UI.**
181
- Sixteen controls in three groups; a save takes effect on the next search.
182
- *Technical: all sixteen fields carry `.volatile()`, so the Host commits them in place instead of remounting the plugin.*
201
+ Twenty controls in four groups; a save takes effect on the next search.
202
+ *Technical: all twenty fields carry `.volatile()`, so the Host commits them in place instead of remounting the plugin.*
183
203
 
184
204
  12. **Failures are explained, not silently empty.**
185
205
  When nothing succeeds, the error lists which engines were tried and why each one failed.
186
206
  *Technical: failures are classified `SEARCH_THROTTLED`, `SEARCH_BLOCKED`, `SEARCH_TIMEOUT`, `SEARCH_NETWORK`, `SEARCH_PARSE`, `SEARCH_ABORTED`; they are return values, not exceptions — only caller cancellation throws.*
187
207
 
208
+ 13. **Packages, repositories, papers and Q&A are one hop away.**
209
+ For "is this npm package still maintained?" or "is there a paper on RAG?", the model can call `domain_search` against eight dedicated sources instead of hoping a web search finds them.
210
+ *Technical: eight keyless sources — `github`, `npm`, `pypi`, `arxiv`, `wikipedia`, `hn` (Hacker News), `stackexchange`, `crates` (crates.io); arguments `query` (required), `sources` (optional array, default = every enabled source), `limit` (optional, default 5, capped at 20). Each source has its own deadline (8000 ms by default), one failure never affects the others, and the failed names come back in `failures`. Measured with `react` at limit 3: github 3, npm 3, pypi 1, arxiv 3, wikipedia 3, hn 3, stackexchange 3, crates 3. Known limits: PyPI has no public search API, so it matches the **exact package name** taken from the first word of the query; Wikipedia searches one language only (`zh` for CJK, otherwise `en`). A host without `ctx.tools` does not register the tool and the rest of the plugin works as usual.*
211
+
212
+ 14. **You can look only at recently published results.**
213
+ Choose "last 24 hours / 7 days / 30 days / 1 year" in Settings and only results inside that window stay — useful for news and releases.
214
+ *Technical: `timeRange` takes `off` (default) / `day` / `week` / `month` / `year` — 24 hours, 7 days, 30 days, 1 year. Publication dates are parsed from each engine's results (Bing's snippet date and `news_dt`, 360's `发贴时间`, DuckDuckGo's timestamp and snippet, AnySearch's `date:` trailer). DuckDuckGo receives the window as `df=d/w/m/y`; Bing's and 360's parameters were measured to have no effect (the page shows the option selected while the results stay the same), so those two and AnySearch filter locally. With a window on, only **dated** results inside it are kept; only when not one dated result falls inside does the undated list come back, so the answer is never empty for that reason alone.*
215
+
188
216
  ## Installing and enabling it
189
217
 
190
218
  ### What you need
@@ -260,6 +288,9 @@ Optionally, add a config row:
260
288
  proxy: "" # empty means auto-detect
261
289
  anysearchApiKey: "" # empty means anonymous access
262
290
  anysearchDomain: "" # or finance / academic / security / legal / code
291
+ timeRange: off # off / day / week / month / year
292
+ domainLimit: 5 # results per domain_search source
293
+ domainTimeoutMs: 8000
263
294
  ```
264
295
 
265
296
  **Where the key goes**: into the profile's `cordis.patch.yml`, never into the repository. The bundle's default config is empty so that publishing it cannot carry your credential along.
@@ -274,7 +305,7 @@ Values saved from the Settings page do not need a restart; see the next section.
274
305
 
275
306
  Any one of these:
276
307
 
277
- 1. **Look at Settings**: Settings → "Web search (Bing)" should show three groups of controls. That means the plugin loaded and the Host accepted its schema.
308
+ 1. **Look at Settings**: Settings → "Web search (Bing)" should show four groups of controls. That means the plugin loaded and the Host accepted its schema.
278
309
  2. **Look at the log**: no warning such as `1 entry did not activate`.
279
310
  3. **Just try it**: ask something that obviously needs the network, such as "search for a DSH plugin tutorial and give me the links". Watch whether `web_search` is called and whether the links are real.
280
311
 
@@ -282,12 +313,13 @@ If the Settings card appears but search still fails, start with [docs/troublesho
282
313
 
283
314
  ## Using the Settings page
284
315
 
285
- Open Settings → "Web search (Bing)". Sixteen controls, arranged in three groups. Each group owns one kind of decision:
316
+ Open Settings → "Web search (Bing)". Twenty controls, arranged in four groups. Each group owns one kind of decision:
286
317
 
287
318
  | Group | Controls | What it decides |
288
319
  | --- | --- | --- |
289
320
  | Engines and transport | 6 | Which engines run, how long to wait, whether to use a proxy, AnySearch credentials and vertical domain |
290
- | Results and ranking | 6 | How many results, how they merge and sort, how much official docs and fresh content count |
321
+ | Results and ranking | 7 | How many results, how far back to look, how they merge and sort, how much official docs and fresh content count |
322
+ | Dedicated search | 3 | Which sources `domain_search` queries, results per source, deadline per source |
291
323
  | Cache and cooldowns | 4 | How long results are remembered, how many, and how long a throttled engine rests |
292
324
 
293
325
  ### Engines and transport (6)
@@ -301,17 +333,26 @@ Open Settings → "Web search (Bing)". Sixteen controls, arranged in three group
301
333
  | AnySearch key | `anysearchApiKey` | empty (anonymous) | Optional. Sent only to `api.anysearch.com`; raises limits and unlocks page extraction |
302
334
  | Vertical domain | `anysearchDomain` | empty (general) | Typical values: `finance` / `academic` / `security` / `legal` / `code`; a dotted value such as `finance.us_stock` selects a sub-domain |
303
335
 
304
- ### Results and ranking (6)
336
+ ### Results and ranking (7)
305
337
 
306
338
  | Control | Key | Default | Plain meaning |
307
339
  | --- | --- | --- | --- |
308
340
  | Results per search | `limit` | `8` | How many results at most, 1–20; a smaller caller budget still wins |
341
+ | Date window | `timeRange` | `off` | `off` disables it; `day` 24 hours, `week` 7 days, `month` 30 days, `year` 1 year. A window keeps only results inside it |
309
342
  | Result blending | `rrfConstant` | `60` | Larger flattens the curve; smaller favours each engine's own top results |
310
343
  | Trusted sites | `authorityDomains` | empty (built-in list) | Extra domains to boost, on top of the built-in official-docs list |
311
344
  | Trusted-site boost | `authorityBoost` | `0.25` | Multiplies an authoritative result by `1 + this`, 0–10 |
312
345
  | Freshness boost | `freshnessBoost` | `0.2` | Boost for newly published content, 0–10, fading linearly to zero |
313
346
  | Freshness window (days) | `freshnessDays` | `30` | Age beyond which the freshness boost is gone |
314
347
 
348
+ ### Dedicated search (3)
349
+
350
+ | Control | Key | Default | Plain meaning |
351
+ | --- | --- | --- | --- |
352
+ | Sources | `domainSources` | all eight | Keep only some of `github`, `npm`, `pypi`, `arxiv`, `wikipedia`, `hn`, `stackexchange`, `crates`, comma-separated; empty means all |
353
+ | Results per source | `domainLimit` | `5` | How many results `domain_search` takes from each source, 1–20 |
354
+ | Deadline per source | `domainTimeoutMs` | `8000` | Per-source deadline in ms, at least 500; raise it to 15000 on a slow network |
355
+
315
356
  ### Cache and cooldowns (4)
316
357
 
317
358
  | Control | Key | Default | Plain meaning |
@@ -323,17 +364,18 @@ Open Settings → "Web search (Bing)". Sixteen controls, arranged in three group
323
364
 
324
365
  ### My advice
325
366
 
326
- **For normal use: leave all sixteen alone. The defaults are the tuned values.**
367
+ **For normal use: leave all twenty alone. The defaults are the tuned values.**
327
368
 
328
- Only two situations are worth changing:
369
+ Only three situations are worth changing:
329
370
 
330
371
  - **Your proxy is not detected**: put the local proxy address in `proxy`, for example `http://127.0.0.1:7890`. HTTP proxies only — swap `socks5://` for the client's HTTP/mixed port.
331
372
  - **You want AnySearch's higher limits or page extraction**: fill in `anysearchApiKey`, and add `anysearchDomain` if you want one vertical index.
373
+ - **You only want `domain_search` to hit a few sources**: narrow `domainSources` to a set such as `github,npm`; empty means all eight.
332
374
 
333
375
  Two things to keep in mind:
334
376
 
335
377
  - A save is immediate and applies to the next search, no restart.
336
- - **Ranking changes can be hidden by the cache**: the cache key holds `lead|wait|limit|engine set|query`, not the ranking knobs. Ask the same question within ten minutes and you get the old order. Rephrase it, or wait ten minutes.
378
+ - **Ranking changes can be hidden by the cache**: the cache key holds `lead|wait|limit|engine set|time window|query`, not the ranking knobs. Ask the same question within ten minutes and you get the old order. Rephrase it, or wait ten minutes. Changing the window changes the key, so an old entry cannot interfere.
337
379
 
338
380
  ## FAQ
339
381
 
@@ -371,6 +413,21 @@ Because it is not the same operation:
371
413
  - It may serve a ten-minute-old cache, and an engine may be sitting out a cooldown;
372
414
  - It knows nothing about your account, location or browsing history, so there is no personalisation.
373
415
 
416
+ ### Why did turning on the time filter return fewer results?
417
+
418
+ Because a window removes results whose date falls outside it. With a window on, only **dated** results inside it are kept; undated results do not count, because nothing proves they are recent — a deliberate trade: better a few results than a page of unknown age posing as "the last 24 hours".
419
+
420
+ The one exception is the fallback: if not a single **dated** result falls inside the window, the undated results come back so you are not left empty-handed. So a windowed search returns somewhere between zero and eight results, not a fixed eight.
421
+
422
+ ### `domain_search` or `web_search` — which one?
423
+
424
+ Depends on whether you want a **page** or an **entry**:
425
+
426
+ - Concepts, news, official sites, tutorials, error messages → `web_search`, which reads the open web;
427
+ - Packages, repositories, papers, Q&A, reference entries → `domain_search`, which asks GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News, Stack Exchange and crates.io directly.
428
+
429
+ They chain: `domain_search` finds the package or paper entry, then `web_fetch` opens its homepage or README for the detail.
430
+
374
431
  ### I changed a setting, why did nothing happen?
375
432
 
376
433
  Three cases:
@@ -404,10 +461,12 @@ web_search call
404
461
  → filter by the hosts allow-list; drop viaProxy engines when no proxy exists
405
462
  → skip engines in cooldown (but never end up with none)
406
463
  → decide the leading region from the query script (ASCII → global, CJK → cn)
407
- → look up the cache (key: lead|wait|limit|sorted engine ids|normalised query)
464
+ → look up the cache (key: lead|wait|limit|sorted engine ids|time window|normalised query)
408
465
  → run the primary tier concurrently, release on the first satisfied condition
409
466
  → if nothing came back, run the fallback tier
467
+ → each engine filters locally by timeRange (DuckDuckGo also gets df pushdown)
410
468
  → fuse per region, rank with RRF, slice to limit
469
+ → filter once more after the merge (undated rows return only when nothing dated is inside)
411
470
  → store in the cache, return { sources, failures, proxy }
412
471
  ```
413
472
 
@@ -429,7 +488,29 @@ score(source) = votes / (rrfConstant + position)
429
488
  - Freshness factor: `1 + freshnessBoost × (1 − ageDays / freshnessDays)`, and exactly 1 outside the window. An unparsable date also counts as 1 — it is not penalised.
430
489
  - Ties break on engine order → rank within that engine → URL string, so identical input always produces an identical order.
431
490
 
432
- **The current gap**: no engine emits `publishedAt` (result pages expose no stable date field), so the freshness boost is effectively idle. It will start working once an engine can supply a date.
491
+ **Freshness now scores real dates**: `publishedAt` is parsed from each engine's results, so `freshnessBoost` / `freshnessDays` are no longer idle. An unparsable or missing date counts as 1 — neither punished nor rewarded. Time-window filtering happens in the engine layer and again after the merge; ranking only scores.
492
+
493
+ ### Publication dates and the time window
494
+
495
+ Each engine writes its date somewhere different; the parsers normalise all of them to ISO 8601 UTC:
496
+
497
+ | Engine | Where the date comes from |
498
+ | --- | --- |
499
+ | Bing | A date at the head of the snippet (`2025年9月25日`, `2 天之前`), or a `news_dt` timestamp element |
500
+ | 360 | Its own `发贴时间:2026年1月22日` stamp |
501
+ | DuckDuckGo | A `result-timestamp` / `result__timestamp` element, or a date in the snippet |
502
+ | AnySearch | The `date: 22 hours ago` trailer appended to the content |
503
+
504
+ Details:
505
+
506
+ - Every absolute date is built with `Date.UTC` and anchored at **noon UTC**, so one date is the same instant in Shanghai, Berlin and Los Angeles and cannot drift into the previous day through the machine's timezone.
507
+ - Relative stamps (`3 days ago`, `3 天前`, `yesterday`) are rolling rather than calendar subtraction, which avoids timezone and daylight-saving questions.
508
+ - The five `timeRange` values map to 24 hours, 7 days, 30 days and 365 days: `day` / `week` / `month` / `year`; `off` disables filtering.
509
+ - **Pushdown versus local filtering**: DuckDuckGo's `df=d/w/m/y` is a real parameter — measured to change the result set. Bing's and 360's time parameters were measured to have no effect (the page shows the option selected while the results stay the same), so those two, like AnySearch, filter locally on the parsed date.
510
+ - **What happens to undated results**: with a window on, only **dated** results inside it are kept; only when not one dated result falls inside does the undated list come back, so the answer is never empty for that reason alone. An unparsable date is treated exactly like a missing one, because "could not parse" is not evidence of age.
511
+ - A future date is not out of bounds: it is still inside the window, and ranking clamps its age to zero.
512
+ - The cache key carries an extra `timeRange` segment when the window is not `off`, so switching windows cannot hit an entry from another window; the `off` key is identical to the previous release.
513
+ - Measured: `latest AI news` returns 8 results with `off` (5 of them dated), and 5 with `day`, all dated.
433
514
 
434
515
  ### Cache and cooldowns
435
516
 
@@ -451,6 +532,27 @@ The direct deadline is deliberately shorter than the tunnel's: a blocked host us
451
532
 
452
533
  Other details: only `http(s)` URLs are accepted; each page keeps at most `1 MiB` (`1048576` bytes), the rest truncated and flagged `truncated`; a non-2xx response is a **result**, not a failure, and comes back with its status code and body; only when all three rungs fail does the provider throw, and the message names each rung's reason.
453
534
 
535
+ ### The domain_search pipeline
536
+
537
+ ```
538
+ domain_search call
539
+ → read config (domainSources / domainLimit / domainTimeoutMs, re-read per call)
540
+ → resolve the proxy (config → environment → system proxy)
541
+ → sources default to the enabled set; limit clamped to 1–20, default domainLimit
542
+ → eight sources in parallel GET, each with its own deadline (domainTimeoutMs)
543
+ → interleave by source, slice to limit
544
+ → return { sources, failures, truncated }
545
+ ```
546
+
547
+ All eight are public, keyless endpoints: GitHub repository search, the npm registry, PyPI, arXiv Atom, the Wikipedia API, Hacker News (Algolia), the Stack Exchange API and crates.io. Each has its own parser; when upstream markup drifts, that source degrades to an empty answer instead of failing the run. A timeout, a 5xx or an unreachable host on one source adds one `failures` line and leaves the others answering. GitHub and crates.io require a `User-Agent`, which every request carries.
548
+
549
+ Two sources need a word of their own:
550
+
551
+ - **PyPI has no public search API**, so this is not full-text search: it takes the first word of the query as an **exact package name**, and a missing package is an empty answer, not a failure.
552
+ - **Wikipedia searches one language only**: a query containing CJK characters uses `zh`, everything else `en`.
553
+
554
+ Registration goes through the host's real tool registry, read with `ctx.get('tools')` rather than `inject` — an inject entry is a hard dependency, so a deployment without a tool service would fail to mount the plugin and `web_search` would disappear with it. When the registry is absent the tool is not registered and the rest of the plugin works as usual. If another plugin already owns the name, it registers as `web_domain_search`.
555
+
454
556
  ### Proxy
455
557
 
456
558
  Node's global `fetch` reads neither the macOS system proxy nor `HTTPS_PROXY` — unless the process started with `NODE_USE_ENV_PROXY=1`, which cannot be switched on from inside a running process. So when a proxy is needed, the plugin speaks to it directly: send `CONNECT`, upgrade with TLS for `https:` targets, write the request by hand, and decode `Content-Length`, chunked, gzip, deflate and brotli, following up to four redirects. `307`/`308` keep the method and body; any other `3xx` downgrades a `POST` to a bodyless `GET`.
@@ -463,16 +565,22 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
463
565
  - **Google is unreachable from this network.** A tunnelled `www.google.com/search` returns a JavaScript shell (94 KB, 1–3 links); headless Chrome is redirected to `google.com.hk` with a reCAPTCHA. `gbv=1`, a consent cookie and a mobile user agent all failed. Eight public SearXNG instances were unusable too (429/403/not JSON).
464
566
  - **It depends on upstream HTML structure.** A redesign breaks a parser; that engine records a reason and yields to the next one rather than silently returning nothing.
465
567
  - **Titles, links and snippets only** — no page text. That is `web_fetch`'s job.
466
- - **No publication date is produced today**, so the freshness boost sits idle.
568
+ - **A time window can only filter results that carry a date**: dates are parsed from result pages, so an engine that prints none leaves that row without an age; when nothing dated falls inside the window the undated list comes back. Bing's and 360's time parameters were measured to have no effect; only DuckDuckGo's `df=` is a real pushdown.
569
+ - **PyPI matches exact package names only**: it has no public search API, so the plugin uses the first word of the query; a typo yields an empty answer.
570
+ - **Wikipedia searches one language only**: `zh` for CJK queries, `en` otherwise.
571
+ - **`domain_search` needs a host tool registry**: without `ctx.tools` it is not registered, and `web_search` is unaffected.
467
572
  - **Anonymous AnySearch caps at 10 results per call.**
468
573
  - **HTTP proxies only** — no `socks5://`.
469
- - **`limit` is capped at 20.**
574
+ - **`limit` is capped at 20, and so is `domainLimit`.**
470
575
 
471
576
  ### Defaults at a glance
472
577
 
473
578
  | Item | Value |
474
579
  | --- | --- |
475
580
  | Result budget `limit` | `8` (1–20) |
581
+ | Time window `timeRange` | `off` (no limit; `day` / `week` / `month` / `year`) |
582
+ | Results per source `domainLimit` | `5` (1–20) |
583
+ | Per-source deadline `domainTimeoutMs` | `8000` ms |
476
584
  | Cache TTL `cacheTtlMs` | `600000` ms = 600 s |
477
585
  | Cache entries `cacheEntries` | `100` |
478
586
  | Per-engine timeout `timeoutMs` | `15000` ms |
@@ -485,14 +593,14 @@ Resolution order: the `proxy` setting → the environment (`DSH_WEB_SEARCH_PROXY
485
593
  ### Development and tests
486
594
 
487
595
  ```sh
488
- node --test test/*.test.ts # the whole suite
596
+ node --test test/*.test.ts # the whole suite (239 tests)
489
597
  node --test test/proxy.test.ts # one file
490
598
  npm run typecheck # tsc --noEmit
491
599
  npm run build # compile src/ to lib/
492
600
  npm run check # typecheck + tests
493
601
  ```
494
602
 
495
- The suite runs offline: parser tests replay captured fixtures from `test/fixtures/`, transport tests open no sockets, and proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer.
603
+ The suite runs offline — 239 tests: parser tests replay captured fixtures from `test/fixtures/` (normal and empty responses for all eight dedicated sources), transport tests open no sockets, proxy tests cover `parseProxy`, `proxyFromEnv` and the pure response framer, and time-window tests pin the date with an injected `now`.
496
604
 
497
605
  CI (`.github/workflows/ci.yml`) runs `npm ci`, `npm run typecheck`, `npm test` and `npm run build` on Node 24 for Ubuntu and macOS, then checks that the build emitted `lib/index.js` and `lib/index.d.ts` and that both `apply` and `Config` import.
498
606