@fffangchenghao/dsh-web-search 0.0.0-stage → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,317 @@
1
+ # Changelog
2
+
3
+ ## 0.11.0
4
+
5
+ ### Added
6
+ - **`domain_search`, a second tool for the sources a web search cannot reach.**
7
+ Eight keyless adapters — GitHub, npm, PyPI, arXiv, Wikipedia, Hacker News,
8
+ Stack Exchange, crates.io — each with its own timeout, each failing alone:
9
+ one dead source never empties the answer, and its name comes back in
10
+ `failures`.
11
+ - **`timeRange`, and publication dates behind it.** Parsers now read dates from
12
+ Bing, 360, the DuckDuckGo timestamp and the AnySearch content trailer;
13
+ DuckDuckGo receives the window as `df=`, the rest are filtered locally, and
14
+ `freshnessBoost` / `freshnessDays` finally score something real instead of
15
+ sitting on a missing field.
16
+
17
+ ### Changed
18
+ - Undated results no longer survive a time window by default. A window keeps the
19
+ dated rows inside it; only when nothing dated falls inside does the undated
20
+ list come back, so "the last day" is not answered with pages of unknown age.
21
+ - The Settings card has twenty controls in four groups, including a new
22
+ dedicated-search group.
23
+
24
+ ### Known limits
25
+ - PyPI publishes no search API, so its source matches the exact package name
26
+ taken from the first word of the query.
27
+ - Wikipedia answers in one language: Chinese/Japanese/Korean questions use zh,
28
+ everything else en.
29
+ - Bing and 360 accept time-filter parameters that the pages display as selected
30
+ while returning the same results, so those two are filtered locally.
31
+
32
+ ## Unreleased
33
+
34
+ ### Added
35
+ - **`domain_search`: key-free vertical search for packages, papers, code, Q&A
36
+ and reference material.** Eight credential-free adapters — GitHub
37
+ repositories, npm, PyPI (exact package name, since PyPI has no public search
38
+ API), arXiv, Wikipedia (zh for CJK queries, en otherwise), Hacker News,
39
+ Stack Exchange and crates.io — run concurrently, interleave by rank and merge
40
+ into the same `{ url, title, snippet, publishedAt }` shape `web_search`
41
+ returns. One source failing, timing out or answering HTTP 500 is reported as
42
+ a failure line and never blocks the others; every request carries a
43
+ User-Agent and its own deadline.
44
+ - Three settings fields, all `.volatile()`: `domainSources` (enabled adapters),
45
+ `domainLimit` (results per source, default 5) and `domainTimeoutMs` (per-source
46
+ deadline, default 8000 ms). The dedicated-sources group is editable from the
47
+ Settings page.
48
+ - The tool registers through the host's real `ToolRuntime.register(definition)`
49
+ API, read with `ctx.get('tools')` rather than `inject` so a deployment without
50
+ a tool registry still mounts the plugin and keeps `web_search`. If another
51
+ plugin owns `domain_search`, it registers as `web_domain_search`.
52
+ - `scripts/verify-tool-registration.mjs` proves the definition against a real
53
+ `ToolRuntime`; `scripts/smoke-verticals.mjs` is the live, non-CI smoke.
54
+ - Offline fixtures and tests for all eight parsers: normal and empty payloads,
55
+ HTTP 500, timeout on an injected fetch, and single-source failure isolation.
56
+
57
+ ## 0.10.0
58
+
59
+ ### Added
60
+ - **Both READMEs are rewritten for people, not for compilers.** A worked
61
+ before/after scene, a concepts section (what a model knows, what an agent is,
62
+ `web_search` versus `web_fetch` with a library analogy, messages and sessions,
63
+ providers, bundles), twelve capability lines, eight FAQ entries, and the
64
+ technical material moved into an appendix.
65
+ - Every Settings control carries a plain-language label and a one-line hint; the
66
+ hint line existed in the card but no dictionary entry ever filled it, and the
67
+ AnySearch fields had no label at all.
68
+
69
+ ### Fixed
70
+ - The two array fields no longer claim to be read-only: they are editable from
71
+ the card as comma separated values.
72
+
73
+ ## 0.9.1
74
+
75
+ ### Fixed
76
+ - **The AnySearch fields were missing from the card.** The client renders the
77
+ controls it knows about from its own field list, and `anysearchApiKey` /
78
+ `anysearchDomain` were never added there, so the API key had no entry — the
79
+ form showed 14 controls where the schema has 16.
80
+ - `max_results` asked AnySearch for 20 while the service caps it at 10 and
81
+ silently returns 10, so the constant now states the real ceiling.
82
+
83
+ ### Changed
84
+ - Every configuration description is in Chinese, and the API key field says what
85
+ an empty value means: anonymous access at a lower rate limit.
86
+ - New [docs/plugin-settings-form.md](docs/plugin-settings-form.md): the four-layer
87
+ chain that decides whether a plugin's configuration reaches Settings, with
88
+ file:line evidence and the four wrong diagnoses this plugin went through.
89
+
90
+ ## 0.9.0
91
+
92
+ ### Added
93
+ - **The settings form works.** All sixteen fields are live-editable from
94
+ Settings, because each one is marked `.volatile()` and the plugin reads the
95
+ current value out of the holder the schema hands it.
96
+
97
+ ### Fixed
98
+ - **`resolveConfig` validated an already-parsed config.** The Loader validates
99
+ a row's config once and hands `apply` the parsed result, in which every
100
+ volatile field is a `{ get }` holder. Running that parsed value back through
101
+ `Config()` re-validated the holders as values and threw
102
+ `$.timeoutMs expected number but got [object Object]`, so the plugin died
103
+ inside `apply` — which the Host reports as "1 entry did not activate". The
104
+ previous three attempts all failed on this same line, which is why marking
105
+ fields, attaching `Config` to the runtime and re-installing each looked
106
+ plausible and each changed nothing.
107
+ - Search re-reads the configuration on every call, so a volatile-only change
108
+ from Settings takes effect without remounting the plugin.
109
+
110
+ All notable changes to `dsh-web-search-bing` are recorded here. The format
111
+ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
112
+ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
113
+
114
+ Version dates were not recorded in this repository's history, so the entries
115
+ below are ordered but undated; only the ordering is authoritative.
116
+
117
+ ## 0.8.0 (unreleased)
118
+
119
+ The fetch half, and AnySearch's paid-tier capabilities.
120
+
121
+ ### Added
122
+
123
+ - **A fetch provider (`http-proxy`), so `web_fetch` works on a restricted
124
+ network.** Measured here: a direct request to `github.com` or
125
+ `html.duckduckgo.com` times out while the same URL answers through the local
126
+ tunnel. The provider tries a direct request first, falls back to the tunnel
127
+ the search side already resolves, and finally — when an AnySearch key is
128
+ configured — to AnySearch's extraction endpoint, which renders the page
129
+ server-side and so also rescues a page that answers 403 to a bare request.
130
+ Select it with `fetchProvider: http-proxy` in the `web` service config.
131
+ - `anysearchApiKey`: sent as a Bearer token to `api.anysearch.com` only. It
132
+ raises the rate limit and unlocks extraction; empty keeps anonymous access.
133
+ Keep it in the profile, not in the bundle: the shipped default is empty.
134
+ - `anysearchDomain`: route the AnySearch engine at one of its vertical indexes
135
+ (`finance`, `academic`, `security`, `legal`, `code`). A dotted value such as
136
+ `finance.us_stock` selects the sub-domain.
137
+ - 9 tests for the fetch ladder and for the credential/domain resolution.
138
+
139
+ ### Fixed
140
+
141
+ - **The settings page rendered nothing: the Host never saw the schema.**
142
+ `dsh-settings` reads a plugin's schema from `entry.fiber.runtime.Config`, and
143
+ for a function-style plugin the runtime *is* the `apply` function, so a named
144
+ `Config` export was unreachable: the settings document dropped the row before
145
+ it looked at the schema at all. The module now exports the plugin as
146
+ `Object.assign(apply, { Config })`, which is what object-style plugins get from
147
+ `static Config`. `test/plugin-entry.test.ts` pins the shape.
148
+ - **The settings page rendered nothing: the configuration was not marked
149
+ live-editable.** `dsh-settings` keeps a namespace only when
150
+ `volatileForm(schema)` yields a form, and that helper keeps a namespace when a
151
+ field carries `meta.volatile`. None of the sixteen fields did, so the Host
152
+ dropped the row before the client ever asked for it — the plugin was loaded,
153
+ its schema was valid, and Settings still had nothing to show. Every field now
154
+ ends in `.volatile()`; `test/config-live.test.ts` restates the Host's filter
155
+ and fails if the mark is dropped again.
156
+ - **The settings page rendered nothing.** Two layers were wrong. The client
157
+ declared `slots` and `locale` but not `configForms`, so Cordis never injected
158
+ the service; and even with the service present, the page read its form once at
159
+ mount, while the Host projects a row's settings namespace asynchronously. The
160
+ page now subscribes to the settings mirror, derives the namespace from the
161
+ projection the Host actually serves (patch id, entry id, package name, then a
162
+ suffix match), and reads the controller for that namespace. When the service
163
+ is present but no served namespace matches, the page names the namespaces it
164
+ does see instead of showing a dead end.
165
+
166
+ ## 0.7.0 (unreleased)
167
+
168
+ The TypeScript rewrite. Behaviour is meant to be unchanged where it was
169
+ measured, and stricter where it was previously guesswork.
170
+
171
+ ### Added
172
+
173
+ - **Reciprocal rank fusion** (`src/rank.ts`): weight shapes *admission*, not
174
+ score. The per-engine lists are walked in a weighted round robin (weight 2
175
+ takes two consecutive rows per round, so it owns roughly twice the slots), and
176
+ the merged walk gives each candidate a 1-based `position`; a source then scores
177
+ `votes / (rrfConstant + position)`, where `votes` is the number of distinct
178
+ engines that returned it, so agreement compounds while one engine's weight does
179
+ not. Authority (`authorityDomains`, `authorityBoost`) and freshness
180
+ (`freshnessBoost`, `freshnessDays`) multiply that score. Ties break
181
+ deterministically on engine index, rank, then URL. Measured with weights 2/1/1
182
+ over lists of 10/10/4 at `limit: 8`, admission splits 4/2/2.
183
+ - **TTL + LRU result cache** (`src/cache.ts`): entries expire after
184
+ `cacheTtlMs` (default 10 minutes) and the least recently used entry is evicted
185
+ past `cacheEntries` (default 100). Keys normalise case and whitespace, and are
186
+ sensitive to `limit`, `wait`, lead region and the participating engine set.
187
+ - **Retry-After aware cooldown** (`src/health.ts`): the service's own wait hint
188
+ wins over the guessed duration; otherwise the cooldown doubles per consecutive
189
+ failure from `baseCooldownMs` up to `maxCooldownMs`. `SEARCH_PARSE` and
190
+ `SEARCH_ABORTED` never cool an engine down.
191
+ - **Typed configuration schema** (`src/config.ts`, schemastery): the loader
192
+ validates the mounted config before `apply` runs, so a bad value fails loudly
193
+ with its path instead of silently using a default.
194
+ - **Fixture-based parser tests** (`test/fixtures/`, `test/parsers.test.ts`):
195
+ captured responses for DuckDuckGo HTML, Bing, 360 and AnySearch, plus the six
196
+ parser regressions (Bing `u=a1` tracker, DuckDuckGo `uddg`, single-quoted
197
+ `class` attribute, 360 `data-mdurl`, AnySearch `code !== 0`, echoed-query
198
+ check).
199
+ - **CI** (`.github/workflows/ci.yml`): Node 24 on Ubuntu and macOS, running
200
+ `npm ci`, `npm run typecheck`, `npm test`, `npm run build`, and a check that the
201
+ build really emits the declared `lib/index.js` entry point.
202
+ - **Proxy transport ported to TypeScript** (`src/proxy.ts`) with tests for
203
+ `parseProxy`, `proxyFromEnv`, the `NO_PROXY` decision and the pure response
204
+ framer (status/headers/body, chunked, gzip, deflate, brotli). `ProxyResponse`
205
+ now carries the final hop's headers so the caller can read `Retry-After`.
206
+
207
+ ### Changed
208
+
209
+ - All implementation moved from `index.js`/`engines.js`/`proxy.js` to
210
+ `src/*.ts`, compiled to `lib/` by `npm run build`; `package.json` now points
211
+ `main` and `types` at `lib/`.
212
+ - The merge is RRF in every case, replacing the fixed round-robin interleave.
213
+ The bilingual region policy still runs first: the region the query language
214
+ belongs to (`global` for ASCII, `cn` for CJK, `both` for either) fills the
215
+ budget, and the other region only appends when it came up short.
216
+ - Cooldown durations are configurable instead of the previous hardcoded five
217
+ minutes, and the failure list says how long an engine will rest and why.
218
+
219
+ ### Fixed
220
+
221
+ - The `wait` setting now reaches the engine run. It was documented, validated and
222
+ offered by the settings card, but `resolveConfig` never carried it into
223
+ `search`, so every call silently used `auto`.
224
+ - A partially specified config no longer throws on activation: `resolveConfig`
225
+ runs its input through the schema first, so a row written as `config: {}` (or
226
+ edited by hand into a fragment) resolves to the documented defaults instead of
227
+ dereferencing an absent key.
228
+ - Weight no longer multiplies every row of an engine. The previous rule let
229
+ AnySearch (weight 2) outscore every other engine at every rank, so a CJK query
230
+ returned 8/8 AnySearch sources and no Chinese results at all; the measured
231
+ split is now 4/2/2 for weights 2/1/1.
232
+ - A `POST` that meets a `3xx` other than `307`/`308` is downgraded to a bodyless
233
+ `GET`; `307`/`308` preserve the method and body (AnySearch needs both paths).
234
+ - A tunnelled response's `Retry-After` is read from the final hop instead of
235
+ being discarded.
236
+
237
+ ## 0.6.0
238
+
239
+ ### Changed
240
+
241
+ - AnySearch now leads CJK queries too (`region: both`), not only ASCII ones.
242
+ A/B measured on the same query set: with domestic engines alone,
243
+ `如何配置 postgres 逻辑复制` was answered with a Baidu Baike page for the
244
+ dictionary word "如何" and a song-lyric page; adding AnySearch returned Cloud
245
+ SQL documentation, a cnblogs walkthrough, the AWS specification guide, Azure
246
+ documentation and a Tencent Cloud article. `Python GIL 是什么` went from
247
+ python.org's home page, its download page and a tutorial site — none of which
248
+ mention the GIL — to explanations of the GIL. `DeepSeek Harness 插件开发` went
249
+ from front pages and an App Store listing to hands-on tutorials.
250
+ - The cost of that is visible: a Chinese query now takes 1.4–2.3 s instead of
251
+ 0.2–0.7 s with domestic engines alone.
252
+
253
+ ## 0.5.0
254
+
255
+ ### Added
256
+
257
+ - **AnySearch anonymous JSON API engine** (`POST api.anysearch.com/v1/search`
258
+ with `{ query, max_results }`, parsed from its `{ code, data: { results } }`
259
+ envelope). It is reachable without a tunnel and is the most precise engine, so
260
+ it takes a weight of 2.
261
+ - Weighted interleave for merging per-engine lists, so a high-precision engine
262
+ holds more slots without silencing the others.
263
+ - `www.`-aware dedupe: `http(s)://www.` and a trailing `/` are the same
264
+ document.
265
+
266
+ ### Changed
267
+
268
+ - Bing global demoted from primary to fallback. Measured on stopword-heavy
269
+ English queries it matched a single token: `how to set up postgres logical
270
+ replication` returned jigsaw-puzzle sites for "set", while the other engines
271
+ stayed on topic. It is worth reaching only when nothing else answered.
272
+
273
+ ## 0.4.0
274
+
275
+ ### Added
276
+
277
+ - **Bilingual region policy**: engines declare `global`, `cn` or `both`; the
278
+ region matching the query script leads the merged list and the other region
279
+ only fills gaps.
280
+ - **`wait` policy**: `auto` (wait for the leading region), `all` (wait for every
281
+ engine) and `fast` (return on the first answer), each with a soft deadline.
282
+ - **Per-engine cooldown** after a block page, so a site that has decided this
283
+ client is a robot is not re-triggered on every search.
284
+ - **Echoed-query check**: a result page must contain one query token of at least
285
+ three characters, otherwise it is a rate-limit or "trending" page belonging to
286
+ somebody else's search and is rejected.
287
+ - DuckDuckGo HTML parser with per-result snippets, alongside the lite endpoint.
288
+
289
+ ## 0.3.0
290
+
291
+ ### Fixed
292
+
293
+ - 360 result URLs: the real target lives in the anchor's `data-mdurl`, not in
294
+ the click-tracker `href`; entries without it are skipped instead of emitting a
295
+ redirect as a source.
296
+ - Single-quoted attributes (`class='result-link'`) are parsed, not only double
297
+ quoted ones.
298
+
299
+ ## 0.2.0
300
+
301
+ ### Fixed
302
+
303
+ - Bing click-tracker links (`/ck/a?...&u=a1<base64url>`) are decoded to the real
304
+ target. The `&` arrives HTML-escaped as `&amp;`, so the value is unescaped
305
+ before the payload is looked for.
306
+ - DuckDuckGo outbound links (`/l/?uddg=<percent-encoded>`) are unwrapped.
307
+ - HTML entities in titles and snippets are decoded, and numeric entities outside
308
+ the known table are dropped rather than leaked into a snippet.
309
+
310
+ ## 0.1.0
311
+
312
+ ### Added
313
+
314
+ - Initial provider: scraped 360 and Bing result pages, registered with the
315
+ `ctx.web` seam as provider id `bing`, returning `{ url, title, snippet }` with
316
+ no API key and no credential.
317
+
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 fangchenghao
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.