@fffangchenghao/dsh-web-search 0.0.0-stage → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,263 @@
1
+ # Changelog
2
+
3
+ ## 0.10.0
4
+
5
+ ### Added
6
+ - **Both READMEs are rewritten for people, not for compilers.** A worked
7
+ before/after scene, a concepts section (what a model knows, what an agent is,
8
+ `web_search` versus `web_fetch` with a library analogy, messages and sessions,
9
+ providers, bundles), twelve capability lines, eight FAQ entries, and the
10
+ technical material moved into an appendix.
11
+ - Every Settings control carries a plain-language label and a one-line hint; the
12
+ hint line existed in the card but no dictionary entry ever filled it, and the
13
+ AnySearch fields had no label at all.
14
+
15
+ ### Fixed
16
+ - The two array fields no longer claim to be read-only: they are editable from
17
+ the card as comma separated values.
18
+
19
+ ## 0.9.1
20
+
21
+ ### Fixed
22
+ - **The AnySearch fields were missing from the card.** The client renders the
23
+ controls it knows about from its own field list, and `anysearchApiKey` /
24
+ `anysearchDomain` were never added there, so the API key had no entry — the
25
+ form showed 14 controls where the schema has 16.
26
+ - `max_results` asked AnySearch for 20 while the service caps it at 10 and
27
+ silently returns 10, so the constant now states the real ceiling.
28
+
29
+ ### Changed
30
+ - Every configuration description is in Chinese, and the API key field says what
31
+ an empty value means: anonymous access at a lower rate limit.
32
+ - New [docs/plugin-settings-form.md](docs/plugin-settings-form.md): the four-layer
33
+ chain that decides whether a plugin's configuration reaches Settings, with
34
+ file:line evidence and the four wrong diagnoses this plugin went through.
35
+
36
+ ## 0.9.0
37
+
38
+ ### Added
39
+ - **The settings form works.** All sixteen fields are live-editable from
40
+ Settings, because each one is marked `.volatile()` and the plugin reads the
41
+ current value out of the holder the schema hands it.
42
+
43
+ ### Fixed
44
+ - **`resolveConfig` validated an already-parsed config.** The Loader validates
45
+ a row's config once and hands `apply` the parsed result, in which every
46
+ volatile field is a `{ get }` holder. Running that parsed value back through
47
+ `Config()` re-validated the holders as values and threw
48
+ `$.timeoutMs expected number but got [object Object]`, so the plugin died
49
+ inside `apply` — which the Host reports as "1 entry did not activate". The
50
+ previous three attempts all failed on this same line, which is why marking
51
+ fields, attaching `Config` to the runtime and re-installing each looked
52
+ plausible and each changed nothing.
53
+ - Search re-reads the configuration on every call, so a volatile-only change
54
+ from Settings takes effect without remounting the plugin.
55
+
56
+ All notable changes to `dsh-web-search-bing` are recorded here. The format
57
+ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
58
+ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
59
+
60
+ Version dates were not recorded in this repository's history, so the entries
61
+ below are ordered but undated; only the ordering is authoritative.
62
+
63
+ ## 0.8.0 (unreleased)
64
+
65
+ The fetch half, and AnySearch's paid-tier capabilities.
66
+
67
+ ### Added
68
+
69
+ - **A fetch provider (`http-proxy`), so `web_fetch` works on a restricted
70
+ network.** Measured here: a direct request to `github.com` or
71
+ `html.duckduckgo.com` times out while the same URL answers through the local
72
+ tunnel. The provider tries a direct request first, falls back to the tunnel
73
+ the search side already resolves, and finally — when an AnySearch key is
74
+ configured — to AnySearch's extraction endpoint, which renders the page
75
+ server-side and so also rescues a page that answers 403 to a bare request.
76
+ Select it with `fetchProvider: http-proxy` in the `web` service config.
77
+ - `anysearchApiKey`: sent as a Bearer token to `api.anysearch.com` only. It
78
+ raises the rate limit and unlocks extraction; empty keeps anonymous access.
79
+ Keep it in the profile, not in the bundle: the shipped default is empty.
80
+ - `anysearchDomain`: route the AnySearch engine at one of its vertical indexes
81
+ (`finance`, `academic`, `security`, `legal`, `code`). A dotted value such as
82
+ `finance.us_stock` selects the sub-domain.
83
+ - 9 tests for the fetch ladder and for the credential/domain resolution.
84
+
85
+ ### Fixed
86
+
87
+ - **The settings page rendered nothing: the Host never saw the schema.**
88
+ `dsh-settings` reads a plugin's schema from `entry.fiber.runtime.Config`, and
89
+ for a function-style plugin the runtime *is* the `apply` function, so a named
90
+ `Config` export was unreachable: the settings document dropped the row before
91
+ it looked at the schema at all. The module now exports the plugin as
92
+ `Object.assign(apply, { Config })`, which is what object-style plugins get from
93
+ `static Config`. `test/plugin-entry.test.ts` pins the shape.
94
+ - **The settings page rendered nothing: the configuration was not marked
95
+ live-editable.** `dsh-settings` keeps a namespace only when
96
+ `volatileForm(schema)` yields a form, and that helper keeps a namespace when a
97
+ field carries `meta.volatile`. None of the sixteen fields did, so the Host
98
+ dropped the row before the client ever asked for it — the plugin was loaded,
99
+ its schema was valid, and Settings still had nothing to show. Every field now
100
+ ends in `.volatile()`; `test/config-live.test.ts` restates the Host's filter
101
+ and fails if the mark is dropped again.
102
+ - **The settings page rendered nothing.** Two layers were wrong. The client
103
+ declared `slots` and `locale` but not `configForms`, so Cordis never injected
104
+ the service; and even with the service present, the page read its form once at
105
+ mount, while the Host projects a row's settings namespace asynchronously. The
106
+ page now subscribes to the settings mirror, derives the namespace from the
107
+ projection the Host actually serves (patch id, entry id, package name, then a
108
+ suffix match), and reads the controller for that namespace. When the service
109
+ is present but no served namespace matches, the page names the namespaces it
110
+ does see instead of showing a dead end.
111
+
112
+ ## 0.7.0 (unreleased)
113
+
114
+ The TypeScript rewrite. Behaviour is meant to be unchanged where it was
115
+ measured, and stricter where it was previously guesswork.
116
+
117
+ ### Added
118
+
119
+ - **Reciprocal rank fusion** (`src/rank.ts`): weight shapes *admission*, not
120
+ score. The per-engine lists are walked in a weighted round robin (weight 2
121
+ takes two consecutive rows per round, so it owns roughly twice the slots), and
122
+ the merged walk gives each candidate a 1-based `position`; a source then scores
123
+ `votes / (rrfConstant + position)`, where `votes` is the number of distinct
124
+ engines that returned it, so agreement compounds while one engine's weight does
125
+ not. Authority (`authorityDomains`, `authorityBoost`) and freshness
126
+ (`freshnessBoost`, `freshnessDays`) multiply that score. Ties break
127
+ deterministically on engine index, rank, then URL. Measured with weights 2/1/1
128
+ over lists of 10/10/4 at `limit: 8`, admission splits 4/2/2.
129
+ - **TTL + LRU result cache** (`src/cache.ts`): entries expire after
130
+ `cacheTtlMs` (default 10 minutes) and the least recently used entry is evicted
131
+ past `cacheEntries` (default 100). Keys normalise case and whitespace, and are
132
+ sensitive to `limit`, `wait`, lead region and the participating engine set.
133
+ - **Retry-After aware cooldown** (`src/health.ts`): the service's own wait hint
134
+ wins over the guessed duration; otherwise the cooldown doubles per consecutive
135
+ failure from `baseCooldownMs` up to `maxCooldownMs`. `SEARCH_PARSE` and
136
+ `SEARCH_ABORTED` never cool an engine down.
137
+ - **Typed configuration schema** (`src/config.ts`, schemastery): the loader
138
+ validates the mounted config before `apply` runs, so a bad value fails loudly
139
+ with its path instead of silently using a default.
140
+ - **Fixture-based parser tests** (`test/fixtures/`, `test/parsers.test.ts`):
141
+ captured responses for DuckDuckGo HTML, Bing, 360 and AnySearch, plus the six
142
+ parser regressions (Bing `u=a1` tracker, DuckDuckGo `uddg`, single-quoted
143
+ `class` attribute, 360 `data-mdurl`, AnySearch `code !== 0`, echoed-query
144
+ check).
145
+ - **CI** (`.github/workflows/ci.yml`): Node 24 on Ubuntu and macOS, running
146
+ `npm ci`, `npm run typecheck`, `npm test`, `npm run build`, and a check that the
147
+ build really emits the declared `lib/index.js` entry point.
148
+ - **Proxy transport ported to TypeScript** (`src/proxy.ts`) with tests for
149
+ `parseProxy`, `proxyFromEnv`, the `NO_PROXY` decision and the pure response
150
+ framer (status/headers/body, chunked, gzip, deflate, brotli). `ProxyResponse`
151
+ now carries the final hop's headers so the caller can read `Retry-After`.
152
+
153
+ ### Changed
154
+
155
+ - All implementation moved from `index.js`/`engines.js`/`proxy.js` to
156
+ `src/*.ts`, compiled to `lib/` by `npm run build`; `package.json` now points
157
+ `main` and `types` at `lib/`.
158
+ - The merge is RRF in every case, replacing the fixed round-robin interleave.
159
+ The bilingual region policy still runs first: the region the query language
160
+ belongs to (`global` for ASCII, `cn` for CJK, `both` for either) fills the
161
+ budget, and the other region only appends when it came up short.
162
+ - Cooldown durations are configurable instead of the previous hardcoded five
163
+ minutes, and the failure list says how long an engine will rest and why.
164
+
165
+ ### Fixed
166
+
167
+ - The `wait` setting now reaches the engine run. It was documented, validated and
168
+ offered by the settings card, but `resolveConfig` never carried it into
169
+ `search`, so every call silently used `auto`.
170
+ - A partially specified config no longer throws on activation: `resolveConfig`
171
+ runs its input through the schema first, so a row written as `config: {}` (or
172
+ edited by hand into a fragment) resolves to the documented defaults instead of
173
+ dereferencing an absent key.
174
+ - Weight no longer multiplies every row of an engine. The previous rule let
175
+ AnySearch (weight 2) outscore every other engine at every rank, so a CJK query
176
+ returned 8/8 AnySearch sources and no Chinese results at all; the measured
177
+ split is now 4/2/2 for weights 2/1/1.
178
+ - A `POST` that meets a `3xx` other than `307`/`308` is downgraded to a bodyless
179
+ `GET`; `307`/`308` preserve the method and body (AnySearch needs both paths).
180
+ - A tunnelled response's `Retry-After` is read from the final hop instead of
181
+ being discarded.
182
+
183
+ ## 0.6.0
184
+
185
+ ### Changed
186
+
187
+ - AnySearch now leads CJK queries too (`region: both`), not only ASCII ones.
188
+ A/B measured on the same query set: with domestic engines alone,
189
+ `如何配置 postgres 逻辑复制` was answered with a Baidu Baike page for the
190
+ dictionary word "如何" and a song-lyric page; adding AnySearch returned Cloud
191
+ SQL documentation, a cnblogs walkthrough, the AWS specification guide, Azure
192
+ documentation and a Tencent Cloud article. `Python GIL 是什么` went from
193
+ python.org's home page, its download page and a tutorial site — none of which
194
+ mention the GIL — to explanations of the GIL. `DeepSeek Harness 插件开发` went
195
+ from front pages and an App Store listing to hands-on tutorials.
196
+ - The cost of that is visible: a Chinese query now takes 1.4–2.3 s instead of
197
+ 0.2–0.7 s with domestic engines alone.
198
+
199
+ ## 0.5.0
200
+
201
+ ### Added
202
+
203
+ - **AnySearch anonymous JSON API engine** (`POST api.anysearch.com/v1/search`
204
+ with `{ query, max_results }`, parsed from its `{ code, data: { results } }`
205
+ envelope). It is reachable without a tunnel and is the most precise engine, so
206
+ it takes a weight of 2.
207
+ - Weighted interleave for merging per-engine lists, so a high-precision engine
208
+ holds more slots without silencing the others.
209
+ - `www.`-aware dedupe: `http(s)://www.` and a trailing `/` are the same
210
+ document.
211
+
212
+ ### Changed
213
+
214
+ - Bing global demoted from primary to fallback. Measured on stopword-heavy
215
+ English queries it matched a single token: `how to set up postgres logical
216
+ replication` returned jigsaw-puzzle sites for "set", while the other engines
217
+ stayed on topic. It is worth reaching only when nothing else answered.
218
+
219
+ ## 0.4.0
220
+
221
+ ### Added
222
+
223
+ - **Bilingual region policy**: engines declare `global`, `cn` or `both`; the
224
+ region matching the query script leads the merged list and the other region
225
+ only fills gaps.
226
+ - **`wait` policy**: `auto` (wait for the leading region), `all` (wait for every
227
+ engine) and `fast` (return on the first answer), each with a soft deadline.
228
+ - **Per-engine cooldown** after a block page, so a site that has decided this
229
+ client is a robot is not re-triggered on every search.
230
+ - **Echoed-query check**: a result page must contain one query token of at least
231
+ three characters, otherwise it is a rate-limit or "trending" page belonging to
232
+ somebody else's search and is rejected.
233
+ - DuckDuckGo HTML parser with per-result snippets, alongside the lite endpoint.
234
+
235
+ ## 0.3.0
236
+
237
+ ### Fixed
238
+
239
+ - 360 result URLs: the real target lives in the anchor's `data-mdurl`, not in
240
+ the click-tracker `href`; entries without it are skipped instead of emitting a
241
+ redirect as a source.
242
+ - Single-quoted attributes (`class='result-link'`) are parsed, not only double
243
+ quoted ones.
244
+
245
+ ## 0.2.0
246
+
247
+ ### Fixed
248
+
249
+ - Bing click-tracker links (`/ck/a?...&u=a1<base64url>`) are decoded to the real
250
+ target. The `&` arrives HTML-escaped as `&amp;`, so the value is unescaped
251
+ before the payload is looked for.
252
+ - DuckDuckGo outbound links (`/l/?uddg=<percent-encoded>`) are unwrapped.
253
+ - HTML entities in titles and snippets are decoded, and numeric entities outside
254
+ the known table are dropped rather than leaked into a snippet.
255
+
256
+ ## 0.1.0
257
+
258
+ ### Added
259
+
260
+ - Initial provider: scraped 360 and Bing result pages, registered with the
261
+ `ctx.web` seam as provider id `bing`, returning `{ url, title, snippet }` with
262
+ no API key and no credential.
263
+
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 fangchenghao
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.