@fffangchenghao/dsh-web-search 0.23.0 → 0.24.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +58 -0
- package/README.md +45 -17
- package/README.zh-CN.md +40 -16
- package/SECURITY.md +11 -6
- package/client.js +5 -5
- package/docs/architecture.md +181 -0
- package/docs/comparison.md +2 -0
- package/docs/engines.md +203 -0
- package/docs/images/web-tools-demo.svg +1 -1
- package/docs/latency-and-sources.md +2 -0
- package/docs/plugin-settings-form.md +160 -0
- package/docs/quickstart.md +6 -0
- package/docs/store-evidence.md +6 -1
- package/docs/troubleshooting.md +225 -0
- package/lib/proxy.d.ts +16 -3
- package/lib/proxy.js +66 -18
- package/locales/en.json +2 -2
- package/locales/zh-CN.json +2 -2
- package/package.json +13 -5
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Architecture
|
|
2
|
+
|
|
3
|
+
How a `web_search` call becomes a list of sources, and which module owns which
|
|
4
|
+
decision. Everything described here exists in `src/*.ts`; the compiled output in
|
|
5
|
+
`lib/` is what `package.json` points at.
|
|
6
|
+
|
|
7
|
+
- [Engines and their quirks](engines.md)
|
|
8
|
+
- [Troubleshooting](troubleshooting.md)
|
|
9
|
+
- [README](../README.md) · [中文说明](../README.zh-CN.md)
|
|
10
|
+
|
|
11
|
+
## Module map
|
|
12
|
+
|
|
13
|
+
| Module | Owns | Exports worth knowing |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| `src/types.ts` | Every contract: `Source`, `HostSpec`, `EngineAnswer`, `SearchOptions`, `SearchOutcome`, `RankOptions`, `CacheOptions`, `HealthOptions`, `SearchErrorCode` | types only, no runtime code |
|
|
16
|
+
| `src/errors.ts` | The one error class and the `Retry-After` parser | `SearchError`, `parseRetryAfter` |
|
|
17
|
+
| `src/config.ts` | The schemastery schema, the defaults and their normalisation | `Config`, `resolveConfig`, `PluginConfig`, `ResolvedConfig` |
|
|
18
|
+
| `src/index.ts` | The plugin entry: `apply` registers the provider with the `web` seam and owns one cache and one health tracker per mount | `apply`, `name`, `inject`, `PROVIDER_ID` |
|
|
19
|
+
| `src/engines.ts` | The engine registry, the region policy, tier scheduling and the cache/cooldown integration | `HOSTS`, `search` |
|
|
20
|
+
| `src/transport.ts` | One engine request, direct or tunnelled, classified into a `SearchError` | `searchHost` |
|
|
21
|
+
| `src/proxy.ts` | Proxy detection and the hand-written `CONNECT`/TLS transport | `parseProxy`, `proxyFromEnv`, `proxyFromSystem`, `resolveProxy`, `proxyGet`, `readResponse` |
|
|
22
|
+
| `src/parsers.ts` | The five payload parsers plus the URL, entity and echoed-query helpers | `resolveHref`, `parseDdgHtml`, `parseDdgLite`, `parseBing`, `parseSo`, `parseAnySearch`, `echoesQuery`, `text` |
|
|
23
|
+
| `src/rank.ts` | Dedupe, weighted interleave, weight-as-admission RRF (votes over merged position), region merge | `dedupe`, `interleave`, `fuse`, `mergeRegions`, `RRF_CONSTANT`, `DEFAULT_AUTHORITY_DOMAINS` |
|
|
24
|
+
| `src/cache.ts` | TTL + LRU result cache with an injectable clock | `cacheKey`, `SearchCache` |
|
|
25
|
+
| `src/health.ts` | Per-engine cooldown, exponential backoff, `Retry-After` override | `EngineHealth` |
|
|
26
|
+
|
|
27
|
+
Dependency direction is one-way and acyclic: `index → engines → {transport,
|
|
28
|
+
rank, cache, health, parsers, proxy}`, every module → `types`, and `proxy` /
|
|
29
|
+
`transport` / `engines` → `errors`. `parsers`, `rank`, `cache` and `health` never
|
|
30
|
+
touch the network; `proxy` never parses a result page.
|
|
31
|
+
|
|
32
|
+
## Data flow
|
|
33
|
+
|
|
34
|
+
1. **Seam call.** The harness calls the provider registered by `apply`
|
|
35
|
+
(`src/index.ts`) with `(request, signal)`. `request.query` must be a non-empty
|
|
36
|
+
string; `request.maxResults`, when a positive integer, caps the budget.
|
|
37
|
+
2. **Config.** The loader has already validated the config section against
|
|
38
|
+
`Config`; `resolveConfig` turns it into the engine layer's shape. `limit`
|
|
39
|
+
becomes `min(maxResults ?? config.limit, config.limit)`.
|
|
40
|
+
3. **Search.** `search(query, signal, options)` in `src/engines.ts`:
|
|
41
|
+
- resolve the proxy (`useProxy: false` forces direct; otherwise
|
|
42
|
+
`resolveProxy(config.proxy)`);
|
|
43
|
+
- filter `HOSTS` by `hostIds`, and drop `viaProxy` engines when no proxy is
|
|
44
|
+
available;
|
|
45
|
+
- split engines into usable / cooling, but never end up with zero engines —
|
|
46
|
+
if everything is cooling, everything is used;
|
|
47
|
+
- compute the leading region from the query script (`global` for ASCII, `cn`
|
|
48
|
+
for CJK);
|
|
49
|
+
- look up the cache with a key built from the normalised query, the budget,
|
|
50
|
+
the wait policy, the lead region and the sorted engine ids.
|
|
51
|
+
4. **Tier run.** Engines of the leading region first, then the rest, split into
|
|
52
|
+
`primary` and `fallback`. `runTier` asks every engine of a tier concurrently
|
|
53
|
+
through `searchHost` and releases on the first of:
|
|
54
|
+
- every engine settled;
|
|
55
|
+
- every engine of the leading region settled (success *or* failure), so a
|
|
56
|
+
failed engine cannot hold the tier open until the deadline;
|
|
57
|
+
- `available() >= limit` sources from at least `needed` contributors across
|
|
58
|
+
**all** regions (two for `auto`), so one slow lead engine cannot keep a
|
|
59
|
+
complete cross-engine answer waiting;
|
|
60
|
+
- the soft deadline (`auto`: 5 s ASCII / 3 s CJK, `fast`: 3.5 s, `all`: 10 s).
|
|
61
|
+
|
|
62
|
+
Engines still running at release are aborted and reported as
|
|
63
|
+
`aborted once enough results had arrived` or `still running at deadline`.
|
|
64
|
+
5. **Per engine.** `searchHost` (`src/transport.ts`) builds the URL from
|
|
65
|
+
`host.build(query)` and picks a route: a domestic engine goes direct first
|
|
66
|
+
and is retried through the tunnel only if that fails; a `viaProxy` engine
|
|
67
|
+
tunnels immediately. Direct requests use global `fetch` with
|
|
68
|
+
`AbortSignal.timeout` combined with the caller's signal; tunnelled requests
|
|
69
|
+
use `proxyGet`.
|
|
70
|
+
6. **Classify.** A non-2xx status becomes `SEARCH_THROTTLED` (429),
|
|
71
|
+
`SEARCH_BLOCKED` (403 and everything else in the 4xx range),
|
|
72
|
+
`SEARCH_NETWORK` (5xx and transport failures) or `SEARCH_TIMEOUT` (a deadline
|
|
73
|
+
that expired). A JSON API validates itself; every scraped page must pass
|
|
74
|
+
`echoesQuery` or it is `SEARCH_BLOCKED`. A parser that throws is
|
|
75
|
+
`SEARCH_PARSE` and is **not** retryable. Cancellation is rethrown, never
|
|
76
|
+
returned.
|
|
77
|
+
7. **Parse.** `host.parse(payload, url)` returns `Source[]` with absolute
|
|
78
|
+
`http(s)` URLs and no empty fields.
|
|
79
|
+
8. **Health.** Each settled engine updates the tracker: success clears its
|
|
80
|
+
streak, failure may start or extend a cooldown.
|
|
81
|
+
9. **Fuse.** `mergeRegions(answers, lead, limit, rank)` fuses the leading group
|
|
82
|
+
with RRF and fills the budget; the other region is fused separately and only
|
|
83
|
+
appended (deduped against the head) when the head came up short. If the
|
|
84
|
+
primary tier produced nothing, the fallback tier runs and the same merge is
|
|
85
|
+
applied. Weight is an *admission* share, not a score multiplier: each engine's
|
|
86
|
+
list is walked in a weighted round robin to give every candidate a 1-based
|
|
87
|
+
position, and the score is `votes / (rrfConstant + position)`, with `votes`
|
|
88
|
+
counting the distinct engines that returned the source.
|
|
89
|
+
10. **Return.** A non-empty result is cached, then returned as
|
|
90
|
+
`{ sources, failures, proxy }`. `index.ts` re-slices to the budget and sets
|
|
91
|
+
`truncated`, or throws an `Error` that lists every engine's failure reason.
|
|
92
|
+
|
|
93
|
+
## Frames and invariants
|
|
94
|
+
|
|
95
|
+
- **Failures are values.** One dead engine degrades into the next one; only
|
|
96
|
+
cancellation escapes as a rejection.
|
|
97
|
+
- **Parsers never throw on markup drift.** They return `[]`, and the transport
|
|
98
|
+
records a parse failure.
|
|
99
|
+
- **Empty fields are omitted, not emptied.** A source never carries `title: ''`.
|
|
100
|
+
- **Determinism.** Fusion ties resolve on engine index, then rank, then URL, so
|
|
101
|
+
the same answers always produce the same order.
|
|
102
|
+
- **No credentials.** Nothing in the flow reads or sends an API key.
|
|
103
|
+
|
|
104
|
+
## Failure codes
|
|
105
|
+
|
|
106
|
+
| Code | Produced by | Cools the engine down |
|
|
107
|
+
| --- | --- | --- |
|
|
108
|
+
| `SEARCH_THROTTLED` | HTTP 429 | yes |
|
|
109
|
+
| `SEARCH_BLOCKED` | HTTP 403, other 4xx, echoed-query mismatch | yes |
|
|
110
|
+
| `SEARCH_NETWORK` | HTTP 5xx, transport error | yes |
|
|
111
|
+
| `SEARCH_TIMEOUT` | a request that hit its deadline | yes |
|
|
112
|
+
| `SEARCH_PARSE` | a parser that threw | no — retrying markup cannot help |
|
|
113
|
+
| `SEARCH_ABORTED` | the caller cancelled | no |
|
|
114
|
+
|
|
115
|
+
The cooldown length comes from `Retry-After` when the service sent one (delta
|
|
116
|
+
seconds or an HTTP date, read from the final redirect hop), otherwise
|
|
117
|
+
`baseCooldownMs × 2^(failures−1)`, capped at `maxCooldownMs`.
|
|
118
|
+
|
|
119
|
+
## Cache key
|
|
120
|
+
|
|
121
|
+
`cacheKey` builds `lead|wait|limit|sorted engine ids|normalised query`. The query
|
|
122
|
+
is trimmed, lowercased and has internal whitespace collapsed; engine order never
|
|
123
|
+
matters because the ids are sorted. The engine ids are the ones the run would use
|
|
124
|
+
*before* the cooldown filter, so an engine entering or leaving a cooldown does not
|
|
125
|
+
invalidate an entry: otherwise every throttle would turn the next identical query
|
|
126
|
+
into a cache miss. Only successful, non-empty results are stored. A hit returns a copy and skips every request. `ttlMs: 0` disables the
|
|
127
|
+
cache entirely; past `maxEntries`, the least recently used entry is evicted.
|
|
128
|
+
|
|
129
|
+
## Proxy transport
|
|
130
|
+
|
|
131
|
+
`proxyGet` opens an HTTP `CONNECT` tunnel to the target host and port, upgrades
|
|
132
|
+
the socket with TLS when the target is `https:`, writes the request itself
|
|
133
|
+
(`accept-encoding: identity`, `connection: close`) and decodes the response:
|
|
134
|
+
`Transfer-Encoding: chunked`, `Content-Length`, `gzip`, `deflate` and `br`.
|
|
135
|
+
Redirects are followed hop by hop, re-opening the tunnel each time, up to four
|
|
136
|
+
hops. `307`/`308` preserve the method and body; any other `3xx` downgrades a
|
|
137
|
+
`POST` to a bodyless `GET`. The final hop's status, headers (lowercased), body
|
|
138
|
+
and URL are returned, which is how the caller can read `Retry-After`.
|
|
139
|
+
|
|
140
|
+
Proxy selection is explicit config → environment (`DSH_WEB_SEARCH_PROXY`,
|
|
141
|
+
`HTTPS_PROXY`, `ALL_PROXY`, `HTTP_PROXY`, both casings) → the OS system proxy
|
|
142
|
+
(macOS `scutil --proxy`, Windows Internet Settings) gated by a 400 ms TCP probe,
|
|
143
|
+
cached for 60 seconds. Only `http://` proxies are
|
|
144
|
+
dialled from detection, because the tunnel itself is a plain `CONNECT`; an
|
|
145
|
+
`https://` value elsewhere in the environment is skipped rather than selected.
|
|
146
|
+
|
|
147
|
+
## Build and runtime layout
|
|
148
|
+
|
|
149
|
+
- `src/**/*.ts` is the source of truth; `npm run build` compiles it to `lib/`
|
|
150
|
+
(`tsconfig.json`, `rootDir: src`, `outDir: lib`).
|
|
151
|
+
- `package.json` points `main` and `types` at `lib/index.js` /
|
|
152
|
+
`lib/index.d.ts`, so a checkout must be built before it can be mounted.
|
|
153
|
+
- Relative imports carry the `.ts` extension and `rewriteRelativeImportExtensions`
|
|
154
|
+
rewrites them on emit; the source is also runnable directly by Node's type
|
|
155
|
+
stripping, which is how the tests run (`npm test`; it passes a quoted glob so
|
|
156
|
+
cmd.exe works too).
|
|
157
|
+
- `types.ts` is types only, so it emits nothing; every other module emits a
|
|
158
|
+
normal ES module.
|
|
159
|
+
- `index.js`, `engines.js` and `proxy.js` are the pre-rewrite JavaScript
|
|
160
|
+
implementation. They are not imported by `src/` and are not part of the
|
|
161
|
+
TypeScript build.
|
|
162
|
+
|
|
163
|
+
## Route memory and engine health
|
|
164
|
+
|
|
165
|
+
`EngineHealth` does two jobs. It keeps the failure cooldown (an engine that
|
|
166
|
+
returned a retryable block rests for 5–30 minutes) and, since 0.21.0, a
|
|
167
|
+
ten-minute route preference. A direct request that ends in `403`/`429` and then
|
|
168
|
+
succeeds through the tunnel records `proxy` for that engine; the next request
|
|
169
|
+
tries the tunnel first, and the direct route stays as a fallback. A later direct
|
|
170
|
+
success clears the preference. The transport test drives this through a real
|
|
171
|
+
HTTP `CONNECT` proxy rather than mocking `proxyGet`.
|
|
172
|
+
|
|
173
|
+
## MCP runner
|
|
174
|
+
|
|
175
|
+
`mcp/server.mjs` is a dependency-free stdio JSON-RPC process. It imports the
|
|
176
|
+
same `lib/engines.js`, `lib/fetch.js`, `lib/research.js` and
|
|
177
|
+
`lib/verticals.js` as the DSH plugin, exposes four tools (`search`, `fetch`,
|
|
178
|
+
`research`, `domain_search`), wraps every output in the untrusted-content
|
|
179
|
+
boundary and never reads a DSH profile. It is not a second search
|
|
180
|
+
implementation; a ranking or transport fix reaches both runtimes after
|
|
181
|
+
`npm run build`.
|
package/docs/comparison.md
CHANGED
|
@@ -6,6 +6,8 @@ whether a provider answers at all, how long it takes, how many distinct domains
|
|
|
6
6
|
it returns, and whether a page on a known official domain appears in the top 8.
|
|
7
7
|
The failures are part of the result.
|
|
8
8
|
|
|
9
|
+
> **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 is a documentation/metadata patch. Neither changes the search, ranking or source-adapter path measured below, so the 0.23.0 numbers remain the latest comparison.
|
|
10
|
+
|
|
9
11
|
## Method
|
|
10
12
|
|
|
11
13
|
- **Corpus:** 30 queries in `bench/compare30/queries.json` — 13 release queries
|
package/docs/engines.md
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# Engines
|
|
2
|
+
|
|
3
|
+
What each engine is asked for, how its answer is parsed, and what was actually
|
|
4
|
+
measured on this network. The registry is `HOSTS` in `src/engines.ts`; the
|
|
5
|
+
parsers are in `src/parsers.ts`; the shared request path is
|
|
6
|
+
`src/transport.ts`.
|
|
7
|
+
|
|
8
|
+
- [Architecture](architecture.md)
|
|
9
|
+
- [Troubleshooting](troubleshooting.md)
|
|
10
|
+
- [README](../README.md) · [中文说明](../README.zh-CN.md)
|
|
11
|
+
|
|
12
|
+
## The shared request
|
|
13
|
+
|
|
14
|
+
Every engine gets the same base headers, built in `src/transport.ts`:
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
|
|
18
|
+
accept-language: zh-CN,zh;q=0.9,en;q=0.8
|
|
19
|
+
accept: application/json (JSON POST engines)
|
|
20
|
+
accept: text/html,application/xhtml+xml (scraped pages)
|
|
21
|
+
content-type: application/json (JSON POST engines only)
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
- **Route.** `anysearch`, `cn.bing` and `so` go direct first and are retried
|
|
25
|
+
through the tunnel only if the direct attempt fails. `ddg-html`, `www.bing` and
|
|
26
|
+
`ddg-lite` are `viaProxy`: they tunnel from the start, and they are filtered out
|
|
27
|
+
of a run when no proxy is available rather than failing slowly.
|
|
28
|
+
- **Deadline.** `timeoutMs` per attempt (default 15 000 ms). Direct requests use
|
|
29
|
+
`AbortSignal.timeout` combined with the caller's signal; tunnelled requests use
|
|
30
|
+
the socket timeouts in `src/proxy.ts`.
|
|
31
|
+
- **Redirects.** Direct requests let `fetch` follow them; tunnelled requests
|
|
32
|
+
follow up to four hops themselves.
|
|
33
|
+
- **Echo check.** Every scraped page must contain one query token of at least
|
|
34
|
+
three characters (`echoesQuery`) or it is treated as a rate-limit page. JSON
|
|
35
|
+
engines set `skipEcho` because their envelope validates itself.
|
|
36
|
+
|
|
37
|
+
## `anysearch` — anonymous JSON API
|
|
38
|
+
|
|
39
|
+
| | |
|
|
40
|
+
| --- | --- |
|
|
41
|
+
| Request | `POST https://api.anysearch.com/v1/search` |
|
|
42
|
+
| Body | `{"query":"<query>","max_results":20}` |
|
|
43
|
+
| Region / weight | `both` / 2 (two slots per fusion round) |
|
|
44
|
+
| Tier | primary |
|
|
45
|
+
|
|
46
|
+
The response is an envelope, not markup:
|
|
47
|
+
|
|
48
|
+
```json
|
|
49
|
+
{ "code": 0, "message": "success", "request_id": "…",
|
|
50
|
+
"data": { "results": [{ "title": "…", "url": "…", "snippet": "…", "content": "…" }], "metadata": { } } }
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
`parseAnySearch` requires `code === 0` and an array at `data.results`; anything
|
|
54
|
+
else — a non-zero code, bad JSON, a missing array — yields `[]` rather than an
|
|
55
|
+
exception. Rows are dropped when `url` does not resolve to an absolute `http(s)`
|
|
56
|
+
URL, and empty `title`/`snippet` fields are omitted. The extra `content` field is
|
|
57
|
+
deliberately ignored: the seam wants a citable title/link/snippet, and the body
|
|
58
|
+
belongs to `web_fetch`.
|
|
59
|
+
|
|
60
|
+
A live capture (see [fixtures](../test/fixtures/README.md)) shows the envelope
|
|
61
|
+
and rows above, plus the service's error shape: an empty query is answered with
|
|
62
|
+
HTTP 400 and `{"code":-1,"message":"Query is required."}`.
|
|
63
|
+
|
|
64
|
+
The endpoint is metered: anonymous calls return at most 10 results, and the
|
|
65
|
+
provider's free tier with a key is documented upstream as 1 000 requests per day.
|
|
66
|
+
That is why the engine asks for 20 rows (`ANYSEARCH_MAX_RESULTS`) but the merge
|
|
67
|
+
sees at most what the service sends.
|
|
68
|
+
|
|
69
|
+
## `ddg-html` — DuckDuckGo HTML
|
|
70
|
+
|
|
71
|
+
| | |
|
|
72
|
+
| --- | --- |
|
|
73
|
+
| Request | `GET https://html.duckduckgo.com/html/?q=<urlencoded query>` |
|
|
74
|
+
| Region / weight | `global` / 1 |
|
|
75
|
+
| Tier / route | primary / tunnel only |
|
|
76
|
+
|
|
77
|
+
The richest English endpoint of the three DuckDuckGo faces here. `parseDdgHtml`
|
|
78
|
+
matches anchors whose `class` is `result__a` (single or double quotes, any
|
|
79
|
+
attribute order), takes the anchor text as the title, and looks for an element
|
|
80
|
+
with `class="result__snippet"` between this anchor and the next one as the
|
|
81
|
+
snippet. Hrefs may be direct or wrapped as `/l/?uddg=<percent-encoded>`, which
|
|
82
|
+
`resolveHref` unwraps.
|
|
83
|
+
|
|
84
|
+
Measured: the live capture on 2026-10-22 serves **direct** hrefs, not the `uddg`
|
|
85
|
+
wrapper that the original implementation was written against. The unwrapper is
|
|
86
|
+
kept because the wrapper still appears, and it is pinned by a synthetic fixture.
|
|
87
|
+
|
|
88
|
+
## `cn.bing` — Bing China
|
|
89
|
+
|
|
90
|
+
| | |
|
|
91
|
+
| --- | --- |
|
|
92
|
+
| Request | `GET https://cn.bing.com/search?q=<urlencoded query>` |
|
|
93
|
+
| Region / weight | `cn` / 1 |
|
|
94
|
+
| Tier / route | primary / direct first, tunnel retry |
|
|
95
|
+
|
|
96
|
+
The Chinese workhorse: the `cn` host targets the mainland index, mixed
|
|
97
|
+
Chinese/English queries work, and the result URLs are direct links. No market
|
|
98
|
+
parameters are sent.
|
|
99
|
+
|
|
100
|
+
## `www.bing` — Bing international (fallback)
|
|
101
|
+
|
|
102
|
+
| | |
|
|
103
|
+
| --- | --- |
|
|
104
|
+
| Request | `GET https://www.bing.com/search?q=<urlencoded>&setmkt=en-US&setlang=en&cc=US` |
|
|
105
|
+
| Region / weight | `global` / 1 |
|
|
106
|
+
| Tier / route | fallback / tunnel only |
|
|
107
|
+
|
|
108
|
+
`parseBing` splits the page on `<li class="b_algo"`, takes the first
|
|
109
|
+
`<h2><a href>` pair as title and URL, and the first `<p>` as the snippet.
|
|
110
|
+
|
|
111
|
+
Links are frequently Bing click trackers — `/ck/a?...&u=a1<base64url>` — where
|
|
112
|
+
the `&` arrives HTML-escaped as `&`. `resolveHref` decodes `&`/`&`
|
|
113
|
+
first, base64url-decodes the payload, and rejects the tracker if no real target
|
|
114
|
+
comes out.
|
|
115
|
+
|
|
116
|
+
**Why it is only a fallback.** Measured on stopword-heavy English queries it
|
|
117
|
+
matched a single token and answered with unrelated pages: `how to set up postgres
|
|
118
|
+
logical replication` returned jigsaw-puzzle sites for "set", while the engines
|
|
119
|
+
above stayed on topic. It is worth reaching only when nothing else answered.
|
|
120
|
+
|
|
121
|
+
## `so` — 360 search
|
|
122
|
+
|
|
123
|
+
| | |
|
|
124
|
+
| --- | --- |
|
|
125
|
+
| Request | `GET https://www.so.com/s?q=<urlencoded query>` |
|
|
126
|
+
| Region / weight | `cn` / 1 |
|
|
127
|
+
| Tier / route | primary / direct first, tunnel retry |
|
|
128
|
+
|
|
129
|
+
`parseSo` splits on `class="res-list"`, finds the heading `<h3 class="res-title…">`
|
|
130
|
+
(the served class value carries a trailing space, and the wrapper is a `<li>` for
|
|
131
|
+
ordinary results and a `<div>` for rich ones, so the element name is not matched),
|
|
132
|
+
then reads the **`data-mdurl`** attribute of the first anchor after it. The
|
|
133
|
+
`href` is a 360 click tracker, which is not a citable source: a result without
|
|
134
|
+
`data-mdurl` is skipped. The snippet comes from `class="res-list-summary"`.
|
|
135
|
+
|
|
136
|
+
## `ddg-lite` — DuckDuckGo Lite (fallback)
|
|
137
|
+
|
|
138
|
+
| | |
|
|
139
|
+
| --- | --- |
|
|
140
|
+
| Request | `GET https://lite.duckduckgo.com/lite/?q=<urlencoded query>` |
|
|
141
|
+
| Region / weight | `global` / 1 |
|
|
142
|
+
| Tier / route | fallback / tunnel only |
|
|
143
|
+
|
|
144
|
+
`parseDdgLite` accepts single- or double-quoted attributes, because this page
|
|
145
|
+
serves `class='result-link'` where Bing serves double quotes. Each hit is
|
|
146
|
+
examined up to the next hit, which is where its sibling
|
|
147
|
+
`<td class='result-snippet'>` lives.
|
|
148
|
+
|
|
149
|
+
## Parser regressions
|
|
150
|
+
|
|
151
|
+
Six failures were paid for once and are now pinned by tests and fixtures:
|
|
152
|
+
|
|
153
|
+
1. a Bing `&u=a1<base64url>` tracker must decode to the real target, not a
|
|
154
|
+
`bing.com/ck/a` URL;
|
|
155
|
+
2. a DuckDuckGo `/l/?uddg=<urlencoded>` wrapper must unwrap to the target;
|
|
156
|
+
3. a single-quoted `class='result-link'` must still be parsed;
|
|
157
|
+
4. 360's `data-mdurl` must be used, not the click tracker;
|
|
158
|
+
5. an AnySearch envelope with `code !== 0` must yield `[]`;
|
|
159
|
+
6. `echoesQuery` must reject a rate-limit/captcha page served for a different
|
|
160
|
+
query.
|
|
161
|
+
|
|
162
|
+
Fixtures: [test/fixtures/README.md](../test/fixtures/README.md) lists every file,
|
|
163
|
+
the exact URL it came from, the query and the capture date (2026-10-22). Two
|
|
164
|
+
files are synthetic and marked as such: the `uddg` wrapper and Bing's
|
|
165
|
+
rate-limited reply, whose block page could not be provoked on demand.
|
|
166
|
+
|
|
167
|
+
## What was measured
|
|
168
|
+
|
|
169
|
+
Timings on this network, with a tunnel on:
|
|
170
|
+
|
|
171
|
+
| Situation | Time |
|
|
172
|
+
| --- | --- |
|
|
173
|
+
| CJK query, `auto` | 1.4–2.3 s |
|
|
174
|
+
| CJK query, domestic engines only (no AnySearch) | 0.2–0.7 s |
|
|
175
|
+
| ASCII query, `auto` | 2.6–3.8 s |
|
|
176
|
+
|
|
177
|
+
Coverage for an English query: 16–30 sources across 12–21 distinct domains, and
|
|
178
|
+
the first 8 slots are 8/8 on-topic foreign documentation. The A/B evidence for
|
|
179
|
+
leading CJK queries with AnySearch is in the [README](../README.md#ab-evidence-cjk-queries-need-anysearch-too).
|
|
180
|
+
|
|
181
|
+
Admission is weighted, scoring is not: with weights 2/1/1 over lists of 10, 10
|
|
182
|
+
and 4 results at `limit: 8`, the 8 slots split 4/2/2 between the engines. The
|
|
183
|
+
previous rule multiplied every row of an engine by its weight, which let
|
|
184
|
+
AnySearch fill all 8 slots and starved the Chinese engines completely (8/0/0).
|
|
185
|
+
|
|
186
|
+
Block pages seen in practice:
|
|
187
|
+
|
|
188
|
+
- 360 served its `访问异常页面` with a captcha under rapid unattended requests;
|
|
189
|
+
- another domestic engine (Shenma, not part of this registry) served an `x5sec`
|
|
190
|
+
punish page during the same measurements;
|
|
191
|
+
- Bing answered repeated rapid requests with a "trending" feed belonging to
|
|
192
|
+
somebody else's search — HTTP 200, so only the echoed-query check catches it.
|
|
193
|
+
|
|
194
|
+
That evidence is why block pages and echoed-query mismatches cool an engine down
|
|
195
|
+
instead of being retried on every search.
|
|
196
|
+
|
|
197
|
+
## What could not be made to work
|
|
198
|
+
|
|
199
|
+
- **Google.** A tunnelled `www.google.com/search` returns a JavaScript shell
|
|
200
|
+
(94 KB, 1–3 links); headless Chrome is redirected to `google.com.hk` and gets a
|
|
201
|
+
reCAPTCHA page. `gbv=1`, a consent cookie and a mobile user agent all failed.
|
|
202
|
+
- **Public SearXNG instances.** Eight were tried; all answered 429, 403 or
|
|
203
|
+
something that was not JSON.
|
|
@@ -52,7 +52,7 @@
|
|
|
52
52
|
<rect x="70" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
|
|
53
53
|
<text x="92" y="651" class="m" font-size="13">Engines</text><text x="92" y="677" class="t" font-size="20" font-weight="700">8 key-free paths</text>
|
|
54
54
|
<rect x="420" y="620" width="330" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
|
|
55
|
-
<text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">
|
|
55
|
+
<text x="442" y="651" class="m" font-size="13">Tests</text><text x="442" y="677" class="t" font-size="20" font-weight="700">690 local cases</text>
|
|
56
56
|
<rect x="770" y="620" width="360" height="76" rx="14" fill="#101d31" stroke="#26364f"/>
|
|
57
57
|
<text x="792" y="651" class="m" font-size="13">Cross-agent</text><text x="792" y="677" class="t" font-size="20" font-weight="700">MCP stdio server</text>
|
|
58
58
|
</g>
|
|
@@ -6,6 +6,8 @@ This note answers three questions from the public 30-query comparison:
|
|
|
6
6
|
2. Can the AnySearch tail and the Chinese Rust miss be fixed?
|
|
7
7
|
3. Does the fix require a vector database or a different retrieval model?
|
|
8
8
|
|
|
9
|
+
> **Version note (2026-11-02):** 0.24.0 added Windows system-proxy discovery, and 0.24.1 is a documentation/metadata patch. The search path is unchanged, so these 0.23.0 measurements still describe 0.24.x.
|
|
10
|
+
|
|
9
11
|
The short answer: Exa is a different architecture, not a faster RRF loop. This
|
|
10
12
|
plugin can borrow tail-tolerance and source-specific facts, but copying Exa's
|
|
11
13
|
semantic index would be a different product. The changes below improve the
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
# 让 DSH 插件的配置出现在「设置」里:四层判定链
|
|
2
|
+
|
|
3
|
+
> 结论先行:一个插件配置项要能在 Settings 里编辑,需要**同时**满足四层条件。
|
|
4
|
+
> 任何一层不满足,症状都完全一样——**设置页空白**。这就是它难 debug 的原因。
|
|
5
|
+
> 本文的 file:line 均来自 `DeepSeek Harness.app/Contents/Resources/app.asar`(2026-10 版本)。
|
|
6
|
+
|
|
7
|
+
## 症状
|
|
8
|
+
|
|
9
|
+
设置页显示插件标题和描述,但下面没有控件;或显示「当前部署尚未提供此插件的设置」。
|
|
10
|
+
插件本身加载正常(工具能用),日志里可能有一行 `warning: 1 entry did not activate`。
|
|
11
|
+
|
|
12
|
+
## 四层判定链
|
|
13
|
+
|
|
14
|
+
### 第 1 层:插件必须真正激活(`fiber.state === 2`)
|
|
15
|
+
|
|
16
|
+
`@deepseek-ai/dsh-settings/lib/index.js` 的 `describe()` 开头:
|
|
17
|
+
|
|
18
|
+
```js
|
|
19
|
+
const schema = this.schema(entry);
|
|
20
|
+
if (schema === undefined || entry.fiber === undefined
|
|
21
|
+
|| entry.fiber.runtime === null || entry.fiber.state !== 2) return [];
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
而 `schema(entry)` 是:
|
|
25
|
+
|
|
26
|
+
```js
|
|
27
|
+
schema(entry) {
|
|
28
|
+
const schema = entry.fiber?.runtime?.Config;
|
|
29
|
+
return schema !== undefined && 'toJSON' in schema ? schema : undefined;
|
|
30
|
+
}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
**要点:schema 挂在「运行时对象」上,不是模块命名空间上。**
|
|
34
|
+
|
|
35
|
+
Loader 的 `unwrapExports`(`cordis-plugin-loader/lib/index.js:669`):
|
|
36
|
+
|
|
37
|
+
```js
|
|
38
|
+
exports = exports.default ?? exports;
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
| 插件导出写法 | 结果 |
|
|
42
|
+
|---|---|
|
|
43
|
+
| `export function apply` + `export { Config }`(无 default) | ✅ `plugin` = 命名空间,`plugin.Config` 命中 |
|
|
44
|
+
| `export default Object.assign(apply, { Config, inject })` | ✅ `plugin` = default,`Config` 命中 |
|
|
45
|
+
| `export default apply` 再另写 `export const Config` | ❌ default 存在 → 命名空间被丢弃 → `runtime.Config` 为 undefined |
|
|
46
|
+
|
|
47
|
+
**函数式插件若用 default 导出,必须把 `Config` 和 `inject` 一起挂上去**;只挂 `Config` 会得到
|
|
48
|
+
`Error: cannot get property "web" without inject`——Cordis 从它挂载的那个对象读 `inject`。
|
|
49
|
+
|
|
50
|
+
### 第 2 层:配置校验必须通过
|
|
51
|
+
|
|
52
|
+
Loader 只校验一次,校验的是 **`entry.options.config`(patch 里的原始值)**:
|
|
53
|
+
|
|
54
|
+
```js
|
|
55
|
+
// cordis/lib/index.js:1355 → :1345 → :956
|
|
56
|
+
const result = runtime.Config['~standard'].validate(config);
|
|
57
|
+
if (result.issues) throw new ValidationError(result.issues);
|
|
58
|
+
return result.value;
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
**校验后的 `result.value` 里,每个 volatile 字段都是 `{ get }` 持有器,不是值。**
|
|
62
|
+
|
|
63
|
+
最容易踩的坑:**别把已经解析过的 config 再喂回 schema**。
|
|
64
|
+
|
|
65
|
+
```ts
|
|
66
|
+
// ❌ 在 apply 内部这样写,必炸
|
|
67
|
+
const settings = Config(config as PluginConfig);
|
|
68
|
+
// ValidationError: $.timeoutMs expected number but got [object Object]
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
这条错误消息**没有** `invalid config:` 前缀,说明它是 schemastery 抛的(你自己的代码),
|
|
72
|
+
却被 Host 报成「entry did not activate」,极易误判成插件加载失败。
|
|
73
|
+
|
|
74
|
+
正确做法:先解包再校验,或干脆只在校验时用 plain 值、输出时优先用调用方的持有器。
|
|
75
|
+
|
|
76
|
+
### 第 3 层:至少一个字段带 `meta.volatile`
|
|
77
|
+
|
|
78
|
+
`lib/types/schema.js:43`:
|
|
79
|
+
|
|
80
|
+
```js
|
|
81
|
+
export function volatileForm(schema) {
|
|
82
|
+
if (schema.meta.volatile) return plainSchema(schema);
|
|
83
|
+
if (schema.type === 'object') {
|
|
84
|
+
// 只保留 volatileForm !== undefined 的子字段
|
|
85
|
+
return Object.keys(dict).length === 0 ? undefined : z.object(dict);
|
|
86
|
+
}
|
|
87
|
+
return undefined;
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
`describe()` 里 `if (form === undefined) return [];`——**一个 volatile 字段都没有,整个命名空间被丢弃**。
|
|
92
|
+
|
|
93
|
+
### 第 4 层:消费方必须 `.get()`
|
|
94
|
+
|
|
95
|
+
一旦标了 `.volatile()`,schema 解析结果是 `{ get: fn }`:
|
|
96
|
+
|
|
97
|
+
```js
|
|
98
|
+
z.object({ timeoutMs: z.number().default(15000).volatile() })({})
|
|
99
|
+
// → { timeoutMs: { get: [Function], [Symbol(cosmokit.volatile.write)]: [Function] } }
|
|
100
|
+
// .get() === 15000
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
所以读取处必须解包:
|
|
104
|
+
|
|
105
|
+
```ts
|
|
106
|
+
const read = (v: unknown) => {
|
|
107
|
+
const holder = v as { get?: unknown } | null;
|
|
108
|
+
return typeof holder?.get === 'function' ? (holder.get as () => unknown)() : v;
|
|
109
|
+
};
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
数组也一样——`{}` 只是持有器的 **JSON 形态**(`get` 是函数、writer 是 symbol)。不标数组反而会让
|
|
113
|
+
设置页保存时报 `Config field "hosts" is not volatile`(`dsh-settings/lib/types/index.js:351`)。
|
|
114
|
+
|
|
115
|
+
## 最小 checklist(新插件抄这个)
|
|
116
|
+
|
|
117
|
+
```ts
|
|
118
|
+
export const Config = z.object({
|
|
119
|
+
someScalar: z.string().default('').description('…').volatile(), // 标量标 volatile
|
|
120
|
+
someArray: z.array(z.string()).default([]).description('…').volatile(), // 数组也标
|
|
121
|
+
});
|
|
122
|
+
|
|
123
|
+
export function apply(ctx, config) {
|
|
124
|
+
// config 里已是持有器:读的时候必须 .get()
|
|
125
|
+
const value = config.someScalar.get();
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
export default Object.assign(apply, { Config, inject }); // Config + inject 一起挂
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
保存/生效语义:**只有 volatile 字段变化**才走 live commit(原地改持有器、不重挂载,
|
|
132
|
+
`cordis-plugin-loader/lib/index.js:382-435`);其他字段变化会整个插件重挂载。因此
|
|
133
|
+
如果插件在 `apply` 里缓存了配置快照,volatile 的实时改动对缓存那份不生效——要么每次调用重读,
|
|
134
|
+
要么接受「需重挂载」。
|
|
135
|
+
|
|
136
|
+
## 诊断顺序(不要跳步)
|
|
137
|
+
|
|
138
|
+
1. **先确认插件激活**:看有没有 `did not activate`,有就先修激活,别动设置页;
|
|
139
|
+
2. **确认 `runtime.Config` 存在**:`describe()` 第一行就依赖它;
|
|
140
|
+
3. **确认 patch 校验通过**:注意报错有没有 `invalid config:` 前缀,没有就是你自己代码抛的;
|
|
141
|
+
4. **确认命名空间被收录**:字段有没有 `meta.volatile`;
|
|
142
|
+
5. **确认客户端读的是正确的 ns 名**:`entry.options.id`(通常等于 patch 里的 `id`);
|
|
143
|
+
6. 最后才怀疑客户端渲染。
|
|
144
|
+
|
|
145
|
+
## 这次踩过的四个误判(供警惕)
|
|
146
|
+
|
|
147
|
+
| 当时的判断 | 为什么看起来合理 | 实际 |
|
|
148
|
+
|---|---|---|
|
|
149
|
+
| 「客户端没 inject `configForms`」 | 页面确实什么都没渲染 | 真实存在但非根因 |
|
|
150
|
+
| 「表单是异步投影,读太早」 | 加订阅后报错消失 | 真实存在但非根因 |
|
|
151
|
+
| 「字段缺 `.volatile()`」 | 官方插件都有这个标记 | 真实存在但非根因 |
|
|
152
|
+
| 「要重装/换安装方式」 | 别的插件能显示,我们是手装的 | 无关 |
|
|
153
|
+
|
|
154
|
+
**根因是第 2 层**:`apply` 内部二次校验已解析的 config,抛在 plugin 内 → 表现为「未激活」。
|
|
155
|
+
四层都修对之后页面才出来——所以每次只修一层,症状看起来完全没变。
|
|
156
|
+
|
|
157
|
+
**教训:先定位「命名空间有没有被 Host 收录」,再改客户端。** 客户端加一行诊断
|
|
158
|
+
(把 `settings.describe()` 返回的 ns 列表打印出来)能在 10 分钟内省掉上面四个误判。
|
|
159
|
+
|
|
160
|
+
> 同一份内容也作为 skill 存放在 `~/.dsh/skills/dsh-plugin-settings-form/SKILL.md`(自包含,触发时一次性加载)。改这里时请同步改那边。
|
package/docs/quickstart.md
CHANGED
|
@@ -13,6 +13,12 @@ dsh plugin --profile defaults add @fffangchenghao/dsh-web-search@latest
|
|
|
13
13
|
Use your profile name instead of `defaults` if you run a named profile. Restart
|
|
14
14
|
DeepSeek Harness once after installation. No API key is required.
|
|
15
15
|
|
|
16
|
+
On Windows the command is identical in PowerShell; the plugin has no native
|
|
17
|
+
addon and needs only Node 22.18+. If `dsh` is not on `PATH`, use the CLI from
|
|
18
|
+
the DSH installation directory or install the npm package from Settings. The
|
|
19
|
+
plugin reads the Windows system proxy from Internet Settings automatically;
|
|
20
|
+
for a PAC-only proxy, set `proxy` or `DSH_WEB_SEARCH_PROXY` explicitly.
|
|
21
|
+
|
|
16
22
|
Verify in **Settings → Plugins → Web Search**:
|
|
17
23
|
|
|
18
24
|
- the engine health section can run a one-probe-per-engine test;
|
package/docs/store-evidence.md
CHANGED
|
@@ -23,10 +23,15 @@ Only `0.2.1-rc.2` is declared, because that is the version actually exercised.
|
|
|
23
23
|
## Environment
|
|
24
24
|
|
|
25
25
|
- macOS (arm64), DeepSeek Harness **0.2.1-rc.2** (`dsh --version`), Node 22.18+.
|
|
26
|
-
- Runtime artifacts: `lib/` is committed to this repository (86 files,
|
|
26
|
+
- Runtime artifacts: `lib/` is committed to this repository (86 files, 956 KiB),
|
|
27
27
|
so the fixed commit's tree contains `lib/index.js` and `lib/index.d.ts`.
|
|
28
28
|
- Disposable profile for the destructive steps: a fresh `DSH_HOME` created with
|
|
29
29
|
`mktemp -d`; the real `~/.dsh` was never written to. Bundled pnpm 11.28.5.
|
|
30
|
+
- **Windows CI:** GitHub Actions `windows-latest`, Node 24. Install, typecheck,
|
|
31
|
+
the full test suite, build and the entry-point import all pass. Run:
|
|
32
|
+
https://github.com/fangchenghao/dsh-web-search/actions/runs/41387433033
|
|
33
|
+
(the matrix also runs on macOS and Ubuntu). This is a CI environment, not a
|
|
34
|
+
claim about every Windows terminal or proxy client.
|
|
30
35
|
|
|
31
36
|
## install — passed
|
|
32
37
|
|