@fffangchenghao/dsh-web-search 0.13.5 → 0.13.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +4 -3
- package/README.zh-CN.md +4 -3
- package/client.js +1 -1
- package/lib/engines.js +5 -0
- package/lib/fetch.d.ts +5 -1
- package/lib/fetch.js +73 -10
- package/lib/markdown.js +35 -11
- package/lib/render.d.ts +1 -1
- package/lib/types.d.ts +9 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,48 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.13.7
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- **A blank query burned seven seconds and six requests.** `search('')` and
|
|
7
|
+
`search(' ')` reached every engine, each failed on its own, and the caller
|
|
8
|
+
got an empty list. Validation now rejects it in under a millisecond with
|
|
9
|
+
`SEARCH_INVALID`.
|
|
10
|
+
- **A PDF arrived as binary noise.** `kindOf()` treated every non-HTML response
|
|
11
|
+
as text, so `application/pdf` came back as 20 000 characters of PDF stream and
|
|
12
|
+
looked like a successful fetch. Binary content types now fail with
|
|
13
|
+
`SEARCH_UNSUPPORTED` and name the type.
|
|
14
|
+
- **A JavaScript shell was a silent success.** `https://github.com/` and
|
|
15
|
+
`https://excalidraw.com/` returned HTTP 200 with zero readable characters, so
|
|
16
|
+
the caller saw an empty page with no explanation. A 200 with no text now
|
|
17
|
+
fails with `SEARCH_EMPTY` and states why, including "no renderer installed".
|
|
18
|
+
Non-2xx responses keep their old behaviour: the status line plus the body.
|
|
19
|
+
- **A layout `<main>` could throw away the article.** Extraction trusted any
|
|
20
|
+
non-empty landmark, so a page whose `<main>` held a breadcrumb produced four
|
|
21
|
+
characters while the prose sat in a sibling container. A landmark now has to
|
|
22
|
+
keep a real share of the page's text, and the body is the last resort.
|
|
23
|
+
- **The fetch ladder stacked deadlines.** Direct (5 s) + tunnel (15 s) + render
|
|
24
|
+
+ extraction meant a dead host could hold a caller for half a minute. The
|
|
25
|
+
whole ladder now shares a 25-second budget and later rungs get only the
|
|
26
|
+
remainder.
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
- `scripts/sweep.mjs` sweeps the ranking knobs over the recorded corpus and
|
|
30
|
+
prints a paired-bootstrap interval against the defaults. `bench/README.md`
|
|
31
|
+
records the result: the best configuration beats the defaults by +0.044 MRR
|
|
32
|
+
with an interval of [−0.004, +0.123], so **the defaults are unchanged**. The
|
|
33
|
+
same page documents live `wait`-policy timings (`auto` keeps the most distinct
|
|
34
|
+
domains).
|
|
35
|
+
|
|
36
|
+
## 0.13.6
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
- **The browser half registered under the old nickname.** The Host matches a
|
|
40
|
+
client bundle by package name, so `__ModuleLoader__.load({ id:
|
|
41
|
+
'dsh-web-search-bing' })` was rejected with "loaded without registering
|
|
42
|
+
@fffangchenghao/dsh-web-search via __ModuleLoader__.load" and the Settings
|
|
43
|
+
page never appeared. The id is the package name now, and a test evaluates the
|
|
44
|
+
bundle against a stub loader and fails if the two ever diverge.
|
|
45
|
+
|
|
3
46
|
## 0.13.5
|
|
4
47
|
|
|
5
48
|
### Documentation
|
package/README.md
CHANGED
|
@@ -478,9 +478,10 @@ Yes — to the engines taking part in that search, which is what searching means
|
|
|
478
478
|
Four usual causes:
|
|
479
479
|
|
|
480
480
|
1. **The site refuses automated access**: a 403 or a verification page. With Playwright installed and content still on the page, `renderJs: auto` tries the local browser first; otherwise, if direct and tunnel both fail, only an AnySearch key enables server-side extraction.
|
|
481
|
-
2. **The content is rendered by JavaScript**: what comes back is an empty shell. With Playwright installed it is rendered automatically; without it, you see the shell.
|
|
481
|
+
2. **The content is rendered by JavaScript**: what comes back is an empty shell. With Playwright installed it is rendered automatically; without it, you see the shell. **If stripping the shell leaves no readable text at all, the fetch fails with `SEARCH_EMPTY` and says why — including "no renderer installed" — instead of returning an empty page as a success.**
|
|
482
482
|
3. **It needs a login or a subscription**: fetching carries no session of yours.
|
|
483
|
-
4. **Direct requests time out**: 5 seconds direct, then the tunnel; if neither works and no key is configured you get an error, and with a key there is one more rung, server-side extraction. The message names each rung's failure.
|
|
483
|
+
4. **Direct requests time out**: 5 seconds direct, then the tunnel; if neither works and no key is configured you get an error, and with a key there is one more rung, server-side extraction. The whole ladder shares a 25-second budget, so later rungs get only what is left. The message names each rung's failure.
|
|
484
|
+
5. **Binary documents**: PDFs, images and archives are refused explicitly (`SEARCH_UNSUPPORTED`) with the content type, rather than handing you a byte stream as prose. To read a PDF, extract its text with another tool first.
|
|
484
485
|
|
|
485
486
|
360's captcha belongs to the first cause, but it used to be especially quiet: from some networks the request 302s to the qcaptcha "unexpected access" page while the HTTP status is still 200. The plugin now records that as `SEARCH_BLOCKED`; querying that engine alone shows `so: bot verification page` instead of zero results, and 360 has been demoted to the fallback tier.
|
|
486
487
|
|
|
@@ -511,7 +512,7 @@ If the page is short enough for `web_fetch` in one pass, you do not need `read_p
|
|
|
511
512
|
First decide whether it is a shell page — HTML with a title and a loading script, whose body appears only after JavaScript runs. Then, in order:
|
|
512
513
|
|
|
513
514
|
1. **If you installed Playwright yourself** (`npm i -g playwright && npx playwright install chromium`): leave `renderJs` on `auto` and reload the plugin, then retry. The plugin recognises the shell and renders it in headless Chromium; `always` forces a render on every fetch, and `off` disables rendering.
|
|
514
|
-
2. **Without Playwright**: nothing is installed automatically
|
|
515
|
+
2. **Without Playwright**: nothing is installed automatically; if the fetch ends up with **no readable text at all**, it fails with `SEARCH_EMPTY` and names the reason — including that no renderer is installed — rather than returning an empty shell as a success. Try another source (a print view, a static mirror, the site's API), or raise `fetchMaxLength` — but that only helps when the text was actually fetched and cut at the tail; it cannot create a body.
|
|
515
516
|
3. **If the page needs a login or a subscription**: rendering cannot rescue it either; fetching carries no session of yours.
|
|
516
517
|
|
|
517
518
|
### Does the plugin secretly install a browser?
|
package/README.zh-CN.md
CHANGED
|
@@ -478,9 +478,10 @@ pnpm install
|
|
|
478
478
|
常见原因有四类:
|
|
479
479
|
|
|
480
480
|
1. **站点拒绝自动访问**:回 403 或验证页。自己装了 Playwright 且页面还有内容时,`renderJs: auto` 会先试本机浏览器;否则直连、隧道都过不去时,只有配了 AnySearch key 才会尝试服务端抽取。
|
|
481
|
-
2. **正文靠 JavaScript 渲染**:抓回来的是空壳 HTML,看不到内容。装了 Playwright
|
|
481
|
+
2. **正文靠 JavaScript 渲染**:抓回来的是空壳 HTML,看不到内容。装了 Playwright 后会自动渲染;没装就只能看到壳。**如果去壳之后一个字都不剩,插件会报错(`SEARCH_EMPTY`)并写明「没有可用渲染器」之类的判定,而不是把空壳当成成功结果返回**。
|
|
482
482
|
3. **要登录或付费**:抓取没有你的登录态。
|
|
483
|
-
4. **直连超时**:会先在 5 秒内试直连,失败转隧道;两个都不通、又没配 key 时报错,配了 key
|
|
483
|
+
4. **直连超时**:会先在 5 秒内试直连,失败转隧道;两个都不通、又没配 key 时报错,配了 key 还会再试服务端抽取。整条梯队总共只有 25 秒预算,后面的档位只能花剩下的时间。报错里会写清楚每一级失败的原因。
|
|
484
|
+
5. **二进制文档**:PDF、图片、压缩包会明确拒绝(`SEARCH_UNSUPPORTED`)并告出内容类型,不会把字节流当成正文塞给你。需要读 PDF 的话,先用别的工具取文本。
|
|
484
485
|
|
|
485
486
|
360 的验证码页属于第 1 类,但它以前特别隐蔽:从部分网络访问会 302 到 qcaptcha 的「访问异常页面」,HTTP 状态却是 200。插件现在会把它记成 `SEARCH_BLOCKED`,单独查该引擎会看到 `so: bot verification page`,而不是 0 条;360 也因此降为兜底档。
|
|
486
487
|
|
|
@@ -511,7 +512,7 @@ pnpm install
|
|
|
511
512
|
先判断它是不是「壳页面」——HTML 里只有标题和加载脚本,正文要等 JavaScript 跑完才出现。然后按顺序试:
|
|
512
513
|
|
|
513
514
|
1. **自己装了 Playwright**(`npm i -g playwright && npx playwright install chromium`):把 `renderJs` 留在 `auto`,重载插件后再抓。插件会识别壳页面,开一个无头 Chromium 把正文渲染出来;`always` 可以强制每次都渲染,`off` 则完全禁用。
|
|
514
|
-
2. **没装 Playwright
|
|
515
|
+
2. **没装 Playwright**:插件不会自动装;如果抓回来**完全没有可读正文**,它会明确报错(`SEARCH_EMPTY`),信息里说明是否因为没有渲染器,而不是返回一个空壳当成功。这时换一个来源(打印版、静态镜像、站点 API),或者把 `fetchMaxLength` 调大——但后者只对「其实抓到了、只是被截断」的情况有用,变不出正文。
|
|
515
516
|
3. **页面本身要登录或付费**:渲染也救不回来,抓取没有你的登录态。
|
|
516
517
|
|
|
517
518
|
### 插件会不会偷偷装浏览器?
|
package/client.js
CHANGED
|
@@ -33,7 +33,7 @@
|
|
|
33
33
|
if (loader === null || typeof loader !== 'object' || typeof loader.load !== 'function') return;
|
|
34
34
|
|
|
35
35
|
loader.load({
|
|
36
|
-
id: 'dsh-web-search
|
|
36
|
+
id: '@fffangchenghao/dsh-web-search',
|
|
37
37
|
factory: function (require) {
|
|
38
38
|
try {
|
|
39
39
|
var React = require('react');
|
package/lib/engines.js
CHANGED
|
@@ -265,6 +265,11 @@ export function resolveHost(host, options) {
|
|
|
265
265
|
return { ...host, body, ...(headers === undefined ? {} : { headers }) };
|
|
266
266
|
}
|
|
267
267
|
export async function search(query, signal, options) {
|
|
268
|
+
// A blank query used to reach every engine, burn seven seconds and come back
|
|
269
|
+
// with six identical "empty query" failures; refuse it before any request.
|
|
270
|
+
if (typeof query !== 'string' || query.trim().length === 0) {
|
|
271
|
+
throw new SearchError('the search query is empty', 'SEARCH_INVALID', false);
|
|
272
|
+
}
|
|
268
273
|
const limit = Math.max(1, Math.min(options.limit, 20));
|
|
269
274
|
const timeoutMs = options.timeoutMs ?? 15000;
|
|
270
275
|
const wait = options.wait ?? 'auto';
|
package/lib/fetch.d.ts
CHANGED
|
@@ -5,10 +5,12 @@ export interface FetchOutcome {
|
|
|
5
5
|
readonly url: string;
|
|
6
6
|
readonly statusCode: number;
|
|
7
7
|
readonly body: {
|
|
8
|
-
readonly kind: 'html' | 'text';
|
|
8
|
+
readonly kind: 'html' | 'text' | 'binary';
|
|
9
9
|
readonly content: string;
|
|
10
10
|
};
|
|
11
11
|
readonly truncated: boolean;
|
|
12
|
+
/** The response's mime type, kept so a rejection can name what it refused. */
|
|
13
|
+
readonly contentType?: string;
|
|
12
14
|
/**
|
|
13
15
|
* How the optional render rung decided, when this provider produced the
|
|
14
16
|
* result. The Host ignores the extra field; it is here so a caller (and a
|
|
@@ -31,6 +33,8 @@ export interface FetchProviderOptions {
|
|
|
31
33
|
id?: string;
|
|
32
34
|
/** Per-attempt deadline in milliseconds. */
|
|
33
35
|
timeoutMs?: number;
|
|
36
|
+
/** Budget for the whole ladder; later rungs get only what is left. */
|
|
37
|
+
overallMs?: number;
|
|
34
38
|
/** Explicit tunnel URL, overriding detection. Empty means detect. */
|
|
35
39
|
proxy?: string;
|
|
36
40
|
/** `false` disables the tunnel rung entirely. */
|
package/lib/fetch.js
CHANGED
|
@@ -44,15 +44,33 @@ const DEFAULT_MAX_BYTES = 1_048_576;
|
|
|
44
44
|
const DIRECT_DEADLINE_MS = 5_000;
|
|
45
45
|
/** Default per-attempt deadline. */
|
|
46
46
|
const DEFAULT_TIMEOUT_MS = 15_000;
|
|
47
|
+
/**
|
|
48
|
+
* Budget for the whole ladder, however many rungs it tries.
|
|
49
|
+
*
|
|
50
|
+
* Each rung has its own deadline, so without a ceiling a blocked host can spend
|
|
51
|
+
* five seconds on a direct attempt and fifteen on each later rung — long enough
|
|
52
|
+
* that a caller waits half a minute for a page that will never arrive.
|
|
53
|
+
*/
|
|
54
|
+
const DEFAULT_OVERALL_MS = 25_000;
|
|
55
|
+
/** Below this much remaining budget a rung is not worth starting. */
|
|
56
|
+
const MIN_RUNG_MS = 750;
|
|
57
|
+
/** Content types whose bytes are documents, not text this plugin can read. */
|
|
58
|
+
const BINARY_TYPE = /^\s*(application\/(pdf|zip|gzip|x-tar|vnd\.|msword|octet-stream|protobuf|x-protobuf)|image\/|audio\/|video\/|font\/)/i;
|
|
47
59
|
/** Pick the body kind from the content type, defaulting to text. */
|
|
48
60
|
function kindOf(contentType) {
|
|
49
61
|
if (contentType === null)
|
|
50
62
|
return 'text';
|
|
51
63
|
const value = contentType.toLowerCase();
|
|
64
|
+
if (BINARY_TYPE.test(value))
|
|
65
|
+
return 'binary';
|
|
52
66
|
if (value.includes('html') || value.includes('xml'))
|
|
53
67
|
return 'html';
|
|
54
68
|
return 'text';
|
|
55
69
|
}
|
|
70
|
+
/** The mime type without parameters, for a message a reader can act on. */
|
|
71
|
+
function mimeOf(contentType) {
|
|
72
|
+
return (contentType ?? '').split(';')[0]?.trim() ?? '';
|
|
73
|
+
}
|
|
56
74
|
/**
|
|
57
75
|
* Cut a body at the byte budget without splitting a surrogate pair.
|
|
58
76
|
* @param text - the decoded body.
|
|
@@ -94,7 +112,13 @@ async function directFetch(url, timeoutMs, signal, userAgent, maxBytes) {
|
|
|
94
112
|
const contentType = response.headers.get('content-type');
|
|
95
113
|
const text = await response.text();
|
|
96
114
|
const { content, truncated } = clamp(text, maxBytes);
|
|
97
|
-
return {
|
|
115
|
+
return {
|
|
116
|
+
url: response.url === '' ? url : response.url,
|
|
117
|
+
statusCode: response.status,
|
|
118
|
+
body: { kind: kindOf(contentType), content },
|
|
119
|
+
contentType: mimeOf(contentType),
|
|
120
|
+
truncated,
|
|
121
|
+
};
|
|
98
122
|
}
|
|
99
123
|
/** Tunnelled request used by the second rung. */
|
|
100
124
|
async function tunnelFetch(url, proxy, timeoutMs, signal, userAgent, maxBytes) {
|
|
@@ -105,10 +129,12 @@ async function tunnelFetch(url, proxy, timeoutMs, signal, userAgent, maxBytes) {
|
|
|
105
129
|
...(signal === undefined ? {} : { signal }),
|
|
106
130
|
});
|
|
107
131
|
const { content, truncated } = clamp(response.text, maxBytes);
|
|
132
|
+
const contentType = response.headers?.['content-type'] ?? null;
|
|
108
133
|
return {
|
|
109
134
|
url,
|
|
110
135
|
statusCode: response.status,
|
|
111
|
-
body: { kind: kindOf(
|
|
136
|
+
body: { kind: kindOf(contentType), content },
|
|
137
|
+
contentType: mimeOf(contentType),
|
|
112
138
|
truncated,
|
|
113
139
|
};
|
|
114
140
|
}
|
|
@@ -146,6 +172,10 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
146
172
|
...overrides,
|
|
147
173
|
};
|
|
148
174
|
const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
|
175
|
+
const overallMs = options.overallMs ?? DEFAULT_OVERALL_MS;
|
|
176
|
+
const startedAt = Date.now();
|
|
177
|
+
/** What is left of the whole-ladder budget. */
|
|
178
|
+
const remaining = () => Math.max(0, overallMs - (Date.now() - startedAt));
|
|
149
179
|
const maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES;
|
|
150
180
|
const userAgent = options.userAgent ?? USER_AGENT;
|
|
151
181
|
const apiKey = (options.anysearchApiKey ?? '').trim();
|
|
@@ -196,6 +226,10 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
196
226
|
const finalize = (outcome, render) => {
|
|
197
227
|
const format = liveFormat();
|
|
198
228
|
const maxLength = liveMaxLength();
|
|
229
|
+
if (outcome.body.kind === 'binary') {
|
|
230
|
+
const mime = outcome.contentType === undefined || outcome.contentType === '' ? 'binary content' : outcome.contentType;
|
|
231
|
+
throw new SearchError(`cannot read ${mime} from ${outcome.url}: this plugin converts HTML and text to Markdown, and a binary document would arrive as noise`, 'SEARCH_UNSUPPORTED', false);
|
|
232
|
+
}
|
|
199
233
|
if (outcome.body.kind === 'html') {
|
|
200
234
|
const rendered = renderFetchText(outcome.body.content, { format, maxLength, baseUrl: outcome.url });
|
|
201
235
|
return {
|
|
@@ -213,6 +247,26 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
213
247
|
render,
|
|
214
248
|
};
|
|
215
249
|
};
|
|
250
|
+
/**
|
|
251
|
+
* Reject an outcome a caller would otherwise read as a successful page.
|
|
252
|
+
*
|
|
253
|
+
* A 4xx/5xx stays a normal result on purpose: the Host renders the status line
|
|
254
|
+
* with the body, and a 403 page often carries the only useful explanation. An
|
|
255
|
+
* empty 200 is different — it reads as success while saying nothing.
|
|
256
|
+
*/
|
|
257
|
+
const assertUsable = (outcome, failures) => {
|
|
258
|
+
if (outcome.statusCode < 400 && outcome.body.content.trim() === '') {
|
|
259
|
+
// The render rung's reason describes the raw bytes, not the extraction
|
|
260
|
+
// result, so never lead with it: "7387 readable characters" next to "empty
|
|
261
|
+
// page" reads like a contradiction.
|
|
262
|
+
const detail = failures.length === 0 ? '' : `; ${failures.join('; ')}`;
|
|
263
|
+
const render = outcome.render?.reason === undefined || outcome.render.reason === ''
|
|
264
|
+
? ''
|
|
265
|
+
: `; renderer: ${outcome.render.reason}`;
|
|
266
|
+
throw new SearchError(`no readable text from ${outcome.url} (HTTP ${outcome.statusCode}): the bytes carried no prose once scripts and markup were removed${render}${detail}`, 'SEARCH_EMPTY', true);
|
|
267
|
+
}
|
|
268
|
+
return outcome;
|
|
269
|
+
};
|
|
216
270
|
return {
|
|
217
271
|
id: options.id ?? 'http-proxy',
|
|
218
272
|
available: () => true,
|
|
@@ -318,7 +372,10 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
318
372
|
// that cannot work would leave the rung that can work with nothing.
|
|
319
373
|
let direct;
|
|
320
374
|
try {
|
|
321
|
-
|
|
375
|
+
const directBudget = Math.min(timeoutMs, DIRECT_DEADLINE_MS, remaining());
|
|
376
|
+
if (directBudget < MIN_RUNG_MS)
|
|
377
|
+
throw new Error('no time left in the fetch budget');
|
|
378
|
+
direct = await deps.direct(target, directBudget, signal, userAgent, maxBytes);
|
|
322
379
|
}
|
|
323
380
|
catch (error) {
|
|
324
381
|
if (cancelled())
|
|
@@ -328,7 +385,7 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
328
385
|
if (direct !== undefined) {
|
|
329
386
|
const rendered = await settle(direct);
|
|
330
387
|
if (rendered !== undefined)
|
|
331
|
-
return finalize(rendered, reportFor());
|
|
388
|
+
return assertUsable(finalize(rendered, reportFor()), failures);
|
|
332
389
|
}
|
|
333
390
|
let proxy;
|
|
334
391
|
if (options.useProxy !== false) {
|
|
@@ -342,7 +399,10 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
342
399
|
if (proxy !== undefined && proxy.length > 0) {
|
|
343
400
|
let tunneled;
|
|
344
401
|
try {
|
|
345
|
-
|
|
402
|
+
const tunnelBudget = Math.min(timeoutMs, remaining());
|
|
403
|
+
if (tunnelBudget < MIN_RUNG_MS)
|
|
404
|
+
throw new Error('no time left in the fetch budget');
|
|
405
|
+
tunneled = await deps.tunnel(target, proxy, tunnelBudget, signal, userAgent, maxBytes);
|
|
346
406
|
}
|
|
347
407
|
catch (error) {
|
|
348
408
|
if (cancelled())
|
|
@@ -352,7 +412,7 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
352
412
|
if (tunneled !== undefined) {
|
|
353
413
|
const rendered = await settle(tunneled);
|
|
354
414
|
if (rendered !== undefined)
|
|
355
|
-
return finalize(rendered, reportFor());
|
|
415
|
+
return assertUsable(finalize(rendered, reportFor()), failures);
|
|
356
416
|
}
|
|
357
417
|
}
|
|
358
418
|
// `always` is also a standalone rescue: a transport that returns 403 or
|
|
@@ -362,18 +422,21 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
362
422
|
const step = await renderStep(undefined);
|
|
363
423
|
lastReport = step.report;
|
|
364
424
|
if (step.done && step.outcome !== undefined)
|
|
365
|
-
return finalize(step.outcome, step.report);
|
|
425
|
+
return assertUsable(finalize(step.outcome, step.report), failures);
|
|
366
426
|
if (!step.done)
|
|
367
427
|
failures.push(`render: ${step.report.reason}`);
|
|
368
428
|
}
|
|
369
429
|
if (apiKey.length > 0) {
|
|
370
430
|
try {
|
|
371
|
-
const
|
|
431
|
+
const extractBudget = Math.min(timeoutMs, remaining());
|
|
432
|
+
if (extractBudget < MIN_RUNG_MS)
|
|
433
|
+
throw new Error('no time left in the fetch budget');
|
|
434
|
+
const extracted = await deps.extract(target, apiKey, extractBudget, signal);
|
|
372
435
|
// Say where the body came from when the browser was tried first.
|
|
373
436
|
const report = lastReport !== undefined && lastReport.attempted && !lastReport.applied
|
|
374
437
|
? makeReport(lastReport.mode, true, false, `${lastReport.reason}; the result came from AnySearch extraction`)
|
|
375
438
|
: reportFor();
|
|
376
|
-
return finalize(extracted, report);
|
|
439
|
+
return assertUsable(finalize(extracted, report), failures);
|
|
377
440
|
}
|
|
378
441
|
catch (error) {
|
|
379
442
|
if (cancelled())
|
|
@@ -387,7 +450,7 @@ export function createFetchProvider(options = {}, overrides = {}) {
|
|
|
387
450
|
const report = lastReport !== undefined && lastReport.attempted && !lastReport.applied
|
|
388
451
|
? makeReport(lastReport.mode, true, false, `${lastReport.reason}; the earlier rung's result was kept`)
|
|
389
452
|
: reportFor();
|
|
390
|
-
return finalize(fallback, report);
|
|
453
|
+
return assertUsable(finalize(fallback, report), failures);
|
|
391
454
|
}
|
|
392
455
|
throw new SearchError(`cannot fetch ${target} (${failures.join('; ')})`, 'SEARCH_NETWORK', true);
|
|
393
456
|
},
|
package/lib/markdown.js
CHANGED
|
@@ -437,12 +437,25 @@ function pickMain(root) {
|
|
|
437
437
|
articleCharacters = characters;
|
|
438
438
|
}
|
|
439
439
|
});
|
|
440
|
-
if (article !== undefined && articleCharacters > 0)
|
|
441
|
-
return article;
|
|
442
440
|
const main = findElement(body, (node) => node.tag === 'main' || (node.attrs['role'] ?? '').toLowerCase() === 'main');
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
441
|
+
const densest = densestContainer(body);
|
|
442
|
+
// A `<main>` or `<article>` that holds almost nothing is a layout shell, not
|
|
443
|
+
// the reading root: github.com returns half a megabyte of HTML whose `main`
|
|
444
|
+
// is a sidebar, which used to convert to zero characters and look like a
|
|
445
|
+
// network failure. Only trust a landmark that keeps a real share of the text.
|
|
446
|
+
const bodyCharacters = characterCount(body);
|
|
447
|
+
const floor = Math.min(2_000, Math.max(120, Math.round(bodyCharacters * 0.25)));
|
|
448
|
+
const landmarks = [article, main];
|
|
449
|
+
for (const landmark of landmarks) {
|
|
450
|
+
if (landmark !== undefined && characterCount(landmark) >= floor)
|
|
451
|
+
return landmark;
|
|
452
|
+
}
|
|
453
|
+
if (characterCount(densest) >= floor)
|
|
454
|
+
return densest;
|
|
455
|
+
// Nothing reaches the floor: return whichever of the three holds the most.
|
|
456
|
+
const ranked = [article, main, densest].filter((node) => node !== undefined);
|
|
457
|
+
ranked.sort((left, right) => characterCount(right) - characterCount(left));
|
|
458
|
+
return ranked[0] ?? body;
|
|
446
459
|
}
|
|
447
460
|
/** Replace a code fragment with a placeholder so clean-up cannot touch it. */
|
|
448
461
|
function stashCode(ctx, text) {
|
|
@@ -795,13 +808,24 @@ export function htmlToMarkdown(html, options = {}) {
|
|
|
795
808
|
const body = findElement(root, (node) => node.tag === 'body') ?? root;
|
|
796
809
|
if (characterCount(body) === 0)
|
|
797
810
|
return '';
|
|
798
|
-
const
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
811
|
+
const renderFrom = (node) => {
|
|
812
|
+
const ctx = {
|
|
813
|
+
format: options.format ?? 'markdown',
|
|
814
|
+
baseUrl: options.baseUrl,
|
|
815
|
+
stash: [],
|
|
816
|
+
};
|
|
817
|
+
return restoreCode(cleanup(renderElement(node, ctx)), ctx).trim();
|
|
803
818
|
};
|
|
804
|
-
|
|
819
|
+
const main = pickMain(root);
|
|
820
|
+
const chosen = renderFrom(main);
|
|
821
|
+
// Last guard: if the landmark produced (almost) nothing while the page holds
|
|
822
|
+
// real prose, read the body instead of returning an empty document.
|
|
823
|
+
if (chosen.length < 120 && characterCount(body) >= 400) {
|
|
824
|
+
const whole = renderFrom(body);
|
|
825
|
+
if (whole.length > chosen.length)
|
|
826
|
+
return whole;
|
|
827
|
+
}
|
|
828
|
+
return chosen;
|
|
805
829
|
}
|
|
806
830
|
/**
|
|
807
831
|
* Apply the output character budget.
|
package/lib/render.d.ts
CHANGED
|
@@ -43,7 +43,7 @@ export declare const SHELL_MARKER_READABLE_LIMIT = 1200;
|
|
|
43
43
|
export interface ShellInput {
|
|
44
44
|
readonly statusCode: number;
|
|
45
45
|
readonly body: {
|
|
46
|
-
readonly kind: 'html' | 'text';
|
|
46
|
+
readonly kind: 'html' | 'text' | 'binary';
|
|
47
47
|
readonly content: string;
|
|
48
48
|
};
|
|
49
49
|
}
|
package/lib/types.d.ts
CHANGED
|
@@ -142,4 +142,12 @@ export interface HealthOptions {
|
|
|
142
142
|
maxCooldownMs?: number;
|
|
143
143
|
}
|
|
144
144
|
/** Business codes a failure can carry. */
|
|
145
|
-
export type SearchErrorCode = 'SEARCH_THROTTLED' | 'SEARCH_BLOCKED' | 'SEARCH_TIMEOUT' | 'SEARCH_NETWORK' | 'SEARCH_PARSE' | 'SEARCH_NETWORK_BLOCKED_BY_POLICY' | 'SEARCH_ABORTED'
|
|
145
|
+
export type SearchErrorCode = 'SEARCH_THROTTLED' | 'SEARCH_BLOCKED' | 'SEARCH_TIMEOUT' | 'SEARCH_NETWORK' | 'SEARCH_PARSE' | 'SEARCH_NETWORK_BLOCKED_BY_POLICY' | 'SEARCH_ABORTED'
|
|
146
|
+
/** The caller sent nonsense (an empty query, a non-HTTP URL). */
|
|
147
|
+
| 'SEARCH_INVALID'
|
|
148
|
+
/** The page answered, but nothing readable came back. */
|
|
149
|
+
| 'SEARCH_EMPTY'
|
|
150
|
+
/** The page answered with a 4xx/5xx status. */
|
|
151
|
+
| 'SEARCH_HTTP_STATUS'
|
|
152
|
+
/** The page is a binary document this plugin cannot convert to text. */
|
|
153
|
+
| 'SEARCH_UNSUPPORTED';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fffangchenghao/dsh-web-search",
|
|
3
|
-
"version": "0.13.
|
|
3
|
+
"version": "0.13.7",
|
|
4
4
|
"description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|