web-forager 1.3.0 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.clinerules/web-forager.md +3 -3
- package/.codebuddy-plugin/marketplace.json +1 -1
- package/.codebuddy-plugin/plugin.json +1 -1
- package/.codex-plugin/marketplace.json +1 -1
- package/.codex-plugin/mcp.json +20 -20
- package/.codex-plugin/plugin.json +1 -1
- package/.continue/rules/web-forager.md +4 -4
- package/.cursor/rules/web-forager.mdc +4 -4
- package/.dsh-plugin/index.js +1 -1
- package/.dsh-plugin/plugin.json +1 -1
- package/.github/copilot-instructions.md +3 -3
- package/.roo/rules/web-forager.md +3 -3
- package/.windsurf/rules/web-forager.md +4 -4
- package/AGENTS.md +10 -3
- package/CHANGELOG.md +508 -454
- package/README.md +499 -477
- package/README_zh.md +498 -477
- package/SKILL.md +115 -15
- package/THIRD_PARTY_LICENSES.txt +5 -0
- package/browser/bridges/playwright-bridge.js +8 -1
- package/browser/launch_chrome.py +193 -144
- package/browser/lib/bili-api.js +12 -1
- package/browser/lib/safety.js +27 -1
- package/browser/lib/session.js +16 -2
- package/browser/lib/untrusted.js +215 -56
- package/browser/mcp_server.js +5 -25
- package/browser/package.json +1 -1
- package/browser/playbooks/search-dig.js +3 -1
- package/browser/playbooks/wechat-dig.js +78 -78
- package/browser/run.js +19 -10
- package/commands/research.md +35 -0
- package/cordis.patch.yml +19 -19
- package/docs/DEPENDENCIES.md +1 -1
- package/docs/INSTALL_COMMANDS.md +4 -4
- package/docs/INSTALL_COMMANDS_zh.md +4 -4
- package/docs/REFERENCE.md +349 -308
- package/docs/REFERENCE_zh.md +342 -309
- package/manifest.json +3 -2
- package/mcp_server.py +383 -130
- package/package.json +4 -2
- package/requirements.txt +7 -4
- package/run.py +61 -5
- package/scripts/bilibili.py +11 -3
- package/scripts/engine.py +1039 -118
- package/scripts/pdf.py +21 -0
- package/scripts/research.py +2087 -0
- package/scripts/scholar.py +33 -6
- package/scripts/sitemap.py +20 -5
- package/scripts/untrusted.py +312 -95
package/CHANGELOG.md
CHANGED
|
@@ -1,454 +1,508 @@
|
|
|
1
|
-
# Changelog
|
|
2
|
-
|
|
3
|
-
## 1.3.
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
###
|
|
8
|
-
|
|
9
|
-
-
|
|
10
|
-
- **
|
|
11
|
-
- **
|
|
12
|
-
-
|
|
13
|
-
- **
|
|
14
|
-
- **
|
|
15
|
-
- **
|
|
16
|
-
- **
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
- **
|
|
26
|
-
- **
|
|
27
|
-
- **
|
|
28
|
-
- **
|
|
29
|
-
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
- `
|
|
34
|
-
-
|
|
35
|
-
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
- **
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
- **
|
|
65
|
-
- **
|
|
66
|
-
-
|
|
67
|
-
- **
|
|
68
|
-
- **
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
-
|
|
75
|
-
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
- **
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
###
|
|
109
|
-
|
|
110
|
-
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
-
|
|
131
|
-
|
|
132
|
-
- `
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
###
|
|
136
|
-
|
|
137
|
-
- **
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
- **
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
- **
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
- **
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
- **
|
|
253
|
-
`
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
`
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
to
|
|
281
|
-
|
|
282
|
-
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
-
|
|
290
|
-
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
-
|
|
325
|
-
|
|
326
|
-
-
|
|
327
|
-
|
|
328
|
-
- The
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
- **
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
-
|
|
445
|
-
|
|
446
|
-
-
|
|
447
|
-
- `--
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.3.1 — 2026-11-17
|
|
4
|
+
|
|
5
|
+
Precision work on the two weakest parts of 1.3.0: the injection screen treated every keyword hit as a reason to withhold, and search relevance was close to binary. This release also adds URL canonicalization, reciprocal-rank fusion, Chinese bigram matching, intent-aware source authority and a broad-recall/focused-evidence `research` action.
|
|
6
|
+
|
|
7
|
+
### Security
|
|
8
|
+
|
|
9
|
+
- **MCP can no longer loosen the boundary.** `respect_robots`, `robots_policy`, `input_gate` and `proxy` were removed from every HTTP tool schema and are force-set to their safe values for agent calls. The local-file `clean` tool (`web_clean`) was removed from MCP entirely: it read an arbitrary path chosen by the model and is now CLI-only. The browser MCP server lost `bridge`, `out`, `root`, `allowWrite`, `inputGate` and `strictRobots`; every MCP playbook run is read-only and always passes `--strict-robots`.
|
|
10
|
+
- **Search and scholarly query endpoints now have an explicit policy.** `--query-robots` (or `WEB_FORAGER_QUERY_ROBOTS`) defaults to `warn`. Bing, 360, Sogou and the arXiv export API disallow their query paths, so the default reads them and records `query_robots: warn` in the engine report; `respect` refuses and `ignore` skips the check, and only a user can choose them. MCP does not expose the switch, though an authorised user can set the variable in the server environment. This closes the previous unconditional `check_robots=False` in the search and scholar paths.
|
|
11
|
+
- **Every redirect hop is checked against robots.txt.** HTTP fetches, URL resolution, the browser bridge and the Bilibili API client now re-check the final target. A public URL that redirects into a disallowed path is refused instead of read.
|
|
12
|
+
- **Rendered pages install a subresource guard.** Python Playwright rendering aborts `file:`/`filesystem:` requests and private HTTP(S) subresources, matching the browser bridge's request filter, so a hostile page cannot use an iframe or image to probe the LAN.
|
|
13
|
+
- **Bounded local extraction.** Sitemap gzip output is capped at 32 MiB through a streaming decompressor, PDF extraction stops at 500 pages or 20 seconds with `truncated` and a warning, and `clean --input` refuses local files over 16 MiB (stdin included). A few kilobytes of gzip can no longer expand into gigabytes.
|
|
14
|
+
- **Synthetic research rows are screened.** Feed headlines, official-index headlines and WeChat snippets now carry `trust: "untrusted"`, `injection_flags` and `injection_findings` like fetched pages, instead of entering evidence unscreened.
|
|
15
|
+
- **Dependency floors follow current advisories.** `certifi>=2026.10.18`, `requests>=2.33.0`, `beautifulsoup4>=4.14.3`, `trafilatura>=2.3.1` and `pypdf>=6.21.4`; the local environment is on pypdf 7.1.0, certifi 2026.10.23, requests 2.34.2 and soupsieve 4.1.1.
|
|
16
|
+
- **Local state is owner-only.** On POSIX the HTTP cache directory and browser profile are `0700`, cache files, surf sessions and browser sessions are `0600`. The cache already refused responses that set cookies and stripped `Authorization` / `Proxy-Authorization` headers; this release adds the permission boundary and documents what leaves the machine. There is no telemetry, account or API key.
|
|
17
|
+
- **CDP is off by default.** `browser/launch_chrome.py` now opens only the dedicated profile; `--cdp` opts in to a loopback CDP port, random by default (`--port 9222` fixes it, and passing `--port` implies `--cdp`). The launcher prints a warning that CDP has no authentication. Without `--cdp`, browser mode reuses the profile through its normal bridge, so the login flow needs no open debug port.
|
|
18
|
+
- **Verbose logging redacts URL query strings and fragments.** `--verbose` keeps the scheme, host and path, so a signed URL cannot write its token to stderr. The full URL remains in the result record the caller requested.
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- **`research` action and `web_research` MCP tool.** A question is first widened across the free engines, then a few distinct, relevant sources are selected by an explicit authority tier and read in full. The result carries `seeds`, cited `evidence` passages (`url`, `title`, `authors`, `published_at`, `date_source`, `passages`), `strategy`, `freshness`, `skipped` and `answerability` (`high`/`medium`/`low`/`none`, plus `source_count`, `view_hosts`, `multi_view`, `stale`). It never invents an answer: thin evidence returns `low` or `none` with the reason.
|
|
23
|
+
- **Intent-aware authority tiers.** `classify_intent` separates technical, news, comparison, policy, howto, research and factual questions. `authority_for` labels each host `primary`, `official`, `academic`, `media`, `reference`, `community`, `low` or `unknown` and weights it by intent; every search result now carries `authority_tier`, `authority_score` and `intent`.
|
|
24
|
+
- **Targeted second pass that survives a broken `site:` operator.** 360 and Bing often ignore `site:` in mainland networks, so research now prefers direct official entry pages (Python asyncio, Go concurrency, Rust, MDN, Microsoft Wi-Fi support), then a site's own public search page (Microsoft Support, MDN), and only then a `site:` query for a non-CJK question. The arXiv API title field remains the paper-shaped route. An official overview page may be followed one same-site link further when that link names the API.
|
|
25
|
+
- **Comparison questions need two full viewpoints.** A comparison is `high` only with multiple independent full sources; a partial WeChat snippet never counts as a viewpoint. When two full authoritative sources already exist for a comparison or technical question, the WeChat-only snippet is skipped instead of being mixed into the evidence.
|
|
26
|
+
- **Cross-language concept matching.** Chinese questions about concurrency, comparisons, connectivity, errors, installation and performance add a small English concept vocabulary for passage matching only. A Chinese “Python 和 Go 高并发” question can therefore match an English Go concurrency page without changing the query sent to the engines.
|
|
27
|
+
- **Publication dates and freshness.** `_normalize_date` reads ISO, Chinese and English metadata/datelines; evidence carries `published_at` and `date_source`, and the payload carries `freshness`. A news answer without a readable date cannot be labelled `high`.
|
|
28
|
+
- **News RSS fallback.** When a news query has no usable engine rows, research reads a public China News Service finance feed and returns dated headlines (`kind: feed_headline`) with an explicit warning. It is a headline/summary, never a fabricated article body. A fetched article that carries no dateline inherits the feed's publication date, so `freshness` is not lost after the full-page read.
|
|
29
|
+
- **Bounded focus reads.** Each research focus fetch is capped at 8 seconds with no retry, and each robots.txt read at 6 seconds, with a 3-second per-address connect timeout. A blocked or dead origin is reported as a skipped fetch instead of consuming the whole call.
|
|
30
|
+
- **CJK bigram matching.** A Chinese sentence is no longer one unmatchable token. Long CJK runs become overlapping character bigrams, while a mixed date token such as `2026年11月` stays whole. Chinese recall that was 0/8 on the priority set now reaches 7–8/8 in local tests.
|
|
31
|
+
- **Search-engine intermediate pages are dropped.** A Sogou/360 result page echoed by another engine is no longer ranked as an answer.
|
|
32
|
+
|
|
33
|
+
- **`search_quality` block.** A search now carries `{query, engine, returned, relevant, off_topic, language_mismatch, engines_answered, engine_errors, engine_status, engine_states, dropped_off_topic, returned_before_filter}` beside the prose warnings, so an agent can decide whether to trust a result list without parsing English. Engine states distinguish `ok`, `empty`, `blocked`, `timeout`, `parse_error` and `error`.
|
|
34
|
+
- **Rank explanations.** Every merged result carries `rank_components` (`coverage`, `title`, `exact`, `rrf`, `authority`, `freshness`, `language_penalty`) and a short `rank_reason`, so one ordering decision can be debugged without printing internal scores.
|
|
35
|
+
- **Canonical URL dedupe.** `utm_*`, `fbclid`, `gclid`, fragments, `www.`, trailing slashes and AMP suffixes no longer make one article look like two results. The display URL is left unchanged; only the dedupe key is normalized.
|
|
36
|
+
- **RRF and exact-title weighting.** A result found by two engines receives a reciprocal-rank-fusion contribution, and an exact title match gets a small bonus over a title that merely contains the phrase. Engine order is still the final tie-break.
|
|
37
|
+
- **Light English stemming for matching.** `injection`/`injections`, `study`/`studies`, `learn`/`learning` meet during relevance checks; the displayed text and query are unchanged.
|
|
38
|
+
- **A quality baseline suite.** `tests/test_quality_baseline.py` fixes the academic-quotation detector case and the Excel INDIRECT search case as regression cases, with a small metrics check on top.
|
|
39
|
+
|
|
40
|
+
### Changed
|
|
41
|
+
|
|
42
|
+
- **Injection findings now carry evidence and a severity.** Every record has `injection_findings: [{code, severity, confidence, span, snippet, context}]` alongside the existing `injection_flags`. `span` is a character range, `severity` is `hard` or `soft`, and `context` is `direct`, `quote` or `example`. The first warning repeats up to three matching snippets, so a false positive can be diagnosed instead of guessed at.
|
|
43
|
+
- **`--input-gate strict` withholds only hard rules.** Forged system tags, secret-exfiltration requests, `curl ... | sh` and `rm -rf /` still reject the body. `OVERRIDE` and `ROLE_SWAP` are soft: a paper quoting an attack template, or a security note discussing "ignore previous instructions", is marked with lower confidence and kept. This directly fixes the arXiv-paper case where a legitimate quotation looked like an attack.
|
|
44
|
+
- **The scan covers the whole body.** The old scanner stopped at 120,000 characters. It now walks overlapping 64k windows, keeps at most three findings per rule, and runs before `max_chars` truncation, so a hostile instruction near the tail of a long page cannot disappear. The browser scanner applies the same rules and limits.
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
|
|
48
|
+
- **A single engine's results are ranked instead of passed through.** `search()` used to return `groups[0][1]` untouched when only one engine answered; every result now goes through the same scorer.
|
|
49
|
+
- **Search relevance is graded, not a 0/1 threshold.** A result must contain the exact phrase or at least two meaningful query terms with half the query covered. Each result carries `coverage`, `matched_terms`, `relevance` and `language_match`. A Chinese tutorial that repeats one English product name is `language_match: false`, ranked last, and dropped when at least three real answers exist. The `Excel INDIRECT` rows are the regression fixture.
|
|
50
|
+
- **Authority beats a content farm on an equal title.** `docs.python.org`, official documentation, academic sources and media are boosted per intent; CSDN, php.cn and document farms are pushed down. The same-title test now puts the official page first.
|
|
51
|
+
- **`search_warnings` reports partial relevance, not only total failure.** It now distinguishes "nothing relevant", "N of M look off-topic" and "no results", and search titles/snippets pass through the same injection scanner as page bodies, carrying their own findings.
|
|
52
|
+
|
|
53
|
+
### Tests
|
|
54
|
+
|
|
55
|
+
- 333 Python tests and 91 browser tests. New coverage: finding evidence and severity, academic-quotation suppression, scanning past 120k, repeated soft hits not hiding a hard rule, strict mode keeping a soft quotation, single-engine ranking, language mismatch, the two-term/phrase relevance rule, dropping off-topic rows when enough real answers exist, search-snippet scanning, canonical URL dedupe, RRF, light stemming, `search_quality` and engine states, rank explanations, the quality baseline corpus, CJK bigram matching, intent/authority tiers, research passage extraction, cross-language concept matching, official comparison entry points, comparison viewpoint rules, date normalization, seed diversity, arXiv author metadata, the WeChat partial-snippet path, redirect robots checks, the query-endpoint policy, the sitemap gzip cap, the PDF page cap, synthetic-row injection screening and the MCP schema restrictions.
|
|
56
|
+
|
|
57
|
+
## 1.3.0 — 2026-11-15
|
|
58
|
+
|
|
59
|
+
A workflow release: a robots policy that can record an authorised override, search that fills its page budget with readable results, parallel engines, and a browser session you can log into once.
|
|
60
|
+
|
|
61
|
+
### Added
|
|
62
|
+
|
|
63
|
+
- **`--robots-policy respect|warn|ignore`.** `respect` is the default and still refuses a disallowed path. `warn` reads it and writes `compliance.robots_overridden: true` plus a warning, for a user who is authorised to read the page. `ignore` is the old `--ignore-robots`, with `robots_checked: false`. Every HTTP record now carries a `compliance` block.
|
|
64
|
+
- **The browser profile launcher** (`browser/launch_chrome.py`) starts Chrome/Chromium/Edge with `~/.web-forager/browser/profile` and a DevTools port, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`.
|
|
65
|
+
- **Storage-state import and export** for browser mode: `--storage-state FILE` loads cookies/localStorage, `--save-storage-state FILE` writes them after the run. A saved file is forced to mode `0600`, a path inside `--root` is refused, and the record says the file carries login state.
|
|
66
|
+
- **`--serial-engines`** keeps a diagnostic run strictly serial; the default now asks engines on different hosts at the same time while each host keeps its one-request-per-second delay.
|
|
67
|
+
- **A `wechat-dig` browser playbook** opens one WeChat article URL in the browser profile and captures its text. When the profile has already passed WeChat's environment check the article is read; when it has not, the playbook reports `blocked` with a hint and does not try to solve the check.
|
|
68
|
+
- **Verification pages are reported instead of recycled.** HTTP mode recognises WeChat's environment check and the common English anti-bot interstitials, raises `VerificationRequired`, and `fetch --query` records the result as `verification_required` with a warning that names the browser route. The CLI exits 9. The bundled bridge also hands the whole `#js_content` article over as one `content` snapshot node, so WeChat paragraphs are not reassembled from leaf spans.
|
|
69
|
+
- **Automatic render fallback for script shells.** When a server answers with a minimal HTML page that carries only a script or redirect, `--render auto` now renders it once with Playwright before extraction. The record says `render_fallback: true`; `fetch --query` allows at most two such renders per query, so one slow site cannot stall a search. This needs no API key and closes the php.cn-style empty-page case.
|
|
70
|
+
- **Free ranking and query repair.** Search results are ordered by query coverage, title match, a small domain prior (docs, papers, government and universities up; document farms down) and snippet freshness. The final list keeps at most two results per host and fills the rest from the overflow. A thin first pass is retried once with a simplified query — quotes and common stopwords removed, or a spaced CJK query joined — and the retry is marked `query_variant` in `engine_report`. No model, key or network service is involved.
|
|
71
|
+
|
|
72
|
+
### Changed
|
|
73
|
+
|
|
74
|
+
- **`max_pages` now means readable pages.** `fetch --query` keeps scanning past a robots-denied or unreadable result (up to a bounded number of attempts), returns the pages it could read, and lists the rest in `skipped` with a code and reason. The response carries a summary warning instead of leaving the agent to discover failures page by page.
|
|
75
|
+
- The `robots_denied` hint now names `--robots-policy warn` for an authorised override and says not to silently switch to a tool that ignores robots. A `verification_required` skip says the same about not solving the check, and points a WeChat link at the `wechat-dig` playbook.
|
|
76
|
+
|
|
77
|
+
### Host compatibility
|
|
78
|
+
|
|
79
|
+
- **Codex now gets a working MCP launch.** The plugin ships `.codex-plugin/mcp.json`, which uses relative `mcp_server.py` and `browser/mcp_server.js` paths with `cwd: "."`; Codex resolves both against the installed plugin directory. It no longer inherits Claude Code's `${CLAUDE_PLUGIN_ROOT}` argument, which Codex leaves literal.
|
|
80
|
+
- **CodeBuddy now declares its MCP servers in the manifest it reads.** `.codebuddy-plugin/plugin.json` carries both `mcpServers` entries with `${CLAUDE_PLUGIN_ROOT}`, so a new CodeBuddy session mounts the HTTP and browser servers. `codebuddy mcp list` can run before plugins are mounted; an empty list there is not a failed install.
|
|
81
|
+
- **dsh now mounts real tools.** `cordis.patch.yml` inserts two `@deepseek-ai/dsh-mcp-client` entries, one per server, and resolves the profile package directory with `dshHomePath(...)`. The previous patch mounted only the no-op `web-forager` module.
|
|
82
|
+
- **The npm package carries every host manifest.** `files` now includes `.claude-plugin/`, `.codex-plugin/`, `.codebuddy-plugin/`, `.dsh-plugin/`, `.mcp.json`, the rule-file directories, `install.py` and `requirements.txt`.
|
|
83
|
+
- The Cursor rule heading now says v1.3.0 instead of v1.0.0.
|
|
84
|
+
|
|
85
|
+
### Documentation
|
|
86
|
+
|
|
87
|
+
- `SKILL.md` and `AGENTS.md` state the rule: a `robots_denied` or `login_required` result is a boundary, not a prompt to re-fetch with a tool that ignores robots.
|
|
88
|
+
- The reference documents the launcher, storage-state switches, the robots policy, the new `compliance` block, the `wechat-dig` playbook and exit 9.
|
|
89
|
+
- Both READMEs were rewritten around a first visit: one pitch, the three install methods, what a run looks like, the security boundary and the legal notice, plus a section on keeping an installed copy current (including the ZCode marketplace-snapshot trap). A DeepSec Shield scan of the release tree reports 0 findings; its agent audit still flags the three signature strings this project defines, which are the scanner's own patterns, not page content.
|
|
90
|
+
|
|
91
|
+
### Tests
|
|
92
|
+
|
|
93
|
+
- **258 Python tests** and **87 browser tests**, all offline. New coverage: the warn policy and its record, clean compliance fields, filling pages past a denial, overlapping engines versus `--serial-engines`, the launcher dry run, saving storage state through the bridge protocol, WeChat verification detection at the engine, the CLI and the search layer, the `wechat-dig` capture and `blocked` paths, the `content` snapshot node, domain/freshness tie-breaks, host diversity, query variants, the shell-render fallback with and without its budget, plus the Codex/CodeBuddy/dsh MCP manifests and the npm host-file list.
|
|
94
|
+
|
|
95
|
+
## 1.2.3 — 2026-11-15
|
|
96
|
+
|
|
97
|
+
A follow-up hardening release after a DeepSec Shield audit of 1.2.2.
|
|
98
|
+
|
|
99
|
+
### Security
|
|
100
|
+
|
|
101
|
+
- **The browser bridge is no longer started through a shell.** `--bridge` and `WEB_FORAGER_BRIDGE` are split into an executable and its arguments and run directly, so a path or environment value cannot append a second command. Quoted paths with spaces still work; pipes, `&&` and redirection are not interpreted.
|
|
102
|
+
- **CI actions are pinned to commit SHAs.** `actions/checkout`, `actions/setup-python` and `actions/setup-node` name the exact commit, so a moved tag cannot change what the workflow runs.
|
|
103
|
+
|
|
104
|
+
### Documentation
|
|
105
|
+
|
|
106
|
+
- The reference states that the bridge command is split on whitespace and started without a shell.
|
|
107
|
+
|
|
108
|
+
### Tests
|
|
109
|
+
|
|
110
|
+
- **228 Python tests** and **81 browser tests**, all offline. New coverage: quoted bridge paths, literal backslashes, an unterminated quote, and a shell metacharacter that must be passed through as an argument rather than executed.
|
|
111
|
+
|
|
112
|
+
## 1.2.2 — 2026-11-15
|
|
113
|
+
|
|
114
|
+
A hardening release that replaces the 1.2.2 build published on 2026-11-12: a request can no longer be aimed at the machine or the LAN, browser state stays in a profile of its own, and fetched text is labelled as third-party data. The legal-notice rewrite and packaging rules from that earlier build are included.
|
|
115
|
+
|
|
116
|
+
### Security
|
|
117
|
+
|
|
118
|
+
- **Private-address guard (SSRF).** Loopback, RFC 1918, link-local (including `169.254.169.254`), CGNAT, multicast, reserved and their IPv6 forms are refused before a socket opens. The host is resolved and checked as written, so decimal IPv4 spellings and IPv6-mapped IPv4 are caught too. The check runs before the request, on every redirect hop (redirects are followed one hop at a time instead of by `requests`), on the robots.txt download, on search-result resolution, and on the URL a rendered page actually lands on. `--allow-private` / `WEB_FORAGER_ALLOW_PRIVATE=1` is the documented opt-out for local fixtures and trusted intranet targets.
|
|
119
|
+
- **Browser navigation is guarded too.** `guardedGoto` checks the requested and the final URL, and the bundled Playwright bridge installs a request filter: every document and subresource the page asks for is resolved, and private targets are aborted. A hostile page can no longer use the open browser to probe `http://127.0.0.1:…` or a cloud metadata endpoint.
|
|
120
|
+
- **Browser state stays in its own profile.** The bundled bridge already kept `~/.web-forager/browser/profile`; the report now states `dedicated_profile` and `shared_browser`, and a CDP attach to a browser you use for everything else adds a warning that every cookie and login in that profile is reachable by the pages a playbook opens.
|
|
121
|
+
- **Fetched text is marked as data.** Every record carries `trust: "untrusted"`. A small scanner (same rules in `scripts/untrusted.py` and `browser/lib/untrusted.js`) names instruction-shaped passages — overrides, fake system tags, `curl … | sh`, secret-exfiltration requests, role swaps — in `injection_flags`, and puts one warning first in the list. `--input-gate strict` withholds the body instead of handing it on; `--input-gate off` skips the scan.
|
|
122
|
+
- **Cookies never enter a record.** The Bilibili cookie is held in memory for the API request and is not written to the record, the session file or an error message; the browser profile directory is reported as a flag, not a path.
|
|
123
|
+
|
|
124
|
+
### Documentation
|
|
125
|
+
|
|
126
|
+
- **Both READMEs were reorganised around a first visit.** The page now opens with what the tool is, what you get and what it refuses, followed by a table of contents, the why section, the three install methods and a quick start split by entry point — chat, terminal, MCP and the optional browser mode — with a "which action do I need" table and copy-paste commands. The first screen carries a one-line notice that links to the full legal terms; the terms themselves moved below the quick start, section by section.
|
|
127
|
+
- **Quick start now holds only the three entry points**: chat, terminal and MCP. The browser playbook table, switches and `--strict-robots` details, and "when the engines are blocked" routes moved to the reference; each is a one-sentence pointer on the landing page. Installation opens with "let your AI install it" as the recommended path for most readers, followed by the plugin-marketplace URL and the per-host commands. The contents is grouped, with acknowledgements, trademarks and licence added.
|
|
128
|
+
- **Reference material moved to `docs/REFERENCE.md` and `docs/REFERENCE_zh.md`**: the full action list and exit-code tables, record fields and `core-1`, workspace output, MCP tool names and protocol checks, requirements, repository layout, test details and extra CLI examples. Every example is its own one-line block, so it can be copied or run without trimming its neighbours. README keeps a three-table quick start, a short exit-code summary and a link.
|
|
129
|
+
- **Long sections fold.** The repository tree, test coverage notes and MCP tool names sit in `<details>` blocks; known limitations are grouped into read-first, edge cases and experimental/high-risk switches.
|
|
130
|
+
- Both READMEs gain a **Security boundaries** section and three limitation bullets (the guard's off switch, the screen's limits, the CDP profile exposure).
|
|
131
|
+
- `AGENTS.md`, `SKILL.md` and the six host rule files state that fetched text is data and must not be executed as instructions.
|
|
132
|
+
- **The README pair carries a six-part legal notice** — purpose, prohibited use, responsibility, no warranty, limitation of liability, redistribution. `SKILL.md`, `AGENTS.md` and the six host rule files carry the short version.
|
|
133
|
+
- **Test fixtures are no longer part of a published archive.** `.gitattributes` marks `tests/` and `browser/tests/` as `export-ignore`, and both npm manifests drop tests from `files`; CI still runs them from the repository clone.
|
|
134
|
+
|
|
135
|
+
### Tests
|
|
136
|
+
|
|
137
|
+
- **228 Python tests** and **75 browser tests**, all offline. New coverage: every private range, a redirect into a metadata endpoint, a two-hop redirect chain, the robots download, the CLI exit code and `--allow-private`, marker detection, strict withholding, and the browser-side guard and scanner.
|
|
138
|
+
|
|
139
|
+
## 1.2.1 — 2026-11-11
|
|
140
|
+
|
|
141
|
+
A search hygiene release: a Chinese query no longer collects dictionary pages for one of its characters, and the README pair was rewritten around what the tool does now.
|
|
142
|
+
|
|
143
|
+
### Fixed
|
|
144
|
+
|
|
145
|
+
- **A mostly-CJK query is no longer handed to Bing as a fallback.** Bing splits a Chinese phrase into single characters, so a search about 反逆向工程 came back with the dictionary page of 反 once the Chinese engines answered thinly. `auto` now stops after 360, Sogou and 搜狗微信; `--engine bing` still asks Bing explicitly.
|
|
146
|
+
- **Dictionary pages that match one character of the query are dropped**, whatever engine returned them. `looks_like_character_page()` recognises a title that shares exactly one character with a longer query and carries a dictionary marker (百科, 拼音, 笔顺, 部首, 组词…), so a result list is not padded with them and `engine_report` counts only usable hits.
|
|
147
|
+
|
|
148
|
+
### Documentation
|
|
149
|
+
|
|
150
|
+
- **Both READMEs were rewritten.** The duplicated mode paragraphs and the second disclaimer block are gone; scope and compliance sit in one section, the two modes face each other in one table, and the CJK search routing, `find`/`feed`, CI and current tool/test counts are stated once. README text now matches the 10 HTTP tools, 4 browser tools and 9 CLI actions that actually ship.
|
|
151
|
+
|
|
152
|
+
### Tests
|
|
153
|
+
|
|
154
|
+
- **174 Python tests** and **60 browser tests**, all offline. New coverage: a CJK query never calls Bing, a character page is dropped from an engine's results, and `looks_like_character_page()` leaves a single-character query and a full-phrase hit alone.
|
|
155
|
+
|
|
156
|
+
## 1.2.0 — 2026-11-09
|
|
157
|
+
|
|
158
|
+
Search is no longer the only way in. A site's own search page and its feed are
|
|
159
|
+
first-class routes now, every engine attempt is reported, and one run can be
|
|
160
|
+
sent through a proxy.
|
|
161
|
+
|
|
162
|
+
### Added
|
|
163
|
+
|
|
164
|
+
- **`--action find` (`web_find`)**: search inside one site through the site's
|
|
165
|
+
own search page. It tries `/?s=`, `/search?q=`, `/?search=` and
|
|
166
|
+
`/search?query=`, ranks the page's links by query coverage, and fetches the
|
|
167
|
+
best ones. This is the route that keeps working when the general engines
|
|
168
|
+
answer with dictionary pages or a verification wall.
|
|
169
|
+
- **`--action feed` (`web_feed`)**: read a site's RSS/Atom feed, discovering it
|
|
170
|
+
from `<link rel=alternate>` when given the site URL instead of the feed URL.
|
|
171
|
+
- **`engine_report`**: every search response now carries one entry per engine
|
|
172
|
+
tried — `ok`, `results`, `ms`, and `error` when it failed, plus `fallback`
|
|
173
|
+
when the engine was asked only because the first answer was thin. "Search is
|
|
174
|
+
broken" becomes "Bing answered, Sogou showed a verification page".
|
|
175
|
+
- **`doctor --probe` (`web_doctor {probe: true}`)**: asks 360, Bing, Sogou, the
|
|
176
|
+
WeChat index and DuckDuckGo once each and reports which ones answer from this
|
|
177
|
+
network, with latency.
|
|
178
|
+
- **`--proxy URL`** and `proxy` on the MCP tools. Requests already honoured
|
|
179
|
+
`HTTPS_PROXY` / `ALL_PROXY`; an explicit proxy now overrides them for one run,
|
|
180
|
+
and `doctor` reports the proxy in use with credentials redacted.
|
|
181
|
+
|
|
182
|
+
### Changed
|
|
183
|
+
|
|
184
|
+
- The MCP server exposes ten HTTP tools instead of eight; the README and
|
|
185
|
+
`SKILL.md` document the two routes that do not depend on a search engine.
|
|
186
|
+
- `doctor` reports the proxy source (`config` or `env:NAME`) without printing a
|
|
187
|
+
password.
|
|
188
|
+
|
|
189
|
+
### Documentation
|
|
190
|
+
|
|
191
|
+
- **The disclaimer forbids building a profile of an identifiable person.**
|
|
192
|
+
Collecting, aggregating or inferring personal information, or using it for
|
|
193
|
+
doxxing, harassment or stalking, is prohibited and is the user's own legal
|
|
194
|
+
responsibility. It appears in `--help`, `manifest.json`, the README pair,
|
|
195
|
+
`SKILL.md` and every host rule file.
|
|
196
|
+
- **The per-host install commands moved to `docs/INSTALL_COMMANDS.md` and
|
|
197
|
+
`docs/INSTALL_COMMANDS_zh.md`.** Methods 1 and 3 stay in the README because
|
|
198
|
+
they are three lines each; Method 2, which is a page of commands, is one link.
|
|
199
|
+
- **The READMEs gained an Acknowledgements section** listing the reporter of
|
|
200
|
+
issue #1 by name, with a link to `docs/ACKNOWLEDGEMENTS.md` /
|
|
201
|
+
`docs/ACKNOWLEDGEMENTS_zh.md` for the details.
|
|
202
|
+
|
|
203
|
+
### Acknowledgements
|
|
204
|
+
|
|
205
|
+
- **NotFatJustRound** ([issue #1](https://github.com/JiamingShi-JsPhys/web-forager/issues/1))
|
|
206
|
+
reported that `web_search` returned every page body twice — once in `pages`,
|
|
207
|
+
once in `results` — which doubled the response and pushed the tail past an
|
|
208
|
+
agent's size limit; the compact `results` index in 1.0.2 came from that report.
|
|
209
|
+
|
|
210
|
+
### Fixed
|
|
211
|
+
|
|
212
|
+
- **Links inside `<article><header>` are no longer skipped.** A results page
|
|
213
|
+
commonly wraps each card's title in an article header; the universal
|
|
214
|
+
"skip `<header>`" rule silently dropped exactly the links a search page
|
|
215
|
+
offers. Page-level headers and navigations are still skipped.
|
|
216
|
+
- `doctor --probe` makes one attempt per engine instead of retrying, so a
|
|
217
|
+
blocking engine costs one timeout rather than three.
|
|
218
|
+
- `.cursor/rules/web-forager.mdc` carried `v1.0.0` in its front matter, so
|
|
219
|
+
`install.py --only cursor` wrote a stale label; it now follows the release
|
|
220
|
+
version.
|
|
221
|
+
|
|
222
|
+
### Tests
|
|
223
|
+
|
|
224
|
+
- **172 Python tests** and **60 browser tests**, all offline. The new suite
|
|
225
|
+
covers site search against a local WordPress-style page, feed discovery from a
|
|
226
|
+
page link and from a direct feed, the per-engine report, the probe, proxy
|
|
227
|
+
wiring and the redaction of proxy credentials.
|
|
228
|
+
|
|
229
|
+
## 1.1.0 — 2026-11-08
|
|
230
|
+
|
|
231
|
+
Browser mode joins web-forager, and an audit pass follows it. The HTTP toolkit
|
|
232
|
+
and the browser layer ship as **one plugin, one name and one version**; the
|
|
233
|
+
former standalone `web-actor` product no longer exists. The empty-result path,
|
|
234
|
+
the bridge's capability claims, the robots.txt audit trail and the shared record
|
|
235
|
+
shape are now honest, and neither README claims coverage the code does not have.
|
|
236
|
+
|
|
237
|
+
### Added
|
|
238
|
+
|
|
239
|
+
- **Browser mode** under `browser/`: a CLI, seven read-only playbooks and an MCP
|
|
240
|
+
server that drive a real browser through a pluggable bridge.
|
|
241
|
+
- **Playbooks**: `bili-feed`, `bili-video`, `bili-chain`, `bili-danmaku`,
|
|
242
|
+
`zhihu-dig`, `xhs-dig` and `search-dig`.
|
|
243
|
+
- **Second MCP server**: `browser/mcp_server.js` exposes `browser_list`,
|
|
244
|
+
`browser_run`, `browser_session` and `browser_doctor`; `.mcp.json`
|
|
245
|
+
registers both servers.
|
|
246
|
+
- **A reference bridge that is found automatically**:
|
|
247
|
+
`browser/bridges/playwright-bridge.js` supports a fresh browser, a persistent
|
|
248
|
+
profile, a headful window and attaching over CDP. `browser/lib/bridge.js`
|
|
249
|
+
looks for `--bridge`, then `WEB_FORAGER_BRIDGE`, then that file, so a first
|
|
250
|
+
run no longer fails with `backend_unavailable` only because nobody exported a
|
|
251
|
+
variable.
|
|
252
|
+
- **A browser profile that persists by default**:
|
|
253
|
+
`~/.web-forager/browser/profile` keeps cookies and logins between runs;
|
|
254
|
+
`WEB_FORAGER_BROWSER_PROFILE=off` restores the throwaway browser. The
|
|
255
|
+
reference bridge also drops the automation banner flag.
|
|
256
|
+
- **Browser records**: the same `metadata / markdown / tables / warnings` shape,
|
|
257
|
+
written to `<WS>/process/web-forager/browser/` and `_broadcast/`.
|
|
258
|
+
- **`/browse` command** for the browser playbooks.
|
|
259
|
+
- **Shared record core (`core-1`)**: `--core` on both CLIs and `core: true` on
|
|
260
|
+
`browser_run` return `schema_version / source / mode / url / title /
|
|
261
|
+
collected_at / metadata / markdown / tables / warnings / extra`, so a mixed
|
|
262
|
+
list of HTTP and browser records can be read uniformly. The same fold is
|
|
263
|
+
available as `engine.normalize_record()` and `normalizeRecord()`.
|
|
264
|
+
- **Browser robots auditing**: `--strict-robots` reads `robots.txt` before every
|
|
265
|
+
navigation and stops with `robots_denied` (exit 3) on a disallowed path.
|
|
266
|
+
Without it the record still states `compliance.robots_checked: false`, so the
|
|
267
|
+
compliance status is auditable either way. `browser/lib/robots.js` implements
|
|
268
|
+
the same RFC 9309 subset as the Python engine.
|
|
269
|
+
- **`degraded` (exit 8) and `blocked` (exit 9)**: a browser run that reads a
|
|
270
|
+
page but finds nothing the playbook looks for now reports `degraded` with its
|
|
271
|
+
records attached, instead of `ok: true` with `captured: 0`. An anti-bot or
|
|
272
|
+
error interstitial reports `blocked`. Both are distinct from `login_required`.
|
|
273
|
+
- **`describe` handshake**: a bridge can declare its capabilities; a bridge that
|
|
274
|
+
does not gets `declared: false` with every capability false. The reference
|
|
275
|
+
Playwright bridge and the test fixtures implement it.
|
|
276
|
+
- **`browser/lib/selectors.js`**: every structural assumption about a
|
|
277
|
+
third-party page lives in one file.
|
|
278
|
+
- **Real-page fixtures**: `browser/tests/fixtures/snapshots/` holds snapshots
|
|
279
|
+
built from live Bilibili pages on 2026-11-07 by
|
|
280
|
+
`browser/tests/tools/html-to-snapshot.py`, with `tests/real-pages.test.js`
|
|
281
|
+
as the offline regression and `npm run test:live` as the opt-in live check.
|
|
282
|
+
- **`--action doctor`** and the **`web_doctor` MCP tool** report every
|
|
283
|
+
dependency with its version or its install command, so an agent that receives
|
|
284
|
+
a thin page can install what is missing instead of guessing.
|
|
285
|
+
- **`node browser/run.js --doctor`** and the **`browser_doctor` MCP tool** do the
|
|
286
|
+
same for the browser stack: Node version, the bridge command in use, whether
|
|
287
|
+
the `playwright` package resolves (searched from the bridge's own directory
|
|
288
|
+
first), whether its Chromium binary is downloaded, and where the profile is.
|
|
289
|
+
- **The Bilibili search API is asked before the result page is read.**
|
|
290
|
+
`lib/bili-api.js` calls Bilibili's public JSON search endpoint (the `buvid3`
|
|
291
|
+
cookie comes from one home-page request) and falls back to scraping the page
|
|
292
|
+
only when the endpoint refuses. `source` in the record says which path ran.
|
|
293
|
+
`WEB_FORAGER_BILI_API=off` keeps the playbook on the page.
|
|
294
|
+
- **`respect_robots` on the MCP tools.** `web_fetch`, `web_search` and
|
|
295
|
+
`web_surf` accept the same switch the CLI spells `--ignore-robots`, so a page
|
|
296
|
+
that robots.txt disallows can be read when the caller is authorised to read
|
|
297
|
+
it. The default is still to check.
|
|
298
|
+
- **Bilibili's JSON search on the direct path.** When robots checks are off
|
|
299
|
+
(`--direct-search` or a global override), `scripts/bilibili.py` asks
|
|
300
|
+
`api.bilibili.com` first — the `buvid3` cookie comes from one home-page
|
|
301
|
+
request, and that request is never answered from the cache because a cached
|
|
302
|
+
anti-bot page supplies no cookie — and scrapes the search page only if the
|
|
303
|
+
API answers `-412` or malformed JSON. Cards read from the page are kept only
|
|
304
|
+
when their text mentions the query, so a recommendation rail can no longer
|
|
305
|
+
pass as search results. `route` in the payload names the host that answered.
|
|
306
|
+
- **The WeChat article index as a search engine (`--engine weixin`).**
|
|
307
|
+
`weixin.sogou.com` indexes article text, which is the one channel that
|
|
308
|
+
normally carries a Chinese person's or institution's name; each result keeps
|
|
309
|
+
its account name and publish date. It joins `auto` for mostly-CJK queries and
|
|
310
|
+
`--engine all`.
|
|
311
|
+
|
|
312
|
+
### Changed
|
|
313
|
+
|
|
314
|
+
- Version 1.1.0 across manifests, `SKILL.md`, CLI and README install tags.
|
|
315
|
+
- The browser layer uses the web-forager name throughout: environment variables
|
|
316
|
+
are `WEB_FORAGER_BRIDGE`, `WEB_FORAGER_BROWSER_HOME`,
|
|
317
|
+
`WEB_FORAGER_BROWSER_HEADFUL`, `WEB_FORAGER_BROWSER_PROFILE` and
|
|
318
|
+
`WEB_FORAGER_BROWSER_CDP`; sessions live in `~/.web-forager/browser/`; record
|
|
319
|
+
`source` is `web-forager-browser`.
|
|
320
|
+
- The plugin manifest gained a `browser` block listing playbooks, backends and
|
|
321
|
+
the browser MCP tools.
|
|
322
|
+
- The User-Agent now carries the real version and the repository URL instead of
|
|
323
|
+
a stale `WebForager/1.0` and a placeholder domain.
|
|
324
|
+
- `bili-danmaku` asks the bridge to observe first and falls back to DOM text
|
|
325
|
+
second; a DOM-derived list is recorded with a warning that it is partial.
|
|
326
|
+
- Duplicate card links now merge, so a captured card keeps its readable title
|
|
327
|
+
instead of the "views danmaku duration" overlay text.
|
|
328
|
+
- The MCP server exposes eight HTTP tools instead of seven.
|
|
329
|
+
- **`auto` widens a thin answer once.** A Latin query that returns fewer than
|
|
330
|
+
three distinct results from 360 and Bing now asks Sogou as well; a CJK query
|
|
331
|
+
whose engines all fail keeps its Bing fallback. A full answer is not widened,
|
|
332
|
+
so the common case costs nothing extra.
|
|
333
|
+
- **`--max-pages 0` returns a search answer without page bodies.** The CLI used
|
|
334
|
+
to floor the value at 1; both the CLI and `web_search` now let a caller ask
|
|
335
|
+
for the result list alone and fetch only what it needs with `--max-chars`.
|
|
336
|
+
- **The MCP search tool carries the same warnings as the CLI.** A response whose
|
|
337
|
+
results do not mention the query text — or that is empty — now says so in
|
|
338
|
+
`warnings` instead of leaving the agent to present dictionary pages as
|
|
339
|
+
findings.
|
|
340
|
+
|
|
341
|
+
### Fixed
|
|
342
|
+
|
|
343
|
+
- A browser run that found nothing no longer reports success.
|
|
344
|
+
- A bridge without `observe` is no longer reported as observing.
|
|
345
|
+
- The Bilibili feed enrichment test no longer reaches the network, so the suite
|
|
346
|
+
is genuinely offline.
|
|
347
|
+
- **`results` no longer repeats every page body.** A search response carried
|
|
348
|
+
each Markdown page twice — once in `pages`, once in `results` — so a caller
|
|
349
|
+
with an output cap received a truncated tail. `results` is now a compact
|
|
350
|
+
index (`url`, `title`, `status`, `content_type`, `markdown_chars`, `tables`,
|
|
351
|
+
`snippet`, `warnings`, `error`); the body stays in `pages` and, for a single
|
|
352
|
+
fetch, at the top level. The old shape is one flag away: `--full-results` on
|
|
353
|
+
the CLI, `fullResults` on the MCP tools.
|
|
354
|
+
- **`auto` no longer mixes Bing into Chinese queries.** Bing treats a quoted
|
|
355
|
+
Chinese phrase as single characters, so dictionary and encyclopaedia pages
|
|
356
|
+
outranked the precise hits. A mostly-CJK query now asks 360, Sogou and the
|
|
357
|
+
WeChat index, and falls back to Bing only if all three are unreachable. Latin
|
|
358
|
+
queries are unchanged.
|
|
359
|
+
- **Navigation waits for the page it lands on.** `goto` now waits for the load
|
|
360
|
+
state and a short settle before returning, so an `evaluate` issued right after
|
|
361
|
+
a redirect no longer races it; `snapshot` retries once on the "execution
|
|
362
|
+
context was destroyed" error that a redirect leaves behind.
|
|
363
|
+
- **The bridge's stderr is kept.** A dead bridge now reports its last few
|
|
364
|
+
stderr lines in the error instead of a bare timeout.
|
|
365
|
+
- **Login walls and anti-bot pages are classified.** A playbook that lands on a
|
|
366
|
+
sign-in or verification URL reports `login_required` or `blocked` with the
|
|
367
|
+
page's real URL, instead of `degraded` with zero captures.
|
|
368
|
+
- **Garbled result titles are dropped.** A title made mostly of replacement or
|
|
369
|
+
control characters — the shape an anti-bot page leaves behind — no longer
|
|
370
|
+
reaches the caller.
|
|
371
|
+
|
|
372
|
+
### Documentation
|
|
373
|
+
|
|
374
|
+
- Browser mode now documents what it costs: `--out` / `out:` to keep records on
|
|
375
|
+
disk, no screenshots by default, `maxScreens` at or below 5 and
|
|
376
|
+
`minInterval` at or above 3000 ms, `bili-danmaku` as experimental, and login
|
|
377
|
+
state through `WEB_FORAGER_BROWSER_CDP` or a persistent profile.
|
|
378
|
+
- HTTP mode is positioned where a search CLI is not: `web_scholar` and
|
|
379
|
+
`web_map`.
|
|
380
|
+
- Weibo and Douban are no longer listed as covered sites: they have no
|
|
381
|
+
site-specific code and are read as ordinary pages.
|
|
382
|
+
- The two modes are described as sharing four core fields, not as writing the
|
|
383
|
+
same record; a field-by-field table replaces the old claim.
|
|
384
|
+
- Baidu's absence, the parser assumptions behind each search engine and the
|
|
385
|
+
absence of site adapters for Zhihu and Xiaohongshu in HTTP mode are documented.
|
|
386
|
+
- Browser exit codes 8 and 9, `--strict-robots`, `--doctor` and automatic
|
|
387
|
+
bridge discovery are documented.
|
|
388
|
+
|
|
389
|
+
### Tests
|
|
390
|
+
|
|
391
|
+
- **157 Python tests** and **60 browser tests**, all offline. The browser suite
|
|
392
|
+
grows from 35 to 60: capabilities, robots parsing and matching, the
|
|
393
|
+
strict-robots round trip, degraded and blocked paths, the core schema, the
|
|
394
|
+
real-page fixtures, the Bilibili JSON client and its page-card filter, the MCP
|
|
395
|
+
search switches and warnings, the thin-result fallback, and the doctor report. The suite
|
|
396
|
+
clears `WEB_FORAGER_BRIDGE`, so a machine that exports the variable still runs
|
|
397
|
+
the no-bridge paths green, and the Bilibili feed test no longer reaches the
|
|
398
|
+
network.
|
|
399
|
+
|
|
400
|
+
### Notes
|
|
401
|
+
|
|
402
|
+
- Browser mode is read-only by default. Like, coin, favourite, follow, share,
|
|
403
|
+
comment, danmaku and publish are denied unless `--allow-write` is passed after
|
|
404
|
+
explicit authorisation.
|
|
405
|
+
- The bundled Playwright bridge still needs `playwright` and its Chromium
|
|
406
|
+
binary installed separately; `--doctor` names whichever is missing. With no
|
|
407
|
+
bridge at all a live browser run exits 3.
|
|
408
|
+
- Danmaku and `observe("danmaku", ms)` are experimental; canvas-rendered text
|
|
409
|
+
still needs OCR.
|
|
410
|
+
|
|
411
|
+
## 1.0.1 — 2026-11-06
|
|
412
|
+
|
|
413
|
+
Patch release for the MCP server on Windows hosts.
|
|
414
|
+
|
|
415
|
+
### Fixed
|
|
416
|
+
|
|
417
|
+
- **MCP stdio encoding**: some hosts (ZCode among them) launch `python` with the
|
|
418
|
+
ANSI code page as the stdio encoding. A response containing a Chinese title
|
|
419
|
+
then raised `UnicodeEncodeError`, closed the server and surfaced as
|
|
420
|
+
"Connection closed" on the tool call. The server now reconfigures stdin and
|
|
421
|
+
stdout to UTF-8 at startup and falls back to `ensure_ascii` escapes when a
|
|
422
|
+
stream cannot be reconfigured.
|
|
423
|
+
|
|
424
|
+
### Tests
|
|
425
|
+
|
|
426
|
+
- Added `tests/test_mcp_stdio.py`, which reproduces the cp1252 sink and checks
|
|
427
|
+
both the reconfigure path and the ASCII fallback. Total: 117 tests.
|
|
428
|
+
|
|
429
|
+
### Verified
|
|
430
|
+
|
|
431
|
+
- Bilibili `feed`, `search`, `video` and `related` were exercised against the
|
|
432
|
+
live site from the CLI, and `web_bilibili` through the MCP server inside
|
|
433
|
+
ZCode. Comments and danmaku text remain out of scope for web-forager.
|
|
434
|
+
|
|
435
|
+
## 1.0.0 — 2026-11-06
|
|
436
|
+
|
|
437
|
+
First stable release. web-forager is a complete public-web intelligence layer
|
|
438
|
+
for agents and analysts, with a CLI, an MCP server and a bilingual skill contract.
|
|
439
|
+
|
|
440
|
+
### Added
|
|
441
|
+
|
|
442
|
+
- **Scholarly APIs**: `--action scholar --source arxiv|crossref|pubmed|doi`
|
|
443
|
+
with normalised records (title, authors, venue, year, DOI, abstract, PDF URL).
|
|
444
|
+
- **Office documents**: DOCX, XLSX and EPUB extraction with the standard
|
|
445
|
+
library only; CSV becomes structured tables.
|
|
446
|
+
- **Feeds**: RSS 2.0, Atom and RSS 1.0 parsing into a Markdown digest.
|
|
447
|
+
- **Sitemap discovery**: `--action map` reads `Sitemap:` lines from robots.txt,
|
|
448
|
+
sitemap indexes and `.xml.gz` files.
|
|
449
|
+
- **MCP server**: `mcp_server.py` speaks newline-delimited JSON-RPC 2.0 and
|
|
450
|
+
exposes `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`,
|
|
451
|
+
`web_bilibili` and `web_clean`; `.mcp.json` wires it into Claude Code.
|
|
452
|
+
- **Search pagination**: `--search-pages N` reads several result pages per
|
|
453
|
+
engine (360, Bing, Sogou).
|
|
454
|
+
- **Context chunks**: `--chunk-size N` adds heading-aware `chunks` to fetch and
|
|
455
|
+
clean output, so an agent can read long pages section by section.
|
|
456
|
+
- **HTTP cache**: `--cache --cache-ttl N --cache-dir DIR` reuses polite GET
|
|
457
|
+
responses; `cached: true` is reported in the record.
|
|
458
|
+
- **Playwright controls**: `--wait-for SELECTOR` and `--scroll N` for lazy or
|
|
459
|
+
delayed content.
|
|
460
|
+
- **Security**: all XML goes through a DOCTYPE/ENTITY guard that uses
|
|
461
|
+
`defusedxml` when available; Office ZIP containers have entry-count and
|
|
462
|
+
expansion limits.
|
|
463
|
+
- **Multi-host adaptation**: `AGENTS.md` is the generic agent contract, with
|
|
464
|
+
matching rule files for Cursor, Windsurf, Cline, Continue, Roo Code and
|
|
465
|
+
GitHub Copilot. `install.py` writes project- or user-scoped rules in one run.
|
|
466
|
+
- **Packaging**: root `package.json`, `index.js` and `cordis.patch.yml` cover
|
|
467
|
+
npm and dsh; `commands/fetch.md`, `surf.md`, `scholar.md` and `map.md` expose
|
|
468
|
+
the actions as plugin commands.
|
|
469
|
+
- **Bilingual documentation**: `README.md` (English) and `README_zh.md`
|
|
470
|
+
(Chinese) mirror each other section for section, with the same three-method
|
|
471
|
+
installation guide, quick start, requirements, repository tree, test summary,
|
|
472
|
+
limitations, issue rules, trademark notice and licence. All documents are
|
|
473
|
+
emoji-free.
|
|
474
|
+
- **Tests**: 115 tests, including installer coverage and host-rule and
|
|
475
|
+
disclaimer checks.
|
|
476
|
+
|
|
477
|
+
### Fixed
|
|
478
|
+
|
|
479
|
+
- **Trafilatura cross-document deduplication**: extraction previously ran with
|
|
480
|
+
`deduplicate=True`, whose process-global cache silently removed sentences
|
|
481
|
+
seen on an earlier page. Long agent sessions could lose a page's opening
|
|
482
|
+
paragraph. Deduplication is now disabled across calls.
|
|
483
|
+
- Sitemap `<loc>` elements were not mapped to the `url` field.
|
|
484
|
+
- RSS/XML payloads no longer trigger the HTML dynamic-rendering heuristic or
|
|
485
|
+
BeautifulSoup's XML warning.
|
|
486
|
+
|
|
487
|
+
### Changed
|
|
488
|
+
|
|
489
|
+
- Version 1.0.0 across all platform manifests, `SKILL.md` and the CLI
|
|
490
|
+
User-Agent.
|
|
491
|
+
- `install.py --global` appends a marked block to `~/.claude/CLAUDE.md` instead
|
|
492
|
+
of replacing the user's global instructions.
|
|
493
|
+
- Documentation presents web-forager as an independent plugin; the `--root`
|
|
494
|
+
output is described as a four-zone workspace without third-party branding.
|
|
495
|
+
|
|
496
|
+
## 0.2.0 — 2026-11-06
|
|
497
|
+
|
|
498
|
+
- 360/Bing/Sogou search with automatic merge and dedupe.
|
|
499
|
+
- Optional PDF extraction through `pypdf` (exit code 7 when absent).
|
|
500
|
+
- `--action surf` with ranked link BFS and resumable `--session`.
|
|
501
|
+
- `--agent` OpenCLI-style success/error envelope.
|
|
502
|
+
|
|
503
|
+
## 0.1.0 — 2026-11-06
|
|
504
|
+
|
|
505
|
+
- Initial clean-room fetcher: robots.txt, polite rate limiting, HTML → Markdown,
|
|
506
|
+
metadata, tables, optional Playwright and a Bilibili adapter.
|
|
507
|
+
- Standalone packaging for Claude Code, Codex, CodeBuddy and DSH, plus
|
|
508
|
+
four-zone `--root` workspace integration.
|