web-forager 1.3.0 → 1.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/.claude-plugin/marketplace.json +1 -1
  2. package/.claude-plugin/plugin.json +1 -1
  3. package/.clinerules/web-forager.md +3 -3
  4. package/.codebuddy-plugin/marketplace.json +1 -1
  5. package/.codebuddy-plugin/plugin.json +1 -1
  6. package/.codex-plugin/marketplace.json +1 -1
  7. package/.codex-plugin/mcp.json +20 -20
  8. package/.codex-plugin/plugin.json +1 -1
  9. package/.continue/rules/web-forager.md +4 -4
  10. package/.cursor/rules/web-forager.mdc +4 -4
  11. package/.dsh-plugin/index.js +1 -1
  12. package/.dsh-plugin/plugin.json +1 -1
  13. package/.github/copilot-instructions.md +3 -3
  14. package/.roo/rules/web-forager.md +3 -3
  15. package/.windsurf/rules/web-forager.md +4 -4
  16. package/AGENTS.md +10 -3
  17. package/CHANGELOG.md +508 -454
  18. package/README.md +499 -477
  19. package/README_zh.md +498 -477
  20. package/SKILL.md +115 -15
  21. package/THIRD_PARTY_LICENSES.txt +5 -0
  22. package/browser/bridges/playwright-bridge.js +8 -1
  23. package/browser/launch_chrome.py +193 -144
  24. package/browser/lib/bili-api.js +12 -1
  25. package/browser/lib/safety.js +27 -1
  26. package/browser/lib/session.js +16 -2
  27. package/browser/lib/untrusted.js +215 -56
  28. package/browser/mcp_server.js +5 -25
  29. package/browser/package.json +1 -1
  30. package/browser/playbooks/search-dig.js +3 -1
  31. package/browser/playbooks/wechat-dig.js +78 -78
  32. package/browser/run.js +19 -10
  33. package/commands/research.md +35 -0
  34. package/cordis.patch.yml +19 -19
  35. package/docs/DEPENDENCIES.md +1 -1
  36. package/docs/INSTALL_COMMANDS.md +4 -4
  37. package/docs/INSTALL_COMMANDS_zh.md +4 -4
  38. package/docs/REFERENCE.md +349 -308
  39. package/docs/REFERENCE_zh.md +342 -309
  40. package/manifest.json +3 -2
  41. package/mcp_server.py +383 -130
  42. package/package.json +4 -2
  43. package/requirements.txt +7 -4
  44. package/run.py +61 -5
  45. package/scripts/bilibili.py +11 -3
  46. package/scripts/engine.py +1039 -118
  47. package/scripts/pdf.py +21 -0
  48. package/scripts/research.py +2087 -0
  49. package/scripts/scholar.py +33 -6
  50. package/scripts/sitemap.py +20 -5
  51. package/scripts/untrusted.py +312 -95
package/CHANGELOG.md CHANGED
@@ -1,454 +1,508 @@
1
- # Changelog
2
-
3
- ## 1.3.0 — 2026-11-15
4
-
5
- A workflow release: a robots policy that can record an authorised override, search that fills its page budget with readable results, parallel engines, and a browser session you can log into once.
6
-
7
- ### Added
8
-
9
- - **`--robots-policy respect|warn|ignore`.** `respect` is the default and still refuses a disallowed path. `warn` reads it and writes `compliance.robots_overridden: true` plus a warning, for a user who is authorised to read the page. `ignore` is the old `--ignore-robots`, with `robots_checked: false`. Every HTTP record now carries a `compliance` block.
10
- - **The browser profile launcher** (`browser/launch_chrome.py`) starts Chrome/Chromium/Edge with `~/.web-forager/browser/profile` and a DevTools port, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`.
11
- - **Storage-state import and export** for browser mode: `--storage-state FILE` loads cookies/localStorage, `--save-storage-state FILE` writes them after the run. A saved file is forced to mode `0600`, a path inside `--root` is refused, and the record says the file carries login state.
12
- - **`--serial-engines`** keeps a diagnostic run strictly serial; the default now asks engines on different hosts at the same time while each host keeps its one-request-per-second delay.
13
- - **A `wechat-dig` browser playbook** opens one WeChat article URL in the browser profile and captures its text. When the profile has already passed WeChat's environment check the article is read; when it has not, the playbook reports `blocked` with a hint and does not try to solve the check.
14
- - **Verification pages are reported instead of recycled.** HTTP mode recognises WeChat's environment check and the common English anti-bot interstitials, raises `VerificationRequired`, and `fetch --query` records the result as `verification_required` with a warning that names the browser route. The CLI exits 9. The bundled bridge also hands the whole `#js_content` article over as one `content` snapshot node, so WeChat paragraphs are not reassembled from leaf spans.
15
- - **Automatic render fallback for script shells.** When a server answers with a minimal HTML page that carries only a script or redirect, `--render auto` now renders it once with Playwright before extraction. The record says `render_fallback: true`; `fetch --query` allows at most two such renders per query, so one slow site cannot stall a search. This needs no API key and closes the php.cn-style empty-page case.
16
- - **Free ranking and query repair.** Search results are ordered by query coverage, title match, a small domain prior (docs, papers, government and universities up; document farms down) and snippet freshness. The final list keeps at most two results per host and fills the rest from the overflow. A thin first pass is retried once with a simplified query — quotes and common stopwords removed, or a spaced CJK query joined — and the retry is marked `query_variant` in `engine_report`. No model, key or network service is involved.
17
-
18
- ### Changed
19
-
20
- - **`max_pages` now means readable pages.** `fetch --query` keeps scanning past a robots-denied or unreadable result (up to a bounded number of attempts), returns the pages it could read, and lists the rest in `skipped` with a code and reason. The response carries a summary warning instead of leaving the agent to discover failures page by page.
21
- - The `robots_denied` hint now names `--robots-policy warn` for an authorised override and says not to silently switch to a tool that ignores robots. A `verification_required` skip says the same about not solving the check, and points a WeChat link at the `wechat-dig` playbook.
22
-
23
- ### Host compatibility
24
-
25
- - **Codex now gets a working MCP launch.** The plugin ships `.codex-plugin/mcp.json`, which uses relative `mcp_server.py` and `browser/mcp_server.js` paths with `cwd: "."`; Codex resolves both against the installed plugin directory. It no longer inherits Claude Code's `${CLAUDE_PLUGIN_ROOT}` argument, which Codex leaves literal.
26
- - **CodeBuddy now declares its MCP servers in the manifest it reads.** `.codebuddy-plugin/plugin.json` carries both `mcpServers` entries with `${CLAUDE_PLUGIN_ROOT}`, so a new CodeBuddy session mounts the HTTP and browser servers. `codebuddy mcp list` can run before plugins are mounted; an empty list there is not a failed install.
27
- - **dsh now mounts real tools.** `cordis.patch.yml` inserts two `@deepseek-ai/dsh-mcp-client` entries, one per server, and resolves the profile package directory with `dshHomePath(...)`. The previous patch mounted only the no-op `web-forager` module.
28
- - **The npm package carries every host manifest.** `files` now includes `.claude-plugin/`, `.codex-plugin/`, `.codebuddy-plugin/`, `.dsh-plugin/`, `.mcp.json`, the rule-file directories, `install.py` and `requirements.txt`.
29
- - The Cursor rule heading now says v1.3.0 instead of v1.0.0.
30
-
31
- ### Documentation
32
-
33
- - `SKILL.md` and `AGENTS.md` state the rule: a `robots_denied` or `login_required` result is a boundary, not a prompt to re-fetch with a tool that ignores robots.
34
- - The reference documents the launcher, storage-state switches, the robots policy, the new `compliance` block, the `wechat-dig` playbook and exit 9.
35
- - Both READMEs were rewritten around a first visit: one pitch, the three install methods, what a run looks like, the security boundary and the legal notice, plus a section on keeping an installed copy current (including the ZCode marketplace-snapshot trap). A DeepSec Shield scan of the release tree reports 0 findings; its agent audit still flags the three signature strings this project defines, which are the scanner's own patterns, not page content.
36
-
37
- ### Tests
38
-
39
- - **258 Python tests** and **87 browser tests**, all offline. New coverage: the warn policy and its record, clean compliance fields, filling pages past a denial, overlapping engines versus `--serial-engines`, the launcher dry run, saving storage state through the bridge protocol, WeChat verification detection at the engine, the CLI and the search layer, the `wechat-dig` capture and `blocked` paths, the `content` snapshot node, domain/freshness tie-breaks, host diversity, query variants, the shell-render fallback with and without its budget, plus the Codex/CodeBuddy/dsh MCP manifests and the npm host-file list.
40
-
41
- ## 1.2.3 — 2026-11-15
42
-
43
- A follow-up hardening release after a DeepSec Shield audit of 1.2.2.
44
-
45
- ### Security
46
-
47
- - **The browser bridge is no longer started through a shell.** `--bridge` and `WEB_FORAGER_BRIDGE` are split into an executable and its arguments and run directly, so a path or environment value cannot append a second command. Quoted paths with spaces still work; pipes, `&&` and redirection are not interpreted.
48
- - **CI actions are pinned to commit SHAs.** `actions/checkout`, `actions/setup-python` and `actions/setup-node` name the exact commit, so a moved tag cannot change what the workflow runs.
49
-
50
- ### Documentation
51
-
52
- - The reference states that the bridge command is split on whitespace and started without a shell.
53
-
54
- ### Tests
55
-
56
- - **228 Python tests** and **81 browser tests**, all offline. New coverage: quoted bridge paths, literal backslashes, an unterminated quote, and a shell metacharacter that must be passed through as an argument rather than executed.
57
-
58
- ## 1.2.2 — 2026-11-15
59
-
60
- A hardening release that replaces the 1.2.2 build published on 2026-11-12: a request can no longer be aimed at the machine or the LAN, browser state stays in a profile of its own, and fetched text is labelled as third-party data. The legal-notice rewrite and packaging rules from that earlier build are included.
61
-
62
- ### Security
63
-
64
- - **Private-address guard (SSRF).** Loopback, RFC 1918, link-local (including `169.254.169.254`), CGNAT, multicast, reserved and their IPv6 forms are refused before a socket opens. The host is resolved and checked as written, so decimal IPv4 spellings and IPv6-mapped IPv4 are caught too. The check runs before the request, on every redirect hop (redirects are followed one hop at a time instead of by `requests`), on the robots.txt download, on search-result resolution, and on the URL a rendered page actually lands on. `--allow-private` / `WEB_FORAGER_ALLOW_PRIVATE=1` is the documented opt-out for local fixtures and trusted intranet targets.
65
- - **Browser navigation is guarded too.** `guardedGoto` checks the requested and the final URL, and the bundled Playwright bridge installs a request filter: every document and subresource the page asks for is resolved, and private targets are aborted. A hostile page can no longer use the open browser to probe `http://127.0.0.1:…` or a cloud metadata endpoint.
66
- - **Browser state stays in its own profile.** The bundled bridge already kept `~/.web-forager/browser/profile`; the report now states `dedicated_profile` and `shared_browser`, and a CDP attach to a browser you use for everything else adds a warning that every cookie and login in that profile is reachable by the pages a playbook opens.
67
- - **Fetched text is marked as data.** Every record carries `trust: "untrusted"`. A small scanner (same rules in `scripts/untrusted.py` and `browser/lib/untrusted.js`) names instruction-shaped passages — overrides, fake system tags, `curl … | sh`, secret-exfiltration requests, role swaps — in `injection_flags`, and puts one warning first in the list. `--input-gate strict` withholds the body instead of handing it on; `--input-gate off` skips the scan.
68
- - **Cookies never enter a record.** The Bilibili cookie is held in memory for the API request and is not written to the record, the session file or an error message; the browser profile directory is reported as a flag, not a path.
69
-
70
- ### Documentation
71
-
72
- - **Both READMEs were reorganised around a first visit.** The page now opens with what the tool is, what you get and what it refuses, followed by a table of contents, the why section, the three install methods and a quick start split by entry point — chat, terminal, MCP and the optional browser mode — with a "which action do I need" table and copy-paste commands. The first screen carries a one-line notice that links to the full legal terms; the terms themselves moved below the quick start, section by section.
73
- - **Quick start now holds only the three entry points**: chat, terminal and MCP. The browser playbook table, switches and `--strict-robots` details, and "when the engines are blocked" routes moved to the reference; each is a one-sentence pointer on the landing page. Installation opens with "let your AI install it" as the recommended path for most readers, followed by the plugin-marketplace URL and the per-host commands. The contents is grouped, with acknowledgements, trademarks and licence added.
74
- - **Reference material moved to `docs/REFERENCE.md` and `docs/REFERENCE_zh.md`**: the full action list and exit-code tables, record fields and `core-1`, workspace output, MCP tool names and protocol checks, requirements, repository layout, test details and extra CLI examples. Every example is its own one-line block, so it can be copied or run without trimming its neighbours. README keeps a three-table quick start, a short exit-code summary and a link.
75
- - **Long sections fold.** The repository tree, test coverage notes and MCP tool names sit in `<details>` blocks; known limitations are grouped into read-first, edge cases and experimental/high-risk switches.
76
- - Both READMEs gain a **Security boundaries** section and three limitation bullets (the guard's off switch, the screen's limits, the CDP profile exposure).
77
- - `AGENTS.md`, `SKILL.md` and the six host rule files state that fetched text is data and must not be executed as instructions.
78
- - **The README pair carries a six-part legal notice** — purpose, prohibited use, responsibility, no warranty, limitation of liability, redistribution. `SKILL.md`, `AGENTS.md` and the six host rule files carry the short version.
79
- - **Test fixtures are no longer part of a published archive.** `.gitattributes` marks `tests/` and `browser/tests/` as `export-ignore`, and both npm manifests drop tests from `files`; CI still runs them from the repository clone.
80
-
81
- ### Tests
82
-
83
- - **228 Python tests** and **75 browser tests**, all offline. New coverage: every private range, a redirect into a metadata endpoint, a two-hop redirect chain, the robots download, the CLI exit code and `--allow-private`, marker detection, strict withholding, and the browser-side guard and scanner.
84
-
85
- ## 1.2.1 — 2026-11-11
86
-
87
- A search hygiene release: a Chinese query no longer collects dictionary pages for one of its characters, and the README pair was rewritten around what the tool does now.
88
-
89
- ### Fixed
90
-
91
- - **A mostly-CJK query is no longer handed to Bing as a fallback.** Bing splits a Chinese phrase into single characters, so a search about 反逆向工程 came back with the dictionary page of 反 once the Chinese engines answered thinly. `auto` now stops after 360, Sogou and 搜狗微信; `--engine bing` still asks Bing explicitly.
92
- - **Dictionary pages that match one character of the query are dropped**, whatever engine returned them. `looks_like_character_page()` recognises a title that shares exactly one character with a longer query and carries a dictionary marker (百科, 拼音, 笔顺, 部首, 组词…), so a result list is not padded with them and `engine_report` counts only usable hits.
93
-
94
- ### Documentation
95
-
96
- - **Both READMEs were rewritten.** The duplicated mode paragraphs and the second disclaimer block are gone; scope and compliance sit in one section, the two modes face each other in one table, and the CJK search routing, `find`/`feed`, CI and current tool/test counts are stated once. README text now matches the 10 HTTP tools, 4 browser tools and 9 CLI actions that actually ship.
97
-
98
- ### Tests
99
-
100
- - **174 Python tests** and **60 browser tests**, all offline. New coverage: a CJK query never calls Bing, a character page is dropped from an engine's results, and `looks_like_character_page()` leaves a single-character query and a full-phrase hit alone.
101
-
102
- ## 1.2.0 — 2026-11-09
103
-
104
- Search is no longer the only way in. A site's own search page and its feed are
105
- first-class routes now, every engine attempt is reported, and one run can be
106
- sent through a proxy.
107
-
108
- ### Added
109
-
110
- - **`--action find` (`web_find`)**: search inside one site through the site's
111
- own search page. It tries `/?s=`, `/search?q=`, `/?search=` and
112
- `/search?query=`, ranks the page's links by query coverage, and fetches the
113
- best ones. This is the route that keeps working when the general engines
114
- answer with dictionary pages or a verification wall.
115
- - **`--action feed` (`web_feed`)**: read a site's RSS/Atom feed, discovering it
116
- from `<link rel=alternate>` when given the site URL instead of the feed URL.
117
- - **`engine_report`**: every search response now carries one entry per engine
118
- tried — `ok`, `results`, `ms`, and `error` when it failed, plus `fallback`
119
- when the engine was asked only because the first answer was thin. "Search is
120
- broken" becomes "Bing answered, Sogou showed a verification page".
121
- - **`doctor --probe` (`web_doctor {probe: true}`)**: asks 360, Bing, Sogou, the
122
- WeChat index and DuckDuckGo once each and reports which ones answer from this
123
- network, with latency.
124
- - **`--proxy URL`** and `proxy` on the MCP tools. Requests already honoured
125
- `HTTPS_PROXY` / `ALL_PROXY`; an explicit proxy now overrides them for one run,
126
- and `doctor` reports the proxy in use with credentials redacted.
127
-
128
- ### Changed
129
-
130
- - The MCP server exposes ten HTTP tools instead of eight; the README and
131
- `SKILL.md` document the two routes that do not depend on a search engine.
132
- - `doctor` reports the proxy source (`config` or `env:NAME`) without printing a
133
- password.
134
-
135
- ### Documentation
136
-
137
- - **The disclaimer forbids building a profile of an identifiable person.**
138
- Collecting, aggregating or inferring personal information, or using it for
139
- doxxing, harassment or stalking, is prohibited and is the user's own legal
140
- responsibility. It appears in `--help`, `manifest.json`, the README pair,
141
- `SKILL.md` and every host rule file.
142
- - **The per-host install commands moved to `docs/INSTALL_COMMANDS.md` and
143
- `docs/INSTALL_COMMANDS_zh.md`.** Methods 1 and 3 stay in the README because
144
- they are three lines each; Method 2, which is a page of commands, is one link.
145
- - **The READMEs gained an Acknowledgements section** listing the reporter of
146
- issue #1 by name, with a link to `docs/ACKNOWLEDGEMENTS.md` /
147
- `docs/ACKNOWLEDGEMENTS_zh.md` for the details.
148
-
149
- ### Acknowledgements
150
-
151
- - **NotFatJustRound** ([issue #1](https://github.com/JiamingShi-JsPhys/web-forager/issues/1))
152
- reported that `web_search` returned every page body twice — once in `pages`,
153
- once in `results` — which doubled the response and pushed the tail past an
154
- agent's size limit; the compact `results` index in 1.0.2 came from that report.
155
-
156
- ### Fixed
157
-
158
- - **Links inside `<article><header>` are no longer skipped.** A results page
159
- commonly wraps each card's title in an article header; the universal
160
- "skip `<header>`" rule silently dropped exactly the links a search page
161
- offers. Page-level headers and navigations are still skipped.
162
- - `doctor --probe` makes one attempt per engine instead of retrying, so a
163
- blocking engine costs one timeout rather than three.
164
- - `.cursor/rules/web-forager.mdc` carried `v1.0.0` in its front matter, so
165
- `install.py --only cursor` wrote a stale label; it now follows the release
166
- version.
167
-
168
- ### Tests
169
-
170
- - **172 Python tests** and **60 browser tests**, all offline. The new suite
171
- covers site search against a local WordPress-style page, feed discovery from a
172
- page link and from a direct feed, the per-engine report, the probe, proxy
173
- wiring and the redaction of proxy credentials.
174
-
175
- ## 1.1.0 — 2026-11-08
176
-
177
- Browser mode joins web-forager, and an audit pass follows it. The HTTP toolkit
178
- and the browser layer ship as **one plugin, one name and one version**; the
179
- former standalone `web-actor` product no longer exists. The empty-result path,
180
- the bridge's capability claims, the robots.txt audit trail and the shared record
181
- shape are now honest, and neither README claims coverage the code does not have.
182
-
183
- ### Added
184
-
185
- - **Browser mode** under `browser/`: a CLI, seven read-only playbooks and an MCP
186
- server that drive a real browser through a pluggable bridge.
187
- - **Playbooks**: `bili-feed`, `bili-video`, `bili-chain`, `bili-danmaku`,
188
- `zhihu-dig`, `xhs-dig` and `search-dig`.
189
- - **Second MCP server**: `browser/mcp_server.js` exposes `browser_list`,
190
- `browser_run`, `browser_session` and `browser_doctor`; `.mcp.json`
191
- registers both servers.
192
- - **A reference bridge that is found automatically**:
193
- `browser/bridges/playwright-bridge.js` supports a fresh browser, a persistent
194
- profile, a headful window and attaching over CDP. `browser/lib/bridge.js`
195
- looks for `--bridge`, then `WEB_FORAGER_BRIDGE`, then that file, so a first
196
- run no longer fails with `backend_unavailable` only because nobody exported a
197
- variable.
198
- - **A browser profile that persists by default**:
199
- `~/.web-forager/browser/profile` keeps cookies and logins between runs;
200
- `WEB_FORAGER_BROWSER_PROFILE=off` restores the throwaway browser. The
201
- reference bridge also drops the automation banner flag.
202
- - **Browser records**: the same `metadata / markdown / tables / warnings` shape,
203
- written to `<WS>/process/web-forager/browser/` and `_broadcast/`.
204
- - **`/browse` command** for the browser playbooks.
205
- - **Shared record core (`core-1`)**: `--core` on both CLIs and `core: true` on
206
- `browser_run` return `schema_version / source / mode / url / title /
207
- collected_at / metadata / markdown / tables / warnings / extra`, so a mixed
208
- list of HTTP and browser records can be read uniformly. The same fold is
209
- available as `engine.normalize_record()` and `normalizeRecord()`.
210
- - **Browser robots auditing**: `--strict-robots` reads `robots.txt` before every
211
- navigation and stops with `robots_denied` (exit 3) on a disallowed path.
212
- Without it the record still states `compliance.robots_checked: false`, so the
213
- compliance status is auditable either way. `browser/lib/robots.js` implements
214
- the same RFC 9309 subset as the Python engine.
215
- - **`degraded` (exit 8) and `blocked` (exit 9)**: a browser run that reads a
216
- page but finds nothing the playbook looks for now reports `degraded` with its
217
- records attached, instead of `ok: true` with `captured: 0`. An anti-bot or
218
- error interstitial reports `blocked`. Both are distinct from `login_required`.
219
- - **`describe` handshake**: a bridge can declare its capabilities; a bridge that
220
- does not gets `declared: false` with every capability false. The reference
221
- Playwright bridge and the test fixtures implement it.
222
- - **`browser/lib/selectors.js`**: every structural assumption about a
223
- third-party page lives in one file.
224
- - **Real-page fixtures**: `browser/tests/fixtures/snapshots/` holds snapshots
225
- built from live Bilibili pages on 2026-11-07 by
226
- `browser/tests/tools/html-to-snapshot.py`, with `tests/real-pages.test.js`
227
- as the offline regression and `npm run test:live` as the opt-in live check.
228
- - **`--action doctor`** and the **`web_doctor` MCP tool** report every
229
- dependency with its version or its install command, so an agent that receives
230
- a thin page can install what is missing instead of guessing.
231
- - **`node browser/run.js --doctor`** and the **`browser_doctor` MCP tool** do the
232
- same for the browser stack: Node version, the bridge command in use, whether
233
- the `playwright` package resolves (searched from the bridge's own directory
234
- first), whether its Chromium binary is downloaded, and where the profile is.
235
- - **The Bilibili search API is asked before the result page is read.**
236
- `lib/bili-api.js` calls Bilibili's public JSON search endpoint (the `buvid3`
237
- cookie comes from one home-page request) and falls back to scraping the page
238
- only when the endpoint refuses. `source` in the record says which path ran.
239
- `WEB_FORAGER_BILI_API=off` keeps the playbook on the page.
240
- - **`respect_robots` on the MCP tools.** `web_fetch`, `web_search` and
241
- `web_surf` accept the same switch the CLI spells `--ignore-robots`, so a page
242
- that robots.txt disallows can be read when the caller is authorised to read
243
- it. The default is still to check.
244
- - **Bilibili's JSON search on the direct path.** When robots checks are off
245
- (`--direct-search` or a global override), `scripts/bilibili.py` asks
246
- `api.bilibili.com` first — the `buvid3` cookie comes from one home-page
247
- request, and that request is never answered from the cache because a cached
248
- anti-bot page supplies no cookie — and scrapes the search page only if the
249
- API answers `-412` or malformed JSON. Cards read from the page are kept only
250
- when their text mentions the query, so a recommendation rail can no longer
251
- pass as search results. `route` in the payload names the host that answered.
252
- - **The WeChat article index as a search engine (`--engine weixin`).**
253
- `weixin.sogou.com` indexes article text, which is the one channel that
254
- normally carries a Chinese person's or institution's name; each result keeps
255
- its account name and publish date. It joins `auto` for mostly-CJK queries and
256
- `--engine all`.
257
-
258
- ### Changed
259
-
260
- - Version 1.1.0 across manifests, `SKILL.md`, CLI and README install tags.
261
- - The browser layer uses the web-forager name throughout: environment variables
262
- are `WEB_FORAGER_BRIDGE`, `WEB_FORAGER_BROWSER_HOME`,
263
- `WEB_FORAGER_BROWSER_HEADFUL`, `WEB_FORAGER_BROWSER_PROFILE` and
264
- `WEB_FORAGER_BROWSER_CDP`; sessions live in `~/.web-forager/browser/`; record
265
- `source` is `web-forager-browser`.
266
- - The plugin manifest gained a `browser` block listing playbooks, backends and
267
- the browser MCP tools.
268
- - The User-Agent now carries the real version and the repository URL instead of
269
- a stale `WebForager/1.0` and a placeholder domain.
270
- - `bili-danmaku` asks the bridge to observe first and falls back to DOM text
271
- second; a DOM-derived list is recorded with a warning that it is partial.
272
- - Duplicate card links now merge, so a captured card keeps its readable title
273
- instead of the "views danmaku duration" overlay text.
274
- - The MCP server exposes eight HTTP tools instead of seven.
275
- - **`auto` widens a thin answer once.** A Latin query that returns fewer than
276
- three distinct results from 360 and Bing now asks Sogou as well; a CJK query
277
- whose engines all fail keeps its Bing fallback. A full answer is not widened,
278
- so the common case costs nothing extra.
279
- - **`--max-pages 0` returns a search answer without page bodies.** The CLI used
280
- to floor the value at 1; both the CLI and `web_search` now let a caller ask
281
- for the result list alone and fetch only what it needs with `--max-chars`.
282
- - **The MCP search tool carries the same warnings as the CLI.** A response whose
283
- results do not mention the query text — or that is empty — now says so in
284
- `warnings` instead of leaving the agent to present dictionary pages as
285
- findings.
286
-
287
- ### Fixed
288
-
289
- - A browser run that found nothing no longer reports success.
290
- - A bridge without `observe` is no longer reported as observing.
291
- - The Bilibili feed enrichment test no longer reaches the network, so the suite
292
- is genuinely offline.
293
- - **`results` no longer repeats every page body.** A search response carried
294
- each Markdown page twice — once in `pages`, once in `results` — so a caller
295
- with an output cap received a truncated tail. `results` is now a compact
296
- index (`url`, `title`, `status`, `content_type`, `markdown_chars`, `tables`,
297
- `snippet`, `warnings`, `error`); the body stays in `pages` and, for a single
298
- fetch, at the top level. The old shape is one flag away: `--full-results` on
299
- the CLI, `fullResults` on the MCP tools.
300
- - **`auto` no longer mixes Bing into Chinese queries.** Bing treats a quoted
301
- Chinese phrase as single characters, so dictionary and encyclopaedia pages
302
- outranked the precise hits. A mostly-CJK query now asks 360, Sogou and the
303
- WeChat index, and falls back to Bing only if all three are unreachable. Latin
304
- queries are unchanged.
305
- - **Navigation waits for the page it lands on.** `goto` now waits for the load
306
- state and a short settle before returning, so an `evaluate` issued right after
307
- a redirect no longer races it; `snapshot` retries once on the "execution
308
- context was destroyed" error that a redirect leaves behind.
309
- - **The bridge's stderr is kept.** A dead bridge now reports its last few
310
- stderr lines in the error instead of a bare timeout.
311
- - **Login walls and anti-bot pages are classified.** A playbook that lands on a
312
- sign-in or verification URL reports `login_required` or `blocked` with the
313
- page's real URL, instead of `degraded` with zero captures.
314
- - **Garbled result titles are dropped.** A title made mostly of replacement or
315
- control characters — the shape an anti-bot page leaves behind — no longer
316
- reaches the caller.
317
-
318
- ### Documentation
319
-
320
- - Browser mode now documents what it costs: `--out` / `out:` to keep records on
321
- disk, no screenshots by default, `maxScreens` at or below 5 and
322
- `minInterval` at or above 3000 ms, `bili-danmaku` as experimental, and login
323
- state through `WEB_FORAGER_BROWSER_CDP` or a persistent profile.
324
- - HTTP mode is positioned where a search CLI is not: `web_scholar` and
325
- `web_map`.
326
- - Weibo and Douban are no longer listed as covered sites: they have no
327
- site-specific code and are read as ordinary pages.
328
- - The two modes are described as sharing four core fields, not as writing the
329
- same record; a field-by-field table replaces the old claim.
330
- - Baidu's absence, the parser assumptions behind each search engine and the
331
- absence of site adapters for Zhihu and Xiaohongshu in HTTP mode are documented.
332
- - Browser exit codes 8 and 9, `--strict-robots`, `--doctor` and automatic
333
- bridge discovery are documented.
334
-
335
- ### Tests
336
-
337
- - **157 Python tests** and **60 browser tests**, all offline. The browser suite
338
- grows from 35 to 60: capabilities, robots parsing and matching, the
339
- strict-robots round trip, degraded and blocked paths, the core schema, the
340
- real-page fixtures, the Bilibili JSON client and its page-card filter, the MCP
341
- search switches and warnings, the thin-result fallback, and the doctor report. The suite
342
- clears `WEB_FORAGER_BRIDGE`, so a machine that exports the variable still runs
343
- the no-bridge paths green, and the Bilibili feed test no longer reaches the
344
- network.
345
-
346
- ### Notes
347
-
348
- - Browser mode is read-only by default. Like, coin, favourite, follow, share,
349
- comment, danmaku and publish are denied unless `--allow-write` is passed after
350
- explicit authorisation.
351
- - The bundled Playwright bridge still needs `playwright` and its Chromium
352
- binary installed separately; `--doctor` names whichever is missing. With no
353
- bridge at all a live browser run exits 3.
354
- - Danmaku and `observe("danmaku", ms)` are experimental; canvas-rendered text
355
- still needs OCR.
356
-
357
- ## 1.0.1 — 2026-11-06
358
-
359
- Patch release for the MCP server on Windows hosts.
360
-
361
- ### Fixed
362
-
363
- - **MCP stdio encoding**: some hosts (ZCode among them) launch `python` with the
364
- ANSI code page as the stdio encoding. A response containing a Chinese title
365
- then raised `UnicodeEncodeError`, closed the server and surfaced as
366
- "Connection closed" on the tool call. The server now reconfigures stdin and
367
- stdout to UTF-8 at startup and falls back to `ensure_ascii` escapes when a
368
- stream cannot be reconfigured.
369
-
370
- ### Tests
371
-
372
- - Added `tests/test_mcp_stdio.py`, which reproduces the cp1252 sink and checks
373
- both the reconfigure path and the ASCII fallback. Total: 117 tests.
374
-
375
- ### Verified
376
-
377
- - Bilibili `feed`, `search`, `video` and `related` were exercised against the
378
- live site from the CLI, and `web_bilibili` through the MCP server inside
379
- ZCode. Comments and danmaku text remain out of scope for web-forager.
380
-
381
- ## 1.0.0 — 2026-11-06
382
-
383
- First stable release. web-forager is a complete public-web intelligence layer
384
- for agents and analysts, with a CLI, an MCP server and a bilingual skill contract.
385
-
386
- ### Added
387
-
388
- - **Scholarly APIs**: `--action scholar --source arxiv|crossref|pubmed|doi`
389
- with normalised records (title, authors, venue, year, DOI, abstract, PDF URL).
390
- - **Office documents**: DOCX, XLSX and EPUB extraction with the standard
391
- library only; CSV becomes structured tables.
392
- - **Feeds**: RSS 2.0, Atom and RSS 1.0 parsing into a Markdown digest.
393
- - **Sitemap discovery**: `--action map` reads `Sitemap:` lines from robots.txt,
394
- sitemap indexes and `.xml.gz` files.
395
- - **MCP server**: `mcp_server.py` speaks newline-delimited JSON-RPC 2.0 and
396
- exposes `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`,
397
- `web_bilibili` and `web_clean`; `.mcp.json` wires it into Claude Code.
398
- - **Search pagination**: `--search-pages N` reads several result pages per
399
- engine (360, Bing, Sogou).
400
- - **Context chunks**: `--chunk-size N` adds heading-aware `chunks` to fetch and
401
- clean output, so an agent can read long pages section by section.
402
- - **HTTP cache**: `--cache --cache-ttl N --cache-dir DIR` reuses polite GET
403
- responses; `cached: true` is reported in the record.
404
- - **Playwright controls**: `--wait-for SELECTOR` and `--scroll N` for lazy or
405
- delayed content.
406
- - **Security**: all XML goes through a DOCTYPE/ENTITY guard that uses
407
- `defusedxml` when available; Office ZIP containers have entry-count and
408
- expansion limits.
409
- - **Multi-host adaptation**: `AGENTS.md` is the generic agent contract, with
410
- matching rule files for Cursor, Windsurf, Cline, Continue, Roo Code and
411
- GitHub Copilot. `install.py` writes project- or user-scoped rules in one run.
412
- - **Packaging**: root `package.json`, `index.js` and `cordis.patch.yml` cover
413
- npm and dsh; `commands/fetch.md`, `surf.md`, `scholar.md` and `map.md` expose
414
- the actions as plugin commands.
415
- - **Bilingual documentation**: `README.md` (English) and `README_zh.md`
416
- (Chinese) mirror each other section for section, with the same three-method
417
- installation guide, quick start, requirements, repository tree, test summary,
418
- limitations, issue rules, trademark notice and licence. All documents are
419
- emoji-free.
420
- - **Tests**: 115 tests, including installer coverage and host-rule and
421
- disclaimer checks.
422
-
423
- ### Fixed
424
-
425
- - **Trafilatura cross-document deduplication**: extraction previously ran with
426
- `deduplicate=True`, whose process-global cache silently removed sentences
427
- seen on an earlier page. Long agent sessions could lose a page's opening
428
- paragraph. Deduplication is now disabled across calls.
429
- - Sitemap `<loc>` elements were not mapped to the `url` field.
430
- - RSS/XML payloads no longer trigger the HTML dynamic-rendering heuristic or
431
- BeautifulSoup's XML warning.
432
-
433
- ### Changed
434
-
435
- - Version 1.0.0 across all platform manifests, `SKILL.md` and the CLI
436
- User-Agent.
437
- - `install.py --global` appends a marked block to `~/.claude/CLAUDE.md` instead
438
- of replacing the user's global instructions.
439
- - Documentation presents web-forager as an independent plugin; the `--root`
440
- output is described as a four-zone workspace without third-party branding.
441
-
442
- ## 0.2.0 — 2026-11-06
443
-
444
- - 360/Bing/Sogou search with automatic merge and dedupe.
445
- - Optional PDF extraction through `pypdf` (exit code 7 when absent).
446
- - `--action surf` with ranked link BFS and resumable `--session`.
447
- - `--agent` OpenCLI-style success/error envelope.
448
-
449
- ## 0.1.0 — 2026-11-06
450
-
451
- - Initial clean-room fetcher: robots.txt, polite rate limiting, HTML → Markdown,
452
- metadata, tables, optional Playwright and a Bilibili adapter.
453
- - Standalone packaging for Claude Code, Codex, CodeBuddy and DSH, plus
454
- four-zone `--root` workspace integration.
1
+ # Changelog
2
+
3
+ ## 1.3.1 — 2026-11-17
4
+
5
+ Precision work on the two weakest parts of 1.3.0: the injection screen treated every keyword hit as a reason to withhold, and search relevance was close to binary. This release also adds URL canonicalization, reciprocal-rank fusion, Chinese bigram matching, intent-aware source authority and a broad-recall/focused-evidence `research` action.
6
+
7
+ ### Security
8
+
9
+ - **MCP can no longer loosen the boundary.** `respect_robots`, `robots_policy`, `input_gate` and `proxy` were removed from every HTTP tool schema and are force-set to their safe values for agent calls. The local-file `clean` tool (`web_clean`) was removed from MCP entirely: it read an arbitrary path chosen by the model and is now CLI-only. The browser MCP server lost `bridge`, `out`, `root`, `allowWrite`, `inputGate` and `strictRobots`; every MCP playbook run is read-only and always passes `--strict-robots`.
10
+ - **Search and scholarly query endpoints now have an explicit policy.** `--query-robots` (or `WEB_FORAGER_QUERY_ROBOTS`) defaults to `warn`. Bing, 360, Sogou and the arXiv export API disallow their query paths, so the default reads them and records `query_robots: warn` in the engine report; `respect` refuses and `ignore` skips the check, and only a user can choose them. MCP does not expose the switch, though an authorised user can set the variable in the server environment. This closes the previous unconditional `check_robots=False` in the search and scholar paths.
11
+ - **Every redirect hop is checked against robots.txt.** HTTP fetches, URL resolution, the browser bridge and the Bilibili API client now re-check the final target. A public URL that redirects into a disallowed path is refused instead of read.
12
+ - **Rendered pages install a subresource guard.** Python Playwright rendering aborts `file:`/`filesystem:` requests and private HTTP(S) subresources, matching the browser bridge's request filter, so a hostile page cannot use an iframe or image to probe the LAN.
13
+ - **Bounded local extraction.** Sitemap gzip output is capped at 32 MiB through a streaming decompressor, PDF extraction stops at 500 pages or 20 seconds with `truncated` and a warning, and `clean --input` refuses local files over 16 MiB (stdin included). A few kilobytes of gzip can no longer expand into gigabytes.
14
+ - **Synthetic research rows are screened.** Feed headlines, official-index headlines and WeChat snippets now carry `trust: "untrusted"`, `injection_flags` and `injection_findings` like fetched pages, instead of entering evidence unscreened.
15
+ - **Dependency floors follow current advisories.** `certifi>=2026.10.18`, `requests>=2.33.0`, `beautifulsoup4>=4.14.3`, `trafilatura>=2.3.1` and `pypdf>=6.21.4`; the local environment is on pypdf 7.1.0, certifi 2026.10.23, requests 2.34.2 and soupsieve 4.1.1.
16
+ - **Local state is owner-only.** On POSIX the HTTP cache directory and browser profile are `0700`, cache files, surf sessions and browser sessions are `0600`. The cache already refused responses that set cookies and stripped `Authorization` / `Proxy-Authorization` headers; this release adds the permission boundary and documents what leaves the machine. There is no telemetry, account or API key.
17
+ - **CDP is off by default.** `browser/launch_chrome.py` now opens only the dedicated profile; `--cdp` opts in to a loopback CDP port, random by default (`--port 9222` fixes it, and passing `--port` implies `--cdp`). The launcher prints a warning that CDP has no authentication. Without `--cdp`, browser mode reuses the profile through its normal bridge, so the login flow needs no open debug port.
18
+ - **Verbose logging redacts URL query strings and fragments.** `--verbose` keeps the scheme, host and path, so a signed URL cannot write its token to stderr. The full URL remains in the result record the caller requested.
19
+
20
+ ### Added
21
+
22
+ - **`research` action and `web_research` MCP tool.** A question is first widened across the free engines, then a few distinct, relevant sources are selected by an explicit authority tier and read in full. The result carries `seeds`, cited `evidence` passages (`url`, `title`, `authors`, `published_at`, `date_source`, `passages`), `strategy`, `freshness`, `skipped` and `answerability` (`high`/`medium`/`low`/`none`, plus `source_count`, `view_hosts`, `multi_view`, `stale`). It never invents an answer: thin evidence returns `low` or `none` with the reason.
23
+ - **Intent-aware authority tiers.** `classify_intent` separates technical, news, comparison, policy, howto, research and factual questions. `authority_for` labels each host `primary`, `official`, `academic`, `media`, `reference`, `community`, `low` or `unknown` and weights it by intent; every search result now carries `authority_tier`, `authority_score` and `intent`.
24
+ - **Targeted second pass that survives a broken `site:` operator.** 360 and Bing often ignore `site:` in mainland networks, so research now prefers direct official entry pages (Python asyncio, Go concurrency, Rust, MDN, Microsoft Wi-Fi support), then a site's own public search page (Microsoft Support, MDN), and only then a `site:` query for a non-CJK question. The arXiv API title field remains the paper-shaped route. An official overview page may be followed one same-site link further when that link names the API.
25
+ - **Comparison questions need two full viewpoints.** A comparison is `high` only with multiple independent full sources; a partial WeChat snippet never counts as a viewpoint. When two full authoritative sources already exist for a comparison or technical question, the WeChat-only snippet is skipped instead of being mixed into the evidence.
26
+ - **Cross-language concept matching.** Chinese questions about concurrency, comparisons, connectivity, errors, installation and performance add a small English concept vocabulary for passage matching only. A Chinese “Python 和 Go 高并发” question can therefore match an English Go concurrency page without changing the query sent to the engines.
27
+ - **Publication dates and freshness.** `_normalize_date` reads ISO, Chinese and English metadata/datelines; evidence carries `published_at` and `date_source`, and the payload carries `freshness`. A news answer without a readable date cannot be labelled `high`.
28
+ - **News RSS fallback.** When a news query has no usable engine rows, research reads a public China News Service finance feed and returns dated headlines (`kind: feed_headline`) with an explicit warning. It is a headline/summary, never a fabricated article body. A fetched article that carries no dateline inherits the feed's publication date, so `freshness` is not lost after the full-page read.
29
+ - **Bounded focus reads.** Each research focus fetch is capped at 8 seconds with no retry, and each robots.txt read at 6 seconds, with a 3-second per-address connect timeout. A blocked or dead origin is reported as a skipped fetch instead of consuming the whole call.
30
+ - **CJK bigram matching.** A Chinese sentence is no longer one unmatchable token. Long CJK runs become overlapping character bigrams, while a mixed date token such as `2026年11月` stays whole. Chinese recall that was 0/8 on the priority set now reaches 7–8/8 in local tests.
31
+ - **Search-engine intermediate pages are dropped.** A Sogou/360 result page echoed by another engine is no longer ranked as an answer.
32
+
33
+ - **`search_quality` block.** A search now carries `{query, engine, returned, relevant, off_topic, language_mismatch, engines_answered, engine_errors, engine_status, engine_states, dropped_off_topic, returned_before_filter}` beside the prose warnings, so an agent can decide whether to trust a result list without parsing English. Engine states distinguish `ok`, `empty`, `blocked`, `timeout`, `parse_error` and `error`.
34
+ - **Rank explanations.** Every merged result carries `rank_components` (`coverage`, `title`, `exact`, `rrf`, `authority`, `freshness`, `language_penalty`) and a short `rank_reason`, so one ordering decision can be debugged without printing internal scores.
35
+ - **Canonical URL dedupe.** `utm_*`, `fbclid`, `gclid`, fragments, `www.`, trailing slashes and AMP suffixes no longer make one article look like two results. The display URL is left unchanged; only the dedupe key is normalized.
36
+ - **RRF and exact-title weighting.** A result found by two engines receives a reciprocal-rank-fusion contribution, and an exact title match gets a small bonus over a title that merely contains the phrase. Engine order is still the final tie-break.
37
+ - **Light English stemming for matching.** `injection`/`injections`, `study`/`studies`, `learn`/`learning` meet during relevance checks; the displayed text and query are unchanged.
38
+ - **A quality baseline suite.** `tests/test_quality_baseline.py` fixes the academic-quotation detector case and the Excel INDIRECT search case as regression cases, with a small metrics check on top.
39
+
40
+ ### Changed
41
+
42
+ - **Injection findings now carry evidence and a severity.** Every record has `injection_findings: [{code, severity, confidence, span, snippet, context}]` alongside the existing `injection_flags`. `span` is a character range, `severity` is `hard` or `soft`, and `context` is `direct`, `quote` or `example`. The first warning repeats up to three matching snippets, so a false positive can be diagnosed instead of guessed at.
43
+ - **`--input-gate strict` withholds only hard rules.** Forged system tags, secret-exfiltration requests, `curl ... | sh` and `rm -rf /` still reject the body. `OVERRIDE` and `ROLE_SWAP` are soft: a paper quoting an attack template, or a security note discussing "ignore previous instructions", is marked with lower confidence and kept. This directly fixes the arXiv-paper case where a legitimate quotation looked like an attack.
44
+ - **The scan covers the whole body.** The old scanner stopped at 120,000 characters. It now walks overlapping 64k windows, keeps at most three findings per rule, and runs before `max_chars` truncation, so a hostile instruction near the tail of a long page cannot disappear. The browser scanner applies the same rules and limits.
45
+
46
+ ### Fixed
47
+
48
+ - **A single engine's results are ranked instead of passed through.** `search()` used to return `groups[0][1]` untouched when only one engine answered; every result now goes through the same scorer.
49
+ - **Search relevance is graded, not a 0/1 threshold.** A result must contain the exact phrase or at least two meaningful query terms with half the query covered. Each result carries `coverage`, `matched_terms`, `relevance` and `language_match`. A Chinese tutorial that repeats one English product name is `language_match: false`, ranked last, and dropped when at least three real answers exist. The `Excel INDIRECT` rows are the regression fixture.
50
+ - **Authority beats a content farm on an equal title.** `docs.python.org`, official documentation, academic sources and media are boosted per intent; CSDN, php.cn and document farms are pushed down. The same-title test now puts the official page first.
51
+ - **`search_warnings` reports partial relevance, not only total failure.** It now distinguishes "nothing relevant", "N of M look off-topic" and "no results", and search titles/snippets pass through the same injection scanner as page bodies, carrying their own findings.
52
+
53
+ ### Tests
54
+
55
+ - 333 Python tests and 91 browser tests. New coverage: finding evidence and severity, academic-quotation suppression, scanning past 120k, repeated soft hits not hiding a hard rule, strict mode keeping a soft quotation, single-engine ranking, language mismatch, the two-term/phrase relevance rule, dropping off-topic rows when enough real answers exist, search-snippet scanning, canonical URL dedupe, RRF, light stemming, `search_quality` and engine states, rank explanations, the quality baseline corpus, CJK bigram matching, intent/authority tiers, research passage extraction, cross-language concept matching, official comparison entry points, comparison viewpoint rules, date normalization, seed diversity, arXiv author metadata, the WeChat partial-snippet path, redirect robots checks, the query-endpoint policy, the sitemap gzip cap, the PDF page cap, synthetic-row injection screening and the MCP schema restrictions.
56
+
57
+ ## 1.3.0 — 2026-11-15
58
+
59
+ A workflow release: a robots policy that can record an authorised override, search that fills its page budget with readable results, parallel engines, and a browser session you can log into once.
60
+
61
+ ### Added
62
+
63
+ - **`--robots-policy respect|warn|ignore`.** `respect` is the default and still refuses a disallowed path. `warn` reads it and writes `compliance.robots_overridden: true` plus a warning, for a user who is authorised to read the page. `ignore` is the old `--ignore-robots`, with `robots_checked: false`. Every HTTP record now carries a `compliance` block.
64
+ - **The browser profile launcher** (`browser/launch_chrome.py`) starts Chrome/Chromium/Edge with `~/.web-forager/browser/profile` and a DevTools port, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`.
65
+ - **Storage-state import and export** for browser mode: `--storage-state FILE` loads cookies/localStorage, `--save-storage-state FILE` writes them after the run. A saved file is forced to mode `0600`, a path inside `--root` is refused, and the record says the file carries login state.
66
+ - **`--serial-engines`** keeps a diagnostic run strictly serial; the default now asks engines on different hosts at the same time while each host keeps its one-request-per-second delay.
67
+ - **A `wechat-dig` browser playbook** opens one WeChat article URL in the browser profile and captures its text. When the profile has already passed WeChat's environment check the article is read; when it has not, the playbook reports `blocked` with a hint and does not try to solve the check.
68
+ - **Verification pages are reported instead of recycled.** HTTP mode recognises WeChat's environment check and the common English anti-bot interstitials, raises `VerificationRequired`, and `fetch --query` records the result as `verification_required` with a warning that names the browser route. The CLI exits 9. The bundled bridge also hands the whole `#js_content` article over as one `content` snapshot node, so WeChat paragraphs are not reassembled from leaf spans.
69
+ - **Automatic render fallback for script shells.** When a server answers with a minimal HTML page that carries only a script or redirect, `--render auto` now renders it once with Playwright before extraction. The record says `render_fallback: true`; `fetch --query` allows at most two such renders per query, so one slow site cannot stall a search. This needs no API key and closes the php.cn-style empty-page case.
70
+ - **Free ranking and query repair.** Search results are ordered by query coverage, title match, a small domain prior (docs, papers, government and universities up; document farms down) and snippet freshness. The final list keeps at most two results per host and fills the rest from the overflow. A thin first pass is retried once with a simplified query — quotes and common stopwords removed, or a spaced CJK query joined — and the retry is marked `query_variant` in `engine_report`. No model, key or network service is involved.
71
+
72
+ ### Changed
73
+
74
+ - **`max_pages` now means readable pages.** `fetch --query` keeps scanning past a robots-denied or unreadable result (up to a bounded number of attempts), returns the pages it could read, and lists the rest in `skipped` with a code and reason. The response carries a summary warning instead of leaving the agent to discover failures page by page.
75
+ - The `robots_denied` hint now names `--robots-policy warn` for an authorised override and says not to silently switch to a tool that ignores robots. A `verification_required` skip says the same about not solving the check, and points a WeChat link at the `wechat-dig` playbook.
76
+
77
+ ### Host compatibility
78
+
79
+ - **Codex now gets a working MCP launch.** The plugin ships `.codex-plugin/mcp.json`, which uses relative `mcp_server.py` and `browser/mcp_server.js` paths with `cwd: "."`; Codex resolves both against the installed plugin directory. It no longer inherits Claude Code's `${CLAUDE_PLUGIN_ROOT}` argument, which Codex leaves literal.
80
+ - **CodeBuddy now declares its MCP servers in the manifest it reads.** `.codebuddy-plugin/plugin.json` carries both `mcpServers` entries with `${CLAUDE_PLUGIN_ROOT}`, so a new CodeBuddy session mounts the HTTP and browser servers. `codebuddy mcp list` can run before plugins are mounted; an empty list there is not a failed install.
81
+ - **dsh now mounts real tools.** `cordis.patch.yml` inserts two `@deepseek-ai/dsh-mcp-client` entries, one per server, and resolves the profile package directory with `dshHomePath(...)`. The previous patch mounted only the no-op `web-forager` module.
82
+ - **The npm package carries every host manifest.** `files` now includes `.claude-plugin/`, `.codex-plugin/`, `.codebuddy-plugin/`, `.dsh-plugin/`, `.mcp.json`, the rule-file directories, `install.py` and `requirements.txt`.
83
+ - The Cursor rule heading now says v1.3.0 instead of v1.0.0.
84
+
85
+ ### Documentation
86
+
87
+ - `SKILL.md` and `AGENTS.md` state the rule: a `robots_denied` or `login_required` result is a boundary, not a prompt to re-fetch with a tool that ignores robots.
88
+ - The reference documents the launcher, storage-state switches, the robots policy, the new `compliance` block, the `wechat-dig` playbook and exit 9.
89
+ - Both READMEs were rewritten around a first visit: one pitch, the three install methods, what a run looks like, the security boundary and the legal notice, plus a section on keeping an installed copy current (including the ZCode marketplace-snapshot trap). A DeepSec Shield scan of the release tree reports 0 findings; its agent audit still flags the three signature strings this project defines, which are the scanner's own patterns, not page content.
90
+
91
+ ### Tests
92
+
93
+ - **258 Python tests** and **87 browser tests**, all offline. New coverage: the warn policy and its record, clean compliance fields, filling pages past a denial, overlapping engines versus `--serial-engines`, the launcher dry run, saving storage state through the bridge protocol, WeChat verification detection at the engine, the CLI and the search layer, the `wechat-dig` capture and `blocked` paths, the `content` snapshot node, domain/freshness tie-breaks, host diversity, query variants, the shell-render fallback with and without its budget, plus the Codex/CodeBuddy/dsh MCP manifests and the npm host-file list.
94
+
95
+ ## 1.2.3 — 2026-11-15
96
+
97
+ A follow-up hardening release after a DeepSec Shield audit of 1.2.2.
98
+
99
+ ### Security
100
+
101
+ - **The browser bridge is no longer started through a shell.** `--bridge` and `WEB_FORAGER_BRIDGE` are split into an executable and its arguments and run directly, so a path or environment value cannot append a second command. Quoted paths with spaces still work; pipes, `&&` and redirection are not interpreted.
102
+ - **CI actions are pinned to commit SHAs.** `actions/checkout`, `actions/setup-python` and `actions/setup-node` name the exact commit, so a moved tag cannot change what the workflow runs.
103
+
104
+ ### Documentation
105
+
106
+ - The reference states that the bridge command is split on whitespace and started without a shell.
107
+
108
+ ### Tests
109
+
110
+ - **228 Python tests** and **81 browser tests**, all offline. New coverage: quoted bridge paths, literal backslashes, an unterminated quote, and a shell metacharacter that must be passed through as an argument rather than executed.
111
+
112
+ ## 1.2.2 — 2026-11-15
113
+
114
+ A hardening release that replaces the 1.2.2 build published on 2026-11-12: a request can no longer be aimed at the machine or the LAN, browser state stays in a profile of its own, and fetched text is labelled as third-party data. The legal-notice rewrite and packaging rules from that earlier build are included.
115
+
116
+ ### Security
117
+
118
+ - **Private-address guard (SSRF).** Loopback, RFC 1918, link-local (including `169.254.169.254`), CGNAT, multicast, reserved and their IPv6 forms are refused before a socket opens. The host is resolved and checked as written, so decimal IPv4 spellings and IPv6-mapped IPv4 are caught too. The check runs before the request, on every redirect hop (redirects are followed one hop at a time instead of by `requests`), on the robots.txt download, on search-result resolution, and on the URL a rendered page actually lands on. `--allow-private` / `WEB_FORAGER_ALLOW_PRIVATE=1` is the documented opt-out for local fixtures and trusted intranet targets.
119
+ - **Browser navigation is guarded too.** `guardedGoto` checks the requested and the final URL, and the bundled Playwright bridge installs a request filter: every document and subresource the page asks for is resolved, and private targets are aborted. A hostile page can no longer use the open browser to probe `http://127.0.0.1:…` or a cloud metadata endpoint.
120
+ - **Browser state stays in its own profile.** The bundled bridge already kept `~/.web-forager/browser/profile`; the report now states `dedicated_profile` and `shared_browser`, and a CDP attach to a browser you use for everything else adds a warning that every cookie and login in that profile is reachable by the pages a playbook opens.
121
+ - **Fetched text is marked as data.** Every record carries `trust: "untrusted"`. A small scanner (same rules in `scripts/untrusted.py` and `browser/lib/untrusted.js`) names instruction-shaped passages — overrides, fake system tags, `curl … | sh`, secret-exfiltration requests, role swaps — in `injection_flags`, and puts one warning first in the list. `--input-gate strict` withholds the body instead of handing it on; `--input-gate off` skips the scan.
122
+ - **Cookies never enter a record.** The Bilibili cookie is held in memory for the API request and is not written to the record, the session file or an error message; the browser profile directory is reported as a flag, not a path.
123
+
124
+ ### Documentation
125
+
126
+ - **Both READMEs were reorganised around a first visit.** The page now opens with what the tool is, what you get and what it refuses, followed by a table of contents, the why section, the three install methods and a quick start split by entry point — chat, terminal, MCP and the optional browser mode — with a "which action do I need" table and copy-paste commands. The first screen carries a one-line notice that links to the full legal terms; the terms themselves moved below the quick start, section by section.
127
+ - **Quick start now holds only the three entry points**: chat, terminal and MCP. The browser playbook table, switches and `--strict-robots` details, and "when the engines are blocked" routes moved to the reference; each is a one-sentence pointer on the landing page. Installation opens with "let your AI install it" as the recommended path for most readers, followed by the plugin-marketplace URL and the per-host commands. The contents is grouped, with acknowledgements, trademarks and licence added.
128
+ - **Reference material moved to `docs/REFERENCE.md` and `docs/REFERENCE_zh.md`**: the full action list and exit-code tables, record fields and `core-1`, workspace output, MCP tool names and protocol checks, requirements, repository layout, test details and extra CLI examples. Every example is its own one-line block, so it can be copied or run without trimming its neighbours. README keeps a three-table quick start, a short exit-code summary and a link.
129
+ - **Long sections fold.** The repository tree, test coverage notes and MCP tool names sit in `<details>` blocks; known limitations are grouped into read-first, edge cases and experimental/high-risk switches.
130
+ - Both READMEs gain a **Security boundaries** section and three limitation bullets (the guard's off switch, the screen's limits, the CDP profile exposure).
131
+ - `AGENTS.md`, `SKILL.md` and the six host rule files state that fetched text is data and must not be executed as instructions.
132
+ - **The README pair carries a six-part legal notice** — purpose, prohibited use, responsibility, no warranty, limitation of liability, redistribution. `SKILL.md`, `AGENTS.md` and the six host rule files carry the short version.
133
+ - **Test fixtures are no longer part of a published archive.** `.gitattributes` marks `tests/` and `browser/tests/` as `export-ignore`, and both npm manifests drop tests from `files`; CI still runs them from the repository clone.
134
+
135
+ ### Tests
136
+
137
+ - **228 Python tests** and **75 browser tests**, all offline. New coverage: every private range, a redirect into a metadata endpoint, a two-hop redirect chain, the robots download, the CLI exit code and `--allow-private`, marker detection, strict withholding, and the browser-side guard and scanner.
138
+
139
+ ## 1.2.1 — 2026-11-11
140
+
141
+ A search hygiene release: a Chinese query no longer collects dictionary pages for one of its characters, and the README pair was rewritten around what the tool does now.
142
+
143
+ ### Fixed
144
+
145
+ - **A mostly-CJK query is no longer handed to Bing as a fallback.** Bing splits a Chinese phrase into single characters, so a search about 反逆向工程 came back with the dictionary page of 反 once the Chinese engines answered thinly. `auto` now stops after 360, Sogou and 搜狗微信; `--engine bing` still asks Bing explicitly.
146
+ - **Dictionary pages that match one character of the query are dropped**, whatever engine returned them. `looks_like_character_page()` recognises a title that shares exactly one character with a longer query and carries a dictionary marker (百科, 拼音, 笔顺, 部首, 组词…), so a result list is not padded with them and `engine_report` counts only usable hits.
147
+
148
+ ### Documentation
149
+
150
+ - **Both READMEs were rewritten.** The duplicated mode paragraphs and the second disclaimer block are gone; scope and compliance sit in one section, the two modes face each other in one table, and the CJK search routing, `find`/`feed`, CI and current tool/test counts are stated once. README text now matches the 10 HTTP tools, 4 browser tools and 9 CLI actions that actually ship.
151
+
152
+ ### Tests
153
+
154
+ - **174 Python tests** and **60 browser tests**, all offline. New coverage: a CJK query never calls Bing, a character page is dropped from an engine's results, and `looks_like_character_page()` leaves a single-character query and a full-phrase hit alone.
155
+
156
+ ## 1.2.0 — 2026-11-09
157
+
158
+ Search is no longer the only way in. A site's own search page and its feed are
159
+ first-class routes now, every engine attempt is reported, and one run can be
160
+ sent through a proxy.
161
+
162
+ ### Added
163
+
164
+ - **`--action find` (`web_find`)**: search inside one site through the site's
165
+ own search page. It tries `/?s=`, `/search?q=`, `/?search=` and
166
+ `/search?query=`, ranks the page's links by query coverage, and fetches the
167
+ best ones. This is the route that keeps working when the general engines
168
+ answer with dictionary pages or a verification wall.
169
+ - **`--action feed` (`web_feed`)**: read a site's RSS/Atom feed, discovering it
170
+ from `<link rel=alternate>` when given the site URL instead of the feed URL.
171
+ - **`engine_report`**: every search response now carries one entry per engine
172
+ tried — `ok`, `results`, `ms`, and `error` when it failed, plus `fallback`
173
+ when the engine was asked only because the first answer was thin. "Search is
174
+ broken" becomes "Bing answered, Sogou showed a verification page".
175
+ - **`doctor --probe` (`web_doctor {probe: true}`)**: asks 360, Bing, Sogou, the
176
+ WeChat index and DuckDuckGo once each and reports which ones answer from this
177
+ network, with latency.
178
+ - **`--proxy URL`** and `proxy` on the MCP tools. Requests already honoured
179
+ `HTTPS_PROXY` / `ALL_PROXY`; an explicit proxy now overrides them for one run,
180
+ and `doctor` reports the proxy in use with credentials redacted.
181
+
182
+ ### Changed
183
+
184
+ - The MCP server exposes ten HTTP tools instead of eight; the README and
185
+ `SKILL.md` document the two routes that do not depend on a search engine.
186
+ - `doctor` reports the proxy source (`config` or `env:NAME`) without printing a
187
+ password.
188
+
189
+ ### Documentation
190
+
191
+ - **The disclaimer forbids building a profile of an identifiable person.**
192
+ Collecting, aggregating or inferring personal information, or using it for
193
+ doxxing, harassment or stalking, is prohibited and is the user's own legal
194
+ responsibility. It appears in `--help`, `manifest.json`, the README pair,
195
+ `SKILL.md` and every host rule file.
196
+ - **The per-host install commands moved to `docs/INSTALL_COMMANDS.md` and
197
+ `docs/INSTALL_COMMANDS_zh.md`.** Methods 1 and 3 stay in the README because
198
+ they are three lines each; Method 2, which is a page of commands, is one link.
199
+ - **The READMEs gained an Acknowledgements section** listing the reporter of
200
+ issue #1 by name, with a link to `docs/ACKNOWLEDGEMENTS.md` /
201
+ `docs/ACKNOWLEDGEMENTS_zh.md` for the details.
202
+
203
+ ### Acknowledgements
204
+
205
+ - **NotFatJustRound** ([issue #1](https://github.com/JiamingShi-JsPhys/web-forager/issues/1))
206
+ reported that `web_search` returned every page body twice — once in `pages`,
207
+ once in `results` — which doubled the response and pushed the tail past an
208
+ agent's size limit; the compact `results` index in 1.0.2 came from that report.
209
+
210
+ ### Fixed
211
+
212
+ - **Links inside `<article><header>` are no longer skipped.** A results page
213
+ commonly wraps each card's title in an article header; the universal
214
+ "skip `<header>`" rule silently dropped exactly the links a search page
215
+ offers. Page-level headers and navigations are still skipped.
216
+ - `doctor --probe` makes one attempt per engine instead of retrying, so a
217
+ blocking engine costs one timeout rather than three.
218
+ - `.cursor/rules/web-forager.mdc` carried `v1.0.0` in its front matter, so
219
+ `install.py --only cursor` wrote a stale label; it now follows the release
220
+ version.
221
+
222
+ ### Tests
223
+
224
+ - **172 Python tests** and **60 browser tests**, all offline. The new suite
225
+ covers site search against a local WordPress-style page, feed discovery from a
226
+ page link and from a direct feed, the per-engine report, the probe, proxy
227
+ wiring and the redaction of proxy credentials.
228
+
229
+ ## 1.1.0 — 2026-11-08
230
+
231
+ Browser mode joins web-forager, and an audit pass follows it. The HTTP toolkit
232
+ and the browser layer ship as **one plugin, one name and one version**; the
233
+ former standalone `web-actor` product no longer exists. The empty-result path,
234
+ the bridge's capability claims, the robots.txt audit trail and the shared record
235
+ shape are now honest, and neither README claims coverage the code does not have.
236
+
237
+ ### Added
238
+
239
+ - **Browser mode** under `browser/`: a CLI, seven read-only playbooks and an MCP
240
+ server that drive a real browser through a pluggable bridge.
241
+ - **Playbooks**: `bili-feed`, `bili-video`, `bili-chain`, `bili-danmaku`,
242
+ `zhihu-dig`, `xhs-dig` and `search-dig`.
243
+ - **Second MCP server**: `browser/mcp_server.js` exposes `browser_list`,
244
+ `browser_run`, `browser_session` and `browser_doctor`; `.mcp.json`
245
+ registers both servers.
246
+ - **A reference bridge that is found automatically**:
247
+ `browser/bridges/playwright-bridge.js` supports a fresh browser, a persistent
248
+ profile, a headful window and attaching over CDP. `browser/lib/bridge.js`
249
+ looks for `--bridge`, then `WEB_FORAGER_BRIDGE`, then that file, so a first
250
+ run no longer fails with `backend_unavailable` only because nobody exported a
251
+ variable.
252
+ - **A browser profile that persists by default**:
253
+ `~/.web-forager/browser/profile` keeps cookies and logins between runs;
254
+ `WEB_FORAGER_BROWSER_PROFILE=off` restores the throwaway browser. The
255
+ reference bridge also drops the automation banner flag.
256
+ - **Browser records**: the same `metadata / markdown / tables / warnings` shape,
257
+ written to `<WS>/process/web-forager/browser/` and `_broadcast/`.
258
+ - **`/browse` command** for the browser playbooks.
259
+ - **Shared record core (`core-1`)**: `--core` on both CLIs and `core: true` on
260
+ `browser_run` return `schema_version / source / mode / url / title /
261
+ collected_at / metadata / markdown / tables / warnings / extra`, so a mixed
262
+ list of HTTP and browser records can be read uniformly. The same fold is
263
+ available as `engine.normalize_record()` and `normalizeRecord()`.
264
+ - **Browser robots auditing**: `--strict-robots` reads `robots.txt` before every
265
+ navigation and stops with `robots_denied` (exit 3) on a disallowed path.
266
+ Without it the record still states `compliance.robots_checked: false`, so the
267
+ compliance status is auditable either way. `browser/lib/robots.js` implements
268
+ the same RFC 9309 subset as the Python engine.
269
+ - **`degraded` (exit 8) and `blocked` (exit 9)**: a browser run that reads a
270
+ page but finds nothing the playbook looks for now reports `degraded` with its
271
+ records attached, instead of `ok: true` with `captured: 0`. An anti-bot or
272
+ error interstitial reports `blocked`. Both are distinct from `login_required`.
273
+ - **`describe` handshake**: a bridge can declare its capabilities; a bridge that
274
+ does not gets `declared: false` with every capability false. The reference
275
+ Playwright bridge and the test fixtures implement it.
276
+ - **`browser/lib/selectors.js`**: every structural assumption about a
277
+ third-party page lives in one file.
278
+ - **Real-page fixtures**: `browser/tests/fixtures/snapshots/` holds snapshots
279
+ built from live Bilibili pages on 2026-11-07 by
280
+ `browser/tests/tools/html-to-snapshot.py`, with `tests/real-pages.test.js`
281
+ as the offline regression and `npm run test:live` as the opt-in live check.
282
+ - **`--action doctor`** and the **`web_doctor` MCP tool** report every
283
+ dependency with its version or its install command, so an agent that receives
284
+ a thin page can install what is missing instead of guessing.
285
+ - **`node browser/run.js --doctor`** and the **`browser_doctor` MCP tool** do the
286
+ same for the browser stack: Node version, the bridge command in use, whether
287
+ the `playwright` package resolves (searched from the bridge's own directory
288
+ first), whether its Chromium binary is downloaded, and where the profile is.
289
+ - **The Bilibili search API is asked before the result page is read.**
290
+ `lib/bili-api.js` calls Bilibili's public JSON search endpoint (the `buvid3`
291
+ cookie comes from one home-page request) and falls back to scraping the page
292
+ only when the endpoint refuses. `source` in the record says which path ran.
293
+ `WEB_FORAGER_BILI_API=off` keeps the playbook on the page.
294
+ - **`respect_robots` on the MCP tools.** `web_fetch`, `web_search` and
295
+ `web_surf` accept the same switch the CLI spells `--ignore-robots`, so a page
296
+ that robots.txt disallows can be read when the caller is authorised to read
297
+ it. The default is still to check.
298
+ - **Bilibili's JSON search on the direct path.** When robots checks are off
299
+ (`--direct-search` or a global override), `scripts/bilibili.py` asks
300
+ `api.bilibili.com` first — the `buvid3` cookie comes from one home-page
301
+ request, and that request is never answered from the cache because a cached
302
+ anti-bot page supplies no cookie — and scrapes the search page only if the
303
+ API answers `-412` or malformed JSON. Cards read from the page are kept only
304
+ when their text mentions the query, so a recommendation rail can no longer
305
+ pass as search results. `route` in the payload names the host that answered.
306
+ - **The WeChat article index as a search engine (`--engine weixin`).**
307
+ `weixin.sogou.com` indexes article text, which is the one channel that
308
+ normally carries a Chinese person's or institution's name; each result keeps
309
+ its account name and publish date. It joins `auto` for mostly-CJK queries and
310
+ `--engine all`.
311
+
312
+ ### Changed
313
+
314
+ - Version 1.1.0 across manifests, `SKILL.md`, CLI and README install tags.
315
+ - The browser layer uses the web-forager name throughout: environment variables
316
+ are `WEB_FORAGER_BRIDGE`, `WEB_FORAGER_BROWSER_HOME`,
317
+ `WEB_FORAGER_BROWSER_HEADFUL`, `WEB_FORAGER_BROWSER_PROFILE` and
318
+ `WEB_FORAGER_BROWSER_CDP`; sessions live in `~/.web-forager/browser/`; record
319
+ `source` is `web-forager-browser`.
320
+ - The plugin manifest gained a `browser` block listing playbooks, backends and
321
+ the browser MCP tools.
322
+ - The User-Agent now carries the real version and the repository URL instead of
323
+ a stale `WebForager/1.0` and a placeholder domain.
324
+ - `bili-danmaku` asks the bridge to observe first and falls back to DOM text
325
+ second; a DOM-derived list is recorded with a warning that it is partial.
326
+ - Duplicate card links now merge, so a captured card keeps its readable title
327
+ instead of the "views danmaku duration" overlay text.
328
+ - The MCP server exposes eight HTTP tools instead of seven.
329
+ - **`auto` widens a thin answer once.** A Latin query that returns fewer than
330
+ three distinct results from 360 and Bing now asks Sogou as well; a CJK query
331
+ whose engines all fail keeps its Bing fallback. A full answer is not widened,
332
+ so the common case costs nothing extra.
333
+ - **`--max-pages 0` returns a search answer without page bodies.** The CLI used
334
+ to floor the value at 1; both the CLI and `web_search` now let a caller ask
335
+ for the result list alone and fetch only what it needs with `--max-chars`.
336
+ - **The MCP search tool carries the same warnings as the CLI.** A response whose
337
+ results do not mention the query text — or that is empty — now says so in
338
+ `warnings` instead of leaving the agent to present dictionary pages as
339
+ findings.
340
+
341
+ ### Fixed
342
+
343
+ - A browser run that found nothing no longer reports success.
344
+ - A bridge without `observe` is no longer reported as observing.
345
+ - The Bilibili feed enrichment test no longer reaches the network, so the suite
346
+ is genuinely offline.
347
+ - **`results` no longer repeats every page body.** A search response carried
348
+ each Markdown page twice — once in `pages`, once in `results` — so a caller
349
+ with an output cap received a truncated tail. `results` is now a compact
350
+ index (`url`, `title`, `status`, `content_type`, `markdown_chars`, `tables`,
351
+ `snippet`, `warnings`, `error`); the body stays in `pages` and, for a single
352
+ fetch, at the top level. The old shape is one flag away: `--full-results` on
353
+ the CLI, `fullResults` on the MCP tools.
354
+ - **`auto` no longer mixes Bing into Chinese queries.** Bing treats a quoted
355
+ Chinese phrase as single characters, so dictionary and encyclopaedia pages
356
+ outranked the precise hits. A mostly-CJK query now asks 360, Sogou and the
357
+ WeChat index, and falls back to Bing only if all three are unreachable. Latin
358
+ queries are unchanged.
359
+ - **Navigation waits for the page it lands on.** `goto` now waits for the load
360
+ state and a short settle before returning, so an `evaluate` issued right after
361
+ a redirect no longer races it; `snapshot` retries once on the "execution
362
+ context was destroyed" error that a redirect leaves behind.
363
+ - **The bridge's stderr is kept.** A dead bridge now reports its last few
364
+ stderr lines in the error instead of a bare timeout.
365
+ - **Login walls and anti-bot pages are classified.** A playbook that lands on a
366
+ sign-in or verification URL reports `login_required` or `blocked` with the
367
+ page's real URL, instead of `degraded` with zero captures.
368
+ - **Garbled result titles are dropped.** A title made mostly of replacement or
369
+ control characters — the shape an anti-bot page leaves behind — no longer
370
+ reaches the caller.
371
+
372
+ ### Documentation
373
+
374
+ - Browser mode now documents what it costs: `--out` / `out:` to keep records on
375
+ disk, no screenshots by default, `maxScreens` at or below 5 and
376
+ `minInterval` at or above 3000 ms, `bili-danmaku` as experimental, and login
377
+ state through `WEB_FORAGER_BROWSER_CDP` or a persistent profile.
378
+ - HTTP mode is positioned where a search CLI is not: `web_scholar` and
379
+ `web_map`.
380
+ - Weibo and Douban are no longer listed as covered sites: they have no
381
+ site-specific code and are read as ordinary pages.
382
+ - The two modes are described as sharing four core fields, not as writing the
383
+ same record; a field-by-field table replaces the old claim.
384
+ - Baidu's absence, the parser assumptions behind each search engine and the
385
+ absence of site adapters for Zhihu and Xiaohongshu in HTTP mode are documented.
386
+ - Browser exit codes 8 and 9, `--strict-robots`, `--doctor` and automatic
387
+ bridge discovery are documented.
388
+
389
+ ### Tests
390
+
391
+ - **157 Python tests** and **60 browser tests**, all offline. The browser suite
392
+ grows from 35 to 60: capabilities, robots parsing and matching, the
393
+ strict-robots round trip, degraded and blocked paths, the core schema, the
394
+ real-page fixtures, the Bilibili JSON client and its page-card filter, the MCP
395
+ search switches and warnings, the thin-result fallback, and the doctor report. The suite
396
+ clears `WEB_FORAGER_BRIDGE`, so a machine that exports the variable still runs
397
+ the no-bridge paths green, and the Bilibili feed test no longer reaches the
398
+ network.
399
+
400
+ ### Notes
401
+
402
+ - Browser mode is read-only by default. Like, coin, favourite, follow, share,
403
+ comment, danmaku and publish are denied unless `--allow-write` is passed after
404
+ explicit authorisation.
405
+ - The bundled Playwright bridge still needs `playwright` and its Chromium
406
+ binary installed separately; `--doctor` names whichever is missing. With no
407
+ bridge at all a live browser run exits 3.
408
+ - Danmaku and `observe("danmaku", ms)` are experimental; canvas-rendered text
409
+ still needs OCR.
410
+
411
+ ## 1.0.1 — 2026-11-06
412
+
413
+ Patch release for the MCP server on Windows hosts.
414
+
415
+ ### Fixed
416
+
417
+ - **MCP stdio encoding**: some hosts (ZCode among them) launch `python` with the
418
+ ANSI code page as the stdio encoding. A response containing a Chinese title
419
+ then raised `UnicodeEncodeError`, closed the server and surfaced as
420
+ "Connection closed" on the tool call. The server now reconfigures stdin and
421
+ stdout to UTF-8 at startup and falls back to `ensure_ascii` escapes when a
422
+ stream cannot be reconfigured.
423
+
424
+ ### Tests
425
+
426
+ - Added `tests/test_mcp_stdio.py`, which reproduces the cp1252 sink and checks
427
+ both the reconfigure path and the ASCII fallback. Total: 117 tests.
428
+
429
+ ### Verified
430
+
431
+ - Bilibili `feed`, `search`, `video` and `related` were exercised against the
432
+ live site from the CLI, and `web_bilibili` through the MCP server inside
433
+ ZCode. Comments and danmaku text remain out of scope for web-forager.
434
+
435
+ ## 1.0.0 — 2026-11-06
436
+
437
+ First stable release. web-forager is a complete public-web intelligence layer
438
+ for agents and analysts, with a CLI, an MCP server and a bilingual skill contract.
439
+
440
+ ### Added
441
+
442
+ - **Scholarly APIs**: `--action scholar --source arxiv|crossref|pubmed|doi`
443
+ with normalised records (title, authors, venue, year, DOI, abstract, PDF URL).
444
+ - **Office documents**: DOCX, XLSX and EPUB extraction with the standard
445
+ library only; CSV becomes structured tables.
446
+ - **Feeds**: RSS 2.0, Atom and RSS 1.0 parsing into a Markdown digest.
447
+ - **Sitemap discovery**: `--action map` reads `Sitemap:` lines from robots.txt,
448
+ sitemap indexes and `.xml.gz` files.
449
+ - **MCP server**: `mcp_server.py` speaks newline-delimited JSON-RPC 2.0 and
450
+ exposes `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`,
451
+ `web_bilibili` and `web_clean`; `.mcp.json` wires it into Claude Code.
452
+ - **Search pagination**: `--search-pages N` reads several result pages per
453
+ engine (360, Bing, Sogou).
454
+ - **Context chunks**: `--chunk-size N` adds heading-aware `chunks` to fetch and
455
+ clean output, so an agent can read long pages section by section.
456
+ - **HTTP cache**: `--cache --cache-ttl N --cache-dir DIR` reuses polite GET
457
+ responses; `cached: true` is reported in the record.
458
+ - **Playwright controls**: `--wait-for SELECTOR` and `--scroll N` for lazy or
459
+ delayed content.
460
+ - **Security**: all XML goes through a DOCTYPE/ENTITY guard that uses
461
+ `defusedxml` when available; Office ZIP containers have entry-count and
462
+ expansion limits.
463
+ - **Multi-host adaptation**: `AGENTS.md` is the generic agent contract, with
464
+ matching rule files for Cursor, Windsurf, Cline, Continue, Roo Code and
465
+ GitHub Copilot. `install.py` writes project- or user-scoped rules in one run.
466
+ - **Packaging**: root `package.json`, `index.js` and `cordis.patch.yml` cover
467
+ npm and dsh; `commands/fetch.md`, `surf.md`, `scholar.md` and `map.md` expose
468
+ the actions as plugin commands.
469
+ - **Bilingual documentation**: `README.md` (English) and `README_zh.md`
470
+ (Chinese) mirror each other section for section, with the same three-method
471
+ installation guide, quick start, requirements, repository tree, test summary,
472
+ limitations, issue rules, trademark notice and licence. All documents are
473
+ emoji-free.
474
+ - **Tests**: 115 tests, including installer coverage and host-rule and
475
+ disclaimer checks.
476
+
477
+ ### Fixed
478
+
479
+ - **Trafilatura cross-document deduplication**: extraction previously ran with
480
+ `deduplicate=True`, whose process-global cache silently removed sentences
481
+ seen on an earlier page. Long agent sessions could lose a page's opening
482
+ paragraph. Deduplication is now disabled across calls.
483
+ - Sitemap `<loc>` elements were not mapped to the `url` field.
484
+ - RSS/XML payloads no longer trigger the HTML dynamic-rendering heuristic or
485
+ BeautifulSoup's XML warning.
486
+
487
+ ### Changed
488
+
489
+ - Version 1.0.0 across all platform manifests, `SKILL.md` and the CLI
490
+ User-Agent.
491
+ - `install.py --global` appends a marked block to `~/.claude/CLAUDE.md` instead
492
+ of replacing the user's global instructions.
493
+ - Documentation presents web-forager as an independent plugin; the `--root`
494
+ output is described as a four-zone workspace without third-party branding.
495
+
496
+ ## 0.2.0 — 2026-11-06
497
+
498
+ - 360/Bing/Sogou search with automatic merge and dedupe.
499
+ - Optional PDF extraction through `pypdf` (exit code 7 when absent).
500
+ - `--action surf` with ranked link BFS and resumable `--session`.
501
+ - `--agent` OpenCLI-style success/error envelope.
502
+
503
+ ## 0.1.0 — 2026-11-06
504
+
505
+ - Initial clean-room fetcher: robots.txt, polite rate limiting, HTML → Markdown,
506
+ metadata, tables, optional Playwright and a Bilibili adapter.
507
+ - Standalone packaging for Claude Code, Codex, CodeBuddy and DSH, plus
508
+ four-zone `--root` workspace integration.