web-forager 0.0.0-stage → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/.claude-plugin/marketplace.json +16 -0
  2. package/.claude-plugin/plugin.json +20 -0
  3. package/.clinerules/web-forager.md +85 -0
  4. package/.codebuddy-plugin/marketplace.json +14 -0
  5. package/.codebuddy-plugin/plugin.json +25 -0
  6. package/.codex-plugin/marketplace.json +12 -0
  7. package/.codex-plugin/mcp.json +20 -0
  8. package/.codex-plugin/plugin.json +15 -0
  9. package/.continue/rules/web-forager.md +90 -0
  10. package/.cursor/rules/web-forager.mdc +80 -0
  11. package/.dsh-plugin/index.js +11 -0
  12. package/.dsh-plugin/plugin.json +5 -0
  13. package/.github/copilot-instructions.md +85 -0
  14. package/.mcp.json +12 -0
  15. package/.roo/rules/web-forager.md +85 -0
  16. package/.windsurf/rules/web-forager.md +90 -0
  17. package/AGENTS.md +112 -0
  18. package/CHANGELOG.md +454 -0
  19. package/LICENSE +21 -0
  20. package/README.md +477 -3
  21. package/README_zh.md +477 -0
  22. package/SKILL.md +351 -0
  23. package/THIRD_PARTY_LICENSES.txt +35 -0
  24. package/browser/bridges/playwright-bridge.js +375 -0
  25. package/browser/launch_chrome.py +144 -0
  26. package/browser/lib/backend.js +302 -0
  27. package/browser/lib/bili-api.js +79 -0
  28. package/browser/lib/bridge.js +83 -0
  29. package/browser/lib/context.js +60 -0
  30. package/browser/lib/doctor.js +97 -0
  31. package/browser/lib/errors.js +17 -0
  32. package/browser/lib/netguard.js +149 -0
  33. package/browser/lib/playbooks.js +30 -0
  34. package/browser/lib/record.js +109 -0
  35. package/browser/lib/robots.js +195 -0
  36. package/browser/lib/safety.js +115 -0
  37. package/browser/lib/selectors.js +80 -0
  38. package/browser/lib/session.js +116 -0
  39. package/browser/lib/snapshot.js +229 -0
  40. package/browser/lib/untrusted.js +56 -0
  41. package/browser/lib/workspace.js +40 -0
  42. package/browser/mcp_server.js +272 -0
  43. package/browser/package.json +40 -0
  44. package/browser/playbooks/bili-chain.js +58 -0
  45. package/browser/playbooks/bili-danmaku.js +102 -0
  46. package/browser/playbooks/bili-feed.js +66 -0
  47. package/browser/playbooks/bili-video.js +38 -0
  48. package/browser/playbooks/search-dig.js +105 -0
  49. package/browser/playbooks/shared.js +138 -0
  50. package/browser/playbooks/wechat-dig.js +78 -0
  51. package/browser/playbooks/xhs-dig.js +89 -0
  52. package/browser/playbooks/zhihu-dig.js +89 -0
  53. package/browser/run.js +432 -0
  54. package/commands/browse.md +53 -0
  55. package/commands/fetch.md +29 -0
  56. package/commands/map.md +24 -0
  57. package/commands/scholar.md +28 -0
  58. package/commands/surf.md +28 -0
  59. package/cordis.patch.yml +19 -0
  60. package/docs/ACKNOWLEDGEMENTS.md +15 -0
  61. package/docs/ACKNOWLEDGEMENTS_zh.md +15 -0
  62. package/docs/DEPENDENCIES.md +40 -0
  63. package/docs/INSTALL_COMMANDS.md +113 -0
  64. package/docs/INSTALL_COMMANDS_zh.md +113 -0
  65. package/docs/REFERENCE.md +308 -0
  66. package/docs/REFERENCE_zh.md +309 -0
  67. package/index.js +7 -0
  68. package/install.py +254 -0
  69. package/manifest.json +97 -0
  70. package/mcp_server.py +597 -0
  71. package/package.json +80 -6
  72. package/requirements.txt +12 -0
  73. package/run.py +852 -0
  74. package/scripts/__init__.py +1 -0
  75. package/scripts/bilibili.py +874 -0
  76. package/scripts/documents.py +437 -0
  77. package/scripts/engine.py +4220 -0
  78. package/scripts/feeds.py +180 -0
  79. package/scripts/pdf.py +135 -0
  80. package/scripts/scholar.py +354 -0
  81. package/scripts/sitemap.py +180 -0
  82. package/scripts/untrusted.py +95 -0
package/CHANGELOG.md ADDED
@@ -0,0 +1,454 @@
1
+ # Changelog
2
+
3
+ ## 1.3.0 — 2026-11-15
4
+
5
+ A workflow release: a robots policy that can record an authorised override, search that fills its page budget with readable results, parallel engines, and a browser session you can log into once.
6
+
7
+ ### Added
8
+
9
+ - **`--robots-policy respect|warn|ignore`.** `respect` is the default and still refuses a disallowed path. `warn` reads it and writes `compliance.robots_overridden: true` plus a warning, for a user who is authorised to read the page. `ignore` is the old `--ignore-robots`, with `robots_checked: false`. Every HTTP record now carries a `compliance` block.
10
+ - **The browser profile launcher** (`browser/launch_chrome.py`) starts Chrome/Chromium/Edge with `~/.web-forager/browser/profile` and a DevTools port, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`.
11
+ - **Storage-state import and export** for browser mode: `--storage-state FILE` loads cookies/localStorage, `--save-storage-state FILE` writes them after the run. A saved file is forced to mode `0600`, a path inside `--root` is refused, and the record says the file carries login state.
12
+ - **`--serial-engines`** keeps a diagnostic run strictly serial; the default now asks engines on different hosts at the same time while each host keeps its one-request-per-second delay.
13
+ - **A `wechat-dig` browser playbook** opens one WeChat article URL in the browser profile and captures its text. When the profile has already passed WeChat's environment check the article is read; when it has not, the playbook reports `blocked` with a hint and does not try to solve the check.
14
+ - **Verification pages are reported instead of recycled.** HTTP mode recognises WeChat's environment check and the common English anti-bot interstitials, raises `VerificationRequired`, and `fetch --query` records the result as `verification_required` with a warning that names the browser route. The CLI exits 9. The bundled bridge also hands the whole `#js_content` article over as one `content` snapshot node, so WeChat paragraphs are not reassembled from leaf spans.
15
+ - **Automatic render fallback for script shells.** When a server answers with a minimal HTML page that carries only a script or redirect, `--render auto` now renders it once with Playwright before extraction. The record says `render_fallback: true`; `fetch --query` allows at most two such renders per query, so one slow site cannot stall a search. This needs no API key and closes the php.cn-style empty-page case.
16
+ - **Free ranking and query repair.** Search results are ordered by query coverage, title match, a small domain prior (docs, papers, government and universities up; document farms down) and snippet freshness. The final list keeps at most two results per host and fills the rest from the overflow. A thin first pass is retried once with a simplified query — quotes and common stopwords removed, or a spaced CJK query joined — and the retry is marked `query_variant` in `engine_report`. No model, key or network service is involved.
17
+
18
+ ### Changed
19
+
20
+ - **`max_pages` now means readable pages.** `fetch --query` keeps scanning past a robots-denied or unreadable result (up to a bounded number of attempts), returns the pages it could read, and lists the rest in `skipped` with a code and reason. The response carries a summary warning instead of leaving the agent to discover failures page by page.
21
+ - The `robots_denied` hint now names `--robots-policy warn` for an authorised override and says not to silently switch to a tool that ignores robots. A `verification_required` skip says the same about not solving the check, and points a WeChat link at the `wechat-dig` playbook.
22
+
23
+ ### Host compatibility
24
+
25
+ - **Codex now gets a working MCP launch.** The plugin ships `.codex-plugin/mcp.json`, which uses relative `mcp_server.py` and `browser/mcp_server.js` paths with `cwd: "."`; Codex resolves both against the installed plugin directory. It no longer inherits Claude Code's `${CLAUDE_PLUGIN_ROOT}` argument, which Codex leaves literal.
26
+ - **CodeBuddy now declares its MCP servers in the manifest it reads.** `.codebuddy-plugin/plugin.json` carries both `mcpServers` entries with `${CLAUDE_PLUGIN_ROOT}`, so a new CodeBuddy session mounts the HTTP and browser servers. `codebuddy mcp list` can run before plugins are mounted; an empty list there is not a failed install.
27
+ - **dsh now mounts real tools.** `cordis.patch.yml` inserts two `@deepseek-ai/dsh-mcp-client` entries, one per server, and resolves the profile package directory with `dshHomePath(...)`. The previous patch mounted only the no-op `web-forager` module.
28
+ - **The npm package carries every host manifest.** `files` now includes `.claude-plugin/`, `.codex-plugin/`, `.codebuddy-plugin/`, `.dsh-plugin/`, `.mcp.json`, the rule-file directories, `install.py` and `requirements.txt`.
29
+ - The Cursor rule heading now says v1.3.0 instead of v1.0.0.
30
+
31
+ ### Documentation
32
+
33
+ - `SKILL.md` and `AGENTS.md` state the rule: a `robots_denied` or `login_required` result is a boundary, not a prompt to re-fetch with a tool that ignores robots.
34
+ - The reference documents the launcher, storage-state switches, the robots policy, the new `compliance` block, the `wechat-dig` playbook and exit 9.
35
+ - Both READMEs were rewritten around a first visit: one pitch, the three install methods, what a run looks like, the security boundary and the legal notice, plus a section on keeping an installed copy current (including the ZCode marketplace-snapshot trap). A DeepSec Shield scan of the release tree reports 0 findings; its agent audit still flags the three signature strings this project defines, which are the scanner's own patterns, not page content.
36
+
37
+ ### Tests
38
+
39
+ - **258 Python tests** and **87 browser tests**, all offline. New coverage: the warn policy and its record, clean compliance fields, filling pages past a denial, overlapping engines versus `--serial-engines`, the launcher dry run, saving storage state through the bridge protocol, WeChat verification detection at the engine, the CLI and the search layer, the `wechat-dig` capture and `blocked` paths, the `content` snapshot node, domain/freshness tie-breaks, host diversity, query variants, the shell-render fallback with and without its budget, plus the Codex/CodeBuddy/dsh MCP manifests and the npm host-file list.
40
+
41
+ ## 1.2.3 — 2026-11-15
42
+
43
+ A follow-up hardening release after a DeepSec Shield audit of 1.2.2.
44
+
45
+ ### Security
46
+
47
+ - **The browser bridge is no longer started through a shell.** `--bridge` and `WEB_FORAGER_BRIDGE` are split into an executable and its arguments and run directly, so a path or environment value cannot append a second command. Quoted paths with spaces still work; pipes, `&&` and redirection are not interpreted.
48
+ - **CI actions are pinned to commit SHAs.** `actions/checkout`, `actions/setup-python` and `actions/setup-node` name the exact commit, so a moved tag cannot change what the workflow runs.
49
+
50
+ ### Documentation
51
+
52
+ - The reference states that the bridge command is split on whitespace and started without a shell.
53
+
54
+ ### Tests
55
+
56
+ - **228 Python tests** and **81 browser tests**, all offline. New coverage: quoted bridge paths, literal backslashes, an unterminated quote, and a shell metacharacter that must be passed through as an argument rather than executed.
57
+
58
+ ## 1.2.2 — 2026-11-15
59
+
60
+ A hardening release that replaces the 1.2.2 build published on 2026-11-12: a request can no longer be aimed at the machine or the LAN, browser state stays in a profile of its own, and fetched text is labelled as third-party data. The legal-notice rewrite and packaging rules from that earlier build are included.
61
+
62
+ ### Security
63
+
64
+ - **Private-address guard (SSRF).** Loopback, RFC 1918, link-local (including `169.254.169.254`), CGNAT, multicast, reserved and their IPv6 forms are refused before a socket opens. The host is resolved and checked as written, so decimal IPv4 spellings and IPv6-mapped IPv4 are caught too. The check runs before the request, on every redirect hop (redirects are followed one hop at a time instead of by `requests`), on the robots.txt download, on search-result resolution, and on the URL a rendered page actually lands on. `--allow-private` / `WEB_FORAGER_ALLOW_PRIVATE=1` is the documented opt-out for local fixtures and trusted intranet targets.
65
+ - **Browser navigation is guarded too.** `guardedGoto` checks the requested and the final URL, and the bundled Playwright bridge installs a request filter: every document and subresource the page asks for is resolved, and private targets are aborted. A hostile page can no longer use the open browser to probe `http://127.0.0.1:…` or a cloud metadata endpoint.
66
+ - **Browser state stays in its own profile.** The bundled bridge already kept `~/.web-forager/browser/profile`; the report now states `dedicated_profile` and `shared_browser`, and a CDP attach to a browser you use for everything else adds a warning that every cookie and login in that profile is reachable by the pages a playbook opens.
67
+ - **Fetched text is marked as data.** Every record carries `trust: "untrusted"`. A small scanner (same rules in `scripts/untrusted.py` and `browser/lib/untrusted.js`) names instruction-shaped passages — overrides, fake system tags, `curl … | sh`, secret-exfiltration requests, role swaps — in `injection_flags`, and puts one warning first in the list. `--input-gate strict` withholds the body instead of handing it on; `--input-gate off` skips the scan.
68
+ - **Cookies never enter a record.** The Bilibili cookie is held in memory for the API request and is not written to the record, the session file or an error message; the browser profile directory is reported as a flag, not a path.
69
+
70
+ ### Documentation
71
+
72
+ - **Both READMEs were reorganised around a first visit.** The page now opens with what the tool is, what you get and what it refuses, followed by a table of contents, the why section, the three install methods and a quick start split by entry point — chat, terminal, MCP and the optional browser mode — with a "which action do I need" table and copy-paste commands. The first screen carries a one-line notice that links to the full legal terms; the terms themselves moved below the quick start, section by section.
73
+ - **Quick start now holds only the three entry points**: chat, terminal and MCP. The browser playbook table, switches and `--strict-robots` details, and "when the engines are blocked" routes moved to the reference; each is a one-sentence pointer on the landing page. Installation opens with "let your AI install it" as the recommended path for most readers, followed by the plugin-marketplace URL and the per-host commands. The contents is grouped, with acknowledgements, trademarks and licence added.
74
+ - **Reference material moved to `docs/REFERENCE.md` and `docs/REFERENCE_zh.md`**: the full action list and exit-code tables, record fields and `core-1`, workspace output, MCP tool names and protocol checks, requirements, repository layout, test details and extra CLI examples. Every example is its own one-line block, so it can be copied or run without trimming its neighbours. README keeps a three-table quick start, a short exit-code summary and a link.
75
+ - **Long sections fold.** The repository tree, test coverage notes and MCP tool names sit in `<details>` blocks; known limitations are grouped into read-first, edge cases and experimental/high-risk switches.
76
+ - Both READMEs gain a **Security boundaries** section and three limitation bullets (the guard's off switch, the screen's limits, the CDP profile exposure).
77
+ - `AGENTS.md`, `SKILL.md` and the six host rule files state that fetched text is data and must not be executed as instructions.
78
+ - **The README pair carries a six-part legal notice** — purpose, prohibited use, responsibility, no warranty, limitation of liability, redistribution. `SKILL.md`, `AGENTS.md` and the six host rule files carry the short version.
79
+ - **Test fixtures are no longer part of a published archive.** `.gitattributes` marks `tests/` and `browser/tests/` as `export-ignore`, and both npm manifests drop tests from `files`; CI still runs them from the repository clone.
80
+
81
+ ### Tests
82
+
83
+ - **228 Python tests** and **75 browser tests**, all offline. New coverage: every private range, a redirect into a metadata endpoint, a two-hop redirect chain, the robots download, the CLI exit code and `--allow-private`, marker detection, strict withholding, and the browser-side guard and scanner.
84
+
85
+ ## 1.2.1 — 2026-11-11
86
+
87
+ A search hygiene release: a Chinese query no longer collects dictionary pages for one of its characters, and the README pair was rewritten around what the tool does now.
88
+
89
+ ### Fixed
90
+
91
+ - **A mostly-CJK query is no longer handed to Bing as a fallback.** Bing splits a Chinese phrase into single characters, so a search about 反逆向工程 came back with the dictionary page of 反 once the Chinese engines answered thinly. `auto` now stops after 360, Sogou and 搜狗微信; `--engine bing` still asks Bing explicitly.
92
+ - **Dictionary pages that match one character of the query are dropped**, whatever engine returned them. `looks_like_character_page()` recognises a title that shares exactly one character with a longer query and carries a dictionary marker (百科, 拼音, 笔顺, 部首, 组词…), so a result list is not padded with them and `engine_report` counts only usable hits.
93
+
94
+ ### Documentation
95
+
96
+ - **Both READMEs were rewritten.** The duplicated mode paragraphs and the second disclaimer block are gone; scope and compliance sit in one section, the two modes face each other in one table, and the CJK search routing, `find`/`feed`, CI and current tool/test counts are stated once. README text now matches the 10 HTTP tools, 4 browser tools and 9 CLI actions that actually ship.
97
+
98
+ ### Tests
99
+
100
+ - **174 Python tests** and **60 browser tests**, all offline. New coverage: a CJK query never calls Bing, a character page is dropped from an engine's results, and `looks_like_character_page()` leaves a single-character query and a full-phrase hit alone.
101
+
102
+ ## 1.2.0 — 2026-11-09
103
+
104
+ Search is no longer the only way in. A site's own search page and its feed are
105
+ first-class routes now, every engine attempt is reported, and one run can be
106
+ sent through a proxy.
107
+
108
+ ### Added
109
+
110
+ - **`--action find` (`web_find`)**: search inside one site through the site's
111
+ own search page. It tries `/?s=`, `/search?q=`, `/?search=` and
112
+ `/search?query=`, ranks the page's links by query coverage, and fetches the
113
+ best ones. This is the route that keeps working when the general engines
114
+ answer with dictionary pages or a verification wall.
115
+ - **`--action feed` (`web_feed`)**: read a site's RSS/Atom feed, discovering it
116
+ from `<link rel=alternate>` when given the site URL instead of the feed URL.
117
+ - **`engine_report`**: every search response now carries one entry per engine
118
+ tried — `ok`, `results`, `ms`, and `error` when it failed, plus `fallback`
119
+ when the engine was asked only because the first answer was thin. "Search is
120
+ broken" becomes "Bing answered, Sogou showed a verification page".
121
+ - **`doctor --probe` (`web_doctor {probe: true}`)**: asks 360, Bing, Sogou, the
122
+ WeChat index and DuckDuckGo once each and reports which ones answer from this
123
+ network, with latency.
124
+ - **`--proxy URL`** and `proxy` on the MCP tools. Requests already honoured
125
+ `HTTPS_PROXY` / `ALL_PROXY`; an explicit proxy now overrides them for one run,
126
+ and `doctor` reports the proxy in use with credentials redacted.
127
+
128
+ ### Changed
129
+
130
+ - The MCP server exposes ten HTTP tools instead of eight; the README and
131
+ `SKILL.md` document the two routes that do not depend on a search engine.
132
+ - `doctor` reports the proxy source (`config` or `env:NAME`) without printing a
133
+ password.
134
+
135
+ ### Documentation
136
+
137
+ - **The disclaimer forbids building a profile of an identifiable person.**
138
+ Collecting, aggregating or inferring personal information, or using it for
139
+ doxxing, harassment or stalking, is prohibited and is the user's own legal
140
+ responsibility. It appears in `--help`, `manifest.json`, the README pair,
141
+ `SKILL.md` and every host rule file.
142
+ - **The per-host install commands moved to `docs/INSTALL_COMMANDS.md` and
143
+ `docs/INSTALL_COMMANDS_zh.md`.** Methods 1 and 3 stay in the README because
144
+ they are three lines each; Method 2, which is a page of commands, is one link.
145
+ - **The READMEs gained an Acknowledgements section** listing the reporter of
146
+ issue #1 by name, with a link to `docs/ACKNOWLEDGEMENTS.md` /
147
+ `docs/ACKNOWLEDGEMENTS_zh.md` for the details.
148
+
149
+ ### Acknowledgements
150
+
151
+ - **NotFatJustRound** ([issue #1](https://github.com/JiamingShi-JsPhys/web-forager/issues/1))
152
+ reported that `web_search` returned every page body twice — once in `pages`,
153
+ once in `results` — which doubled the response and pushed the tail past an
154
+ agent's size limit; the compact `results` index in 1.0.2 came from that report.
155
+
156
+ ### Fixed
157
+
158
+ - **Links inside `<article><header>` are no longer skipped.** A results page
159
+ commonly wraps each card's title in an article header; the universal
160
+ "skip `<header>`" rule silently dropped exactly the links a search page
161
+ offers. Page-level headers and navigations are still skipped.
162
+ - `doctor --probe` makes one attempt per engine instead of retrying, so a
163
+ blocking engine costs one timeout rather than three.
164
+ - `.cursor/rules/web-forager.mdc` carried `v1.0.0` in its front matter, so
165
+ `install.py --only cursor` wrote a stale label; it now follows the release
166
+ version.
167
+
168
+ ### Tests
169
+
170
+ - **172 Python tests** and **60 browser tests**, all offline. The new suite
171
+ covers site search against a local WordPress-style page, feed discovery from a
172
+ page link and from a direct feed, the per-engine report, the probe, proxy
173
+ wiring and the redaction of proxy credentials.
174
+
175
+ ## 1.1.0 — 2026-11-08
176
+
177
+ Browser mode joins web-forager, and an audit pass follows it. The HTTP toolkit
178
+ and the browser layer ship as **one plugin, one name and one version**; the
179
+ former standalone `web-actor` product no longer exists. The empty-result path,
180
+ the bridge's capability claims, the robots.txt audit trail and the shared record
181
+ shape are now honest, and neither README claims coverage the code does not have.
182
+
183
+ ### Added
184
+
185
+ - **Browser mode** under `browser/`: a CLI, seven read-only playbooks and an MCP
186
+ server that drive a real browser through a pluggable bridge.
187
+ - **Playbooks**: `bili-feed`, `bili-video`, `bili-chain`, `bili-danmaku`,
188
+ `zhihu-dig`, `xhs-dig` and `search-dig`.
189
+ - **Second MCP server**: `browser/mcp_server.js` exposes `browser_list`,
190
+ `browser_run`, `browser_session` and `browser_doctor`; `.mcp.json`
191
+ registers both servers.
192
+ - **A reference bridge that is found automatically**:
193
+ `browser/bridges/playwright-bridge.js` supports a fresh browser, a persistent
194
+ profile, a headful window and attaching over CDP. `browser/lib/bridge.js`
195
+ looks for `--bridge`, then `WEB_FORAGER_BRIDGE`, then that file, so a first
196
+ run no longer fails with `backend_unavailable` only because nobody exported a
197
+ variable.
198
+ - **A browser profile that persists by default**:
199
+ `~/.web-forager/browser/profile` keeps cookies and logins between runs;
200
+ `WEB_FORAGER_BROWSER_PROFILE=off` restores the throwaway browser. The
201
+ reference bridge also drops the automation banner flag.
202
+ - **Browser records**: the same `metadata / markdown / tables / warnings` shape,
203
+ written to `<WS>/process/web-forager/browser/` and `_broadcast/`.
204
+ - **`/browse` command** for the browser playbooks.
205
+ - **Shared record core (`core-1`)**: `--core` on both CLIs and `core: true` on
206
+ `browser_run` return `schema_version / source / mode / url / title /
207
+ collected_at / metadata / markdown / tables / warnings / extra`, so a mixed
208
+ list of HTTP and browser records can be read uniformly. The same fold is
209
+ available as `engine.normalize_record()` and `normalizeRecord()`.
210
+ - **Browser robots auditing**: `--strict-robots` reads `robots.txt` before every
211
+ navigation and stops with `robots_denied` (exit 3) on a disallowed path.
212
+ Without it the record still states `compliance.robots_checked: false`, so the
213
+ compliance status is auditable either way. `browser/lib/robots.js` implements
214
+ the same RFC 9309 subset as the Python engine.
215
+ - **`degraded` (exit 8) and `blocked` (exit 9)**: a browser run that reads a
216
+ page but finds nothing the playbook looks for now reports `degraded` with its
217
+ records attached, instead of `ok: true` with `captured: 0`. An anti-bot or
218
+ error interstitial reports `blocked`. Both are distinct from `login_required`.
219
+ - **`describe` handshake**: a bridge can declare its capabilities; a bridge that
220
+ does not gets `declared: false` with every capability false. The reference
221
+ Playwright bridge and the test fixtures implement it.
222
+ - **`browser/lib/selectors.js`**: every structural assumption about a
223
+ third-party page lives in one file.
224
+ - **Real-page fixtures**: `browser/tests/fixtures/snapshots/` holds snapshots
225
+ built from live Bilibili pages on 2026-11-07 by
226
+ `browser/tests/tools/html-to-snapshot.py`, with `tests/real-pages.test.js`
227
+ as the offline regression and `npm run test:live` as the opt-in live check.
228
+ - **`--action doctor`** and the **`web_doctor` MCP tool** report every
229
+ dependency with its version or its install command, so an agent that receives
230
+ a thin page can install what is missing instead of guessing.
231
+ - **`node browser/run.js --doctor`** and the **`browser_doctor` MCP tool** do the
232
+ same for the browser stack: Node version, the bridge command in use, whether
233
+ the `playwright` package resolves (searched from the bridge's own directory
234
+ first), whether its Chromium binary is downloaded, and where the profile is.
235
+ - **The Bilibili search API is asked before the result page is read.**
236
+ `lib/bili-api.js` calls Bilibili's public JSON search endpoint (the `buvid3`
237
+ cookie comes from one home-page request) and falls back to scraping the page
238
+ only when the endpoint refuses. `source` in the record says which path ran.
239
+ `WEB_FORAGER_BILI_API=off` keeps the playbook on the page.
240
+ - **`respect_robots` on the MCP tools.** `web_fetch`, `web_search` and
241
+ `web_surf` accept the same switch the CLI spells `--ignore-robots`, so a page
242
+ that robots.txt disallows can be read when the caller is authorised to read
243
+ it. The default is still to check.
244
+ - **Bilibili's JSON search on the direct path.** When robots checks are off
245
+ (`--direct-search` or a global override), `scripts/bilibili.py` asks
246
+ `api.bilibili.com` first — the `buvid3` cookie comes from one home-page
247
+ request, and that request is never answered from the cache because a cached
248
+ anti-bot page supplies no cookie — and scrapes the search page only if the
249
+ API answers `-412` or malformed JSON. Cards read from the page are kept only
250
+ when their text mentions the query, so a recommendation rail can no longer
251
+ pass as search results. `route` in the payload names the host that answered.
252
+ - **The WeChat article index as a search engine (`--engine weixin`).**
253
+ `weixin.sogou.com` indexes article text, which is the one channel that
254
+ normally carries a Chinese person's or institution's name; each result keeps
255
+ its account name and publish date. It joins `auto` for mostly-CJK queries and
256
+ `--engine all`.
257
+
258
+ ### Changed
259
+
260
+ - Version 1.1.0 across manifests, `SKILL.md`, CLI and README install tags.
261
+ - The browser layer uses the web-forager name throughout: environment variables
262
+ are `WEB_FORAGER_BRIDGE`, `WEB_FORAGER_BROWSER_HOME`,
263
+ `WEB_FORAGER_BROWSER_HEADFUL`, `WEB_FORAGER_BROWSER_PROFILE` and
264
+ `WEB_FORAGER_BROWSER_CDP`; sessions live in `~/.web-forager/browser/`; record
265
+ `source` is `web-forager-browser`.
266
+ - The plugin manifest gained a `browser` block listing playbooks, backends and
267
+ the browser MCP tools.
268
+ - The User-Agent now carries the real version and the repository URL instead of
269
+ a stale `WebForager/1.0` and a placeholder domain.
270
+ - `bili-danmaku` asks the bridge to observe first and falls back to DOM text
271
+ second; a DOM-derived list is recorded with a warning that it is partial.
272
+ - Duplicate card links now merge, so a captured card keeps its readable title
273
+ instead of the "views danmaku duration" overlay text.
274
+ - The MCP server exposes eight HTTP tools instead of seven.
275
+ - **`auto` widens a thin answer once.** A Latin query that returns fewer than
276
+ three distinct results from 360 and Bing now asks Sogou as well; a CJK query
277
+ whose engines all fail keeps its Bing fallback. A full answer is not widened,
278
+ so the common case costs nothing extra.
279
+ - **`--max-pages 0` returns a search answer without page bodies.** The CLI used
280
+ to floor the value at 1; both the CLI and `web_search` now let a caller ask
281
+ for the result list alone and fetch only what it needs with `--max-chars`.
282
+ - **The MCP search tool carries the same warnings as the CLI.** A response whose
283
+ results do not mention the query text — or that is empty — now says so in
284
+ `warnings` instead of leaving the agent to present dictionary pages as
285
+ findings.
286
+
287
+ ### Fixed
288
+
289
+ - A browser run that found nothing no longer reports success.
290
+ - A bridge without `observe` is no longer reported as observing.
291
+ - The Bilibili feed enrichment test no longer reaches the network, so the suite
292
+ is genuinely offline.
293
+ - **`results` no longer repeats every page body.** A search response carried
294
+ each Markdown page twice — once in `pages`, once in `results` — so a caller
295
+ with an output cap received a truncated tail. `results` is now a compact
296
+ index (`url`, `title`, `status`, `content_type`, `markdown_chars`, `tables`,
297
+ `snippet`, `warnings`, `error`); the body stays in `pages` and, for a single
298
+ fetch, at the top level. The old shape is one flag away: `--full-results` on
299
+ the CLI, `fullResults` on the MCP tools.
300
+ - **`auto` no longer mixes Bing into Chinese queries.** Bing treats a quoted
301
+ Chinese phrase as single characters, so dictionary and encyclopaedia pages
302
+ outranked the precise hits. A mostly-CJK query now asks 360, Sogou and the
303
+ WeChat index, and falls back to Bing only if all three are unreachable. Latin
304
+ queries are unchanged.
305
+ - **Navigation waits for the page it lands on.** `goto` now waits for the load
306
+ state and a short settle before returning, so an `evaluate` issued right after
307
+ a redirect no longer races it; `snapshot` retries once on the "execution
308
+ context was destroyed" error that a redirect leaves behind.
309
+ - **The bridge's stderr is kept.** A dead bridge now reports its last few
310
+ stderr lines in the error instead of a bare timeout.
311
+ - **Login walls and anti-bot pages are classified.** A playbook that lands on a
312
+ sign-in or verification URL reports `login_required` or `blocked` with the
313
+ page's real URL, instead of `degraded` with zero captures.
314
+ - **Garbled result titles are dropped.** A title made mostly of replacement or
315
+ control characters — the shape an anti-bot page leaves behind — no longer
316
+ reaches the caller.
317
+
318
+ ### Documentation
319
+
320
+ - Browser mode now documents what it costs: `--out` / `out:` to keep records on
321
+ disk, no screenshots by default, `maxScreens` at or below 5 and
322
+ `minInterval` at or above 3000 ms, `bili-danmaku` as experimental, and login
323
+ state through `WEB_FORAGER_BROWSER_CDP` or a persistent profile.
324
+ - HTTP mode is positioned where a search CLI is not: `web_scholar` and
325
+ `web_map`.
326
+ - Weibo and Douban are no longer listed as covered sites: they have no
327
+ site-specific code and are read as ordinary pages.
328
+ - The two modes are described as sharing four core fields, not as writing the
329
+ same record; a field-by-field table replaces the old claim.
330
+ - Baidu's absence, the parser assumptions behind each search engine and the
331
+ absence of site adapters for Zhihu and Xiaohongshu in HTTP mode are documented.
332
+ - Browser exit codes 8 and 9, `--strict-robots`, `--doctor` and automatic
333
+ bridge discovery are documented.
334
+
335
+ ### Tests
336
+
337
+ - **157 Python tests** and **60 browser tests**, all offline. The browser suite
338
+ grows from 35 to 60: capabilities, robots parsing and matching, the
339
+ strict-robots round trip, degraded and blocked paths, the core schema, the
340
+ real-page fixtures, the Bilibili JSON client and its page-card filter, the MCP
341
+ search switches and warnings, the thin-result fallback, and the doctor report. The suite
342
+ clears `WEB_FORAGER_BRIDGE`, so a machine that exports the variable still runs
343
+ the no-bridge paths green, and the Bilibili feed test no longer reaches the
344
+ network.
345
+
346
+ ### Notes
347
+
348
+ - Browser mode is read-only by default. Like, coin, favourite, follow, share,
349
+ comment, danmaku and publish are denied unless `--allow-write` is passed after
350
+ explicit authorisation.
351
+ - The bundled Playwright bridge still needs `playwright` and its Chromium
352
+ binary installed separately; `--doctor` names whichever is missing. With no
353
+ bridge at all a live browser run exits 3.
354
+ - Danmaku and `observe("danmaku", ms)` are experimental; canvas-rendered text
355
+ still needs OCR.
356
+
357
+ ## 1.0.1 — 2026-11-06
358
+
359
+ Patch release for the MCP server on Windows hosts.
360
+
361
+ ### Fixed
362
+
363
+ - **MCP stdio encoding**: some hosts (ZCode among them) launch `python` with the
364
+ ANSI code page as the stdio encoding. A response containing a Chinese title
365
+ then raised `UnicodeEncodeError`, closed the server and surfaced as
366
+ "Connection closed" on the tool call. The server now reconfigures stdin and
367
+ stdout to UTF-8 at startup and falls back to `ensure_ascii` escapes when a
368
+ stream cannot be reconfigured.
369
+
370
+ ### Tests
371
+
372
+ - Added `tests/test_mcp_stdio.py`, which reproduces the cp1252 sink and checks
373
+ both the reconfigure path and the ASCII fallback. Total: 117 tests.
374
+
375
+ ### Verified
376
+
377
+ - Bilibili `feed`, `search`, `video` and `related` were exercised against the
378
+ live site from the CLI, and `web_bilibili` through the MCP server inside
379
+ ZCode. Comments and danmaku text remain out of scope for web-forager.
380
+
381
+ ## 1.0.0 — 2026-11-06
382
+
383
+ First stable release. web-forager is a complete public-web intelligence layer
384
+ for agents and analysts, with a CLI, an MCP server and a bilingual skill contract.
385
+
386
+ ### Added
387
+
388
+ - **Scholarly APIs**: `--action scholar --source arxiv|crossref|pubmed|doi`
389
+ with normalised records (title, authors, venue, year, DOI, abstract, PDF URL).
390
+ - **Office documents**: DOCX, XLSX and EPUB extraction with the standard
391
+ library only; CSV becomes structured tables.
392
+ - **Feeds**: RSS 2.0, Atom and RSS 1.0 parsing into a Markdown digest.
393
+ - **Sitemap discovery**: `--action map` reads `Sitemap:` lines from robots.txt,
394
+ sitemap indexes and `.xml.gz` files.
395
+ - **MCP server**: `mcp_server.py` speaks newline-delimited JSON-RPC 2.0 and
396
+ exposes `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`,
397
+ `web_bilibili` and `web_clean`; `.mcp.json` wires it into Claude Code.
398
+ - **Search pagination**: `--search-pages N` reads several result pages per
399
+ engine (360, Bing, Sogou).
400
+ - **Context chunks**: `--chunk-size N` adds heading-aware `chunks` to fetch and
401
+ clean output, so an agent can read long pages section by section.
402
+ - **HTTP cache**: `--cache --cache-ttl N --cache-dir DIR` reuses polite GET
403
+ responses; `cached: true` is reported in the record.
404
+ - **Playwright controls**: `--wait-for SELECTOR` and `--scroll N` for lazy or
405
+ delayed content.
406
+ - **Security**: all XML goes through a DOCTYPE/ENTITY guard that uses
407
+ `defusedxml` when available; Office ZIP containers have entry-count and
408
+ expansion limits.
409
+ - **Multi-host adaptation**: `AGENTS.md` is the generic agent contract, with
410
+ matching rule files for Cursor, Windsurf, Cline, Continue, Roo Code and
411
+ GitHub Copilot. `install.py` writes project- or user-scoped rules in one run.
412
+ - **Packaging**: root `package.json`, `index.js` and `cordis.patch.yml` cover
413
+ npm and dsh; `commands/fetch.md`, `surf.md`, `scholar.md` and `map.md` expose
414
+ the actions as plugin commands.
415
+ - **Bilingual documentation**: `README.md` (English) and `README_zh.md`
416
+ (Chinese) mirror each other section for section, with the same three-method
417
+ installation guide, quick start, requirements, repository tree, test summary,
418
+ limitations, issue rules, trademark notice and licence. All documents are
419
+ emoji-free.
420
+ - **Tests**: 115 tests, including installer coverage and host-rule and
421
+ disclaimer checks.
422
+
423
+ ### Fixed
424
+
425
+ - **Trafilatura cross-document deduplication**: extraction previously ran with
426
+ `deduplicate=True`, whose process-global cache silently removed sentences
427
+ seen on an earlier page. Long agent sessions could lose a page's opening
428
+ paragraph. Deduplication is now disabled across calls.
429
+ - Sitemap `<loc>` elements were not mapped to the `url` field.
430
+ - RSS/XML payloads no longer trigger the HTML dynamic-rendering heuristic or
431
+ BeautifulSoup's XML warning.
432
+
433
+ ### Changed
434
+
435
+ - Version 1.0.0 across all platform manifests, `SKILL.md` and the CLI
436
+ User-Agent.
437
+ - `install.py --global` appends a marked block to `~/.claude/CLAUDE.md` instead
438
+ of replacing the user's global instructions.
439
+ - Documentation presents web-forager as an independent plugin; the `--root`
440
+ output is described as a four-zone workspace without third-party branding.
441
+
442
+ ## 0.2.0 — 2026-11-06
443
+
444
+ - 360/Bing/Sogou search with automatic merge and dedupe.
445
+ - Optional PDF extraction through `pypdf` (exit code 7 when absent).
446
+ - `--action surf` with ranked link BFS and resumable `--session`.
447
+ - `--agent` OpenCLI-style success/error envelope.
448
+
449
+ ## 0.1.0 — 2026-11-06
450
+
451
+ - Initial clean-room fetcher: robots.txt, polite rate limiting, HTML → Markdown,
452
+ metadata, tables, optional Playwright and a Bilibili adapter.
453
+ - Standalone packaging for Claude Code, Codex, CodeBuddy and DSH, plus
454
+ four-zone `--root` workspace integration.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Jiaming Shi
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.