web-forager 1.3.0 → 1.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/.claude-plugin/marketplace.json +1 -1
  2. package/.claude-plugin/plugin.json +1 -1
  3. package/.clinerules/web-forager.md +3 -3
  4. package/.codebuddy-plugin/marketplace.json +1 -1
  5. package/.codebuddy-plugin/plugin.json +1 -1
  6. package/.codex-plugin/marketplace.json +1 -1
  7. package/.codex-plugin/mcp.json +20 -20
  8. package/.codex-plugin/plugin.json +1 -1
  9. package/.continue/rules/web-forager.md +4 -4
  10. package/.cursor/rules/web-forager.mdc +4 -4
  11. package/.dsh-plugin/index.js +1 -1
  12. package/.dsh-plugin/plugin.json +1 -1
  13. package/.github/copilot-instructions.md +3 -3
  14. package/.roo/rules/web-forager.md +3 -3
  15. package/.windsurf/rules/web-forager.md +4 -4
  16. package/AGENTS.md +10 -3
  17. package/CHANGELOG.md +508 -454
  18. package/README.md +499 -477
  19. package/README_zh.md +498 -477
  20. package/SKILL.md +115 -15
  21. package/THIRD_PARTY_LICENSES.txt +5 -0
  22. package/browser/bridges/playwright-bridge.js +8 -1
  23. package/browser/launch_chrome.py +193 -144
  24. package/browser/lib/bili-api.js +12 -1
  25. package/browser/lib/safety.js +27 -1
  26. package/browser/lib/session.js +16 -2
  27. package/browser/lib/untrusted.js +215 -56
  28. package/browser/mcp_server.js +5 -25
  29. package/browser/package.json +1 -1
  30. package/browser/playbooks/search-dig.js +3 -1
  31. package/browser/playbooks/wechat-dig.js +78 -78
  32. package/browser/run.js +19 -10
  33. package/commands/research.md +35 -0
  34. package/cordis.patch.yml +19 -19
  35. package/docs/DEPENDENCIES.md +1 -1
  36. package/docs/INSTALL_COMMANDS.md +4 -4
  37. package/docs/INSTALL_COMMANDS_zh.md +4 -4
  38. package/docs/REFERENCE.md +349 -308
  39. package/docs/REFERENCE_zh.md +342 -309
  40. package/manifest.json +3 -2
  41. package/mcp_server.py +383 -130
  42. package/package.json +4 -2
  43. package/requirements.txt +7 -4
  44. package/run.py +61 -5
  45. package/scripts/bilibili.py +11 -3
  46. package/scripts/engine.py +1039 -118
  47. package/scripts/pdf.py +21 -0
  48. package/scripts/research.py +2087 -0
  49. package/scripts/scholar.py +33 -6
  50. package/scripts/sitemap.py +20 -5
  51. package/scripts/untrusted.py +312 -95
package/docs/REFERENCE.md CHANGED
@@ -1,308 +1,349 @@
1
- # web-forager reference
2
-
3
- > Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
4
-
5
- **English** · [简体中文](REFERENCE_zh.md)
6
-
7
- ---
8
-
9
- ## MCP servers
10
-
11
- <details>
12
- <summary>Tool names</summary>
13
-
14
- HTTP tools: `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_clean`, `web_doctor`, `web_find`, `web_feed`. Browser tools: `browser_list`, `browser_run`, `browser_session`, `browser_doctor`.
15
-
16
- </details>
17
-
18
- Test the protocol directly:
19
-
20
- ```bash
21
- printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
22
- '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
23
- ```
24
-
25
- ```bash
26
- printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
27
- '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
28
- ```
29
-
30
- ## Browser mode
31
-
32
- The browser layer lives in `browser/` and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless `--allow-write` is passed after the user authorises that single action.
33
-
34
- ```bash
35
- npm install playwright && npx playwright install chromium
36
- node browser/run.js --doctor
37
- node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
38
- node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
39
- node browser/run.js --playbook wechat-dig --url "https://mp.weixin.qq.com/s/..." --agent
40
- ```
41
-
42
- | Playbook | Does | Requires |
43
- |---|---|---|
44
- | `bili-feed` | Home feed, scroll and capture unseen cards | none |
45
- | `bili-video` | One video and its visible comments | `--url` or `--bvid` |
46
- | `bili-chain` | Follow the next-episode control and resume from the session | `--url`, `--bvid`, or a chain |
47
- | `bili-danmaku` | Timestamped danmaku, preferring bridge `observe` over DOM text (experimental) | `--url` or `--bvid` |
48
- | `zhihu-dig` | Zhihu search, rank links, capture the best unseen page | `--query` |
49
- | `xhs-dig` | Xiaohongshu public notes, rank `/explore` links, capture one | `--query` |
50
- | `search-dig` | Bilibili search, rank cards, open the best unseen result | `--query` |
51
- | `wechat-dig` | One WeChat article; reports the environment-check wall instead of recycling it | `--url` |
52
-
53
- WeChat articles are the one case HTTP mode cannot finish. `mp.weixin.qq.com` disallows `/s` in robots.txt, and even with `--robots-policy warn` the article URL answers with an environment-check page or a captcha redirect instead of the body; the HTTP record reports `verification_required` (exit 9) rather than handing back the error page as content. `wechat-dig` opens the same URL in a real browser profile. If that profile has already passed WeChat's check, the article is captured; if not, the playbook reports `blocked` and does not attempt the challenge.
54
-
55
- The bundled bridge is picked up automatically once `playwright` and its Chromium binary are installed; `--bridge` or `WEB_FORAGER_BRIDGE` points at a bridge elsewhere. The command is split on whitespace and started without a shell, so quote a path that contains spaces and pass arguments as usual; pipes, `&&` and redirection are not interpreted. `WEB_FORAGER_BROWSER_HEADFUL=1` shows the window, `WEB_FORAGER_BROWSER_PROFILE=/path` reuses a persistent profile (this is what carries login state for the `extension` backend), and `WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222` attaches to a browser you already control (`cdp`). A bridge declares its real capabilities through an optional `describe` call; one that does not implement it is reported as `declared: false` with every capability false.
56
-
57
- To sign in once, run `python browser/launch_chrome.py` (it defaults to `~/.web-forager/browser/profile`, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`), then pass `--backend cdp`. `--storage-state FILE` loads a Playwright state file before the run; `--save-storage-state FILE` writes one after it, with mode `0600` and a refusal if the path is inside `--root`. That file contains cookies and logins: treat it as a credential, keep it out of git, and do not copy it to a shared machine.
58
-
59
- `--strict-robots` fetches `robots.txt` before every navigation and stops with `robots_denied` (exit 3) on a disallowed path. Without it the record says so: `compliance.robots_checked` is `false`. A run that reaches the site but finds nothing readable is `degraded` (exit 8) with its records attached; an anti-bot or error interstitial is `blocked` (exit 9); a site that asks for credentials is `login_required` (exit 6).
60
-
61
- A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with `--out result.json` (or `out:` in MCP), no screenshots are taken unless asked, `--max-screens 5` and `--min-interval 3000` keep a walk polite, and a long walk is better split into runs that resume from `--session`.
62
-
63
- ---
64
-
65
- ## When the engines are blocked
66
-
67
- Search defaults to `auto`. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with `--engine bing`.
68
-
69
- Two routes need no search engine at all:
70
-
71
- ```bash
72
- python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
73
- python run.py --action feed --url https://the-decoder.com/ --num-results 10
74
- ```
75
-
76
- `find` tries the site's own search URLs (`/?s=`, `/search?q=` and two more), ranks the links by the query and fetches the best ones. `feed` discovers the site's RSS/Atom `<link rel=alternate>` and reads it. `doctor --probe` asks every engine once and reports which ones answer from this network; `doctor` also reports the proxy in use with its credentials removed. Requests honour `HTTPS_PROXY` / `ALL_PROXY`, and `--proxy URL` overrides them for one run.
77
-
78
- ---
79
-
80
- ## More CLI examples
81
-
82
- Each command has its own block so it can be copied without trimming the others.
83
-
84
- **Fetch one Bilibili video.**
85
-
86
- ```bash
87
- python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
88
- ```
89
-
90
- **Fetch a page that needs JavaScript rendering.**
91
-
92
- ```bash
93
- python run.py --action fetch --url "https://www.zhihu.com/" --render auto
94
- ```
95
-
96
- **Surf a site and resume the trail later.**
97
-
98
- ```bash
99
- python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
100
- ```
101
-
102
- **Clean a local PDF into Markdown.**
103
-
104
- ```bash
105
- python run.py --action clean --input notes.pdf --format markdown
106
- ```
107
-
108
- **List up to 500 page URLs from a site's sitemaps.**
109
-
110
- ```bash
111
- python run.py --action map --url "https://example.org" --limit 500
112
- ```
113
-
114
- **Read the Bilibili home feed.**
115
-
116
- ```bash
117
- python run.py --action bili --mode feed --max-items 10
118
- ```
119
-
120
- ---
121
-
122
- ## Records and output
123
-
124
- ### One record per page
125
-
126
- | Field | HTTP mode | Browser mode |
127
- |---|---|---|
128
- | `metadata`, `markdown`, `tables`, `warnings` | yes | yes |
129
- | `trust`, `injection_flags` | yes | yes |
130
- | `url`, `title` | yes (`url` and `final_url`) | yes |
131
- | `status`, `content_type`, `document_kind`, `cached`, `rendered`, `render_fallback`, `fetched_at`, `chunks`, `links` | yes | no |
132
- | `schema_version`, `captured_at`, `playbook`, `backend`, `compliance`, `viewport_evidence`, `interaction_trace` | no | yes |
133
-
134
- `render_fallback` is `true` when the server answered with a minimal script shell and HTTP mode rendered it once with Playwright before extraction. A `fetch --query` allows at most two such renders, and each one adds several seconds.
135
-
136
- Search results are ordered by query coverage, title match, a small domain prior, snippet freshness and engine order. The final list keeps at most two results per host, so one dominant host cannot crowd out every other source; when too few hosts answered, the overflow fills the remaining slots. A thin first pass is retried once with a simplified query (quotes and common stopwords removed, or a spaced CJK query joined); that retry appears in `engine_report` with `query_variant`.
137
-
138
- `--core` returns the shared shape instead:
139
- ```json
140
- {
141
- "schema_version": "core-1",
142
- "source": "web-forager",
143
- "mode": "http",
144
- "url": "https://example.org/article",
145
- "title": "An article",
146
- "collected_at": "2026-11-07T10:00:00Z",
147
- "metadata": {},
148
- "markdown": "...",
149
- "tables": [],
150
- "warnings": [],
151
- "extra": { "status": 200, "cached": false }
152
- }
153
- ```
154
-
155
- `mode` is `http` or `browser`, `source` names the producer, and `extra` keeps every field the two modes do not share. The same fold is available as a library call: `engine.normalize_record()` in Python and `normalizeRecord()` in `browser/lib/record.js`.
156
-
157
- `results` is an index, not a second copy of every page. For a search it carries `url`, `title`, `snippet`, `status`, `content_type`, `markdown_chars`, `tables` and any `warnings` or `error`; the Markdown body lives once, in `pages`, and a single `fetch` keeps it at the top level. `--full-results` (CLI) or `fullResults: true` (MCP) restores the old shape.
158
-
159
- ### Workspace output
160
-
161
- ```bash
162
- python run.py --action fetch --url "https://example.org/article" --root <WS>
163
- ```
164
-
165
- Writes only:
166
-
167
- ```text
168
- <WS>/process/web-forager/web-forager_<timestamp>_<slug>.json
169
- <WS>/process/web-forager/web-forager_<timestamp>_<slug>.md
170
- <WS>/_broadcast/web-forager_<timestamp>_<slug>.json
171
- ```
172
-
173
- `material/` and `results/` belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under `<WS>/process/web-forager/browser/` and `<WS>/_broadcast/`.
174
-
175
- ---
176
-
177
- ---
178
-
179
- ## Actions and exit codes
180
-
181
- | Action | Purpose |
182
- |---|---|
183
- | `fetch` | one URL or keyword query → pages |
184
- | `find` | search inside one site through the site's own search page |
185
- | `feed` | read a site's RSS/Atom feed, discovering it from the page |
186
- | `clean` | local or remote HTML/PDF/DOCX/XLSX/EPUB/CSV → Markdown |
187
- | `bili` | Bilibili feed / video / related / search / surf |
188
- | `surf` | search or seed URL, then ranked link BFS |
189
- | `scholar` | arXiv / Crossref / PubMed / DOI records |
190
- | `map` | page URLs from robots.txt and sitemaps |
191
- | `doctor` | dependency report: what is installed, what to install |
192
-
193
- | Exit code | Meaning (both modes unless overridden below) |
194
- |---|---|
195
- | 0 | success |
196
- | 2 | argument or URL error |
197
- | 3 | robots.txt denied the request, or no browser bridge configured |
198
- | 4 | network, HTTP or navigation failure |
199
- | 5 | extraction or capture failed |
200
- | 6 | Playwright requested but unavailable |
201
- | 7 | PDF requested but `pypdf` unavailable |
202
- | 9 | verification or anti-bot page instead of content (`verification_required` in HTTP mode, `blocked` in browser mode) |
203
- | 130 | interrupted |
204
-
205
- Browser mode reuses that table and changes or adds only these codes:
206
-
207
- | Exit code | Browser meaning |
208
- |---|---|
209
- | 2 | also: unknown backend |
210
- | 6 | `login_required`: the site asked for credentials |
211
- | 7 | `confirmation_required`: a write-like action was denied |
212
- | 8 | `degraded`: the page was read, but the playbook found nothing it looks for |
213
- | 9 | `blocked`: an anti-bot or error interstitial replaced the page |
214
-
215
- `--agent` keeps these exit codes and prints a `{"ok":false,"error":{"code","message","hint"}}` envelope on stdout. `doctor` reports every dependency with its version or the command that installs it; a missing Playwright returns exit code 6, and a missing `pypdf` returns exit code 7.
216
-
217
- ---
218
-
219
- ---
220
-
221
- ## Requirements
222
-
223
- Python 3.10+. Core runtime: `requests`, `beautifulsoup4`, `trafilatura`. Optional: `playwright` (JS rendering), `pypdf` (PDF). `defusedxml` is used when already installed but is never required — XML with an internal DTD subset or any ENTITY declaration is rejected regardless. Browser mode additionally needs Node.js 18+ and a bridge command; `browser/bridges/playwright-bridge.js` needs `playwright` installed with npm. No third-party code is vendored; the full per-capability matrix lives in [DEPENDENCIES.md](DEPENDENCIES.md).
224
-
225
- ---
226
-
227
- ---
228
-
229
- ## Repository layout
230
-
231
- <details>
232
- <summary>Full tree</summary>
233
-
234
- ```text
235
- web-forager/
236
- ├── .claude-plugin/ Claude Code manifest + marketplace
237
- ├── .codex-plugin/ Codex manifest + marketplace
238
- ├── .codebuddy-plugin/ CodeBuddy manifest + marketplace
239
- ├── .dsh-plugin/ DSH manifest + CJS entry
240
- ├── .cursor/rules/ Cursor rule file
241
- ├── .windsurf/rules/ Windsurf rule file
242
- ├── .clinerules/ Cline rule file
243
- ├── .continue/rules/ Continue rule file
244
- ├── .roo/rules/ Roo Code rule file
245
- ├── .github/ Copilot instructions + Actions CI
246
- ├── AGENTS.md generic agent contract
247
- ├── install.py one-command rule-file installer
248
- ├── manifest.json plugin manifest
249
- ├── .mcp.json MCP stdio registration
250
- ├── package.json / index.js npm + dsh packaging
251
- ├── cordis.patch.yml dsh bundle patch
252
- ├── run.py CLI: nine actions
253
- ├── mcp_server.py MCP stdio server (10 HTTP tools)
254
- ├── browser/ browser mode (Node 18+)
255
- │ ├── run.js CLI: eight read-only playbooks
256
- │ ├── mcp_server.js MCP stdio server (4 browser tools)
257
- │ ├── lib/ bridge backend, session, record, safety, selectors, robots, snapshot
258
- │ ├── playbooks/ bili-* / zhihu-dig / xhs-dig / search-dig / wechat-dig
259
- │ ├── bridges/ optional Playwright reference bridge
260
- │ └── tests/ fake backends, real-page snapshots, live smoke
261
- ├── SKILL.md Claude Code skill contract
262
- ├── commands/ /fetch /surf /scholar /map /browse
263
- ├── docs/ install commands, acknowledgements, dependency matrix
264
- ├── scripts/
265
- │ ├── engine.py robots, fetch, search, cache, chunks, Markdown, surf, find, feed
266
- │ ├── pdf.py PDF text (optional pypdf)
267
- │ ├── documents.py DOCX / XLSX / EPUB / CSV
268
- │ ├── feeds.py RSS / Atom / RSS 1.0
269
- │ ├── scholar.py arXiv / Crossref / PubMed / DOI
270
- │ ├── sitemap.py sitemap discovery
271
- │ └── bilibili.py Bilibili adapter
272
- ├── tests/ offline HTTP fixtures
273
- ├── requirements.txt
274
- ├── THIRD_PARTY_LICENSES.txt
275
- ├── CHANGELOG.md
276
- ├── README.md · README_zh.md
277
- └── LICENSE MIT
278
- ```
279
-
280
- </details>
281
-
282
- ---
283
-
284
- ---
285
-
286
- ## Tests
287
-
288
- ```bash
289
- python -m pytest tests -q
290
- npm run test:browser
291
- ```
292
-
293
- That is **228 Python tests** plus **75 browser tests**, all passing and fully offline.
294
-
295
- <details>
296
- <summary>What the suites cover</summary>
297
-
298
- `conftest.py` starts a temporary local HTTP server serving robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages, a site-search page and multi-hop surf pages. The Python suite covers robots denial, extraction, tables, documents, feeds, cache, search pagination and routing, chunking, MCP, surf sessions, installer behaviour, host-rule and disclaimer checks, the shared `core-1` record, and every exit code. The browser suite uses a fake backend plus fake bridges that speak the wire protocol, so the CLI runs end to end without a browser; it covers capability declaration, robots.txt matching, the `--strict-robots` round trip, the private-address guard, the untrusted-content screen, and the degraded and blocked paths over snapshots recorded from live pages.
299
-
300
- </details>
301
-
302
- CI runs both suites on every push to `main`, on pull requests and on version tags: Python on Ubuntu 3.10/3.12 and Windows 3.12, browser mode on Node 20/22. A live check is opt-in and never runs in CI:
303
-
304
- ```bash
305
- WEB_FORAGER_LIVE=1 WEB_FORAGER_BRIDGE="node browser/bridges/playwright-bridge.js" npm run test:live
306
- ```
307
-
308
- ---
1
+ # web-forager reference
2
+
3
+ > Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
4
+
5
+ **English** · [简体中文](REFERENCE_zh.md)
6
+
7
+ ---
8
+
9
+ ## MCP servers
10
+
11
+ <details>
12
+ <summary>Tool names</summary>
13
+
14
+ HTTP tools: `web_fetch`, `web_search`, `web_research`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_doctor`, `web_find`, `web_feed`. Browser tools: `browser_list`, `browser_run`, `browser_session`, `browser_doctor`.
15
+
16
+ MCP calls are always read-only and keep the target-page boundary: no tool parameter can change `respect_robots`, `robots_policy`, `input_gate`, the proxy, the browser bridge or a write switch, and the local file `clean` action is not exposed over MCP. Search-engine and scholarly query endpoints have a separate user-level policy (`--query-robots` on the CLI, `WEB_FORAGER_QUERY_ROBOTS` in the server environment), default `warn`: a disallowed query path is read and the engine report names the override; `respect` refuses it. The agent cannot change this through a tool call.
17
+
18
+ </details>
19
+
20
+ Test the protocol directly:
21
+
22
+ ```bash
23
+ printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
24
+ '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
25
+ ```
26
+
27
+ ```bash
28
+ printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
29
+ '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
30
+ ```
31
+
32
+ ## Browser mode
33
+
34
+ The browser layer lives in `browser/` and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless `--allow-write` is passed after the user authorises that single action.
35
+
36
+ ```bash
37
+ npm install playwright && npx playwright install chromium
38
+ node browser/run.js --doctor
39
+ node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
40
+ node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
41
+ node browser/run.js --playbook wechat-dig --url "https://mp.weixin.qq.com/s/..." --agent
42
+ ```
43
+
44
+ | Playbook | Does | Requires |
45
+ |---|---|---|
46
+ | `bili-feed` | Home feed, scroll and capture unseen cards | none |
47
+ | `bili-video` | One video and its visible comments | `--url` or `--bvid` |
48
+ | `bili-chain` | Follow the next-episode control and resume from the session | `--url`, `--bvid`, or a chain |
49
+ | `bili-danmaku` | Timestamped danmaku, preferring bridge `observe` over DOM text (experimental) | `--url` or `--bvid` |
50
+ | `zhihu-dig` | Zhihu search, rank links, capture the best unseen page | `--query` |
51
+ | `xhs-dig` | Xiaohongshu public notes, rank `/explore` links, capture one | `--query` |
52
+ | `search-dig` | Bilibili search, rank cards, open the best unseen result | `--query` |
53
+ | `wechat-dig` | One WeChat article; reports the environment-check wall instead of recycling it | `--url` |
54
+
55
+ WeChat articles are the one case HTTP mode cannot finish. `mp.weixin.qq.com` disallows `/s` in robots.txt, and even with `--robots-policy warn` the article URL answers with an environment-check page or a captcha redirect instead of the body; the HTTP record reports `verification_required` (exit 9) rather than handing back the error page as content. `wechat-dig` opens the same URL in a real browser profile. If that profile has already passed WeChat's check, the article is captured; if not, the playbook reports `blocked` and does not attempt the challenge.
56
+
57
+ The bundled bridge is picked up automatically once `playwright` and its Chromium binary are installed; `--bridge` or `WEB_FORAGER_BRIDGE` points at a bridge elsewhere. The command is split on whitespace and started without a shell, so quote a path that contains spaces and pass arguments as usual; pipes, `&&` and redirection are not interpreted. `WEB_FORAGER_BROWSER_HEADFUL=1` shows the window, `WEB_FORAGER_BROWSER_PROFILE=/path` reuses a persistent profile (this is what carries login state for the `extension` backend), and `WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222` attaches to a browser you already control (`cdp`). A bridge declares its real capabilities through an optional `describe` call; one that does not implement it is reported as `declared: false` with every capability false.
58
+
59
+ To sign in once, run `python browser/launch_chrome.py` (it defaults to `~/.web-forager/browser/profile`, and CDP is **off** by default; `--cdp` opts in to a loopback CDP port, random by default or fixed with `--port 9222`). Without `--cdp`, run browser mode with the default backend and it reuses the profile. With `--cdp`, pass `--backend cdp`. CDP itself has no authentication: keep the port local and close the browser when finished. `--storage-state FILE` loads a Playwright state file before the run; `--save-storage-state FILE` writes one after it, with mode `0600` and a refusal if the path is inside `--root`. That file contains cookies and logins: treat it as a credential, keep it out of git, and do not copy it to a shared machine.
60
+
61
+ `--strict-robots` fetches `robots.txt` before every navigation and stops with `robots_denied` (exit 3) on a disallowed path. Without it the record says so: `compliance.robots_checked` is `false`. A run that reaches the site but finds nothing readable is `degraded` (exit 8) with its records attached; an anti-bot or error interstitial is `blocked` (exit 9); a site that asks for credentials is `login_required` (exit 6).
62
+
63
+ A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with `--out result.json` (or `out:` in MCP), no screenshots are taken unless asked, `--max-screens 5` and `--min-interval 3000` keep a walk polite, and a long walk is better split into runs that resume from `--session`.
64
+
65
+ ---
66
+
67
+ ## When the engines are blocked
68
+
69
+ Search defaults to `auto`. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with `--engine bing`.
70
+
71
+ Two routes need no search engine at all:
72
+
73
+ ```bash
74
+ python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
75
+ python run.py --action feed --url https://the-decoder.com/ --num-results 10
76
+ ```
77
+
78
+ `find` tries the site's own search URLs (`/?s=`, `/search?q=` and two more), ranks the links by the query and fetches the best ones. `feed` discovers the site's RSS/Atom `<link rel=alternate>` and reads it. `doctor --probe` asks every engine once and reports which ones answer from this network; `doctor` also reports the proxy in use with its credentials removed. Requests honour `HTTPS_PROXY` / `ALL_PROXY`, and `--proxy URL` overrides them for one run.
79
+
80
+ ---
81
+
82
+ ## More CLI examples
83
+
84
+ Each command has its own block so it can be copied without trimming the others.
85
+
86
+ **Fetch one Bilibili video.**
87
+
88
+ ```bash
89
+ python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
90
+ ```
91
+
92
+ **Fetch a page that needs JavaScript rendering.**
93
+
94
+ ```bash
95
+ python run.py --action fetch --url "https://www.zhihu.com/" --render auto
96
+ ```
97
+
98
+ **Surf a site and resume the trail later.**
99
+
100
+ ```bash
101
+ python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
102
+ ```
103
+
104
+ **Clean a local PDF into Markdown.**
105
+
106
+ ```bash
107
+ python run.py --action clean --input notes.pdf --format markdown
108
+ ```
109
+
110
+ **List up to 500 page URLs from a site's sitemaps.**
111
+
112
+ ```bash
113
+ python run.py --action map --url "https://example.org" --limit 500
114
+ ```
115
+
116
+ **Read the Bilibili home feed.**
117
+
118
+ ```bash
119
+ python run.py --action bili --mode feed --max-items 10
120
+ ```
121
+
122
+ ---
123
+
124
+ ## Records and output
125
+
126
+ ### One record per page
127
+
128
+ | Field | HTTP mode | Browser mode |
129
+ |---|---|---|
130
+ | `metadata`, `markdown`, `tables`, `warnings` | yes | yes |
131
+ | `trust`, `injection_flags`, `injection_findings` | yes | yes |
132
+ | `url`, `title` | yes (`url` and `final_url`) | yes |
133
+ | `status`, `content_type`, `document_kind`, `cached`, `rendered`, `render_fallback`, `fetched_at`, `chunks`, `links` | yes | no |
134
+ | `schema_version`, `captured_at`, `playbook`, `backend`, `compliance`, `viewport_evidence`, `interaction_trace` | no | yes |
135
+
136
+ `render_fallback` is `true` when the server answered with a minimal script shell and HTTP mode rendered it once with Playwright before extraction. A `fetch --query` allows at most two such renders, and each one adds several seconds.
137
+
138
+ `injection_findings` is a list of `{code, severity, confidence, span, snippet, context}`. `span` is a character range in the scanned text, `severity` is `hard` or `soft`, and `context` is `direct`, `quote` or `example`. Hard rules describe an action (forged system tag, secret exfiltration, `curl ... | sh`, `rm -rf /`); `--input-gate strict` withholds a body only when one matches. Soft rules describe override or role-swap language and never withhold on their own, because papers and security notes quote them. The scanner walks the whole body in overlapping windows, before `max_chars` truncation, and the first warning repeats up to three matching snippets.
139
+
140
+ Search results are ordered by query coverage, phrase and title match, a small domain prior, snippet freshness and engine order — including when only one engine answered. Each result carries `coverage`, `matched_terms`, `relevance` and `language_match`. A result is relevant when it contains the exact phrase or at least two meaningful query terms with half the query covered; a wrong-language page that repeats one product name is `language_match: false`. Off-topic rows are ranked last, and are dropped when at least three relevant results exist; otherwise a warning names how many were kept. The final list keeps at most two results per host, so one dominant host cannot crowd out every other source; when too few hosts answered, the overflow fills the remaining slots. A thin first pass is retried once with a simplified query (quotes and common stopwords removed, or a spaced CJK query joined); that retry appears in `engine_report` with `query_variant`. Search titles and snippets pass through the same injection scanner, and a matching result carries its own `injection_flags` and `injection_findings`.
141
+
142
+ Before ranking, duplicate URLs are canonicalized: `utm_*`, `fbclid`, fragments, `www.`, trailing slashes and `/amp` no longer make one article look like two results. A result found by more than one engine receives a reciprocal-rank-fusion contribution, and an exact title match gets a small bonus over a title that merely contains the phrase. Light English stemming is used for matching only, so `injection` meets `injections` without changing the displayed query or text. The search payload also carries `search_quality: {query, engine, returned, relevant, off_topic, language_mismatch, engines_answered, engine_errors, dropped_off_topic, returned_before_filter}`, a machine-readable summary for agents that do not want to parse the prose warnings.
143
+
144
+ `--core` returns the shared shape instead:
145
+ ```json
146
+ {
147
+ "schema_version": "core-1",
148
+ "source": "web-forager",
149
+ "mode": "http",
150
+ "url": "https://example.org/article",
151
+ "title": "An article",
152
+ "collected_at": "2026-11-07T10:00:00Z",
153
+ "metadata": {},
154
+ "markdown": "...",
155
+ "tables": [],
156
+ "warnings": [],
157
+ "extra": { "status": 200, "cached": false }
158
+ }
159
+ ```
160
+
161
+ `mode` is `http` or `browser`, `source` names the producer, and `extra` keeps every field the two modes do not share. The same fold is available as a library call: `engine.normalize_record()` in Python and `normalizeRecord()` in `browser/lib/record.js`.
162
+
163
+ `results` is an index, not a second copy of every page. For a search it carries `url`, `title`, `snippet`, `status`, `content_type`, `markdown_chars`, `tables` and any `warnings` or `error`; the Markdown body lives once, in `pages`, and a single `fetch` keeps it at the top level. `--full-results` (CLI) or `fullResults: true` (MCP) restores the old shape.
164
+
165
+ ### Workspace output
166
+
167
+ ```bash
168
+ python run.py --action fetch --url "https://example.org/article" --root <WS>
169
+ ```
170
+
171
+ Writes only:
172
+
173
+ ```text
174
+ <WS>/process/web-forager/web-forager_<timestamp>_<slug>.json
175
+ <WS>/process/web-forager/web-forager_<timestamp>_<slug>.md
176
+ <WS>/_broadcast/web-forager_<timestamp>_<slug>.json
177
+ ```
178
+
179
+ `material/` and `results/` belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under `<WS>/process/web-forager/browser/` and `<WS>/_broadcast/`.
180
+
181
+ ---
182
+
183
+ ---
184
+
185
+ ## Research output
186
+
187
+ `research --query` runs two stages: broad recall across the free engines, then
188
+ focused reading of a few distinct hosts. The payload adds:
189
+
190
+ | Field | Meaning |
191
+ |---|---|
192
+ | `intent` | `{intent, confidence, signals}` from a deterministic keyword classifier |
193
+ | `strategy` | `{breadth, search_pages, targeted_site, targeted_sites, targeted_by_site, official_entries, focus_*}` |
194
+ | `seeds` | selected results with `authority_tier`, `authority_score`, `reason` |
195
+ | `evidence` | `[{url, title, authority_tier, authors, published_at, date_source, passages, partial?}]` |
196
+ | `freshness` | `{newest, oldest, dated_sources, undated_sources, newest_age_days, all_stale}` |
197
+ | `answerability` | `{level, coverage, authority_level, partial_only, source_count, host_count, view_hosts, multi_view, stale, reason}` |
198
+ | `skipped` | pages not used, with `code` (`robots_denied`, `browser_required`, `browser_required_low_value`, `no_matching_passage`, …) |
199
+
200
+ If the broad pool has no relevant primary/official/academic source, the second
201
+ pass prefers a direct official entry page (Python asyncio, Go concurrency, Rust,
202
+ MDN, Microsoft Wi-Fi support), then the site's own search page (Microsoft
203
+ Support, MDN), and only then a `site:` engine query for a non-CJK question.
204
+ Direct entries and host search pages do not depend on an engine honouring
205
+ `site:`. A news query with no usable engine rows falls back to a public finance
206
+ RSS channel (China News Service); a dated headline and summary are labelled
207
+ `feed_headline`, never expanded into a fabricated article. When the article body
208
+ has no dateline, the fetched page inherits the feed's publication date. Each
209
+ focus fetch is capped at 8 seconds with no retry, and robots.txt at 6 seconds,
210
+ so one dead origin cannot consume the whole call. A comparison needs at
211
+ least two independent full-source viewpoints;
212
+ a WeChat snippet never counts as one. When two full authoritative sources are
213
+ already present for a comparison or technical question, a WeChat-only snippet is
214
+ skipped rather than mixed in. A WeChat link that HTTP mode cannot read still
215
+ yields a `partial` search-snippet record with an explicit warning, never a
216
+ fabricated full article. Every passage is quoted, untrusted page text.
217
+
218
+ ## Actions and exit codes
219
+
220
+ | Action | Purpose |
221
+ |---|---|
222
+ | `fetch` | one URL or keyword query → pages |
223
+ | `research` | broad recall → focused, cited evidence passages |
224
+ | `find` | search inside one site through the site's own search page |
225
+ | `feed` | read a site's RSS/Atom feed, discovering it from the page |
226
+ | `clean` | local or remote HTML/PDF/DOCX/XLSX/EPUB/CSV → Markdown |
227
+ | `bili` | Bilibili feed / video / related / search / surf |
228
+ | `surf` | search or seed URL, then ranked link BFS |
229
+ | `scholar` | arXiv / Crossref / PubMed / DOI records |
230
+ | `map` | page URLs from robots.txt and sitemaps |
231
+ | `doctor` | dependency report: what is installed, what to install |
232
+
233
+ | Exit code | Meaning (both modes unless overridden below) |
234
+ |---|---|
235
+ | 0 | success |
236
+ | 2 | argument or URL error |
237
+ | 3 | robots.txt denied the request, or no browser bridge configured |
238
+ | 4 | network, HTTP or navigation failure |
239
+ | 5 | extraction or capture failed |
240
+ | 6 | Playwright requested but unavailable |
241
+ | 7 | PDF requested but `pypdf` unavailable |
242
+ | 9 | verification or anti-bot page instead of content (`verification_required` in HTTP mode, `blocked` in browser mode) |
243
+ | 130 | interrupted |
244
+
245
+ Browser mode reuses that table and changes or adds only these codes:
246
+
247
+ | Exit code | Browser meaning |
248
+ |---|---|
249
+ | 2 | also: unknown backend |
250
+ | 6 | `login_required`: the site asked for credentials |
251
+ | 7 | `confirmation_required`: a write-like action was denied |
252
+ | 8 | `degraded`: the page was read, but the playbook found nothing it looks for |
253
+ | 9 | `blocked`: an anti-bot or error interstitial replaced the page |
254
+
255
+ `--agent` keeps these exit codes and prints a `{"ok":false,"error":{"code","message","hint"}}` envelope on stdout. `doctor` reports every dependency with its version or the command that installs it; a missing Playwright returns exit code 6, and a missing `pypdf` returns exit code 7.
256
+
257
+ ---
258
+
259
+ ---
260
+
261
+ ## Requirements
262
+
263
+ Python 3.10+. Core runtime: `requests`, `beautifulsoup4`, `trafilatura`. Optional: `playwright` (JS rendering), `pypdf` (PDF). `defusedxml` is used when already installed but is never required — XML with an internal DTD subset or any ENTITY declaration is rejected regardless. Browser mode additionally needs Node.js 18+ and a bridge command; `browser/bridges/playwright-bridge.js` needs `playwright` installed with npm. No third-party code is vendored; the full per-capability matrix lives in [DEPENDENCIES.md](DEPENDENCIES.md).
264
+
265
+ ---
266
+
267
+ ---
268
+
269
+ ## Repository layout
270
+
271
+ <details>
272
+ <summary>Full tree</summary>
273
+
274
+ ```text
275
+ web-forager/
276
+ ├── .claude-plugin/ Claude Code manifest + marketplace
277
+ ├── .codex-plugin/ Codex manifest + marketplace
278
+ ├── .codebuddy-plugin/ CodeBuddy manifest + marketplace
279
+ ├── .dsh-plugin/ DSH manifest + CJS entry
280
+ ├── .cursor/rules/ Cursor rule file
281
+ ├── .windsurf/rules/ Windsurf rule file
282
+ ├── .clinerules/ Cline rule file
283
+ ├── .continue/rules/ Continue rule file
284
+ ├── .roo/rules/ Roo Code rule file
285
+ ├── .github/ Copilot instructions + Actions CI
286
+ ├── AGENTS.md generic agent contract
287
+ ├── install.py one-command rule-file installer
288
+ ├── manifest.json plugin manifest
289
+ ├── .mcp.json MCP stdio registration
290
+ ├── package.json / index.js npm + dsh packaging
291
+ ├── cordis.patch.yml dsh bundle patch
292
+ ├── run.py CLI: nine actions
293
+ ├── mcp_server.py MCP stdio server (10 HTTP tools)
294
+ ├── browser/ browser mode (Node 18+)
295
+ │ ├── run.js CLI: eight read-only playbooks
296
+ │ ├── mcp_server.js MCP stdio server (4 browser tools)
297
+ │ ├── lib/ bridge backend, session, record, safety, selectors, robots, snapshot
298
+ │ ├── playbooks/ bili-* / zhihu-dig / xhs-dig / search-dig / wechat-dig
299
+ │ ├── bridges/ optional Playwright reference bridge
300
+ │ └── tests/ fake backends, real-page snapshots, live smoke
301
+ ├── SKILL.md Claude Code skill contract
302
+ ├── commands/ /fetch /research /surf /scholar /map /browse
303
+ ├── docs/ install commands, acknowledgements, dependency matrix
304
+ ├── scripts/
305
+ │ ├── engine.py robots, fetch, search, cache, chunks, Markdown, surf, find, feed
306
+ │ ├── research.py broad recall, authority selection, cited evidence passages
307
+ │ ├── pdf.py PDF text (optional pypdf)
308
+ │ ├── documents.py DOCX / XLSX / EPUB / CSV
309
+ │ ├── feeds.py RSS / Atom / RSS 1.0
310
+ │ ├── scholar.py arXiv / Crossref / PubMed / DOI
311
+ │ ├── sitemap.py sitemap discovery
312
+ │ └── bilibili.py Bilibili adapter
313
+ ├── tests/ offline HTTP fixtures
314
+ ├── requirements.txt
315
+ ├── THIRD_PARTY_LICENSES.txt
316
+ ├── CHANGELOG.md
317
+ ├── README.md · README_zh.md
318
+ └── LICENSE MIT
319
+ ```
320
+
321
+ </details>
322
+
323
+ ---
324
+
325
+ ---
326
+
327
+ ## Tests
328
+
329
+ ```bash
330
+ python -m pytest tests -q
331
+ npm run test:browser
332
+ ```
333
+
334
+ That is **333 Python tests** plus **91 browser tests**, all passing and fully offline.
335
+
336
+ <details>
337
+ <summary>What the suites cover</summary>
338
+
339
+ `conftest.py` starts a temporary local HTTP server serving robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages, a site-search page and multi-hop surf pages. The Python suite covers robots denial, extraction, tables, documents, feeds, cache, search pagination and routing, chunking, MCP, surf sessions, installer behaviour, host-rule and disclaimer checks, the shared `core-1` record, and every exit code. The browser suite uses a fake backend plus fake bridges that speak the wire protocol, so the CLI runs end to end without a browser; it covers capability declaration, robots.txt matching, the `--strict-robots` round trip, the private-address guard, the untrusted-content screen, and the degraded and blocked paths over snapshots recorded from live pages.
340
+
341
+ </details>
342
+
343
+ CI runs both suites on every push to `main`, on pull requests and on version tags: Python on Ubuntu 3.10/3.12 and Windows 3.12, browser mode on Node 20/22. A live check is opt-in and never runs in CI:
344
+
345
+ ```bash
346
+ WEB_FORAGER_LIVE=1 WEB_FORAGER_BRIDGE="node browser/bridges/playwright-bridge.js" npm run test:live
347
+ ```
348
+
349
+ ---