web-forager 0.0.0-stage → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/.claude-plugin/marketplace.json +16 -0
  2. package/.claude-plugin/plugin.json +20 -0
  3. package/.clinerules/web-forager.md +85 -0
  4. package/.codebuddy-plugin/marketplace.json +14 -0
  5. package/.codebuddy-plugin/plugin.json +25 -0
  6. package/.codex-plugin/marketplace.json +12 -0
  7. package/.codex-plugin/mcp.json +20 -0
  8. package/.codex-plugin/plugin.json +15 -0
  9. package/.continue/rules/web-forager.md +90 -0
  10. package/.cursor/rules/web-forager.mdc +80 -0
  11. package/.dsh-plugin/index.js +11 -0
  12. package/.dsh-plugin/plugin.json +5 -0
  13. package/.github/copilot-instructions.md +85 -0
  14. package/.mcp.json +12 -0
  15. package/.roo/rules/web-forager.md +85 -0
  16. package/.windsurf/rules/web-forager.md +90 -0
  17. package/AGENTS.md +112 -0
  18. package/CHANGELOG.md +454 -0
  19. package/LICENSE +21 -0
  20. package/README.md +477 -3
  21. package/README_zh.md +477 -0
  22. package/SKILL.md +351 -0
  23. package/THIRD_PARTY_LICENSES.txt +35 -0
  24. package/browser/bridges/playwright-bridge.js +375 -0
  25. package/browser/launch_chrome.py +144 -0
  26. package/browser/lib/backend.js +302 -0
  27. package/browser/lib/bili-api.js +79 -0
  28. package/browser/lib/bridge.js +83 -0
  29. package/browser/lib/context.js +60 -0
  30. package/browser/lib/doctor.js +97 -0
  31. package/browser/lib/errors.js +17 -0
  32. package/browser/lib/netguard.js +149 -0
  33. package/browser/lib/playbooks.js +30 -0
  34. package/browser/lib/record.js +109 -0
  35. package/browser/lib/robots.js +195 -0
  36. package/browser/lib/safety.js +115 -0
  37. package/browser/lib/selectors.js +80 -0
  38. package/browser/lib/session.js +116 -0
  39. package/browser/lib/snapshot.js +229 -0
  40. package/browser/lib/untrusted.js +56 -0
  41. package/browser/lib/workspace.js +40 -0
  42. package/browser/mcp_server.js +272 -0
  43. package/browser/package.json +40 -0
  44. package/browser/playbooks/bili-chain.js +58 -0
  45. package/browser/playbooks/bili-danmaku.js +102 -0
  46. package/browser/playbooks/bili-feed.js +66 -0
  47. package/browser/playbooks/bili-video.js +38 -0
  48. package/browser/playbooks/search-dig.js +105 -0
  49. package/browser/playbooks/shared.js +138 -0
  50. package/browser/playbooks/wechat-dig.js +78 -0
  51. package/browser/playbooks/xhs-dig.js +89 -0
  52. package/browser/playbooks/zhihu-dig.js +89 -0
  53. package/browser/run.js +432 -0
  54. package/commands/browse.md +53 -0
  55. package/commands/fetch.md +29 -0
  56. package/commands/map.md +24 -0
  57. package/commands/scholar.md +28 -0
  58. package/commands/surf.md +28 -0
  59. package/cordis.patch.yml +19 -0
  60. package/docs/ACKNOWLEDGEMENTS.md +15 -0
  61. package/docs/ACKNOWLEDGEMENTS_zh.md +15 -0
  62. package/docs/DEPENDENCIES.md +40 -0
  63. package/docs/INSTALL_COMMANDS.md +113 -0
  64. package/docs/INSTALL_COMMANDS_zh.md +113 -0
  65. package/docs/REFERENCE.md +308 -0
  66. package/docs/REFERENCE_zh.md +309 -0
  67. package/index.js +7 -0
  68. package/install.py +254 -0
  69. package/manifest.json +97 -0
  70. package/mcp_server.py +597 -0
  71. package/package.json +80 -6
  72. package/requirements.txt +12 -0
  73. package/run.py +852 -0
  74. package/scripts/__init__.py +1 -0
  75. package/scripts/bilibili.py +874 -0
  76. package/scripts/documents.py +437 -0
  77. package/scripts/engine.py +4220 -0
  78. package/scripts/feeds.py +180 -0
  79. package/scripts/pdf.py +135 -0
  80. package/scripts/scholar.py +354 -0
  81. package/scripts/sitemap.py +180 -0
  82. package/scripts/untrusted.py +95 -0
@@ -0,0 +1,40 @@
1
+ # Dependency matrix
2
+
3
+ `web-forager` starts with the standard library plus three runtime packages.
4
+ Everything else is optional and degrades with a clear exit code and hint.
5
+
6
+ | Capability | Package | Licence | Required? | Missing behaviour |
7
+ |---|---|---|---|---|
8
+ | HTTP, robots, search, surf, cache | `requests` | Apache-2.0 | yes | exit 1 with import error |
9
+ | HTML parsing and fallback Markdown | `beautifulsoup4` | MIT | yes | exit 1 with import error |
10
+ | Preferred article extraction | `trafilatura` | Apache-2.0 | yes | falls back to BeautifulSoup |
11
+ | JavaScript rendering | `playwright` | Apache-2.0 | optional | `--render always` exits 6; `auto` skips with a warning |
12
+ | PDF text and metadata | `pypdf` | BSD-3-Clause | optional | PDF input exits 7 with `pip install pypdf` |
13
+ | Hardened XML parsing | `defusedxml` | PSF-2.0 | optional | DOCTYPE internal subsets and all ENTITY declarations are rejected anyway |
14
+ | DOCX / XLSX / EPUB / CSV | standard library | — | built in | — |
15
+ | RSS / Atom / RSS 1.0 | standard library | — | built in | — |
16
+ | Sitemap parsing (`.xml`, `.xml.gz`) | standard library | — | built in | — |
17
+ | arXiv / Crossref / PubMed / DOI | `requests` only | — | built in | — |
18
+ | MCP stdio server (HTTP) | standard library | — | built in | — |
19
+ | Browser mode CLI and MCP | Node.js >= 18, standard library only | — | optional | browser runs exit 3 without a bridge |
20
+ | Browser bridge (reference) | `playwright` (npm) | Apache-2.0 | optional | bridge exits 3 with an install hint |
21
+
22
+ Install the required set:
23
+
24
+ ```bash
25
+ pip install -r requirements.txt
26
+ ```
27
+
28
+ Enable optional capabilities:
29
+
30
+ ```bash
31
+ pip install playwright && playwright install chromium # JS rendering
32
+ pip install pypdf # PDF text
33
+ ```
34
+
35
+ Browser mode needs Node.js 18 or newer and a bridge command in
36
+ `WEB_FORAGER_BRIDGE`. The reference bridge in `browser/bridges/` additionally
37
+ needs `npm install playwright`; the core browser CLI and MCP server use only the
38
+ Node standard library.
39
+
40
+ No GPL or AGPL package is used. See [`THIRD_PARTY_LICENSES.txt`](../THIRD_PARTY_LICENSES.txt).
@@ -0,0 +1,113 @@
1
+ # Install commands
2
+
3
+ Every per-host command, the `install.py` flags and the optional dependencies. Methods 1 and 2 are short enough to stay in the README: [web-forager](../README.md) · [简体中文](INSTALL_COMMANDS_zh.md)
4
+
5
+ ## Method 3: install with commands
6
+
7
+ **Claude Code** — from your shell (v2.1.292+), marketplace and install in one step:
8
+
9
+ ```bash
10
+ claude plugin install web-forager --marketplace https://github.com/JiamingShi-JsPhys/web-forager.git
11
+ ```
12
+
13
+ **Claude Code** — from inside a session (v2.1.275+), add the marketplace from the URL first, pinned to the release tag:
14
+
15
+ ```text
16
+ /plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.0
17
+ ```
18
+
19
+ Then install the entry it lists:
20
+
21
+ ```text
22
+ /plugin install web-forager@web-forager-marketplace
23
+ ```
24
+
25
+ **Codex** — register the URL as a Git marketplace; `--ref` pins the release tag, and an SSH URL or a local path works in place of the HTTPS one:
26
+
27
+ ```bash
28
+ codex plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git --ref v1.3.0
29
+ ```
30
+
31
+ **Codex** — install the plugin from that marketplace:
32
+
33
+ ```bash
34
+ codex plugin add web-forager@web-forager-marketplace
35
+ ```
36
+
37
+ **dsh (DeepSeek Harness)** — installs from the same repository URL straight into a profile:
38
+
39
+ ```bash
40
+ dsh plugin --profile web add github:JiamingShi-JsPhys/web-forager#v1.3.0
41
+ ```
42
+
43
+ **CodeBuddy** — add the URL as a Git marketplace:
44
+
45
+ ```text
46
+ /plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.0
47
+ ```
48
+
49
+ Then install the entry it lists:
50
+
51
+ ```text
52
+ /plugin install web-forager@web-forager-marketplace
53
+ ```
54
+
55
+ **Host MCP notes.** Codex reads `.codex-plugin/mcp.json`, whose relative paths resolve against the installed plugin directory rather than `${CLAUDE_PLUGIN_ROOT}`. CodeBuddy reads `mcpServers` from `.codebuddy-plugin/plugin.json`; run `/reload-plugins` or start a new session after install, because `codebuddy mcp list` can run before plugins are mounted and show none. dsh registers both servers through `@deepseek-ai/dsh-mcp-client` in `cordis.patch.yml`; start a new profile session after install. All three expose the same ten HTTP tools and four browser tools.
56
+
57
+ Cursor, Windsurf, Cline's IDE extension, Continue, Roo Code, GitHub Copilot and opencode each read a rule file from their own location, and none of them accepts this repository as a plugin — Cline's `cline plugin install` takes only JS/TS plugin packages, Cursor's plugin format is a different bundle, and `opencode plugin <module>` installs npm modules. **No vendor ships an installer that understands another vendor's rule file**, so this repository ships one:
58
+
59
+ ```bash
60
+ git clone https://github.com/JiamingShi-JsPhys/web-forager.git && python web-forager/install.py --project <project> --global
61
+ ```
62
+
63
+ That single command writes every rule file at once:
64
+
65
+ - `--project <project>` covers the project-scoped files — Cursor, Windsurf, Cline, Continue, Roo Code, GitHub Copilot, and the generic `AGENTS.md` that Codex, Cursor, Copilot agents and opencode all read.
66
+ - `--global` covers the user-scoped files — `~/.config/opencode/AGENTS.md`, `$CODEX_HOME/AGENTS.md` (default `~/.codex/AGENTS.md`), and, for Claude Code, a marked block appended to `~/.claude/CLAUDE.md` when `~/.claude` already exists — and skips any host this machine does not have.
67
+
68
+ Re-running it is safe: identical files report `unchanged`, and a rule file that has drifted is left alone unless you pass `--force`.
69
+
70
+ List every destination without writing anything:
71
+
72
+ ```bash
73
+ python web-forager/install.py --list
74
+ ```
75
+
76
+ Preview the run:
77
+
78
+ ```bash
79
+ python web-forager/install.py --project <project> --global --dry-run
80
+ ```
81
+
82
+ Install for one host only:
83
+
84
+ ```bash
85
+ python web-forager/install.py --project <project> --only cursor,windsurf
86
+ ```
87
+
88
+ Keep the cloned repository somewhere the agent can reach: every rule file names `run.py`, `mcp_server.py` and `SKILL.md`, so the host agent calls the same engine as a plugin host. The rule file describes *how to call* the tool — it does not vendor the code.
89
+
90
+ **opencode without copying** — opencode can also read the rule file in place through `opencode.json`'s `instructions` array, which needs no copy and can be committed for a whole team:
91
+
92
+ ```json
93
+ {
94
+ "$schema": "https://opencode.ai/config.json",
95
+ "instructions": ["../web-forager/AGENTS.md"]
96
+ }
97
+ ```
98
+
99
+ **Manual alternative** — if you would rather not run the script, copy the file yourself to the destination the script would have used:
100
+
101
+ ```text
102
+ <clone>/AGENTS.md → <project>/AGENTS.md
103
+ ```
104
+
105
+ ## Optional capabilities
106
+
107
+ ```bash
108
+ pip install -r requirements.txt # core: requests, bs4, trafilatura
109
+ pip install playwright && playwright install chromium # JS rendering
110
+ pip install pypdf # PDF text
111
+ ```
112
+
113
+ DOCX/XLSX/EPUB/CSV, RSS/Atom, sitemap parsing and both MCP servers need **no extra Python package**. Browser mode needs Node.js 18 or newer; a browser bridge is supplied by you, and `browser/bridges/playwright-bridge.js` additionally needs `npm install playwright`. See [docs/DEPENDENCIES.md](DEPENDENCIES.md).
@@ -0,0 +1,113 @@
1
+ # 安装命令
2
+
3
+ 各宿主的完整命令、`install.py` 的全部参数与可选依赖。方法一与方法二较短,留在 README:[web-forager](../README_zh.md) · [English](INSTALL_COMMANDS.md)
4
+
5
+ ## 方法三:用命令安装
6
+
7
+ **Claude Code** —— 在终端执行(v2.1.292+),一步完成注册与安装:
8
+
9
+ ```bash
10
+ claude plugin install web-forager --marketplace https://github.com/JiamingShi-JsPhys/web-forager.git
11
+ ```
12
+
13
+ **Claude Code** —— 在会话内执行(v2.1.275+),先用 URL 添加市场,并锁定发布 tag:
14
+
15
+ ```text
16
+ /plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.0
17
+ ```
18
+
19
+ 再安装它列出的条目:
20
+
21
+ ```text
22
+ /plugin install web-forager@web-forager-marketplace
23
+ ```
24
+
25
+ **Codex** —— 把 URL 注册为 Git 市场;`--ref` 锁定发布 tag,HTTPS 换成 SSH 或本地路径同样可用:
26
+
27
+ ```bash
28
+ codex plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git --ref v1.3.0
29
+ ```
30
+
31
+ **Codex** —— 从该市场安装插件:
32
+
33
+ ```bash
34
+ codex plugin add web-forager@web-forager-marketplace
35
+ ```
36
+
37
+ **dsh(DeepSeek Harness)** —— 直接从同一个仓库 URL 安装进指定 profile:
38
+
39
+ ```bash
40
+ dsh plugin --profile web add github:JiamingShi-JsPhys/web-forager#v1.3.0
41
+ ```
42
+
43
+ **CodeBuddy** —— 先把 URL 加为 Git 市场:
44
+
45
+ ```text
46
+ /plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.0
47
+ ```
48
+
49
+ 再安装它列出的条目:
50
+
51
+ ```text
52
+ /plugin install web-forager@web-forager-marketplace
53
+ ```
54
+
55
+ **各宿主的 MCP 说明。** Codex 读取 `.codex-plugin/mcp.json`,其中的相对路径按已安装插件目录解析,不使用 `${CLAUDE_PLUGIN_ROOT}`。CodeBuddy 从 `.codebuddy-plugin/plugin.json` 读取 `mcpServers`;安装后运行 `/reload-plugins` 或新开会话,因为 `codebuddy mcp list` 可能在插件挂载前运行而显示为空。dsh 通过 `cordis.patch.yml` 里的 `@deepseek-ai/dsh-mcp-client` 注册两个服务;安装后新开 profile 会话。三者的工具集相同:十个 HTTP 工具与四个浏览器工具。
56
+
57
+ Cursor、Windsurf、Cline 的 IDE 扩展、Continue、Roo Code、GitHub Copilot 与 opencode 各自从自己的位置读取规则文件,而且它们都不接受本仓库作为插件——Cline 的 `cline plugin install` 只接受 JS/TS 插件包,Cursor 的插件格式是另一套打包,`opencode plugin <module>` 装的是 npm 模块。**没有任何厂商提供能识别别家规则文件的安装器**,所以本仓库自带一个:
58
+
59
+ ```bash
60
+ git clone https://github.com/JiamingShi-JsPhys/web-forager.git && python web-forager/install.py --project <项目> --global
61
+ ```
62
+
63
+ 这一条命令会一次性写好全部规则文件:
64
+
65
+ - `--project <项目>` 负责项目级文件——Cursor、Windsurf、Cline、Continue、Roo Code、GitHub Copilot,以及 Codex、Cursor、Copilot Agent 与 opencode 都会读的通用 `AGENTS.md`。
66
+ - `--global` 负责用户级文件——`~/.config/opencode/AGENTS.md`、`$CODEX_HOME/AGENTS.md`(默认 `~/.codex/AGENTS.md`),以及对 Claude Code 在 `~/.claude` 已存在时向 `~/.claude/CLAUDE.md` 追加的带标记块——并自动跳过本机没有安装的宿主。
67
+
68
+ 重复运行是安全的:内容相同的文件报告 `unchanged`,已经改动过的规则文件在你不加 `--force` 时保持原样。
69
+
70
+ 先看清单,不写任何文件:
71
+
72
+ ```bash
73
+ python web-forager/install.py --list
74
+ ```
75
+
76
+ 预演一次:
77
+
78
+ ```bash
79
+ python web-forager/install.py --project <项目> --global --dry-run
80
+ ```
81
+
82
+ 只装某一个宿主:
83
+
84
+ ```bash
85
+ python web-forager/install.py --project <项目> --only cursor,windsurf
86
+ ```
87
+
88
+ 克隆下来的仓库要放在 Agent 能访问到的位置:每个规则文件都点名了 `run.py`、`mcp_server.py` 与 `SKILL.md`,宿主 Agent 因此可以像插件宿主一样调用同一个引擎。规则文件只描述“怎么调用”,并不内置代码。
89
+
90
+ **opencode 也可以完全不复制** —— 用 `opencode.json` 的 `instructions` 数组原地指向规则文件,无需复制,还能随项目提交给整个团队:
91
+
92
+ ```json
93
+ {
94
+ "$schema": "https://opencode.ai/config.json",
95
+ "instructions": ["../web-forager/AGENTS.md"]
96
+ }
97
+ ```
98
+
99
+ **不想跑脚本时的手动方式** —— 自己复制,目标路径就是脚本会写入的那一处:
100
+
101
+ ```text
102
+ <克隆目录>/AGENTS.md → <项目>/AGENTS.md
103
+ ```
104
+
105
+ ## 可选能力
106
+
107
+ ```bash
108
+ pip install -r requirements.txt # 核心:requests、bs4、trafilatura
109
+ pip install playwright && playwright install chromium # JS 渲染
110
+ pip install pypdf # PDF 文本
111
+ ```
112
+
113
+ DOCX/XLSX/EPUB/CSV、RSS/Atom、sitemap 解析与两个 MCP 服务**无需额外 Python 依赖**。浏览器模式需要 Node.js 18 或更新;浏览器桥由你提供,`browser/bridges/playwright-bridge.js` 另需 `npm install playwright`。详见 [docs/DEPENDENCIES.md](DEPENDENCIES.md)。
@@ -0,0 +1,308 @@
1
+ # web-forager reference
2
+
3
+ > Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
4
+
5
+ **English** · [简体中文](REFERENCE_zh.md)
6
+
7
+ ---
8
+
9
+ ## MCP servers
10
+
11
+ <details>
12
+ <summary>Tool names</summary>
13
+
14
+ HTTP tools: `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_clean`, `web_doctor`, `web_find`, `web_feed`. Browser tools: `browser_list`, `browser_run`, `browser_session`, `browser_doctor`.
15
+
16
+ </details>
17
+
18
+ Test the protocol directly:
19
+
20
+ ```bash
21
+ printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
22
+ '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
23
+ ```
24
+
25
+ ```bash
26
+ printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
27
+ '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
28
+ ```
29
+
30
+ ## Browser mode
31
+
32
+ The browser layer lives in `browser/` and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless `--allow-write` is passed after the user authorises that single action.
33
+
34
+ ```bash
35
+ npm install playwright && npx playwright install chromium
36
+ node browser/run.js --doctor
37
+ node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
38
+ node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
39
+ node browser/run.js --playbook wechat-dig --url "https://mp.weixin.qq.com/s/..." --agent
40
+ ```
41
+
42
+ | Playbook | Does | Requires |
43
+ |---|---|---|
44
+ | `bili-feed` | Home feed, scroll and capture unseen cards | none |
45
+ | `bili-video` | One video and its visible comments | `--url` or `--bvid` |
46
+ | `bili-chain` | Follow the next-episode control and resume from the session | `--url`, `--bvid`, or a chain |
47
+ | `bili-danmaku` | Timestamped danmaku, preferring bridge `observe` over DOM text (experimental) | `--url` or `--bvid` |
48
+ | `zhihu-dig` | Zhihu search, rank links, capture the best unseen page | `--query` |
49
+ | `xhs-dig` | Xiaohongshu public notes, rank `/explore` links, capture one | `--query` |
50
+ | `search-dig` | Bilibili search, rank cards, open the best unseen result | `--query` |
51
+ | `wechat-dig` | One WeChat article; reports the environment-check wall instead of recycling it | `--url` |
52
+
53
+ WeChat articles are the one case HTTP mode cannot finish. `mp.weixin.qq.com` disallows `/s` in robots.txt, and even with `--robots-policy warn` the article URL answers with an environment-check page or a captcha redirect instead of the body; the HTTP record reports `verification_required` (exit 9) rather than handing back the error page as content. `wechat-dig` opens the same URL in a real browser profile. If that profile has already passed WeChat's check, the article is captured; if not, the playbook reports `blocked` and does not attempt the challenge.
54
+
55
+ The bundled bridge is picked up automatically once `playwright` and its Chromium binary are installed; `--bridge` or `WEB_FORAGER_BRIDGE` points at a bridge elsewhere. The command is split on whitespace and started without a shell, so quote a path that contains spaces and pass arguments as usual; pipes, `&&` and redirection are not interpreted. `WEB_FORAGER_BROWSER_HEADFUL=1` shows the window, `WEB_FORAGER_BROWSER_PROFILE=/path` reuses a persistent profile (this is what carries login state for the `extension` backend), and `WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222` attaches to a browser you already control (`cdp`). A bridge declares its real capabilities through an optional `describe` call; one that does not implement it is reported as `declared: false` with every capability false.
56
+
57
+ To sign in once, run `python browser/launch_chrome.py` (it defaults to `~/.web-forager/browser/profile`, prints the `WEB_FORAGER_BROWSER_CDP` value, and supports `--dry-run`), then pass `--backend cdp`. `--storage-state FILE` loads a Playwright state file before the run; `--save-storage-state FILE` writes one after it, with mode `0600` and a refusal if the path is inside `--root`. That file contains cookies and logins: treat it as a credential, keep it out of git, and do not copy it to a shared machine.
58
+
59
+ `--strict-robots` fetches `robots.txt` before every navigation and stops with `robots_denied` (exit 3) on a disallowed path. Without it the record says so: `compliance.robots_checked` is `false`. A run that reaches the site but finds nothing readable is `degraded` (exit 8) with its records attached; an anti-bot or error interstitial is `blocked` (exit 9); a site that asks for credentials is `login_required` (exit 6).
60
+
61
+ A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with `--out result.json` (or `out:` in MCP), no screenshots are taken unless asked, `--max-screens 5` and `--min-interval 3000` keep a walk polite, and a long walk is better split into runs that resume from `--session`.
62
+
63
+ ---
64
+
65
+ ## When the engines are blocked
66
+
67
+ Search defaults to `auto`. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with `--engine bing`.
68
+
69
+ Two routes need no search engine at all:
70
+
71
+ ```bash
72
+ python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
73
+ python run.py --action feed --url https://the-decoder.com/ --num-results 10
74
+ ```
75
+
76
+ `find` tries the site's own search URLs (`/?s=`, `/search?q=` and two more), ranks the links by the query and fetches the best ones. `feed` discovers the site's RSS/Atom `<link rel=alternate>` and reads it. `doctor --probe` asks every engine once and reports which ones answer from this network; `doctor` also reports the proxy in use with its credentials removed. Requests honour `HTTPS_PROXY` / `ALL_PROXY`, and `--proxy URL` overrides them for one run.
77
+
78
+ ---
79
+
80
+ ## More CLI examples
81
+
82
+ Each command has its own block so it can be copied without trimming the others.
83
+
84
+ **Fetch one Bilibili video.**
85
+
86
+ ```bash
87
+ python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
88
+ ```
89
+
90
+ **Fetch a page that needs JavaScript rendering.**
91
+
92
+ ```bash
93
+ python run.py --action fetch --url "https://www.zhihu.com/" --render auto
94
+ ```
95
+
96
+ **Surf a site and resume the trail later.**
97
+
98
+ ```bash
99
+ python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
100
+ ```
101
+
102
+ **Clean a local PDF into Markdown.**
103
+
104
+ ```bash
105
+ python run.py --action clean --input notes.pdf --format markdown
106
+ ```
107
+
108
+ **List up to 500 page URLs from a site's sitemaps.**
109
+
110
+ ```bash
111
+ python run.py --action map --url "https://example.org" --limit 500
112
+ ```
113
+
114
+ **Read the Bilibili home feed.**
115
+
116
+ ```bash
117
+ python run.py --action bili --mode feed --max-items 10
118
+ ```
119
+
120
+ ---
121
+
122
+ ## Records and output
123
+
124
+ ### One record per page
125
+
126
+ | Field | HTTP mode | Browser mode |
127
+ |---|---|---|
128
+ | `metadata`, `markdown`, `tables`, `warnings` | yes | yes |
129
+ | `trust`, `injection_flags` | yes | yes |
130
+ | `url`, `title` | yes (`url` and `final_url`) | yes |
131
+ | `status`, `content_type`, `document_kind`, `cached`, `rendered`, `render_fallback`, `fetched_at`, `chunks`, `links` | yes | no |
132
+ | `schema_version`, `captured_at`, `playbook`, `backend`, `compliance`, `viewport_evidence`, `interaction_trace` | no | yes |
133
+
134
+ `render_fallback` is `true` when the server answered with a minimal script shell and HTTP mode rendered it once with Playwright before extraction. A `fetch --query` allows at most two such renders, and each one adds several seconds.
135
+
136
+ Search results are ordered by query coverage, title match, a small domain prior, snippet freshness and engine order. The final list keeps at most two results per host, so one dominant host cannot crowd out every other source; when too few hosts answered, the overflow fills the remaining slots. A thin first pass is retried once with a simplified query (quotes and common stopwords removed, or a spaced CJK query joined); that retry appears in `engine_report` with `query_variant`.
137
+
138
+ `--core` returns the shared shape instead:
139
+ ```json
140
+ {
141
+ "schema_version": "core-1",
142
+ "source": "web-forager",
143
+ "mode": "http",
144
+ "url": "https://example.org/article",
145
+ "title": "An article",
146
+ "collected_at": "2026-11-07T10:00:00Z",
147
+ "metadata": {},
148
+ "markdown": "...",
149
+ "tables": [],
150
+ "warnings": [],
151
+ "extra": { "status": 200, "cached": false }
152
+ }
153
+ ```
154
+
155
+ `mode` is `http` or `browser`, `source` names the producer, and `extra` keeps every field the two modes do not share. The same fold is available as a library call: `engine.normalize_record()` in Python and `normalizeRecord()` in `browser/lib/record.js`.
156
+
157
+ `results` is an index, not a second copy of every page. For a search it carries `url`, `title`, `snippet`, `status`, `content_type`, `markdown_chars`, `tables` and any `warnings` or `error`; the Markdown body lives once, in `pages`, and a single `fetch` keeps it at the top level. `--full-results` (CLI) or `fullResults: true` (MCP) restores the old shape.
158
+
159
+ ### Workspace output
160
+
161
+ ```bash
162
+ python run.py --action fetch --url "https://example.org/article" --root <WS>
163
+ ```
164
+
165
+ Writes only:
166
+
167
+ ```text
168
+ <WS>/process/web-forager/web-forager_<timestamp>_<slug>.json
169
+ <WS>/process/web-forager/web-forager_<timestamp>_<slug>.md
170
+ <WS>/_broadcast/web-forager_<timestamp>_<slug>.json
171
+ ```
172
+
173
+ `material/` and `results/` belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under `<WS>/process/web-forager/browser/` and `<WS>/_broadcast/`.
174
+
175
+ ---
176
+
177
+ ---
178
+
179
+ ## Actions and exit codes
180
+
181
+ | Action | Purpose |
182
+ |---|---|
183
+ | `fetch` | one URL or keyword query → pages |
184
+ | `find` | search inside one site through the site's own search page |
185
+ | `feed` | read a site's RSS/Atom feed, discovering it from the page |
186
+ | `clean` | local or remote HTML/PDF/DOCX/XLSX/EPUB/CSV → Markdown |
187
+ | `bili` | Bilibili feed / video / related / search / surf |
188
+ | `surf` | search or seed URL, then ranked link BFS |
189
+ | `scholar` | arXiv / Crossref / PubMed / DOI records |
190
+ | `map` | page URLs from robots.txt and sitemaps |
191
+ | `doctor` | dependency report: what is installed, what to install |
192
+
193
+ | Exit code | Meaning (both modes unless overridden below) |
194
+ |---|---|
195
+ | 0 | success |
196
+ | 2 | argument or URL error |
197
+ | 3 | robots.txt denied the request, or no browser bridge configured |
198
+ | 4 | network, HTTP or navigation failure |
199
+ | 5 | extraction or capture failed |
200
+ | 6 | Playwright requested but unavailable |
201
+ | 7 | PDF requested but `pypdf` unavailable |
202
+ | 9 | verification or anti-bot page instead of content (`verification_required` in HTTP mode, `blocked` in browser mode) |
203
+ | 130 | interrupted |
204
+
205
+ Browser mode reuses that table and changes or adds only these codes:
206
+
207
+ | Exit code | Browser meaning |
208
+ |---|---|
209
+ | 2 | also: unknown backend |
210
+ | 6 | `login_required`: the site asked for credentials |
211
+ | 7 | `confirmation_required`: a write-like action was denied |
212
+ | 8 | `degraded`: the page was read, but the playbook found nothing it looks for |
213
+ | 9 | `blocked`: an anti-bot or error interstitial replaced the page |
214
+
215
+ `--agent` keeps these exit codes and prints a `{"ok":false,"error":{"code","message","hint"}}` envelope on stdout. `doctor` reports every dependency with its version or the command that installs it; a missing Playwright returns exit code 6, and a missing `pypdf` returns exit code 7.
216
+
217
+ ---
218
+
219
+ ---
220
+
221
+ ## Requirements
222
+
223
+ Python 3.10+. Core runtime: `requests`, `beautifulsoup4`, `trafilatura`. Optional: `playwright` (JS rendering), `pypdf` (PDF). `defusedxml` is used when already installed but is never required — XML with an internal DTD subset or any ENTITY declaration is rejected regardless. Browser mode additionally needs Node.js 18+ and a bridge command; `browser/bridges/playwright-bridge.js` needs `playwright` installed with npm. No third-party code is vendored; the full per-capability matrix lives in [DEPENDENCIES.md](DEPENDENCIES.md).
224
+
225
+ ---
226
+
227
+ ---
228
+
229
+ ## Repository layout
230
+
231
+ <details>
232
+ <summary>Full tree</summary>
233
+
234
+ ```text
235
+ web-forager/
236
+ ├── .claude-plugin/ Claude Code manifest + marketplace
237
+ ├── .codex-plugin/ Codex manifest + marketplace
238
+ ├── .codebuddy-plugin/ CodeBuddy manifest + marketplace
239
+ ├── .dsh-plugin/ DSH manifest + CJS entry
240
+ ├── .cursor/rules/ Cursor rule file
241
+ ├── .windsurf/rules/ Windsurf rule file
242
+ ├── .clinerules/ Cline rule file
243
+ ├── .continue/rules/ Continue rule file
244
+ ├── .roo/rules/ Roo Code rule file
245
+ ├── .github/ Copilot instructions + Actions CI
246
+ ├── AGENTS.md generic agent contract
247
+ ├── install.py one-command rule-file installer
248
+ ├── manifest.json plugin manifest
249
+ ├── .mcp.json MCP stdio registration
250
+ ├── package.json / index.js npm + dsh packaging
251
+ ├── cordis.patch.yml dsh bundle patch
252
+ ├── run.py CLI: nine actions
253
+ ├── mcp_server.py MCP stdio server (10 HTTP tools)
254
+ ├── browser/ browser mode (Node 18+)
255
+ │ ├── run.js CLI: eight read-only playbooks
256
+ │ ├── mcp_server.js MCP stdio server (4 browser tools)
257
+ │ ├── lib/ bridge backend, session, record, safety, selectors, robots, snapshot
258
+ │ ├── playbooks/ bili-* / zhihu-dig / xhs-dig / search-dig / wechat-dig
259
+ │ ├── bridges/ optional Playwright reference bridge
260
+ │ └── tests/ fake backends, real-page snapshots, live smoke
261
+ ├── SKILL.md Claude Code skill contract
262
+ ├── commands/ /fetch /surf /scholar /map /browse
263
+ ├── docs/ install commands, acknowledgements, dependency matrix
264
+ ├── scripts/
265
+ │ ├── engine.py robots, fetch, search, cache, chunks, Markdown, surf, find, feed
266
+ │ ├── pdf.py PDF text (optional pypdf)
267
+ │ ├── documents.py DOCX / XLSX / EPUB / CSV
268
+ │ ├── feeds.py RSS / Atom / RSS 1.0
269
+ │ ├── scholar.py arXiv / Crossref / PubMed / DOI
270
+ │ ├── sitemap.py sitemap discovery
271
+ │ └── bilibili.py Bilibili adapter
272
+ ├── tests/ offline HTTP fixtures
273
+ ├── requirements.txt
274
+ ├── THIRD_PARTY_LICENSES.txt
275
+ ├── CHANGELOG.md
276
+ ├── README.md · README_zh.md
277
+ └── LICENSE MIT
278
+ ```
279
+
280
+ </details>
281
+
282
+ ---
283
+
284
+ ---
285
+
286
+ ## Tests
287
+
288
+ ```bash
289
+ python -m pytest tests -q
290
+ npm run test:browser
291
+ ```
292
+
293
+ That is **228 Python tests** plus **75 browser tests**, all passing and fully offline.
294
+
295
+ <details>
296
+ <summary>What the suites cover</summary>
297
+
298
+ `conftest.py` starts a temporary local HTTP server serving robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages, a site-search page and multi-hop surf pages. The Python suite covers robots denial, extraction, tables, documents, feeds, cache, search pagination and routing, chunking, MCP, surf sessions, installer behaviour, host-rule and disclaimer checks, the shared `core-1` record, and every exit code. The browser suite uses a fake backend plus fake bridges that speak the wire protocol, so the CLI runs end to end without a browser; it covers capability declaration, robots.txt matching, the `--strict-robots` round trip, the private-address guard, the untrusted-content screen, and the degraded and blocked paths over snapshots recorded from live pages.
299
+
300
+ </details>
301
+
302
+ CI runs both suites on every push to `main`, on pull requests and on version tags: Python on Ubuntu 3.10/3.12 and Windows 3.12, browser mode on Node 20/22. A live check is opt-in and never runs in CI:
303
+
304
+ ```bash
305
+ WEB_FORAGER_LIVE=1 WEB_FORAGER_BRIDGE="node browser/bridges/playwright-bridge.js" npm run test:live
306
+ ```
307
+
308
+ ---