web-forager 0.0.0-stage → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +16 -0
- package/.claude-plugin/plugin.json +20 -0
- package/.clinerules/web-forager.md +85 -0
- package/.codebuddy-plugin/marketplace.json +14 -0
- package/.codebuddy-plugin/plugin.json +25 -0
- package/.codex-plugin/marketplace.json +12 -0
- package/.codex-plugin/mcp.json +20 -0
- package/.codex-plugin/plugin.json +15 -0
- package/.continue/rules/web-forager.md +90 -0
- package/.cursor/rules/web-forager.mdc +80 -0
- package/.dsh-plugin/index.js +11 -0
- package/.dsh-plugin/plugin.json +5 -0
- package/.github/copilot-instructions.md +85 -0
- package/.mcp.json +12 -0
- package/.roo/rules/web-forager.md +85 -0
- package/.windsurf/rules/web-forager.md +90 -0
- package/AGENTS.md +119 -0
- package/CHANGELOG.md +508 -0
- package/LICENSE +21 -0
- package/README.md +498 -2
- package/README_zh.md +498 -0
- package/SKILL.md +451 -0
- package/THIRD_PARTY_LICENSES.txt +40 -0
- package/browser/bridges/playwright-bridge.js +382 -0
- package/browser/launch_chrome.py +193 -0
- package/browser/lib/backend.js +302 -0
- package/browser/lib/bili-api.js +90 -0
- package/browser/lib/bridge.js +83 -0
- package/browser/lib/context.js +60 -0
- package/browser/lib/doctor.js +97 -0
- package/browser/lib/errors.js +17 -0
- package/browser/lib/netguard.js +149 -0
- package/browser/lib/playbooks.js +30 -0
- package/browser/lib/record.js +109 -0
- package/browser/lib/robots.js +195 -0
- package/browser/lib/safety.js +141 -0
- package/browser/lib/selectors.js +80 -0
- package/browser/lib/session.js +130 -0
- package/browser/lib/snapshot.js +229 -0
- package/browser/lib/untrusted.js +215 -0
- package/browser/lib/workspace.js +40 -0
- package/browser/mcp_server.js +252 -0
- package/browser/package.json +40 -0
- package/browser/playbooks/bili-chain.js +58 -0
- package/browser/playbooks/bili-danmaku.js +102 -0
- package/browser/playbooks/bili-feed.js +66 -0
- package/browser/playbooks/bili-video.js +38 -0
- package/browser/playbooks/search-dig.js +107 -0
- package/browser/playbooks/shared.js +138 -0
- package/browser/playbooks/wechat-dig.js +78 -0
- package/browser/playbooks/xhs-dig.js +89 -0
- package/browser/playbooks/zhihu-dig.js +89 -0
- package/browser/run.js +441 -0
- package/commands/browse.md +53 -0
- package/commands/fetch.md +29 -0
- package/commands/map.md +24 -0
- package/commands/research.md +35 -0
- package/commands/scholar.md +28 -0
- package/commands/surf.md +28 -0
- package/cordis.patch.yml +19 -0
- package/docs/ACKNOWLEDGEMENTS.md +15 -0
- package/docs/ACKNOWLEDGEMENTS_zh.md +15 -0
- package/docs/DEPENDENCIES.md +40 -0
- package/docs/INSTALL_COMMANDS.md +113 -0
- package/docs/INSTALL_COMMANDS_zh.md +113 -0
- package/docs/REFERENCE.md +349 -0
- package/docs/REFERENCE_zh.md +342 -0
- package/index.js +7 -0
- package/install.py +254 -0
- package/manifest.json +98 -0
- package/mcp_server.py +850 -0
- package/package.json +82 -6
- package/requirements.txt +15 -0
- package/run.py +908 -0
- package/scripts/__init__.py +1 -0
- package/scripts/bilibili.py +882 -0
- package/scripts/documents.py +437 -0
- package/scripts/engine.py +5141 -0
- package/scripts/feeds.py +180 -0
- package/scripts/pdf.py +156 -0
- package/scripts/research.py +2087 -0
- package/scripts/scholar.py +381 -0
- package/scripts/sitemap.py +195 -0
- package/scripts/untrusted.py +312 -0
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# 致谢
|
|
2
|
+
|
|
3
|
+
这里记录报告过问题、并因此改变了这个项目的人:谁、报告了什么、改了什么。README 只列名字,细节放在这一页。
|
|
4
|
+
|
|
5
|
+
[English](ACKNOWLEDGEMENTS.md) · **简体中文**
|
|
6
|
+
|
|
7
|
+
## NotFatJustRound
|
|
8
|
+
|
|
9
|
+
[issue #1](https://github.com/JiamingShi-JsPhys/web-forager/issues/1)。
|
|
10
|
+
|
|
11
|
+
`web_search` 把每个页面的正文返回了两遍(`pages` 与 `results` 各一份),响应体积翻倍并超出智能体的大小上限。1.0.2 起新增的精简 `results` 索引、以及之后对输出体积的处理,都来自这份报告。
|
|
12
|
+
|
|
13
|
+
## AI 辅助
|
|
14
|
+
|
|
15
|
+
Claude(Anthropic)在本项目的部分环节中提供了一些辅助,主要是代码片段、测试、文档与发布说明方面的修改建议。所有建议都经审阅,只有通过测试、检查或实际验证的才会采纳;最终内容由作者定稿。许多建议经过大幅改写,也有一些未被采用。
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Dependency matrix
|
|
2
|
+
|
|
3
|
+
`web-forager` starts with the standard library plus three runtime packages.
|
|
4
|
+
Everything else is optional and degrades with a clear exit code and hint.
|
|
5
|
+
|
|
6
|
+
| Capability | Package | Licence | Required? | Missing behaviour |
|
|
7
|
+
|---|---|---|---|---|
|
|
8
|
+
| HTTP, robots, search, surf, cache | `requests`, `certifi` | Apache-2.0, MPL-2.0 | yes | exit 1 with import error |
|
|
9
|
+
| HTML parsing and fallback Markdown | `beautifulsoup4` | MIT | yes | exit 1 with import error |
|
|
10
|
+
| Preferred article extraction | `trafilatura` | Apache-2.0 | yes | falls back to BeautifulSoup |
|
|
11
|
+
| JavaScript rendering | `playwright` | Apache-2.0 | optional | `--render always` exits 6; `auto` skips with a warning |
|
|
12
|
+
| PDF text and metadata | `pypdf` | BSD-3-Clause | optional | PDF input exits 7 with `pip install pypdf` |
|
|
13
|
+
| Hardened XML parsing | `defusedxml` | PSF-2.0 | optional | DOCTYPE internal subsets and all ENTITY declarations are rejected anyway |
|
|
14
|
+
| DOCX / XLSX / EPUB / CSV | standard library | — | built in | — |
|
|
15
|
+
| RSS / Atom / RSS 1.0 | standard library | — | built in | — |
|
|
16
|
+
| Sitemap parsing (`.xml`, `.xml.gz`) | standard library | — | built in | — |
|
|
17
|
+
| arXiv / Crossref / PubMed / DOI | `requests` only | — | built in | — |
|
|
18
|
+
| MCP stdio server (HTTP) | standard library | — | built in | — |
|
|
19
|
+
| Browser mode CLI and MCP | Node.js >= 18, standard library only | — | optional | browser runs exit 3 without a bridge |
|
|
20
|
+
| Browser bridge (reference) | `playwright` (npm) | Apache-2.0 | optional | bridge exits 3 with an install hint |
|
|
21
|
+
|
|
22
|
+
Install the required set:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
pip install -r requirements.txt
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Enable optional capabilities:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
pip install playwright && playwright install chromium # JS rendering
|
|
32
|
+
pip install pypdf # PDF text
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Browser mode needs Node.js 18 or newer and a bridge command in
|
|
36
|
+
`WEB_FORAGER_BRIDGE`. The reference bridge in `browser/bridges/` additionally
|
|
37
|
+
needs `npm install playwright`; the core browser CLI and MCP server use only the
|
|
38
|
+
Node standard library.
|
|
39
|
+
|
|
40
|
+
No GPL or AGPL package is used. See [`THIRD_PARTY_LICENSES.txt`](../THIRD_PARTY_LICENSES.txt).
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# Install commands
|
|
2
|
+
|
|
3
|
+
Every per-host command, the `install.py` flags and the optional dependencies. Methods 1 and 2 are short enough to stay in the README: [web-forager](../README.md) · [简体中文](INSTALL_COMMANDS_zh.md)
|
|
4
|
+
|
|
5
|
+
## Method 3: install with commands
|
|
6
|
+
|
|
7
|
+
**Claude Code** — from your shell (v2.1.292+), marketplace and install in one step:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
claude plugin install web-forager --marketplace https://github.com/JiamingShi-JsPhys/web-forager.git
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
**Claude Code** — from inside a session (v2.1.275+), add the marketplace from the URL first, pinned to the release tag:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
/plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.1
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Then install the entry it lists:
|
|
20
|
+
|
|
21
|
+
```text
|
|
22
|
+
/plugin install web-forager@web-forager-marketplace
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Codex** — register the URL as a Git marketplace; `--ref` pins the release tag, and an SSH URL or a local path works in place of the HTTPS one:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
codex plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git --ref v1.3.1
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Codex** — install the plugin from that marketplace:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
codex plugin add web-forager@web-forager-marketplace
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**dsh (DeepSeek Harness)** — installs from the same repository URL straight into a profile:
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
dsh plugin --profile web add github:JiamingShi-JsPhys/web-forager#v1.3.1
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
**CodeBuddy** — add the URL as a Git marketplace:
|
|
44
|
+
|
|
45
|
+
```text
|
|
46
|
+
/plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.1
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Then install the entry it lists:
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
/plugin install web-forager@web-forager-marketplace
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**Host MCP notes.** Codex reads `.codex-plugin/mcp.json`, whose relative paths resolve against the installed plugin directory rather than `${CLAUDE_PLUGIN_ROOT}`. CodeBuddy reads `mcpServers` from `.codebuddy-plugin/plugin.json`; run `/reload-plugins` or start a new session after install, because `codebuddy mcp list` can run before plugins are mounted and show none. dsh registers both servers through `@deepseek-ai/dsh-mcp-client` in `cordis.patch.yml`; start a new profile session after install. All three expose the same ten HTTP tools and four browser tools.
|
|
56
|
+
|
|
57
|
+
Cursor, Windsurf, Cline's IDE extension, Continue, Roo Code, GitHub Copilot and opencode each read a rule file from their own location, and none of them accepts this repository as a plugin — Cline's `cline plugin install` takes only JS/TS plugin packages, Cursor's plugin format is a different bundle, and `opencode plugin <module>` installs npm modules. **No vendor ships an installer that understands another vendor's rule file**, so this repository ships one:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
git clone https://github.com/JiamingShi-JsPhys/web-forager.git && python web-forager/install.py --project <project> --global
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
That single command writes every rule file at once:
|
|
64
|
+
|
|
65
|
+
- `--project <project>` covers the project-scoped files — Cursor, Windsurf, Cline, Continue, Roo Code, GitHub Copilot, and the generic `AGENTS.md` that Codex, Cursor, Copilot agents and opencode all read.
|
|
66
|
+
- `--global` covers the user-scoped files — `~/.config/opencode/AGENTS.md`, `$CODEX_HOME/AGENTS.md` (default `~/.codex/AGENTS.md`), and, for Claude Code, a marked block appended to `~/.claude/CLAUDE.md` when `~/.claude` already exists — and skips any host this machine does not have.
|
|
67
|
+
|
|
68
|
+
Re-running it is safe: identical files report `unchanged`, and a rule file that has drifted is left alone unless you pass `--force`.
|
|
69
|
+
|
|
70
|
+
List every destination without writing anything:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
python web-forager/install.py --list
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Preview the run:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
python web-forager/install.py --project <project> --global --dry-run
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Install for one host only:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
python web-forager/install.py --project <project> --only cursor,windsurf
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Keep the cloned repository somewhere the agent can reach: every rule file names `run.py`, `mcp_server.py` and `SKILL.md`, so the host agent calls the same engine as a plugin host. The rule file describes *how to call* the tool — it does not vendor the code.
|
|
89
|
+
|
|
90
|
+
**opencode without copying** — opencode can also read the rule file in place through `opencode.json`'s `instructions` array, which needs no copy and can be committed for a whole team:
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
{
|
|
94
|
+
"$schema": "https://opencode.ai/config.json",
|
|
95
|
+
"instructions": ["../web-forager/AGENTS.md"]
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**Manual alternative** — if you would rather not run the script, copy the file yourself to the destination the script would have used:
|
|
100
|
+
|
|
101
|
+
```text
|
|
102
|
+
<clone>/AGENTS.md → <project>/AGENTS.md
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
## Optional capabilities
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
pip install -r requirements.txt # core: requests, bs4, trafilatura
|
|
109
|
+
pip install playwright && playwright install chromium # JS rendering
|
|
110
|
+
pip install pypdf # PDF text
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
DOCX/XLSX/EPUB/CSV, RSS/Atom, sitemap parsing and both MCP servers need **no extra Python package**. Browser mode needs Node.js 18 or newer; a browser bridge is supplied by you, and `browser/bridges/playwright-bridge.js` additionally needs `npm install playwright`. See [docs/DEPENDENCIES.md](DEPENDENCIES.md).
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# 安装命令
|
|
2
|
+
|
|
3
|
+
各宿主的完整命令、`install.py` 的全部参数与可选依赖。方法一与方法二较短,留在 README:[web-forager](../README_zh.md) · [English](INSTALL_COMMANDS.md)
|
|
4
|
+
|
|
5
|
+
## 方法三:用命令安装
|
|
6
|
+
|
|
7
|
+
**Claude Code** —— 在终端执行(v2.1.292+),一步完成注册与安装:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
claude plugin install web-forager --marketplace https://github.com/JiamingShi-JsPhys/web-forager.git
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
**Claude Code** —— 在会话内执行(v2.1.275+),先用 URL 添加市场,并锁定发布 tag:
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
/plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.1
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
再安装它列出的条目:
|
|
20
|
+
|
|
21
|
+
```text
|
|
22
|
+
/plugin install web-forager@web-forager-marketplace
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Codex** —— 把 URL 注册为 Git 市场;`--ref` 锁定发布 tag,HTTPS 换成 SSH 或本地路径同样可用:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
codex plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git --ref v1.3.1
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Codex** —— 从该市场安装插件:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
codex plugin add web-forager@web-forager-marketplace
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**dsh(DeepSeek Harness)** —— 直接从同一个仓库 URL 安装进指定 profile:
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
dsh plugin --profile web add github:JiamingShi-JsPhys/web-forager#v1.3.1
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
**CodeBuddy** —— 先把 URL 加为 Git 市场:
|
|
44
|
+
|
|
45
|
+
```text
|
|
46
|
+
/plugin marketplace add https://github.com/JiamingShi-JsPhys/web-forager.git#v1.3.1
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
再安装它列出的条目:
|
|
50
|
+
|
|
51
|
+
```text
|
|
52
|
+
/plugin install web-forager@web-forager-marketplace
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**各宿主的 MCP 说明。** Codex 读取 `.codex-plugin/mcp.json`,其中的相对路径按已安装插件目录解析,不使用 `${CLAUDE_PLUGIN_ROOT}`。CodeBuddy 从 `.codebuddy-plugin/plugin.json` 读取 `mcpServers`;安装后运行 `/reload-plugins` 或新开会话,因为 `codebuddy mcp list` 可能在插件挂载前运行而显示为空。dsh 通过 `cordis.patch.yml` 里的 `@deepseek-ai/dsh-mcp-client` 注册两个服务;安装后新开 profile 会话。三者的工具集相同:十个 HTTP 工具与四个浏览器工具。
|
|
56
|
+
|
|
57
|
+
Cursor、Windsurf、Cline 的 IDE 扩展、Continue、Roo Code、GitHub Copilot 与 opencode 各自从自己的位置读取规则文件,而且它们都不接受本仓库作为插件——Cline 的 `cline plugin install` 只接受 JS/TS 插件包,Cursor 的插件格式是另一套打包,`opencode plugin <module>` 装的是 npm 模块。**没有任何厂商提供能识别别家规则文件的安装器**,所以本仓库自带一个:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
git clone https://github.com/JiamingShi-JsPhys/web-forager.git && python web-forager/install.py --project <项目> --global
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
这一条命令会一次性写好全部规则文件:
|
|
64
|
+
|
|
65
|
+
- `--project <项目>` 负责项目级文件——Cursor、Windsurf、Cline、Continue、Roo Code、GitHub Copilot,以及 Codex、Cursor、Copilot Agent 与 opencode 都会读的通用 `AGENTS.md`。
|
|
66
|
+
- `--global` 负责用户级文件——`~/.config/opencode/AGENTS.md`、`$CODEX_HOME/AGENTS.md`(默认 `~/.codex/AGENTS.md`),以及对 Claude Code 在 `~/.claude` 已存在时向 `~/.claude/CLAUDE.md` 追加的带标记块——并自动跳过本机没有安装的宿主。
|
|
67
|
+
|
|
68
|
+
重复运行是安全的:内容相同的文件报告 `unchanged`,已经改动过的规则文件在你不加 `--force` 时保持原样。
|
|
69
|
+
|
|
70
|
+
先看清单,不写任何文件:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
python web-forager/install.py --list
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
预演一次:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
python web-forager/install.py --project <项目> --global --dry-run
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
只装某一个宿主:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
python web-forager/install.py --project <项目> --only cursor,windsurf
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
克隆下来的仓库要放在 Agent 能访问到的位置:每个规则文件都点名了 `run.py`、`mcp_server.py` 与 `SKILL.md`,宿主 Agent 因此可以像插件宿主一样调用同一个引擎。规则文件只描述“怎么调用”,并不内置代码。
|
|
89
|
+
|
|
90
|
+
**opencode 也可以完全不复制** —— 用 `opencode.json` 的 `instructions` 数组原地指向规则文件,无需复制,还能随项目提交给整个团队:
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
{
|
|
94
|
+
"$schema": "https://opencode.ai/config.json",
|
|
95
|
+
"instructions": ["../web-forager/AGENTS.md"]
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**不想跑脚本时的手动方式** —— 自己复制,目标路径就是脚本会写入的那一处:
|
|
100
|
+
|
|
101
|
+
```text
|
|
102
|
+
<克隆目录>/AGENTS.md → <项目>/AGENTS.md
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
## 可选能力
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
pip install -r requirements.txt # 核心:requests、bs4、trafilatura
|
|
109
|
+
pip install playwright && playwright install chromium # JS 渲染
|
|
110
|
+
pip install pypdf # PDF 文本
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
DOCX/XLSX/EPUB/CSV、RSS/Atom、sitemap 解析与两个 MCP 服务**无需额外 Python 依赖**。浏览器模式需要 Node.js 18 或更新;浏览器桥由你提供,`browser/bridges/playwright-bridge.js` 另需 `npm install playwright`。详见 [docs/DEPENDENCIES.md](DEPENDENCIES.md)。
|
|
@@ -0,0 +1,349 @@
|
|
|
1
|
+
# web-forager reference
|
|
2
|
+
|
|
3
|
+
> Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
|
|
4
|
+
|
|
5
|
+
**English** · [简体中文](REFERENCE_zh.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## MCP servers
|
|
10
|
+
|
|
11
|
+
<details>
|
|
12
|
+
<summary>Tool names</summary>
|
|
13
|
+
|
|
14
|
+
HTTP tools: `web_fetch`, `web_search`, `web_research`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_doctor`, `web_find`, `web_feed`. Browser tools: `browser_list`, `browser_run`, `browser_session`, `browser_doctor`.
|
|
15
|
+
|
|
16
|
+
MCP calls are always read-only and keep the target-page boundary: no tool parameter can change `respect_robots`, `robots_policy`, `input_gate`, the proxy, the browser bridge or a write switch, and the local file `clean` action is not exposed over MCP. Search-engine and scholarly query endpoints have a separate user-level policy (`--query-robots` on the CLI, `WEB_FORAGER_QUERY_ROBOTS` in the server environment), default `warn`: a disallowed query path is read and the engine report names the override; `respect` refuses it. The agent cannot change this through a tool call.
|
|
17
|
+
|
|
18
|
+
</details>
|
|
19
|
+
|
|
20
|
+
Test the protocol directly:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
|
24
|
+
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
|
29
|
+
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Browser mode
|
|
33
|
+
|
|
34
|
+
The browser layer lives in `browser/` and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless `--allow-write` is passed after the user authorises that single action.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
npm install playwright && npx playwright install chromium
|
|
38
|
+
node browser/run.js --doctor
|
|
39
|
+
node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
|
|
40
|
+
node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
|
|
41
|
+
node browser/run.js --playbook wechat-dig --url "https://mp.weixin.qq.com/s/..." --agent
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
| Playbook | Does | Requires |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| `bili-feed` | Home feed, scroll and capture unseen cards | none |
|
|
47
|
+
| `bili-video` | One video and its visible comments | `--url` or `--bvid` |
|
|
48
|
+
| `bili-chain` | Follow the next-episode control and resume from the session | `--url`, `--bvid`, or a chain |
|
|
49
|
+
| `bili-danmaku` | Timestamped danmaku, preferring bridge `observe` over DOM text (experimental) | `--url` or `--bvid` |
|
|
50
|
+
| `zhihu-dig` | Zhihu search, rank links, capture the best unseen page | `--query` |
|
|
51
|
+
| `xhs-dig` | Xiaohongshu public notes, rank `/explore` links, capture one | `--query` |
|
|
52
|
+
| `search-dig` | Bilibili search, rank cards, open the best unseen result | `--query` |
|
|
53
|
+
| `wechat-dig` | One WeChat article; reports the environment-check wall instead of recycling it | `--url` |
|
|
54
|
+
|
|
55
|
+
WeChat articles are the one case HTTP mode cannot finish. `mp.weixin.qq.com` disallows `/s` in robots.txt, and even with `--robots-policy warn` the article URL answers with an environment-check page or a captcha redirect instead of the body; the HTTP record reports `verification_required` (exit 9) rather than handing back the error page as content. `wechat-dig` opens the same URL in a real browser profile. If that profile has already passed WeChat's check, the article is captured; if not, the playbook reports `blocked` and does not attempt the challenge.
|
|
56
|
+
|
|
57
|
+
The bundled bridge is picked up automatically once `playwright` and its Chromium binary are installed; `--bridge` or `WEB_FORAGER_BRIDGE` points at a bridge elsewhere. The command is split on whitespace and started without a shell, so quote a path that contains spaces and pass arguments as usual; pipes, `&&` and redirection are not interpreted. `WEB_FORAGER_BROWSER_HEADFUL=1` shows the window, `WEB_FORAGER_BROWSER_PROFILE=/path` reuses a persistent profile (this is what carries login state for the `extension` backend), and `WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222` attaches to a browser you already control (`cdp`). A bridge declares its real capabilities through an optional `describe` call; one that does not implement it is reported as `declared: false` with every capability false.
|
|
58
|
+
|
|
59
|
+
To sign in once, run `python browser/launch_chrome.py` (it defaults to `~/.web-forager/browser/profile`, and CDP is **off** by default; `--cdp` opts in to a loopback CDP port, random by default or fixed with `--port 9222`). Without `--cdp`, run browser mode with the default backend and it reuses the profile. With `--cdp`, pass `--backend cdp`. CDP itself has no authentication: keep the port local and close the browser when finished. `--storage-state FILE` loads a Playwright state file before the run; `--save-storage-state FILE` writes one after it, with mode `0600` and a refusal if the path is inside `--root`. That file contains cookies and logins: treat it as a credential, keep it out of git, and do not copy it to a shared machine.
|
|
60
|
+
|
|
61
|
+
`--strict-robots` fetches `robots.txt` before every navigation and stops with `robots_denied` (exit 3) on a disallowed path. Without it the record says so: `compliance.robots_checked` is `false`. A run that reaches the site but finds nothing readable is `degraded` (exit 8) with its records attached; an anti-bot or error interstitial is `blocked` (exit 9); a site that asks for credentials is `login_required` (exit 6).
|
|
62
|
+
|
|
63
|
+
A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with `--out result.json` (or `out:` in MCP), no screenshots are taken unless asked, `--max-screens 5` and `--min-interval 3000` keep a walk polite, and a long walk is better split into runs that resume from `--session`.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## When the engines are blocked
|
|
68
|
+
|
|
69
|
+
Search defaults to `auto`. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with `--engine bing`.
|
|
70
|
+
|
|
71
|
+
Two routes need no search engine at all:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
|
|
75
|
+
python run.py --action feed --url https://the-decoder.com/ --num-results 10
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
`find` tries the site's own search URLs (`/?s=`, `/search?q=` and two more), ranks the links by the query and fetches the best ones. `feed` discovers the site's RSS/Atom `<link rel=alternate>` and reads it. `doctor --probe` asks every engine once and reports which ones answer from this network; `doctor` also reports the proxy in use with its credentials removed. Requests honour `HTTPS_PROXY` / `ALL_PROXY`, and `--proxy URL` overrides them for one run.
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## More CLI examples
|
|
83
|
+
|
|
84
|
+
Each command has its own block so it can be copied without trimming the others.
|
|
85
|
+
|
|
86
|
+
**Fetch one Bilibili video.**
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
**Fetch a page that needs JavaScript rendering.**
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
python run.py --action fetch --url "https://www.zhihu.com/" --render auto
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
**Surf a site and resume the trail later.**
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**Clean a local PDF into Markdown.**
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
python run.py --action clean --input notes.pdf --format markdown
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
**List up to 500 page URLs from a site's sitemaps.**
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
python run.py --action map --url "https://example.org" --limit 500
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
**Read the Bilibili home feed.**
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
python run.py --action bili --mode feed --max-items 10
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## Records and output
|
|
125
|
+
|
|
126
|
+
### One record per page
|
|
127
|
+
|
|
128
|
+
| Field | HTTP mode | Browser mode |
|
|
129
|
+
|---|---|---|
|
|
130
|
+
| `metadata`, `markdown`, `tables`, `warnings` | yes | yes |
|
|
131
|
+
| `trust`, `injection_flags`, `injection_findings` | yes | yes |
|
|
132
|
+
| `url`, `title` | yes (`url` and `final_url`) | yes |
|
|
133
|
+
| `status`, `content_type`, `document_kind`, `cached`, `rendered`, `render_fallback`, `fetched_at`, `chunks`, `links` | yes | no |
|
|
134
|
+
| `schema_version`, `captured_at`, `playbook`, `backend`, `compliance`, `viewport_evidence`, `interaction_trace` | no | yes |
|
|
135
|
+
|
|
136
|
+
`render_fallback` is `true` when the server answered with a minimal script shell and HTTP mode rendered it once with Playwright before extraction. A `fetch --query` allows at most two such renders, and each one adds several seconds.
|
|
137
|
+
|
|
138
|
+
`injection_findings` is a list of `{code, severity, confidence, span, snippet, context}`. `span` is a character range in the scanned text, `severity` is `hard` or `soft`, and `context` is `direct`, `quote` or `example`. Hard rules describe an action (forged system tag, secret exfiltration, `curl ... | sh`, `rm -rf /`); `--input-gate strict` withholds a body only when one matches. Soft rules describe override or role-swap language and never withhold on their own, because papers and security notes quote them. The scanner walks the whole body in overlapping windows, before `max_chars` truncation, and the first warning repeats up to three matching snippets.
|
|
139
|
+
|
|
140
|
+
Search results are ordered by query coverage, phrase and title match, a small domain prior, snippet freshness and engine order — including when only one engine answered. Each result carries `coverage`, `matched_terms`, `relevance` and `language_match`. A result is relevant when it contains the exact phrase or at least two meaningful query terms with half the query covered; a wrong-language page that repeats one product name is `language_match: false`. Off-topic rows are ranked last, and are dropped when at least three relevant results exist; otherwise a warning names how many were kept. The final list keeps at most two results per host, so one dominant host cannot crowd out every other source; when too few hosts answered, the overflow fills the remaining slots. A thin first pass is retried once with a simplified query (quotes and common stopwords removed, or a spaced CJK query joined); that retry appears in `engine_report` with `query_variant`. Search titles and snippets pass through the same injection scanner, and a matching result carries its own `injection_flags` and `injection_findings`.
|
|
141
|
+
|
|
142
|
+
Before ranking, duplicate URLs are canonicalized: `utm_*`, `fbclid`, fragments, `www.`, trailing slashes and `/amp` no longer make one article look like two results. A result found by more than one engine receives a reciprocal-rank-fusion contribution, and an exact title match gets a small bonus over a title that merely contains the phrase. Light English stemming is used for matching only, so `injection` meets `injections` without changing the displayed query or text. The search payload also carries `search_quality: {query, engine, returned, relevant, off_topic, language_mismatch, engines_answered, engine_errors, dropped_off_topic, returned_before_filter}`, a machine-readable summary for agents that do not want to parse the prose warnings.
|
|
143
|
+
|
|
144
|
+
`--core` returns the shared shape instead:
|
|
145
|
+
```json
|
|
146
|
+
{
|
|
147
|
+
"schema_version": "core-1",
|
|
148
|
+
"source": "web-forager",
|
|
149
|
+
"mode": "http",
|
|
150
|
+
"url": "https://example.org/article",
|
|
151
|
+
"title": "An article",
|
|
152
|
+
"collected_at": "2026-11-07T10:00:00Z",
|
|
153
|
+
"metadata": {},
|
|
154
|
+
"markdown": "...",
|
|
155
|
+
"tables": [],
|
|
156
|
+
"warnings": [],
|
|
157
|
+
"extra": { "status": 200, "cached": false }
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
`mode` is `http` or `browser`, `source` names the producer, and `extra` keeps every field the two modes do not share. The same fold is available as a library call: `engine.normalize_record()` in Python and `normalizeRecord()` in `browser/lib/record.js`.
|
|
162
|
+
|
|
163
|
+
`results` is an index, not a second copy of every page. For a search it carries `url`, `title`, `snippet`, `status`, `content_type`, `markdown_chars`, `tables` and any `warnings` or `error`; the Markdown body lives once, in `pages`, and a single `fetch` keeps it at the top level. `--full-results` (CLI) or `fullResults: true` (MCP) restores the old shape.
|
|
164
|
+
|
|
165
|
+
### Workspace output
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
python run.py --action fetch --url "https://example.org/article" --root <WS>
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Writes only:
|
|
172
|
+
|
|
173
|
+
```text
|
|
174
|
+
<WS>/process/web-forager/web-forager_<timestamp>_<slug>.json
|
|
175
|
+
<WS>/process/web-forager/web-forager_<timestamp>_<slug>.md
|
|
176
|
+
<WS>/_broadcast/web-forager_<timestamp>_<slug>.json
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`material/` and `results/` belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under `<WS>/process/web-forager/browser/` and `<WS>/_broadcast/`.
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Research output
|
|
186
|
+
|
|
187
|
+
`research --query` runs two stages: broad recall across the free engines, then
|
|
188
|
+
focused reading of a few distinct hosts. The payload adds:
|
|
189
|
+
|
|
190
|
+
| Field | Meaning |
|
|
191
|
+
|---|---|
|
|
192
|
+
| `intent` | `{intent, confidence, signals}` from a deterministic keyword classifier |
|
|
193
|
+
| `strategy` | `{breadth, search_pages, targeted_site, targeted_sites, targeted_by_site, official_entries, focus_*}` |
|
|
194
|
+
| `seeds` | selected results with `authority_tier`, `authority_score`, `reason` |
|
|
195
|
+
| `evidence` | `[{url, title, authority_tier, authors, published_at, date_source, passages, partial?}]` |
|
|
196
|
+
| `freshness` | `{newest, oldest, dated_sources, undated_sources, newest_age_days, all_stale}` |
|
|
197
|
+
| `answerability` | `{level, coverage, authority_level, partial_only, source_count, host_count, view_hosts, multi_view, stale, reason}` |
|
|
198
|
+
| `skipped` | pages not used, with `code` (`robots_denied`, `browser_required`, `browser_required_low_value`, `no_matching_passage`, …) |
|
|
199
|
+
|
|
200
|
+
If the broad pool has no relevant primary/official/academic source, the second
|
|
201
|
+
pass prefers a direct official entry page (Python asyncio, Go concurrency, Rust,
|
|
202
|
+
MDN, Microsoft Wi-Fi support), then the site's own search page (Microsoft
|
|
203
|
+
Support, MDN), and only then a `site:` engine query for a non-CJK question.
|
|
204
|
+
Direct entries and host search pages do not depend on an engine honouring
|
|
205
|
+
`site:`. A news query with no usable engine rows falls back to a public finance
|
|
206
|
+
RSS channel (China News Service); a dated headline and summary are labelled
|
|
207
|
+
`feed_headline`, never expanded into a fabricated article. When the article body
|
|
208
|
+
has no dateline, the fetched page inherits the feed's publication date. Each
|
|
209
|
+
focus fetch is capped at 8 seconds with no retry, and robots.txt at 6 seconds,
|
|
210
|
+
so one dead origin cannot consume the whole call. A comparison needs at
|
|
211
|
+
least two independent full-source viewpoints;
|
|
212
|
+
a WeChat snippet never counts as one. When two full authoritative sources are
|
|
213
|
+
already present for a comparison or technical question, a WeChat-only snippet is
|
|
214
|
+
skipped rather than mixed in. A WeChat link that HTTP mode cannot read still
|
|
215
|
+
yields a `partial` search-snippet record with an explicit warning, never a
|
|
216
|
+
fabricated full article. Every passage is quoted, untrusted page text.
|
|
217
|
+
|
|
218
|
+
## Actions and exit codes
|
|
219
|
+
|
|
220
|
+
| Action | Purpose |
|
|
221
|
+
|---|---|
|
|
222
|
+
| `fetch` | one URL or keyword query → pages |
|
|
223
|
+
| `research` | broad recall → focused, cited evidence passages |
|
|
224
|
+
| `find` | search inside one site through the site's own search page |
|
|
225
|
+
| `feed` | read a site's RSS/Atom feed, discovering it from the page |
|
|
226
|
+
| `clean` | local or remote HTML/PDF/DOCX/XLSX/EPUB/CSV → Markdown |
|
|
227
|
+
| `bili` | Bilibili feed / video / related / search / surf |
|
|
228
|
+
| `surf` | search or seed URL, then ranked link BFS |
|
|
229
|
+
| `scholar` | arXiv / Crossref / PubMed / DOI records |
|
|
230
|
+
| `map` | page URLs from robots.txt and sitemaps |
|
|
231
|
+
| `doctor` | dependency report: what is installed, what to install |
|
|
232
|
+
|
|
233
|
+
| Exit code | Meaning (both modes unless overridden below) |
|
|
234
|
+
|---|---|
|
|
235
|
+
| 0 | success |
|
|
236
|
+
| 2 | argument or URL error |
|
|
237
|
+
| 3 | robots.txt denied the request, or no browser bridge configured |
|
|
238
|
+
| 4 | network, HTTP or navigation failure |
|
|
239
|
+
| 5 | extraction or capture failed |
|
|
240
|
+
| 6 | Playwright requested but unavailable |
|
|
241
|
+
| 7 | PDF requested but `pypdf` unavailable |
|
|
242
|
+
| 9 | verification or anti-bot page instead of content (`verification_required` in HTTP mode, `blocked` in browser mode) |
|
|
243
|
+
| 130 | interrupted |
|
|
244
|
+
|
|
245
|
+
Browser mode reuses that table and changes or adds only these codes:
|
|
246
|
+
|
|
247
|
+
| Exit code | Browser meaning |
|
|
248
|
+
|---|---|
|
|
249
|
+
| 2 | also: unknown backend |
|
|
250
|
+
| 6 | `login_required`: the site asked for credentials |
|
|
251
|
+
| 7 | `confirmation_required`: a write-like action was denied |
|
|
252
|
+
| 8 | `degraded`: the page was read, but the playbook found nothing it looks for |
|
|
253
|
+
| 9 | `blocked`: an anti-bot or error interstitial replaced the page |
|
|
254
|
+
|
|
255
|
+
`--agent` keeps these exit codes and prints a `{"ok":false,"error":{"code","message","hint"}}` envelope on stdout. `doctor` reports every dependency with its version or the command that installs it; a missing Playwright returns exit code 6, and a missing `pypdf` returns exit code 7.
|
|
256
|
+
|
|
257
|
+
---
|
|
258
|
+
|
|
259
|
+
---
|
|
260
|
+
|
|
261
|
+
## Requirements
|
|
262
|
+
|
|
263
|
+
Python 3.10+. Core runtime: `requests`, `beautifulsoup4`, `trafilatura`. Optional: `playwright` (JS rendering), `pypdf` (PDF). `defusedxml` is used when already installed but is never required — XML with an internal DTD subset or any ENTITY declaration is rejected regardless. Browser mode additionally needs Node.js 18+ and a bridge command; `browser/bridges/playwright-bridge.js` needs `playwright` installed with npm. No third-party code is vendored; the full per-capability matrix lives in [DEPENDENCIES.md](DEPENDENCIES.md).
|
|
264
|
+
|
|
265
|
+
---
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Repository layout
|
|
270
|
+
|
|
271
|
+
<details>
|
|
272
|
+
<summary>Full tree</summary>
|
|
273
|
+
|
|
274
|
+
```text
|
|
275
|
+
web-forager/
|
|
276
|
+
├── .claude-plugin/ Claude Code manifest + marketplace
|
|
277
|
+
├── .codex-plugin/ Codex manifest + marketplace
|
|
278
|
+
├── .codebuddy-plugin/ CodeBuddy manifest + marketplace
|
|
279
|
+
├── .dsh-plugin/ DSH manifest + CJS entry
|
|
280
|
+
├── .cursor/rules/ Cursor rule file
|
|
281
|
+
├── .windsurf/rules/ Windsurf rule file
|
|
282
|
+
├── .clinerules/ Cline rule file
|
|
283
|
+
├── .continue/rules/ Continue rule file
|
|
284
|
+
├── .roo/rules/ Roo Code rule file
|
|
285
|
+
├── .github/ Copilot instructions + Actions CI
|
|
286
|
+
├── AGENTS.md generic agent contract
|
|
287
|
+
├── install.py one-command rule-file installer
|
|
288
|
+
├── manifest.json plugin manifest
|
|
289
|
+
├── .mcp.json MCP stdio registration
|
|
290
|
+
├── package.json / index.js npm + dsh packaging
|
|
291
|
+
├── cordis.patch.yml dsh bundle patch
|
|
292
|
+
├── run.py CLI: nine actions
|
|
293
|
+
├── mcp_server.py MCP stdio server (10 HTTP tools)
|
|
294
|
+
├── browser/ browser mode (Node 18+)
|
|
295
|
+
│ ├── run.js CLI: eight read-only playbooks
|
|
296
|
+
│ ├── mcp_server.js MCP stdio server (4 browser tools)
|
|
297
|
+
│ ├── lib/ bridge backend, session, record, safety, selectors, robots, snapshot
|
|
298
|
+
│ ├── playbooks/ bili-* / zhihu-dig / xhs-dig / search-dig / wechat-dig
|
|
299
|
+
│ ├── bridges/ optional Playwright reference bridge
|
|
300
|
+
│ └── tests/ fake backends, real-page snapshots, live smoke
|
|
301
|
+
├── SKILL.md Claude Code skill contract
|
|
302
|
+
├── commands/ /fetch /research /surf /scholar /map /browse
|
|
303
|
+
├── docs/ install commands, acknowledgements, dependency matrix
|
|
304
|
+
├── scripts/
|
|
305
|
+
│ ├── engine.py robots, fetch, search, cache, chunks, Markdown, surf, find, feed
|
|
306
|
+
│ ├── research.py broad recall, authority selection, cited evidence passages
|
|
307
|
+
│ ├── pdf.py PDF text (optional pypdf)
|
|
308
|
+
│ ├── documents.py DOCX / XLSX / EPUB / CSV
|
|
309
|
+
│ ├── feeds.py RSS / Atom / RSS 1.0
|
|
310
|
+
│ ├── scholar.py arXiv / Crossref / PubMed / DOI
|
|
311
|
+
│ ├── sitemap.py sitemap discovery
|
|
312
|
+
│ └── bilibili.py Bilibili adapter
|
|
313
|
+
├── tests/ offline HTTP fixtures
|
|
314
|
+
├── requirements.txt
|
|
315
|
+
├── THIRD_PARTY_LICENSES.txt
|
|
316
|
+
├── CHANGELOG.md
|
|
317
|
+
├── README.md · README_zh.md
|
|
318
|
+
└── LICENSE MIT
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
</details>
|
|
322
|
+
|
|
323
|
+
---
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
## Tests
|
|
328
|
+
|
|
329
|
+
```bash
|
|
330
|
+
python -m pytest tests -q
|
|
331
|
+
npm run test:browser
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
That is **333 Python tests** plus **91 browser tests**, all passing and fully offline.
|
|
335
|
+
|
|
336
|
+
<details>
|
|
337
|
+
<summary>What the suites cover</summary>
|
|
338
|
+
|
|
339
|
+
`conftest.py` starts a temporary local HTTP server serving robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages, a site-search page and multi-hop surf pages. The Python suite covers robots denial, extraction, tables, documents, feeds, cache, search pagination and routing, chunking, MCP, surf sessions, installer behaviour, host-rule and disclaimer checks, the shared `core-1` record, and every exit code. The browser suite uses a fake backend plus fake bridges that speak the wire protocol, so the CLI runs end to end without a browser; it covers capability declaration, robots.txt matching, the `--strict-robots` round trip, the private-address guard, the untrusted-content screen, and the degraded and blocked paths over snapshots recorded from live pages.
|
|
340
|
+
|
|
341
|
+
</details>
|
|
342
|
+
|
|
343
|
+
CI runs both suites on every push to `main`, on pull requests and on version tags: Python on Ubuntu 3.10/3.12 and Windows 3.12, browser mode on Node 20/22. A live check is opt-in and never runs in CI:
|
|
344
|
+
|
|
345
|
+
```bash
|
|
346
|
+
WEB_FORAGER_LIVE=1 WEB_FORAGER_BRIDGE="node browser/bridges/playwright-bridge.js" npm run test:live
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
---
|