web-forager 1.3.0 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.clinerules/web-forager.md +3 -3
- package/.codebuddy-plugin/marketplace.json +1 -1
- package/.codebuddy-plugin/plugin.json +1 -1
- package/.codex-plugin/marketplace.json +1 -1
- package/.codex-plugin/mcp.json +20 -20
- package/.codex-plugin/plugin.json +1 -1
- package/.continue/rules/web-forager.md +4 -4
- package/.cursor/rules/web-forager.mdc +4 -4
- package/.dsh-plugin/index.js +1 -1
- package/.dsh-plugin/plugin.json +1 -1
- package/.github/copilot-instructions.md +3 -3
- package/.roo/rules/web-forager.md +3 -3
- package/.windsurf/rules/web-forager.md +4 -4
- package/AGENTS.md +10 -3
- package/CHANGELOG.md +508 -454
- package/README.md +499 -477
- package/README_zh.md +498 -477
- package/SKILL.md +115 -15
- package/THIRD_PARTY_LICENSES.txt +5 -0
- package/browser/bridges/playwright-bridge.js +8 -1
- package/browser/launch_chrome.py +193 -144
- package/browser/lib/bili-api.js +12 -1
- package/browser/lib/safety.js +27 -1
- package/browser/lib/session.js +16 -2
- package/browser/lib/untrusted.js +215 -56
- package/browser/mcp_server.js +5 -25
- package/browser/package.json +1 -1
- package/browser/playbooks/search-dig.js +3 -1
- package/browser/playbooks/wechat-dig.js +78 -78
- package/browser/run.js +19 -10
- package/commands/research.md +35 -0
- package/cordis.patch.yml +19 -19
- package/docs/DEPENDENCIES.md +1 -1
- package/docs/INSTALL_COMMANDS.md +4 -4
- package/docs/INSTALL_COMMANDS_zh.md +4 -4
- package/docs/REFERENCE.md +349 -308
- package/docs/REFERENCE_zh.md +342 -309
- package/manifest.json +3 -2
- package/mcp_server.py +383 -130
- package/package.json +4 -2
- package/requirements.txt +7 -4
- package/run.py +61 -5
- package/scripts/bilibili.py +11 -3
- package/scripts/engine.py +1039 -118
- package/scripts/pdf.py +21 -0
- package/scripts/research.py +2087 -0
- package/scripts/scholar.py +33 -6
- package/scripts/sitemap.py +20 -5
- package/scripts/untrusted.py +312 -95
package/docs/REFERENCE.md
CHANGED
|
@@ -1,308 +1,349 @@
|
|
|
1
|
-
# web-forager reference
|
|
2
|
-
|
|
3
|
-
> Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
|
|
4
|
-
|
|
5
|
-
**English** · [简体中文](REFERENCE_zh.md)
|
|
6
|
-
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
## MCP servers
|
|
10
|
-
|
|
11
|
-
<details>
|
|
12
|
-
<summary>Tool names</summary>
|
|
13
|
-
|
|
14
|
-
HTTP tools: `web_fetch`, `web_search`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
node browser/run.js --
|
|
39
|
-
node browser/run.js --playbook
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
|
45
|
-
|
|
46
|
-
| `bili-
|
|
47
|
-
| `bili-
|
|
48
|
-
| `
|
|
49
|
-
| `
|
|
50
|
-
| `
|
|
51
|
-
| `
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
```
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
|
129
|
-
|
|
130
|
-
| `
|
|
131
|
-
| `
|
|
132
|
-
| `
|
|
133
|
-
|
|
134
|
-
`
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
`--
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
"
|
|
148
|
-
"
|
|
149
|
-
"
|
|
150
|
-
"
|
|
151
|
-
"
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
```
|
|
168
|
-
<WS
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
|
194
|
-
|
|
195
|
-
|
|
|
196
|
-
|
|
|
197
|
-
|
|
|
198
|
-
|
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
├──
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
npm
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
1
|
+
# web-forager reference
|
|
2
|
+
|
|
3
|
+
> Part of [web-forager](../README.md). This page holds the lookup material: records, actions, exit codes, requirements, repository layout and tests.
|
|
4
|
+
|
|
5
|
+
**English** · [简体中文](REFERENCE_zh.md)
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## MCP servers
|
|
10
|
+
|
|
11
|
+
<details>
|
|
12
|
+
<summary>Tool names</summary>
|
|
13
|
+
|
|
14
|
+
HTTP tools: `web_fetch`, `web_search`, `web_research`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_doctor`, `web_find`, `web_feed`. Browser tools: `browser_list`, `browser_run`, `browser_session`, `browser_doctor`.
|
|
15
|
+
|
|
16
|
+
MCP calls are always read-only and keep the target-page boundary: no tool parameter can change `respect_robots`, `robots_policy`, `input_gate`, the proxy, the browser bridge or a write switch, and the local file `clean` action is not exposed over MCP. Search-engine and scholarly query endpoints have a separate user-level policy (`--query-robots` on the CLI, `WEB_FORAGER_QUERY_ROBOTS` in the server environment), default `warn`: a disallowed query path is read and the engine report names the override; `respect` refuses it. The agent cannot change this through a tool call.
|
|
17
|
+
|
|
18
|
+
</details>
|
|
19
|
+
|
|
20
|
+
Test the protocol directly:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
|
24
|
+
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | python mcp_server.py
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
|
29
|
+
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | node browser/mcp_server.js
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Browser mode
|
|
33
|
+
|
|
34
|
+
The browser layer lives in `browser/` and needs Node.js 18 or newer. It is read-only by default: like, coin, favourite, follow, share, comment, danmaku and publish are denied unless `--allow-write` is passed after the user authorises that single action.
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
npm install playwright && npx playwright install chromium
|
|
38
|
+
node browser/run.js --doctor
|
|
39
|
+
node browser/run.js --playbook bili-feed --session bili-01 --max-items 8 --agent
|
|
40
|
+
node browser/run.js --playbook zhihu-dig --query "reinforcement learning" --agent
|
|
41
|
+
node browser/run.js --playbook wechat-dig --url "https://mp.weixin.qq.com/s/..." --agent
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
| Playbook | Does | Requires |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| `bili-feed` | Home feed, scroll and capture unseen cards | none |
|
|
47
|
+
| `bili-video` | One video and its visible comments | `--url` or `--bvid` |
|
|
48
|
+
| `bili-chain` | Follow the next-episode control and resume from the session | `--url`, `--bvid`, or a chain |
|
|
49
|
+
| `bili-danmaku` | Timestamped danmaku, preferring bridge `observe` over DOM text (experimental) | `--url` or `--bvid` |
|
|
50
|
+
| `zhihu-dig` | Zhihu search, rank links, capture the best unseen page | `--query` |
|
|
51
|
+
| `xhs-dig` | Xiaohongshu public notes, rank `/explore` links, capture one | `--query` |
|
|
52
|
+
| `search-dig` | Bilibili search, rank cards, open the best unseen result | `--query` |
|
|
53
|
+
| `wechat-dig` | One WeChat article; reports the environment-check wall instead of recycling it | `--url` |
|
|
54
|
+
|
|
55
|
+
WeChat articles are the one case HTTP mode cannot finish. `mp.weixin.qq.com` disallows `/s` in robots.txt, and even with `--robots-policy warn` the article URL answers with an environment-check page or a captcha redirect instead of the body; the HTTP record reports `verification_required` (exit 9) rather than handing back the error page as content. `wechat-dig` opens the same URL in a real browser profile. If that profile has already passed WeChat's check, the article is captured; if not, the playbook reports `blocked` and does not attempt the challenge.
|
|
56
|
+
|
|
57
|
+
The bundled bridge is picked up automatically once `playwright` and its Chromium binary are installed; `--bridge` or `WEB_FORAGER_BRIDGE` points at a bridge elsewhere. The command is split on whitespace and started without a shell, so quote a path that contains spaces and pass arguments as usual; pipes, `&&` and redirection are not interpreted. `WEB_FORAGER_BROWSER_HEADFUL=1` shows the window, `WEB_FORAGER_BROWSER_PROFILE=/path` reuses a persistent profile (this is what carries login state for the `extension` backend), and `WEB_FORAGER_BROWSER_CDP=http://127.0.0.1:9222` attaches to a browser you already control (`cdp`). A bridge declares its real capabilities through an optional `describe` call; one that does not implement it is reported as `declared: false` with every capability false.
|
|
58
|
+
|
|
59
|
+
To sign in once, run `python browser/launch_chrome.py` (it defaults to `~/.web-forager/browser/profile`, and CDP is **off** by default; `--cdp` opts in to a loopback CDP port, random by default or fixed with `--port 9222`). Without `--cdp`, run browser mode with the default backend and it reuses the profile. With `--cdp`, pass `--backend cdp`. CDP itself has no authentication: keep the port local and close the browser when finished. `--storage-state FILE` loads a Playwright state file before the run; `--save-storage-state FILE` writes one after it, with mode `0600` and a refusal if the path is inside `--root`. That file contains cookies and logins: treat it as a credential, keep it out of git, and do not copy it to a shared machine.
|
|
60
|
+
|
|
61
|
+
`--strict-robots` fetches `robots.txt` before every navigation and stops with `robots_denied` (exit 3) on a disallowed path. Without it the record says so: `compliance.robots_checked` is `false`. A run that reaches the site but finds nothing readable is `degraded` (exit 8) with its records attached; an anti-bot or error interstitial is `blocked` (exit 9); a site that asks for credentials is `login_required` (exit 6).
|
|
62
|
+
|
|
63
|
+
A browser run is the most expensive thing the plugin does, so its defaults are small: records go to disk with `--out result.json` (or `out:` in MCP), no screenshots are taken unless asked, `--max-screens 5` and `--min-interval 3000` keep a walk polite, and a long walk is better split into runs that resume from `--session`.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## When the engines are blocked
|
|
68
|
+
|
|
69
|
+
Search defaults to `auto`. A mostly-CJK query goes to 360, Sogou and Sogou WeChat — the article index is the one channel that normally carries a Chinese person's or institution's name — and is not handed to Bing, which splits such a phrase into single characters and answers with dictionary pages. Those pages are dropped from any engine's results. An agent that wants Bing for a Chinese query can still ask with `--engine bing`.
|
|
70
|
+
|
|
71
|
+
Two routes need no search engine at all:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
python run.py --action find --site the-decoder.com --query "OpenAI watermark" --max-pages 3
|
|
75
|
+
python run.py --action feed --url https://the-decoder.com/ --num-results 10
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
`find` tries the site's own search URLs (`/?s=`, `/search?q=` and two more), ranks the links by the query and fetches the best ones. `feed` discovers the site's RSS/Atom `<link rel=alternate>` and reads it. `doctor --probe` asks every engine once and reports which ones answer from this network; `doctor` also reports the proxy in use with its credentials removed. Requests honour `HTTPS_PROXY` / `ALL_PROXY`, and `--proxy URL` overrides them for one run.
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## More CLI examples
|
|
83
|
+
|
|
84
|
+
Each command has its own block so it can be copied without trimming the others.
|
|
85
|
+
|
|
86
|
+
**Fetch one Bilibili video.**
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
python run.py --action fetch --url "https://www.bilibili.com/video/BV1xx411c7mD"
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
**Fetch a page that needs JavaScript rendering.**
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
python run.py --action fetch --url "https://www.zhihu.com/" --render auto
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
**Surf a site and resume the trail later.**
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
python run.py --action surf --url "https://www.zhihu.com/" --depth 1 --max-pages 5 --session zhihu-01
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**Clean a local PDF into Markdown.**
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
python run.py --action clean --input notes.pdf --format markdown
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
**List up to 500 page URLs from a site's sitemaps.**
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
python run.py --action map --url "https://example.org" --limit 500
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
**Read the Bilibili home feed.**
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
python run.py --action bili --mode feed --max-items 10
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## Records and output
|
|
125
|
+
|
|
126
|
+
### One record per page
|
|
127
|
+
|
|
128
|
+
| Field | HTTP mode | Browser mode |
|
|
129
|
+
|---|---|---|
|
|
130
|
+
| `metadata`, `markdown`, `tables`, `warnings` | yes | yes |
|
|
131
|
+
| `trust`, `injection_flags`, `injection_findings` | yes | yes |
|
|
132
|
+
| `url`, `title` | yes (`url` and `final_url`) | yes |
|
|
133
|
+
| `status`, `content_type`, `document_kind`, `cached`, `rendered`, `render_fallback`, `fetched_at`, `chunks`, `links` | yes | no |
|
|
134
|
+
| `schema_version`, `captured_at`, `playbook`, `backend`, `compliance`, `viewport_evidence`, `interaction_trace` | no | yes |
|
|
135
|
+
|
|
136
|
+
`render_fallback` is `true` when the server answered with a minimal script shell and HTTP mode rendered it once with Playwright before extraction. A `fetch --query` allows at most two such renders, and each one adds several seconds.
|
|
137
|
+
|
|
138
|
+
`injection_findings` is a list of `{code, severity, confidence, span, snippet, context}`. `span` is a character range in the scanned text, `severity` is `hard` or `soft`, and `context` is `direct`, `quote` or `example`. Hard rules describe an action (forged system tag, secret exfiltration, `curl ... | sh`, `rm -rf /`); `--input-gate strict` withholds a body only when one matches. Soft rules describe override or role-swap language and never withhold on their own, because papers and security notes quote them. The scanner walks the whole body in overlapping windows, before `max_chars` truncation, and the first warning repeats up to three matching snippets.
|
|
139
|
+
|
|
140
|
+
Search results are ordered by query coverage, phrase and title match, a small domain prior, snippet freshness and engine order — including when only one engine answered. Each result carries `coverage`, `matched_terms`, `relevance` and `language_match`. A result is relevant when it contains the exact phrase or at least two meaningful query terms with half the query covered; a wrong-language page that repeats one product name is `language_match: false`. Off-topic rows are ranked last, and are dropped when at least three relevant results exist; otherwise a warning names how many were kept. The final list keeps at most two results per host, so one dominant host cannot crowd out every other source; when too few hosts answered, the overflow fills the remaining slots. A thin first pass is retried once with a simplified query (quotes and common stopwords removed, or a spaced CJK query joined); that retry appears in `engine_report` with `query_variant`. Search titles and snippets pass through the same injection scanner, and a matching result carries its own `injection_flags` and `injection_findings`.
|
|
141
|
+
|
|
142
|
+
Before ranking, duplicate URLs are canonicalized: `utm_*`, `fbclid`, fragments, `www.`, trailing slashes and `/amp` no longer make one article look like two results. A result found by more than one engine receives a reciprocal-rank-fusion contribution, and an exact title match gets a small bonus over a title that merely contains the phrase. Light English stemming is used for matching only, so `injection` meets `injections` without changing the displayed query or text. The search payload also carries `search_quality: {query, engine, returned, relevant, off_topic, language_mismatch, engines_answered, engine_errors, dropped_off_topic, returned_before_filter}`, a machine-readable summary for agents that do not want to parse the prose warnings.
|
|
143
|
+
|
|
144
|
+
`--core` returns the shared shape instead:
|
|
145
|
+
```json
|
|
146
|
+
{
|
|
147
|
+
"schema_version": "core-1",
|
|
148
|
+
"source": "web-forager",
|
|
149
|
+
"mode": "http",
|
|
150
|
+
"url": "https://example.org/article",
|
|
151
|
+
"title": "An article",
|
|
152
|
+
"collected_at": "2026-11-07T10:00:00Z",
|
|
153
|
+
"metadata": {},
|
|
154
|
+
"markdown": "...",
|
|
155
|
+
"tables": [],
|
|
156
|
+
"warnings": [],
|
|
157
|
+
"extra": { "status": 200, "cached": false }
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
`mode` is `http` or `browser`, `source` names the producer, and `extra` keeps every field the two modes do not share. The same fold is available as a library call: `engine.normalize_record()` in Python and `normalizeRecord()` in `browser/lib/record.js`.
|
|
162
|
+
|
|
163
|
+
`results` is an index, not a second copy of every page. For a search it carries `url`, `title`, `snippet`, `status`, `content_type`, `markdown_chars`, `tables` and any `warnings` or `error`; the Markdown body lives once, in `pages`, and a single `fetch` keeps it at the top level. `--full-results` (CLI) or `fullResults: true` (MCP) restores the old shape.
|
|
164
|
+
|
|
165
|
+
### Workspace output
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
python run.py --action fetch --url "https://example.org/article" --root <WS>
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Writes only:
|
|
172
|
+
|
|
173
|
+
```text
|
|
174
|
+
<WS>/process/web-forager/web-forager_<timestamp>_<slug>.json
|
|
175
|
+
<WS>/process/web-forager/web-forager_<timestamp>_<slug>.md
|
|
176
|
+
<WS>/_broadcast/web-forager_<timestamp>_<slug>.json
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`material/` and `results/` belong to the user and the orchestrator; web-forager never writes there. Browser captures follow the same discipline under `<WS>/process/web-forager/browser/` and `<WS>/_broadcast/`.
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Research output
|
|
186
|
+
|
|
187
|
+
`research --query` runs two stages: broad recall across the free engines, then
|
|
188
|
+
focused reading of a few distinct hosts. The payload adds:
|
|
189
|
+
|
|
190
|
+
| Field | Meaning |
|
|
191
|
+
|---|---|
|
|
192
|
+
| `intent` | `{intent, confidence, signals}` from a deterministic keyword classifier |
|
|
193
|
+
| `strategy` | `{breadth, search_pages, targeted_site, targeted_sites, targeted_by_site, official_entries, focus_*}` |
|
|
194
|
+
| `seeds` | selected results with `authority_tier`, `authority_score`, `reason` |
|
|
195
|
+
| `evidence` | `[{url, title, authority_tier, authors, published_at, date_source, passages, partial?}]` |
|
|
196
|
+
| `freshness` | `{newest, oldest, dated_sources, undated_sources, newest_age_days, all_stale}` |
|
|
197
|
+
| `answerability` | `{level, coverage, authority_level, partial_only, source_count, host_count, view_hosts, multi_view, stale, reason}` |
|
|
198
|
+
| `skipped` | pages not used, with `code` (`robots_denied`, `browser_required`, `browser_required_low_value`, `no_matching_passage`, …) |
|
|
199
|
+
|
|
200
|
+
If the broad pool has no relevant primary/official/academic source, the second
|
|
201
|
+
pass prefers a direct official entry page (Python asyncio, Go concurrency, Rust,
|
|
202
|
+
MDN, Microsoft Wi-Fi support), then the site's own search page (Microsoft
|
|
203
|
+
Support, MDN), and only then a `site:` engine query for a non-CJK question.
|
|
204
|
+
Direct entries and host search pages do not depend on an engine honouring
|
|
205
|
+
`site:`. A news query with no usable engine rows falls back to a public finance
|
|
206
|
+
RSS channel (China News Service); a dated headline and summary are labelled
|
|
207
|
+
`feed_headline`, never expanded into a fabricated article. When the article body
|
|
208
|
+
has no dateline, the fetched page inherits the feed's publication date. Each
|
|
209
|
+
focus fetch is capped at 8 seconds with no retry, and robots.txt at 6 seconds,
|
|
210
|
+
so one dead origin cannot consume the whole call. A comparison needs at
|
|
211
|
+
least two independent full-source viewpoints;
|
|
212
|
+
a WeChat snippet never counts as one. When two full authoritative sources are
|
|
213
|
+
already present for a comparison or technical question, a WeChat-only snippet is
|
|
214
|
+
skipped rather than mixed in. A WeChat link that HTTP mode cannot read still
|
|
215
|
+
yields a `partial` search-snippet record with an explicit warning, never a
|
|
216
|
+
fabricated full article. Every passage is quoted, untrusted page text.
|
|
217
|
+
|
|
218
|
+
## Actions and exit codes
|
|
219
|
+
|
|
220
|
+
| Action | Purpose |
|
|
221
|
+
|---|---|
|
|
222
|
+
| `fetch` | one URL or keyword query → pages |
|
|
223
|
+
| `research` | broad recall → focused, cited evidence passages |
|
|
224
|
+
| `find` | search inside one site through the site's own search page |
|
|
225
|
+
| `feed` | read a site's RSS/Atom feed, discovering it from the page |
|
|
226
|
+
| `clean` | local or remote HTML/PDF/DOCX/XLSX/EPUB/CSV → Markdown |
|
|
227
|
+
| `bili` | Bilibili feed / video / related / search / surf |
|
|
228
|
+
| `surf` | search or seed URL, then ranked link BFS |
|
|
229
|
+
| `scholar` | arXiv / Crossref / PubMed / DOI records |
|
|
230
|
+
| `map` | page URLs from robots.txt and sitemaps |
|
|
231
|
+
| `doctor` | dependency report: what is installed, what to install |
|
|
232
|
+
|
|
233
|
+
| Exit code | Meaning (both modes unless overridden below) |
|
|
234
|
+
|---|---|
|
|
235
|
+
| 0 | success |
|
|
236
|
+
| 2 | argument or URL error |
|
|
237
|
+
| 3 | robots.txt denied the request, or no browser bridge configured |
|
|
238
|
+
| 4 | network, HTTP or navigation failure |
|
|
239
|
+
| 5 | extraction or capture failed |
|
|
240
|
+
| 6 | Playwright requested but unavailable |
|
|
241
|
+
| 7 | PDF requested but `pypdf` unavailable |
|
|
242
|
+
| 9 | verification or anti-bot page instead of content (`verification_required` in HTTP mode, `blocked` in browser mode) |
|
|
243
|
+
| 130 | interrupted |
|
|
244
|
+
|
|
245
|
+
Browser mode reuses that table and changes or adds only these codes:
|
|
246
|
+
|
|
247
|
+
| Exit code | Browser meaning |
|
|
248
|
+
|---|---|
|
|
249
|
+
| 2 | also: unknown backend |
|
|
250
|
+
| 6 | `login_required`: the site asked for credentials |
|
|
251
|
+
| 7 | `confirmation_required`: a write-like action was denied |
|
|
252
|
+
| 8 | `degraded`: the page was read, but the playbook found nothing it looks for |
|
|
253
|
+
| 9 | `blocked`: an anti-bot or error interstitial replaced the page |
|
|
254
|
+
|
|
255
|
+
`--agent` keeps these exit codes and prints a `{"ok":false,"error":{"code","message","hint"}}` envelope on stdout. `doctor` reports every dependency with its version or the command that installs it; a missing Playwright returns exit code 6, and a missing `pypdf` returns exit code 7.
|
|
256
|
+
|
|
257
|
+
---
|
|
258
|
+
|
|
259
|
+
---
|
|
260
|
+
|
|
261
|
+
## Requirements
|
|
262
|
+
|
|
263
|
+
Python 3.10+. Core runtime: `requests`, `beautifulsoup4`, `trafilatura`. Optional: `playwright` (JS rendering), `pypdf` (PDF). `defusedxml` is used when already installed but is never required — XML with an internal DTD subset or any ENTITY declaration is rejected regardless. Browser mode additionally needs Node.js 18+ and a bridge command; `browser/bridges/playwright-bridge.js` needs `playwright` installed with npm. No third-party code is vendored; the full per-capability matrix lives in [DEPENDENCIES.md](DEPENDENCIES.md).
|
|
264
|
+
|
|
265
|
+
---
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Repository layout
|
|
270
|
+
|
|
271
|
+
<details>
|
|
272
|
+
<summary>Full tree</summary>
|
|
273
|
+
|
|
274
|
+
```text
|
|
275
|
+
web-forager/
|
|
276
|
+
├── .claude-plugin/ Claude Code manifest + marketplace
|
|
277
|
+
├── .codex-plugin/ Codex manifest + marketplace
|
|
278
|
+
├── .codebuddy-plugin/ CodeBuddy manifest + marketplace
|
|
279
|
+
├── .dsh-plugin/ DSH manifest + CJS entry
|
|
280
|
+
├── .cursor/rules/ Cursor rule file
|
|
281
|
+
├── .windsurf/rules/ Windsurf rule file
|
|
282
|
+
├── .clinerules/ Cline rule file
|
|
283
|
+
├── .continue/rules/ Continue rule file
|
|
284
|
+
├── .roo/rules/ Roo Code rule file
|
|
285
|
+
├── .github/ Copilot instructions + Actions CI
|
|
286
|
+
├── AGENTS.md generic agent contract
|
|
287
|
+
├── install.py one-command rule-file installer
|
|
288
|
+
├── manifest.json plugin manifest
|
|
289
|
+
├── .mcp.json MCP stdio registration
|
|
290
|
+
├── package.json / index.js npm + dsh packaging
|
|
291
|
+
├── cordis.patch.yml dsh bundle patch
|
|
292
|
+
├── run.py CLI: nine actions
|
|
293
|
+
├── mcp_server.py MCP stdio server (10 HTTP tools)
|
|
294
|
+
├── browser/ browser mode (Node 18+)
|
|
295
|
+
│ ├── run.js CLI: eight read-only playbooks
|
|
296
|
+
│ ├── mcp_server.js MCP stdio server (4 browser tools)
|
|
297
|
+
│ ├── lib/ bridge backend, session, record, safety, selectors, robots, snapshot
|
|
298
|
+
│ ├── playbooks/ bili-* / zhihu-dig / xhs-dig / search-dig / wechat-dig
|
|
299
|
+
│ ├── bridges/ optional Playwright reference bridge
|
|
300
|
+
│ └── tests/ fake backends, real-page snapshots, live smoke
|
|
301
|
+
├── SKILL.md Claude Code skill contract
|
|
302
|
+
├── commands/ /fetch /research /surf /scholar /map /browse
|
|
303
|
+
├── docs/ install commands, acknowledgements, dependency matrix
|
|
304
|
+
├── scripts/
|
|
305
|
+
│ ├── engine.py robots, fetch, search, cache, chunks, Markdown, surf, find, feed
|
|
306
|
+
│ ├── research.py broad recall, authority selection, cited evidence passages
|
|
307
|
+
│ ├── pdf.py PDF text (optional pypdf)
|
|
308
|
+
│ ├── documents.py DOCX / XLSX / EPUB / CSV
|
|
309
|
+
│ ├── feeds.py RSS / Atom / RSS 1.0
|
|
310
|
+
│ ├── scholar.py arXiv / Crossref / PubMed / DOI
|
|
311
|
+
│ ├── sitemap.py sitemap discovery
|
|
312
|
+
│ └── bilibili.py Bilibili adapter
|
|
313
|
+
├── tests/ offline HTTP fixtures
|
|
314
|
+
├── requirements.txt
|
|
315
|
+
├── THIRD_PARTY_LICENSES.txt
|
|
316
|
+
├── CHANGELOG.md
|
|
317
|
+
├── README.md · README_zh.md
|
|
318
|
+
└── LICENSE MIT
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
</details>
|
|
322
|
+
|
|
323
|
+
---
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
## Tests
|
|
328
|
+
|
|
329
|
+
```bash
|
|
330
|
+
python -m pytest tests -q
|
|
331
|
+
npm run test:browser
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
That is **333 Python tests** plus **91 browser tests**, all passing and fully offline.
|
|
335
|
+
|
|
336
|
+
<details>
|
|
337
|
+
<summary>What the suites cover</summary>
|
|
338
|
+
|
|
339
|
+
`conftest.py` starts a temporary local HTTP server serving robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages, a site-search page and multi-hop surf pages. The Python suite covers robots denial, extraction, tables, documents, feeds, cache, search pagination and routing, chunking, MCP, surf sessions, installer behaviour, host-rule and disclaimer checks, the shared `core-1` record, and every exit code. The browser suite uses a fake backend plus fake bridges that speak the wire protocol, so the CLI runs end to end without a browser; it covers capability declaration, robots.txt matching, the `--strict-robots` round trip, the private-address guard, the untrusted-content screen, and the degraded and blocked paths over snapshots recorded from live pages.
|
|
340
|
+
|
|
341
|
+
</details>
|
|
342
|
+
|
|
343
|
+
CI runs both suites on every push to `main`, on pull requests and on version tags: Python on Ubuntu 3.10/3.12 and Windows 3.12, browser mode on Node 20/22. A live check is opt-in and never runs in CI:
|
|
344
|
+
|
|
345
|
+
```bash
|
|
346
|
+
WEB_FORAGER_LIVE=1 WEB_FORAGER_BRIDGE="node browser/bridges/playwright-bridge.js" npm run test:live
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
---
|