web-forager 0.0.0-stage → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +16 -0
- package/.claude-plugin/plugin.json +20 -0
- package/.clinerules/web-forager.md +85 -0
- package/.codebuddy-plugin/marketplace.json +14 -0
- package/.codebuddy-plugin/plugin.json +25 -0
- package/.codex-plugin/marketplace.json +12 -0
- package/.codex-plugin/mcp.json +20 -0
- package/.codex-plugin/plugin.json +15 -0
- package/.continue/rules/web-forager.md +90 -0
- package/.cursor/rules/web-forager.mdc +80 -0
- package/.dsh-plugin/index.js +11 -0
- package/.dsh-plugin/plugin.json +5 -0
- package/.github/copilot-instructions.md +85 -0
- package/.mcp.json +12 -0
- package/.roo/rules/web-forager.md +85 -0
- package/.windsurf/rules/web-forager.md +90 -0
- package/AGENTS.md +119 -0
- package/CHANGELOG.md +508 -0
- package/LICENSE +21 -0
- package/README.md +498 -2
- package/README_zh.md +498 -0
- package/SKILL.md +451 -0
- package/THIRD_PARTY_LICENSES.txt +40 -0
- package/browser/bridges/playwright-bridge.js +382 -0
- package/browser/launch_chrome.py +193 -0
- package/browser/lib/backend.js +302 -0
- package/browser/lib/bili-api.js +90 -0
- package/browser/lib/bridge.js +83 -0
- package/browser/lib/context.js +60 -0
- package/browser/lib/doctor.js +97 -0
- package/browser/lib/errors.js +17 -0
- package/browser/lib/netguard.js +149 -0
- package/browser/lib/playbooks.js +30 -0
- package/browser/lib/record.js +109 -0
- package/browser/lib/robots.js +195 -0
- package/browser/lib/safety.js +141 -0
- package/browser/lib/selectors.js +80 -0
- package/browser/lib/session.js +130 -0
- package/browser/lib/snapshot.js +229 -0
- package/browser/lib/untrusted.js +215 -0
- package/browser/lib/workspace.js +40 -0
- package/browser/mcp_server.js +252 -0
- package/browser/package.json +40 -0
- package/browser/playbooks/bili-chain.js +58 -0
- package/browser/playbooks/bili-danmaku.js +102 -0
- package/browser/playbooks/bili-feed.js +66 -0
- package/browser/playbooks/bili-video.js +38 -0
- package/browser/playbooks/search-dig.js +107 -0
- package/browser/playbooks/shared.js +138 -0
- package/browser/playbooks/wechat-dig.js +78 -0
- package/browser/playbooks/xhs-dig.js +89 -0
- package/browser/playbooks/zhihu-dig.js +89 -0
- package/browser/run.js +441 -0
- package/commands/browse.md +53 -0
- package/commands/fetch.md +29 -0
- package/commands/map.md +24 -0
- package/commands/research.md +35 -0
- package/commands/scholar.md +28 -0
- package/commands/surf.md +28 -0
- package/cordis.patch.yml +19 -0
- package/docs/ACKNOWLEDGEMENTS.md +15 -0
- package/docs/ACKNOWLEDGEMENTS_zh.md +15 -0
- package/docs/DEPENDENCIES.md +40 -0
- package/docs/INSTALL_COMMANDS.md +113 -0
- package/docs/INSTALL_COMMANDS_zh.md +113 -0
- package/docs/REFERENCE.md +349 -0
- package/docs/REFERENCE_zh.md +342 -0
- package/index.js +7 -0
- package/install.py +254 -0
- package/manifest.json +98 -0
- package/mcp_server.py +850 -0
- package/package.json +82 -6
- package/requirements.txt +15 -0
- package/run.py +908 -0
- package/scripts/__init__.py +1 -0
- package/scripts/bilibili.py +882 -0
- package/scripts/documents.py +437 -0
- package/scripts/engine.py +5141 -0
- package/scripts/feeds.py +180 -0
- package/scripts/pdf.py +156 -0
- package/scripts/research.py +2087 -0
- package/scripts/scholar.py +381 -0
- package/scripts/sitemap.py +195 -0
- package/scripts/untrusted.py +312 -0
package/README.md
CHANGED
|
@@ -1,3 +1,499 @@
|
|
|
1
|
-
#
|
|
1
|
+
# web-forager
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Read the public web politely, and hand an agent one clean record it can trust.**
|
|
4
|
+
|
|
5
|
+
Search, fetch, extract and browse live in one engine: 360 / Bing / Sogou / Sogou WeChat for discovery; HTML, PDF, Office, EPUB, CSV, feeds and sitemaps for reading; a real browser for pages that only exist after scripts or a login. It refuses what it should not touch, and every record says what was skipped and why.
|
|
6
|
+
|
|
7
|
+
**English** · [简体中文](README_zh.md)
|
|
8
|
+
|
|
9
|
+
**Version:** 1.3.1 | **License:** MIT | **Python:** 3.10+ | **Node:** 18+ (browser mode)
|
|
10
|
+
|
|
11
|
+
> For learning and information gathering only. Read the [full legal notice](#full-legal-notice) before relying on any output.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Contents
|
|
16
|
+
|
|
17
|
+
**Start**
|
|
18
|
+
|
|
19
|
+
- [Why web-forager](#why-web-forager)
|
|
20
|
+
- [Install and download](#install-and-download)
|
|
21
|
+
- [Quick start](#quick-start)
|
|
22
|
+
- [Things you can just ask](#things-you-can-just-ask)
|
|
23
|
+
|
|
24
|
+
**Going deeper**
|
|
25
|
+
|
|
26
|
+
- [Two modes, one record](#two-modes-one-record)
|
|
27
|
+
- [How a run flows](#how-a-run-flows)
|
|
28
|
+
- [What a run looks like](#what-a-run-looks-like)
|
|
29
|
+
- [Repository layout](#repository-layout)
|
|
30
|
+
- [Full reference](#full-reference)
|
|
31
|
+
- [Security boundary](#security-boundary)
|
|
32
|
+
- [Scope and compliance](#scope-and-compliance)
|
|
33
|
+
|
|
34
|
+
**Other**
|
|
35
|
+
|
|
36
|
+
- [Requirements](#requirements)
|
|
37
|
+
- [Tests](#tests)
|
|
38
|
+
- [Full legal notice](#full-legal-notice)
|
|
39
|
+
- [Known limitations](#known-limitations)
|
|
40
|
+
- [Issues and feedback](#issues-and-feedback)
|
|
41
|
+
- [Acknowledgements](#acknowledgements)
|
|
42
|
+
- [Trademarks](#trademarks)
|
|
43
|
+
- [License and author](#license-and-author)
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Why web-forager
|
|
48
|
+
|
|
49
|
+
A search API returns links and stops. A scraper reads pages and ignores the rules. An agent needs both halves, plus a contract it can trust. This repository puts all three in one engine, with no API key and no hosted index in the middle.
|
|
50
|
+
|
|
51
|
+
- **One engine, every content type.** HTML, PDF, DOCX, XLSX, EPUB, CSV, RSS/Atom, sitemaps, Bilibili and rendered pages all come back as the same four fields — `metadata`, `markdown`, `tables`, `warnings` — so the caller never branches on the source.
|
|
52
|
+
- **Compliance is inside the fetcher, not in a README promise.** robots.txt, a per-origin delay, response-size caps, redirect limits and a contactable User-Agent apply to every request, including each search, sitemap and surf hop.
|
|
53
|
+
- **It degrades instead of crashing.** A missing bridge is exit 3, missing Playwright is exit 6, missing `pypdf` is exit 7, a site without a sitemap is an empty result with a note, and a verification page is reported as `verification_required` rather than recycled as content.
|
|
54
|
+
- **It tells the truth about what it read.** `engine_report` names every engine tried with its latency and error, `skipped` names every result that could not be read, `degraded` means the page was read but the playbook found nothing, and fetched text carries `trust: "untrusted"` plus `injection_findings` — code, severity, character span and matching snippet — when it looks like a prompt.
|
|
55
|
+
- **Results are ordered even when one engine answers.** Coverage, title and phrase match, a small domain prior and snippet freshness decide the order; duplicate URLs are canonicalized (`utm_*`, fragments, `www.`, trailing slash), multi-engine hits get an RRF contribution, and light stemming lets `injection` meet `injections`. A result needs the exact phrase or at least two meaningful query terms; a Chinese tutorial that merely repeats one English product name is marked `language_match: false` and dropped when enough real answers exist. The list keeps at most two results per host and fills from the overflow when too few hosts answered; a thin first pass retries once with a simplified query. A `search_quality` block counts relevant, off-topic and language-mismatched rows.
|
|
56
|
+
- **A script shell is rendered, within a budget.** When a server returns almost no HTML, `--render auto` renders it once with Playwright before extraction; `fetch --query` and `surf` cap that at two pages per run.
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## Install and download
|
|
61
|
+
|
|
62
|
+
Install this repository as **one plugin**. The CLI, both MCP servers and `SKILL.md` share a single engine and a single robots/cache pipeline; splitting the tree breaks that.
|
|
63
|
+
|
|
64
|
+
**Most people take method one: let an AI install it.** A host with a plugin marketplace takes method two; a terminal install, or several hosts at once, takes method three.
|
|
65
|
+
|
|
66
|
+
## Method 1: let your AI install it
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
Visit the GitHub repository `https://github.com/JiamingShi-JsPhys/web-forager`, clone it, and run `python install.py --project . --global`. If your host has a plugin system, install the whole thing as a single plugin instead; that script covers the hosts that read rule files.
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Method 2: install from the plugin marketplace by URL
|
|
73
|
+
|
|
74
|
+
Every host below has its own plugin marketplace. Register this repository's Git URL, then install the plugin it lists. Registering the same URL twice is harmless — the second call reports that it is already known.
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
https://github.com/JiamingShi-JsPhys/web-forager
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Append `#v1.3.1` to pin the install to this release. Drop it to follow the default branch.
|
|
81
|
+
|
|
82
|
+
## Method 3: install with commands
|
|
83
|
+
|
|
84
|
+
Claude Code, Codex, dsh and CodeBuddy take a one-line plugin command; Cursor, Windsurf, Cline, Continue, Roo Code, GitHub Copilot and opencode read a rule file, and one script writes all of them. Every command, every flag and the optional dependencies live on one page: **[docs/INSTALL_COMMANDS.md](docs/INSTALL_COMMANDS.md)**.
|
|
85
|
+
|
|
86
|
+
Each plugin host gets its own MCP launch file: Claude Code reads `.mcp.json`, Codex reads `.codex-plugin/mcp.json`, CodeBuddy reads the `mcpServers` block in `.codebuddy-plugin/plugin.json`, and dsh reads the two `dsh-mcp-client` entries in `cordis.patch.yml`. After installing in CodeBuddy or dsh, start a new session so the servers mount; `codebuddy mcp list` can run before plugins load and is not a health check.
|
|
87
|
+
|
|
88
|
+
### Keeping an installed copy current
|
|
89
|
+
|
|
90
|
+
A host copies the plugin into a cache of its own, and that copy is what actually runs. A fixed bug keeps running until the cache is refreshed — and a marketplace backed by a local folder may hold a snapshot rather than the live directory.
|
|
91
|
+
|
|
92
|
+
For ZCode, refresh the marketplace index first, then the plugin:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
zcode plugins marketplace update web-forager-marketplace
|
|
96
|
+
zcode plugins update web-forager@web-forager-marketplace
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
If the version number did not change between two installs, `update` may treat the cache as current; reinstall it instead:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
zcode plugins uninstall web-forager@web-forager-marketplace --force
|
|
103
|
+
zcode plugins install web-forager@web-forager-marketplace --yes
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Claude Code users can run `claude plugin update web-forager`. After any update, `python run.py --action doctor` reports what the running copy can actually see.
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## Quick start
|
|
111
|
+
|
|
112
|
+
Three ways in, plus an optional browser mode. Pick the one that matches where you already are.
|
|
113
|
+
|
|
114
|
+
### From the chat box
|
|
115
|
+
|
|
116
|
+
The skill is registered under the `web-forager` namespace, so it is one `/` away:
|
|
117
|
+
|
|
118
|
+
```text
|
|
119
|
+
/web-forager:web-forager collect https://www.bilibili.com/video/BV1xx411c7mD as Markdown
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Say what you need after the name; the skill picks the action and returns Markdown, metadata, tables and warnings. A page that robots.txt disallows is refused, not fetched.
|
|
123
|
+
|
|
124
|
+
Five commands cover the usual jobs:
|
|
125
|
+
|
|
126
|
+
| Command | Does |
|
|
127
|
+
|---|---|
|
|
128
|
+
| `/web-forager:fetch` | Fetch one URL, or search a keyword and read the results |
|
|
129
|
+
| `/web-forager:surf` | Search or open a seed URL, then follow ranked in-page links with a resumable session |
|
|
130
|
+
| `/web-forager:scholar` | Query arXiv, Crossref or PubMed, or resolve a DOI |
|
|
131
|
+
| `/web-forager:map` | Discover page URLs from robots.txt Sitemap lines, sitemap indexes and `.xml.gz` files |
|
|
132
|
+
| `/web-forager:browse` | Run a read-only browser playbook through the browser MCP server |
|
|
133
|
+
|
|
134
|
+
```text
|
|
135
|
+
/web-forager:fetch https://example.org/article
|
|
136
|
+
/web-forager:scholar 10.1038/nature14539
|
|
137
|
+
/web-forager:browse read this WeChat article: https://mp.weixin.qq.com/s/...
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
### From a terminal
|
|
141
|
+
|
|
142
|
+
Install the Python dependencies once:
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
pip install -r requirements.txt
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Read one page as Markdown, metadata and tables:
|
|
149
|
+
|
|
150
|
+
```bash
|
|
151
|
+
python run.py --action fetch --url "https://example.org/article"
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Search, then read the most relevant pages:
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
python run.py --action fetch --query "city walk guide" --num-results 8 --max-pages 3
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Broad recall, then cited evidence only:
|
|
161
|
+
|
|
162
|
+
```bash
|
|
163
|
+
python run.py --action research --query "Python 3.12 asyncio.TaskGroup default exception handling" --focus 3
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`research` ranks sources by relevance and an explicit authority tier, makes one
|
|
167
|
+
narrow `site:` or arXiv-title query when the broad pool has no authoritative
|
|
168
|
+
answer, reads a few distinct hosts, and returns the matching passages with their
|
|
169
|
+
URLs plus an `answerability` verdict. It says `none`/`low` instead of inventing.
|
|
170
|
+
|
|
171
|
+
A paper, without touching a result page:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
python run.py --action scholar --source doi --query "10.1038/nature14539"
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Search defaults to `auto`: a mostly-CJK query goes to 360, Sogou and Sogou WeChat, a Latin query to 360 and Bing. If the engines are blocked, the [full reference](docs/REFERENCE.md#when-the-engines-are-blocked) shows the site-search and feed routes that need no engine at all.
|
|
178
|
+
|
|
179
|
+
Not sure which action you need?
|
|
180
|
+
|
|
181
|
+
| You have | Use |
|
|
182
|
+
|---|---|
|
|
183
|
+
| one URL or a keyword | `--action fetch` |
|
|
184
|
+
| a question that needs cited evidence | `--action research` |
|
|
185
|
+
| a site you trust, no working search engine | `--action find --site <HOST> --query <TEXT>` |
|
|
186
|
+
| a blog or news site | `--action feed --url <SITE>` |
|
|
187
|
+
| a PDF, DOCX, XLSX, EPUB or CSV | `--action clean --input <FILE>` |
|
|
188
|
+
| a topic to explore link by link | `--action surf --query <TEXT>` |
|
|
189
|
+
| a DOI, arXiv id or paper title | `--action scholar` |
|
|
190
|
+
| a site whose pages you want to enumerate | `--action map --url <SITE>` |
|
|
191
|
+
| Bilibili | `--action bili` |
|
|
192
|
+
| "why is this not working" | `--action doctor [--probe]` |
|
|
193
|
+
|
|
194
|
+
Add `--agent` to any action for a machine-readable `{"ok":true,"data":...}` envelope. Add `--out result.json` to keep the full record on disk, or `--root <WS>` to publish it into a workspace (`<WS>/process/web-forager/` and `<WS>/_broadcast/`); see [records and output](docs/REFERENCE.md#records-and-output).
|
|
195
|
+
|
|
196
|
+
### From an MCP client
|
|
197
|
+
|
|
198
|
+
```json
|
|
199
|
+
{
|
|
200
|
+
"mcpServers": {
|
|
201
|
+
"web-forager": {
|
|
202
|
+
"command": "python",
|
|
203
|
+
"args": ["/absolute/path/to/web-forager/mcp_server.py"]
|
|
204
|
+
},
|
|
205
|
+
"web-forager-browser": {
|
|
206
|
+
"command": "node",
|
|
207
|
+
"args": ["/absolute/path/to/web-forager/browser/mcp_server.js"]
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
}
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
The HTTP server exposes ten tools (`web_fetch`, `web_search`, `web_research`, `web_surf`, `web_scholar`, `web_map`, `web_bilibili`, `web_doctor`, `web_find`, `web_feed`); the browser server exposes four (`browser_list`, `browser_run`, `browser_session`, `browser_doctor`).
|
|
214
|
+
|
|
215
|
+
MCP calls cannot relax the safety boundary: no tool parameter turns robots off, changes the injection gate, sets a proxy, supplies a browser bridge or allows a write action. The local file `clean` action is CLI-only and is not an MCP tool. One switch keeps a search cheap: `max_pages: 0` returns the result list with no page bodies. `web_search` also returns `engine_report` and `warnings` when the results do not mention the query.
|
|
216
|
+
|
|
217
|
+
### Browser mode (optional)
|
|
218
|
+
|
|
219
|
+
Needs Node.js 18+ and a bridge command, and is read-only by default. Sign in once with `python browser/launch_chrome.py` (CDP is off by default; it opens the dedicated profile), then run the browser with the default backend, which reuses that profile. If you specifically need DevTools attach, start the launcher with `--cdp` and pass `--backend cdp`. The eight playbooks, their switches, `--strict-robots`, storage-state and the cost controls are in the [full reference](docs/REFERENCE.md#browser-mode). `wechat-dig` is the one to reach for when an HTTP search found a WeChat article but robots.txt and the environment check stopped the read.
|
|
220
|
+
|
|
221
|
+
---
|
|
222
|
+
|
|
223
|
+
## Things you can just ask
|
|
224
|
+
|
|
225
|
+
| Say it your way | Routes to | You get back |
|
|
226
|
+
|---|---|---|
|
|
227
|
+
| "Read this article and keep the tables" | `fetch --url` | Markdown, metadata, tables and any extraction warnings |
|
|
228
|
+
| "Find recent writing on X" | `fetch --query` | Ranked, de-duplicated results, optionally the top pages |
|
|
229
|
+
| "What does this site publish?" | `map --url` | Page URLs from robots.txt Sitemap lines and sitemap indexes |
|
|
230
|
+
| "Follow this topic from that page" | `surf --query` or `--url` | A resumable link-by-link walk, one state change per cycle |
|
|
231
|
+
| "Find papers on X" | `scholar --source arxiv` | arXiv metadata and abstracts, not a scraped result page |
|
|
232
|
+
| "What is this DOI?" | `scholar --source doi` | One normalised bibliographic record |
|
|
233
|
+
| "Read this PDF / Word file" | `clean --input` | The same Markdown record as an HTML page |
|
|
234
|
+
| "Why is this failing?" | `doctor --probe` | Dependency versions and which search engines answer here |
|
|
235
|
+
| "Read this WeChat article" | `browse` + `wechat-dig` | The article, or an honest `blocked` when the profile has not passed the check |
|
|
236
|
+
| "Open the page behind my login" | `browse` + `--backend cdp` | What the page shows to that profile, with the shared-profile warning |
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## Two modes, one record
|
|
241
|
+
|
|
242
|
+
| | HTTP mode | Browser mode |
|
|
243
|
+
|---|---|---|
|
|
244
|
+
| Reads | what the server returns | what a person sees once scripts run and login state applies |
|
|
245
|
+
| Entry points | `run.py`, `mcp_server.py` | `browser/run.js`, `browser/mcp_server.js` |
|
|
246
|
+
| Needs | Python 3.10+ | Node.js 18+ and a bridge command |
|
|
247
|
+
| Covers | search, HTML/PDF/Office/feeds, sitemaps, surf, Bilibili, scholarly APIs | eight read-only playbooks, rendered pages, danmaku, WeChat articles |
|
|
248
|
+
|
|
249
|
+
Both modes agree on four core fields — `metadata`, `markdown`, `tables` and `warnings` — and extend them differently. `--core` on either CLI, or `core: true` on `browser_run`, folds a record onto one documented shape ([records and output](docs/REFERENCE.md#records-and-output)).
|
|
250
|
+
|
|
251
|
+
Only Bilibili has a site-specific HTTP adapter. Zhihu, Xiaohongshu, Weibo and Douban are read as ordinary pages in HTTP mode; Zhihu and Xiaohongshu add browser playbooks, and Weibo and Douban have none. WeChat articles are the one case HTTP mode regularly cannot finish: robots.txt disallows the article path, and the page itself answers with an environment check. `wechat-dig` opens it in a browser profile instead, and reports `blocked` when that profile has not passed the check rather than solving it.
|
|
252
|
+
|
|
253
|
+
---
|
|
254
|
+
|
|
255
|
+
## How a run flows
|
|
256
|
+
|
|
257
|
+
Every request enters through the same `HttpFetcher`: robots policy, cache, retry/backoff, per-origin rate limiter and size cap. The chosen action routes the payload to the matching extractor, and all extractors converge on one Markdown record.
|
|
258
|
+
|
|
259
|
+
```mermaid
|
|
260
|
+
flowchart LR
|
|
261
|
+
IN["URL / keyword / DOI / file"] --> SEL{"choose action"}
|
|
262
|
+
SEL -->|one page| F["fetch"]
|
|
263
|
+
SEL -->|one site| FD["find / feed"]
|
|
264
|
+
SEL -->|local file| C["clean"]
|
|
265
|
+
SEL -->|papers| S["scholar"]
|
|
266
|
+
SEL -->|site inventory| M["map"]
|
|
267
|
+
SEL -->|exploration| W["surf"]
|
|
268
|
+
SEL -->|Bilibili| B["bili"]
|
|
269
|
+
|
|
270
|
+
F --> SEARCH["search providers<br/>360 / Bing / Sogou / WeChat"]
|
|
271
|
+
FD --> HTTP
|
|
272
|
+
C --> EXT["HTML / PDF / DOCX / XLSX / EPUB / CSV"]
|
|
273
|
+
S --> API["arXiv / Crossref / PubMed"]
|
|
274
|
+
M --> SM["robots Sitemap / sitemap index / .xml.gz"]
|
|
275
|
+
W --> HTTP
|
|
276
|
+
B --> HTTP
|
|
277
|
+
SEARCH --> RANK["rank + diversify<br/>coverage, phrase, language, freshness"]
|
|
278
|
+
RANK --> HTTP["robots.txt check<br/>per-origin rate limit"]
|
|
279
|
+
HTTP --> EXT
|
|
280
|
+
EXT --> MD["Markdown + metadata + tables + chunks"]
|
|
281
|
+
MD --> OUT["JSON / --agent envelope / --root workspace / MCP"]
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
`surf` only follows `<a href>` links that already exist in the page; it never constructs guessed URLs, submits forms or executes page JavaScript. `map` reads only published sitemaps, and `scholar` calls documented APIs instead of scraping result pages.
|
|
285
|
+
|
|
286
|
+
Browser mode is a second loop beside this one: the bridge supplies a page, `browser/run.js` reads a DOM snapshot, the session decides what is new, the playbook picks the next navigation step and a capture is written. Only one state-changing action runs per observation cycle, and write-like controls are denied by default.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## What a run looks like
|
|
291
|
+
|
|
292
|
+
A search that can read every result:
|
|
293
|
+
|
|
294
|
+
```text
|
|
295
|
+
$ python run.py --action fetch --query "city walk guide" --num-results 8 --max-pages 3
|
|
296
|
+
mode: search
|
|
297
|
+
pages: 3/3 readable
|
|
298
|
+
skipped: none
|
|
299
|
+
engine_report: 360 (0), Sogou (7, 0.7 s), WeChat (8, 0.5 s)
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
A search where one source refuses:
|
|
303
|
+
|
|
304
|
+
```text
|
|
305
|
+
skipped: 5
|
|
306
|
+
robots_denied https://mp.weixin.qq.com/s?...
|
|
307
|
+
robots_denied https://mp.weixin.qq.com/s?...
|
|
308
|
+
warnings:
|
|
309
|
+
robots.txt denied the skipped paths. Use --robots-policy warn only when you are authorised to read them; the record will mark the override.
|
|
310
|
+
WeChat article links cannot be read in HTTP mode ... open one with browser mode: browser_run with playbook "wechat-dig".
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
The first is a success. The second is still a success — it says what it could not do, instead of returning an error page as the article.
|
|
314
|
+
|
|
315
|
+
---
|
|
316
|
+
|
|
317
|
+
## Repository layout
|
|
318
|
+
|
|
319
|
+
```text
|
|
320
|
+
web-forager/
|
|
321
|
+
├── run.py # one CLI entry: fetch, research, find, feed, clean, surf, scholar, map, bili, doctor
|
|
322
|
+
├── mcp_server.py # stdio JSON-RPC, ten HTTP tools
|
|
323
|
+
├── scripts/ # engine, extractors, search parsers, robots, cache, untrusted screen
|
|
324
|
+
├── browser/ # Node browser mode: run.js, MCP server, playbooks, bridge, profile launcher
|
|
325
|
+
├── commands/ # slash commands: fetch, surf, scholar, map, browse
|
|
326
|
+
├── docs/ # reference, install commands, dependencies, acknowledgements
|
|
327
|
+
├── tests/ # Python suite and offline fixtures
|
|
328
|
+
└── browser/tests/ # browser suite: fake backend, fake bridges, recorded snapshots
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
Per-directory detail, every action and flag, record shapes and the full exit-code table: **[docs/REFERENCE.md](docs/REFERENCE.md)**.
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## Full reference
|
|
336
|
+
|
|
337
|
+
Records, the full action list, exit codes, requirements, repository layout and test details live in **[docs/REFERENCE.md](docs/REFERENCE.md)**, so this page stays a quick start.
|
|
338
|
+
|
|
339
|
+
| Exit code | Means |
|
|
340
|
+
|---|---|
|
|
341
|
+
| `0` | the action finished; unreadable results are listed in `skipped` |
|
|
342
|
+
| `2` | argument or URL error |
|
|
343
|
+
| `3` | robots.txt denied the request, or no browser bridge configured |
|
|
344
|
+
| `4` | network, HTTP or navigation failure |
|
|
345
|
+
| `5` | extraction or capture failed |
|
|
346
|
+
| `6` | Playwright requested but unavailable, or a login wall in browser mode |
|
|
347
|
+
| `7` | PDF requested but `pypdf` unavailable |
|
|
348
|
+
| `8` | `degraded`: the page was read but the playbook found nothing |
|
|
349
|
+
| `9` | `blocked` or `verification_required`: an anti-bot or verification page replaced the content |
|
|
350
|
+
| `10` | the query was refused because it asked to profile a private person |
|
|
351
|
+
| `130` | interrupted |
|
|
352
|
+
|
|
353
|
+
`--agent` keeps these codes and writes `{"ok":false,"error":{"code","message","hint"}}` on stdout.
|
|
354
|
+
|
|
355
|
+
---
|
|
356
|
+
|
|
357
|
+
## Security boundary
|
|
358
|
+
|
|
359
|
+
The tool makes the requests a prompt or an agent asks for, so the boundaries are in the code rather than left to the caller's memory.
|
|
360
|
+
|
|
361
|
+
- **Private targets are refused.** Loopback, RFC 1918, link-local (including `169.254.169.254`), CGNAT and their IPv6 forms are rejected before a socket opens, and again on every redirect hop, so a public URL cannot bounce the client into the LAN. `--allow-private` exists for local fixtures and trusted intranet tools and is off by default. The browser side installs the same guard as a request filter, so a hostile page cannot probe `http://127.0.0.1:...`.
|
|
362
|
+
- **robots.txt is a policy, not a lock.** The default `--robots-policy respect` refuses a disallowed path. `warn` reads it and writes `compliance.robots_overridden: true`; `ignore` records that robots was not consulted. A denial is reported to the caller — it is not a cue to retry the same URL with a tool that ignores robots.
|
|
363
|
+
- **Query endpoints have their own policy.** Search engines and scholarly APIs are not target pages, so they use `--query-robots` (or `WEB_FORAGER_QUERY_ROBOTS`), default `warn`. Bing, 360, Sogou and the arXiv export API disallow their query paths, so the default reads them and records `query_robots: warn` in the engine report; `respect` refuses, and `ignore` skips the check. Only a user can choose, and MCP never accepts the switch from the model.
|
|
364
|
+
- **The browser keeps its own profile.** The bundled bridge stores state under `~/.web-forager/browser/profile`, not in the browser you use every day. `browser/launch_chrome.py` starts Chrome on that profile, and `--save-storage-state` writes a Playwright state file with mode `0600` (refused inside `--root`). Attaching over CDP to a personal browser still works, but the run is marked `shared_browser: true` with a warning: every cookie and login in that profile is reachable by the pages a playbook opens.
|
|
365
|
+
- **Fetched text is data, and says so.** Every record carries `trust: "untrusted"`. When a page contains instruction-shaped text, `injection_findings` records the code, severity, character span and a short matching snippet; the first warning repeats up to three of those snippets. Hard rules (forged system tags, secret-exfiltration requests, `curl ... | sh`, `rm -rf /`) can make `--input-gate strict` withhold the body. Soft rules ("ignore previous instructions", role swaps) are reported but never withhold on their own, because papers and security notes quote them legitimately. The scan covers the whole body in overlapping windows, before `max_chars` truncation. This is a screen, not a sandbox: the host must still keep untrusted content, private data and outbound requests out of one autonomous loop.
|
|
366
|
+
|
|
367
|
+
### Privacy and local state
|
|
368
|
+
|
|
369
|
+
- **No telemetry, no account, no API key.** The plugin does not send usage data anywhere; it makes the requests you ask for to the selected search engine, scholarly API or site. Your query and the target URL necessarily leave the machine for that request.
|
|
370
|
+
- **On-disk state is local.** With caching on, page bodies and response metadata live under `~/.web-forager/cache` for one hour; responses that set cookies are not cached, and `Authorization` / `Proxy-Authorization` headers are stripped. On POSIX the cache directory is `0700` and its files `0600`.
|
|
371
|
+
- **Sessions and browser state are local too.** Surf sessions (`~/.web-forager/sessions`) contain visited URLs; browser sessions live under `~/.web-forager/browser/sessions`; `--save-storage-state` writes cookies with mode `0600`; the dedicated browser profile is `0700`. `--verbose` logs the scheme, host and path only; the query string and fragment are dropped, so a signed URL does not put its token on stderr.
|
|
372
|
+
- **The DevTools port is off by default.** `browser/launch_chrome.py` opens only the dedicated profile unless you pass `--cdp`; then it binds Chrome's CDP port to loopback and picks a random free port (`--port 9222` makes it fixed). Any local process can still attach while it is open, so keep it on loopback and close the browser when you are finished.
|
|
373
|
+
- **Personal data is not a target.** The doxxing guard refuses explicit person-profiling queries and warns on a bare name; it is intent-based, not a guarantee. Do not use the tool to collect or assemble private information about an identifiable person.
|
|
374
|
+
|
|
375
|
+
---
|
|
376
|
+
|
|
377
|
+
## Scope and compliance
|
|
378
|
+
|
|
379
|
+
The core engine is an original clean-room implementation: no source is copied from Crawlee, Scrapy, Colly, Katana or any other crawling framework, and no GPL/AGPL dependency is used. The tool **never bypasses a login, CAPTCHA, Cloudflare challenge or paywall**, and it does not run concurrent request floods. `robots.txt` (RFC 9309 subset) is checked before every target page according to `--robots-policy`, each request carries a contactable information-gathering User-Agent, and the default rate is one request per second per origin — search, sitemap and surf hops included.
|
|
380
|
+
|
|
381
|
+
Where a site answers with a verification page, the record says `verification_required` and stops. Where a browser profile has not passed a site's check, the playbook says `blocked` and stops. Neither path tries to satisfy the challenge.
|
|
382
|
+
|
|
383
|
+
---
|
|
384
|
+
|
|
385
|
+
## Requirements
|
|
386
|
+
|
|
387
|
+
Python 3.10+ and the packages in `requirements.txt` (`requests`, `beautifulsoup4`, `trafilatura`). Optional: `playwright` for JavaScript rendering, `pypdf` for PDF text. Browser mode additionally needs Node.js 18+ and a bridge command; the bundled bridge needs `playwright` installed with npm and its Chromium binary.
|
|
388
|
+
|
|
389
|
+
```bash
|
|
390
|
+
pip install playwright && playwright install chromium
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
A missing optional package never crashes a run: Playwright is exit code 6, `pypdf` is exit code 7, and the record carries an install hint. The per-capability matrix is in [DEPENDENCIES.md](DEPENDENCIES.md).
|
|
394
|
+
|
|
395
|
+
## Tests
|
|
396
|
+
|
|
397
|
+
```bash
|
|
398
|
+
python -m pytest
|
|
399
|
+
node --test "browser/tests/*.test.js"
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
333 Python tests and 91 browser tests, all offline. The Python suite starts a temporary local HTTP server for robots.txt, articles, PDF/Office files, feeds, sitemaps, search-result pages and multi-hop surf pages; the browser suite uses a fake backend and fake bridges that speak the same wire protocol, plus snapshots recorded from live pages. CI runs both from a repository clone; release archives exclude the test folders.
|
|
403
|
+
|
|
404
|
+
## Full legal notice
|
|
405
|
+
|
|
406
|
+
### What this tool is for
|
|
407
|
+
|
|
408
|
+
web-forager automates reading publicly reachable pages, for learning, research, monitoring and personal knowledge work only.
|
|
409
|
+
|
|
410
|
+
### What it must not be used for
|
|
411
|
+
|
|
412
|
+
- Identifying, locating, profiling or contacting a private individual, or assembling scattered records into a picture of a person.
|
|
413
|
+
- Harassment, stalking, intimidation, doxxing, or any invasion of another person's privacy.
|
|
414
|
+
- Bypassing logins, CAPTCHAs, paywalls or access controls, or placing load on a site beyond the built-in rate limit.
|
|
415
|
+
- Sending malicious concurrent requests, or any denial-of-service-style access, to a target site.
|
|
416
|
+
- Any use that breaks the law where you are, or the terms of the site you are reading.
|
|
417
|
+
|
|
418
|
+
### Your responsibility
|
|
419
|
+
|
|
420
|
+
It is the end user's responsibility to obey all applicable local, state and federal laws and the Terms of Service and copyright of every site reached. The authors do not grant permission to access anything you are not entitled to read, and do not verify that a given use is lawful. The user bears full legal responsibility for how this software is used.
|
|
421
|
+
|
|
422
|
+
### No warranty
|
|
423
|
+
|
|
424
|
+
The software is provided "as is", without warranty of any kind, express or implied, as stated in the [MIT License](LICENSE). It may fail, return incomplete or outdated content, or change behaviour between versions.
|
|
425
|
+
|
|
426
|
+
### Limitation of liability
|
|
427
|
+
|
|
428
|
+
To the maximum extent permitted by law, the authors and contributors assume no liability and are not responsible for any misuse of this software or for any damage, loss or legal consequence arising from it. Fetched text comes from third-party sites and is not endorsed or checked by this project.
|
|
429
|
+
|
|
430
|
+
### If you redistribute it
|
|
431
|
+
|
|
432
|
+
Keep this disclaimer and the licence notice with any copy you publish or fork.
|
|
433
|
+
|
|
434
|
+
---
|
|
435
|
+
|
|
436
|
+
## Known limitations
|
|
437
|
+
|
|
438
|
+
### Read before use
|
|
439
|
+
|
|
440
|
+
- **This is a low-frequency tool, not a crawler fleet.** Requests are serial and rate-limited per origin (one per second by default); `surf`, search and sitemap reads do not fan out concurrently. It is unsuitable for large-scale harvesting.
|
|
441
|
+
- **PDF extraction reads the text layer only.** Scanned PDFs need OCR, which this plugin does not do.
|
|
442
|
+
- **Login walls are not bypassed.** `zhihu-dig` and `xhs-dig` stop with `login_required`; reuse a logged-in profile through your bridge.
|
|
443
|
+
- **WeChat articles are not read in HTTP mode.** robots.txt disallows `/s`, and with an override the URL returns an environment check. The record says `verification_required`; `wechat-dig` can read the article only in a profile that has already passed that check.
|
|
444
|
+
- **Default search records a robots override.** Bing, 360, Sogou and the arXiv export API disallow their query paths. `--query-robots` defaults to `warn`, so search works but the engine report names the override; set `respect` to refuse a disallowed endpoint, or `ignore` to skip the check.
|
|
445
|
+
- **robots.txt unreachable means deny.** That is the conservative RFC behaviour. `--ignore-robots` is only for resources you are explicitly authorised to access. MCP browser runs always pass `--strict-robots`; CLI browser runs consult robots.txt under `--strict-robots`, and every HTTP and browser redirect hop is re-checked. The record always states whether the check was made (`compliance.robots_checked`).
|
|
446
|
+
|
|
447
|
+
### Edge cases
|
|
448
|
+
|
|
449
|
+
- **Scholarly APIs return metadata and abstracts.** They do not return full papers; respect each platform's terms.
|
|
450
|
+
- **`--render auto` is a heuristic, plus one bounded fallback.** Short visible text plus SPA markers triggers rendering; a minimal script shell triggers one extra render. Complex sites may still need `--render always`, and `--wait-for` / `--scroll` only work when Playwright is installed.
|
|
451
|
+
- **Search depends on public result pages.** Availability varies by network, and every parser (`parse_360_html`, `parse_bing_html`, `parse_sogou_html`, `parse_weixin_html`, `parse_ddg_html`) encodes an assumption about the result-page markup. Each has an offline fixture; a redesign breaks the fixture test rather than silently returning nothing. **Baidu is not supported**: its result page is heavily guarded and no parser is shipped. Sogou may show a human-verification page, and `--engine all` can fail partially. A Chinese query never falls back to Bing, and dictionary pages that match one character of a query are dropped.
|
|
452
|
+
- **Only Bilibili has a site-specific HTTP adapter.** The other sites named above are read as ordinary pages, which is enough for public article pages and not enough for logged-in feeds.
|
|
453
|
+
- **Browser mode needs a working bridge.** `playwright` and its Chromium binary are installed separately, `--doctor` names whichever is missing, and with no bridge at all a live run exits 3.
|
|
454
|
+
- **`surf` follows ordinary links only.** It does not click buttons, execute JavaScript or handle login walls.
|
|
455
|
+
- **DNS is resolved twice.** The guard checks the addresses DNS returns and the HTTP client then resolves again to connect. A hostile resolver that answers differently between the two calls (rebinding) is outside what this check can stop; pair it with an egress policy if that is your threat model.
|
|
456
|
+
|
|
457
|
+
### Experimental features and high-risk switches
|
|
458
|
+
|
|
459
|
+
- **Danmaku is experimental.** Bridge `observe("danmaku", ms)` is preferred; DOM text is a fallback that assumes one visible `mm:ss text` layout, and the record says so in its warnings. Canvas-rendered text still needs OCR.
|
|
460
|
+
- **Browser selectors are heuristics.** Every pattern about a third-party page lives in `browser/lib/selectors.js` and is regression-tested against snapshots derived from live pages (`browser/tests/fixtures/snapshots/`, recorded 2026-11-07). That catches drift after the fact, not before it; `npm run test:live` re-checks the live sites when you set `WEB_FORAGER_LIVE=1` and a bridge.
|
|
461
|
+
- **The injection screen is a screen, not a sandbox.** It matches known phrasings and will miss paraphrases. `trust: "untrusted"` and the host's own permission model are what contain a hostile page; the flag list only makes the boundary visible.
|
|
462
|
+
- **The private-address guard has an off switch.** `--allow-private` (or `WEB_FORAGER_ALLOW_PRIVATE=1`) exists for local fixtures and trusted intranet targets and disables the loopback/LAN/metadata check. Never set it for input you do not control.
|
|
463
|
+
- **CDP attach exposes the profile you attach to.** The bundled bridge uses a dedicated profile, but `WEB_FORAGER_BROWSER_CDP` reuses a browser you already run, with every cookie and login in it.
|
|
464
|
+
|
|
465
|
+
---
|
|
466
|
+
|
|
467
|
+
## Issues and feedback
|
|
468
|
+
|
|
469
|
+
**Ordinary bugs and feature requests**: the repository's [Issues](https://github.com/JiamingShi-JsPhys/web-forager/issues) page, with reproduction steps, expected behaviour, actual behaviour and your environment (operating system, Python version).
|
|
470
|
+
|
|
471
|
+
**Security vulnerabilities (this category only)**: do **not** use the public issue tracker — email `jiamingshi_git@outlook.com` instead.
|
|
472
|
+
|
|
473
|
+
- **Subject**: must carry the prefix `report_web-forager:` plus a short summary.
|
|
474
|
+
- Example: `report_web-forager: path traversal can read outside the target folder`
|
|
475
|
+
- **Body**: must include detailed reproduction steps and the environment.
|
|
476
|
+
|
|
477
|
+
*Contributors who file clear, reproducible issues or pull requests are credited in the next release's notes and in [Acknowledgements](#acknowledgements).*
|
|
478
|
+
|
|
479
|
+
## Acknowledgements
|
|
480
|
+
|
|
481
|
+
**NotFatJustRound** — early testing and feedback.
|
|
482
|
+
|
|
483
|
+
**DeepSec** — the code-security auditor this repository is scanned with. It found the two command-injection paths closed in 1.2.3 and the supply-chain issue in the same release.
|
|
484
|
+
|
|
485
|
+
**Claude** (Anthropic) — some assistance on parts of this project: code-snippet suggestions, and suggested changes to the tests, documentation and release notes. Every suggestion was reviewed and adopted only where a test, a check or a direct verification could confirm it; the author made the final call on the wording.
|
|
486
|
+
|
|
487
|
+
[Full acknowledgements](docs/ACKNOWLEDGEMENTS.md)
|
|
488
|
+
|
|
489
|
+
## Trademarks
|
|
490
|
+
|
|
491
|
+
"Bilibili", "Zhihu", "Xiaohongshu", "arXiv", "Crossref", "PubMed", "Claude", "Codex", "CodeBuddy", "ZCode" and "DeepSeek" are trademarks or registered trademarks of their respective owners. web-forager is an independent, third-party tool with no affiliation, endorsement or sponsorship from any of them. All requests go to public pages or documented APIs; users are responsible for complying with each platform's terms.
|
|
492
|
+
|
|
493
|
+
---
|
|
494
|
+
|
|
495
|
+
## License and author
|
|
496
|
+
|
|
497
|
+
MIT. See [LICENSE](LICENSE) and [THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt).
|
|
498
|
+
|
|
499
|
+
**Author and maintainer**: Jiaming Shi
|