@aliasmode/anyfetch 0.7.1 → 0.8.1

package/README.md

@@ -8,26 +8,27 @@ Agent harnesses fetch pages with a bot identity, and a large part of the web ans
 
 | 97 popular sites that block AI agents | Pages read |
 | --- | --- |
-| **Anyfetch** | **75** |
-| curl | 47 |
-| Reference MCP fetch server | 41 |
+| **Anyfetch** | **82** |
+| **Anyfetch with a [NobleProxy](https://nobleproxy.com/?utm_source=anyfetch&utm_medium=readme&utm_campaign=anyfetch) residential IP** | **87** |
+| curl | 46 |
+| Reference MCP fetch server | 39 |
 | Claude Code built-in `WebFetch` | 36 |
 
-Anyfetch read 21 sites that no other tool in the table could read. Among them: The New York Times, Reuters, Bloomberg, The Washington Post, Medium, Reddit, Stack Overflow and YouTube.
+Anyfetch read 28 sites that no other tool in the table could read. Among them: The New York Times, Reuters, Bloomberg, The Washington Post, The Economist, Medium, Reddit, Stack Overflow, IMDb and YouTube.
 
-| 30 research questions, each asked twice | Answered correctly both times |
+| 33 research questions, each asked twice | Answered correctly both times |
 | --- | --- |
-| **Agent with Anyfetch** | **30** |
-| Same agent with Claude Code built-in `WebSearch` and `WebFetch` | 25 |
+| **Agent with Anyfetch** | **32** |
+| Same agent with Claude Code built-in `WebSearch` and `WebFetch` | 27 |
 
-Ten of the questions are about a page on a site that blocks AI agents. The built-in fetch could not read any of those 10 pages. Its search found the answer in another source for 5 of them. Anyfetch read all 10. The questions were written from pages that Anyfetch can read, so this group shows what the built-in tools miss.
+Thirteen of the questions are about a page on a site that blocks AI agents. The built-in fetch could not read any of those 13 pages. Its search found the answer in another source for 7 of them. Anyfetch answered 12 in both runs. The questions were written from pages that Anyfetch can read, so this group shows what the built-in tools miss.
 
 | One page, started fresh | Time | Peak memory |
 | --- | --- | --- |
 | **Anyfetch** | **0.19 s** | **17 MB** |
 | Headless Chromium | 1.15 s | 544 MB |
 
-Measured on 2026-10-11 from a cloud server, with no proxy. See [how we measured](#benchmark) and how to run it again.
+Sites: measured on 2026-10-12 with version 0.8.0. Questions: measured on 2026-10-12 with version 0.7.1. Time and memory: measured on 2026-10-11. All from a cloud server, with no proxy except in the proxy row. See [how we measured](#benchmark) and how to run it again.
 
 ## What your agent gets
 
@@ -36,9 +37,10 @@ Measured on 2026-10-11 from a cloud server, with no proxy. See [how we measured]
 - **Answers, not whole pages.** Give a question with the URL, and get only the parts of the page that match it. The Python 3.15 release notes are 218,000 characters; the answer to one question about them came back in 2,200.
 - **Clean Markdown.** Links, tables and code blocks stay intact. Long pages come in parts.
 - **PDFs.** Papers and reports come back as text, page by page.
-- **Sites that need JavaScript.** Anyfetch reads the data inside the page, and uses the site's public API or feed for Stack Overflow, Reddit, X posts, PyPI and crates.io. No browser runs. For pages that hold nothing without their scripts, it can use Lightpanda, an optional small engine.
-- **News and answers.** Full articles on Reuters and Bloomberg, and the answers on Quora, from the data that these sites send with the page.
-- **Browser-grade requests.** Safari's TLS, HTTP/2 and header fingerprint. A refused fetch is retried automatically, as the same browser or as a second one.
+- **Sites that need JavaScript.** Anyfetch reads the data inside the page, and uses the site's public API or feed for Stack Overflow, Reddit, IMDb, X posts, PyPI and crates.io. A Reddit thread comes with every comment and its score. No browser runs. For pages that hold nothing without their scripts, it can use Lightpanda, an optional small engine.
+- **News and answers.** Full articles on Reuters and Bloomberg, the latest articles of The Economist, and the answers on Quora, from the data that these sites send.
+- **Home pages and listings.** A page with no single article comes back as its headlines and links, in page order.
+- **Browser-grade requests.** Safari's TLS, HTTP/2 and header fingerprint. A refused fetch is retried automatically, as the same browser or as a second one. Bot checks that are one fixed step, as on Reddit and Behance, are passed.
 - **Cloudflare checks.** When a site insists on a JavaScript check, your own Firefox passes it once. After that, Anyfetch reads the site with plain requests.
 - **Fast on repeat.** Several pages in one call, fetched at the same time. A page fetched in the last 15 minutes comes from a local copy.
 - **Lightweight.** A 16 MB Rust binary. No bundled browser, no JavaScript engine, no Python, no Node.
@@ -322,7 +324,9 @@ host:port:username:password
 
 Anyfetch then sends every search and fetch through that HTTP proxy. The file holds the password in plain text, so keep it private: `chmod 600 ~/.config/anyfetch/proxy`. Delete the file to go direct again.
 
-Need a proxy? [NobleProxy](https://nobleproxy.com/?utm_source=anyfetch&utm_medium=readme&utm_campaign=anyfetch) is our own service: static residential IPs that only you use, with unlimited bandwidth and HTTP support.
+Some sites accept your own address and refuse the proxy's address. To reach both kinds, add a second line with the word `direct`. Anyfetch then tries the proxy first, and asks the site directly when the proxy route is refused.
+
+Need a proxy? Use [NobleProxy](https://nobleproxy.com/?utm_source=anyfetch&utm_medium=readme&utm_campaign=anyfetch), our own service. In our test, one NobleProxy address took Anyfetch from 82 to 87 of the 97 sites, with the same result in every run. It opened The Wall Street Journal, Tripadvisor, Yelp, Etsy and Bandcamp, which refuse a server address. You get a static residential IP that only you use, with unlimited bandwidth and HTTP support.
 
 ## What it does not do
 
@@ -334,18 +338,36 @@ Does your agent need a real browser, with logins and saved sessions? Use [AliasM
 
 The numbers at the top come from two scripts and one question set in [`bench/`](bench).
 
-- **Pages read.** `bench/bench.py` reads each of the 97 sites in [`bench/sites.txt`](bench/sites.txt) once with each tool. A page counts as read when the tool returns at least 500 characters of text and the text is not a bot check. The same rule is applied to every tool. Anyfetch ran three times and read 73, 76 and 75 pages; the table shows the middle run. Claude Code's `WebFetch` cannot be scripted, so an agent called it once per site and recorded if real content came back.
+- **Pages read.** `bench/bench.py` reads each of the 97 sites in [`bench/sites.txt`](bench/sites.txt) once with each tool. A page counts as read when the tool returns at least 500 characters of text and the text is not a bot check. The same rule is applied to every tool. Anyfetch ran three times and read 82, 81 and 82 pages; the table shows the middle run. Through a [NobleProxy](https://nobleproxy.com/?utm_source=anyfetch&utm_medium=readme&utm_campaign=anyfetch) static residential IP, a fixed address of a US cable provider, with the `direct` line in the proxy file, it read 87 pages in each of three runs. The proxy added The Wall Street Journal, Tripadvisor, Yelp, Etsy and Bandcamp. The other tools were not tested through the proxy. Claude Code's `WebFetch` cannot be scripted, so an agent called it once per site and recorded if real content came back.
 - **Time and memory.** `bench/speed.py` reads 10 pages with each program, started fresh for every page, and reports the median.
 
-- **Research questions.** [`bench/tasks.json`](bench/tasks.json) has 30 questions. Each has one exact answer and the sentence on the source page that proves it. The pages are from July to October 2026, so a model cannot answer from memory. A Claude Sonnet agent answered the questions with Anyfetch only, and again with the built-in `WebSearch` and `WebFetch` of Claude Code only. The instructions were the same: at most 8 web calls for each question, and the answer `UNKNOWN` when the tools do not give it. Anyfetch ran with no proxy, no Firefox and no Lightpanda. Each tool ran twice, and a question counts when both runs are correct.
+- **Research questions.** [`bench/tasks.json`](bench/tasks.json) has 33 questions. Each has one exact answer and the sentence on the source page that proves it. The pages are from July to October 2026, so a model cannot answer from memory. A Claude Sonnet agent answered the questions with Anyfetch only, and again with the built-in `WebSearch` and `WebFetch` of Claude Code only. The instructions were the same: at most 8 web calls for each question, and the answer `UNKNOWN` when the tools do not give it. Anyfetch ran with no proxy, no Firefox and no Lightpanda. Each tool ran twice, and a question counts when both runs are correct.
 
 | Question group | Anyfetch | Built-in tools |
 | --- | --- | --- |
-| A page on a site that blocks AI agents | 10 of 10 | 5 of 10 |
+| A page on a site that blocks AI agents | 12 of 13 | 7 of 13 |
 | A page on an open site | 10 of 10 | 10 of 10 |
 | No URL: search first, then read | 10 of 10 | 10 of 10 |
 
-In each single run the built-in tools answered 26 questions, but not the same 26. Three things to know about this test. The question writer used Anyfetch to check each answer, so the first group has only pages that Anyfetch can read. Four more sites on the writer's list (Reuters, Bloomberg, Financial Times and Quora) have no question, because it could not read an article page there. The site test above reads the home page of Reuters and Bloomberg, which Anyfetch can read. Since version 0.7.0, Anyfetch also reads article pages on Reuters and Bloomberg and the answers on Quora. The Financial Times needs a subscription. One Anyfetch test agent stopped after its first question, so its 5 questions were run again. Every answer is in [`bench/results/tasks.tsv`](bench/results/tasks.tsv).
+In single runs, Anyfetch answered 32 and 33 questions, and the built-in tools 28 and 27. The one Anyfetch miss was a Reddit thread: Reddit limited the rate of the test machine that day, and in one run Anyfetch read a link from a comment in place of the link of the post.
+
+Three things to know about this test. The question writer used Anyfetch to check each answer, so the first group has only pages that Anyfetch can read. The Financial Times was on the writer's list and has no question: its articles need a subscription. Thirty of the questions are one day older than the test; 28 of their answers were checked on the source page again that day. Every answer is in [`bench/results/tasks.tsv`](bench/results/tasks.tsv).
+
+We also read the same 97 sites with eight other tools, each with its default settings and no API key:
+
+| Tool | Pages read |
+| --- | --- |
+| **Anyfetch** | **82** |
+| Exa MCP (hosted, answers from stored copies) | 78 |
+| Scrapling, stealth browser | 71 |
+| Scrapling, plain HTTP | 69 |
+| Hermes Agent `web_extract` (hosted readers) | 68 |
+| Crawl4AI | 66 |
+| Jina Reader | 61 |
+| Playwright MCP | 52 |
+| DuckDuckGo MCP server | 49 |
+
+How each one ran, and what to know before you compare, is in [`bench/other-tools/NOTES.md`](bench/other-tools/NOTES.md). The full ranking with every site is on [aliasmode.com](https://aliasmode.com/anyfetch/best-web-fetch-tools-for-ai-agents/).
 
 The per-site results are in [`bench/results/`](bench/results). Sites change their rules often, and results differ between networks, so run it yourself:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aliasmode/anyfetch",
-  "version": "0.7.1",
+  "version": "0.8.1",
   "description": "Web search and web fetch for AI agents, with no API keys. Reads the sites that refuse AI agents.",
   "keywords": ["mcp", "mcp-server", "web-search", "web-fetch", "ai-agents", "claude-code", "codex", "cursor"],
   "license": "Apache-2.0",
@@ -11,10 +11,10 @@
   "files": ["run.js"],
   "engines": { "node": ">=16" },
   "optionalDependencies": {
-    "@aliasmode/anyfetch-darwin-arm64": "0.7.1",
-    "@aliasmode/anyfetch-darwin-x64": "0.7.1",
-    "@aliasmode/anyfetch-linux-arm64": "0.7.1",
-    "@aliasmode/anyfetch-linux-x64": "0.7.1",
-    "@aliasmode/anyfetch-win32-x64": "0.7.1"
+    "@aliasmode/anyfetch-darwin-arm64": "0.8.1",
+    "@aliasmode/anyfetch-darwin-x64": "0.8.1",
+    "@aliasmode/anyfetch-linux-arm64": "0.8.1",
+    "@aliasmode/anyfetch-linux-x64": "0.8.1",
+    "@aliasmode/anyfetch-win32-x64": "0.8.1"
   }
 }