donsetch 4.10.5 → 5.0.0-beta.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +13 -12
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -2,6 +2,8 @@
2
2
 
3
3
  **The web, for AI agents.** Fetch, search, crawl and screenshot from one local binary. Zero API keys, zero accounts.
4
4
 
5
+ **v5 beta:** a new fetch engine. No browser in the fetch path: a wall is answered by the right client identity, the site's own challenge program run in-process, or the site's own public interface.
6
+
5
7
  [![Release](https://img.shields.io/github/v/release/dondai44423/donsetch?color=00d4aa&style=flat-square)](https://github.com/dondai44423/donsetch/releases)
6
8
  [![CI](https://img.shields.io/github/actions/workflow/status/dondai44423/donsetch/ci.yml?label=CI&style=flat-square)](https://github.com/dondai44423/donsetch/actions/workflows/ci.yml)
7
9
  [![npm downloads](https://img.shields.io/npm/dm/donsetch?color=cb3837&style=flat-square&label=downloads)](https://www.npmjs.com/package/donsetch)
@@ -10,7 +12,7 @@
10
12
 
11
13
  ![DonSeTch, the web, for AI agents](https://raw.githubusercontent.com/dondai44423/donsetch/master/assets/herobanner.png)
12
14
 
13
- DonSeTch gives any AI agent full web research from a single local process: four tools, one Rust binary, no API keys, no accounts, nothing to configure. The transport is built from scratch (no hyper, no Playwright, no Selenium), which is why the fetch tier is fast *and* stealthy and the tool schema fits in ~2.4k tokens.
15
+ DonSeTch gives any AI agent full web research from a single local process: four tools, one Rust binary, no API keys, no accounts, nothing to configure. Fetching launches no browser (no Playwright, no Selenium, no headless Chrome): pages are read as a real client reads them, at curl's cost, and all four tool schemas fit in ~2k tokens.
14
16
 
15
17
  Works with every MCP client (Claude Code, Cursor, OpenCode, Pi, Hermes) and as a standalone CLI.
16
18
 
@@ -18,13 +20,12 @@ Works with every MCP client (Claude Code, Cursor, OpenCode, Pi, Hermes) and as a
18
20
 
19
21
  | | What it does |
20
22
  |---|---|
21
- | **Real Chrome TLS** | Drives Chrome's own BoringSSL natively: your ClientHello IS Chrome's, ML-DSA signature algorithms included. Emergent from the real engine, not a faked table that rots. |
22
- | **Temporal stealth** | TLS session resumption, 304 revalidation, persistent cookies, connection pooling, TCP Fast Open. The loudest remaining bot tell, and nobody else fakes it. |
23
- | **Solve-and-bounce** | The browser solves the challenge and hands cookies to tier 1, which then fetches at full speed. The browser almost never fetches content. |
23
+ | **Real clients, whole** | Eight client identities (Chrome on Linux, Windows and Android, Firefox, Safari, an Android app client, a plain client), each carrying one real profile's TLS, HTTP/2, header order and client hints, chosen per site. The Chrome identities match real Chrome's JA4 and HTTP/2 fingerprint exactly. |
24
+ | **Challenges, in one process** | A wall that needs the page's own program gets it run in an embedded V8 against a small bounded host, and the credential it mints is spent by the same identity. AWS WAF passes cold in 2-5 s. No browser, no rendering. |
24
25
  | **Keyless search** | 10+ backends in parallel, fused by cross-engine consensus plus local semantic reranking. No keys, $0 forever. BYOK is optional. |
25
26
  | **Pixel-fusion PDF** | Glyphs and rendered pixels come from the same stream and are fused deterministically, with a per-region trust audit. Scanned PDFs auto-OCR. |
26
- | **Token control** | MCP results use compact, unguessable link and search handles; one-shot CLI search prints reusable URLs. `focus`, `toc`, `section`, `must_contain` and `since_last` cut a page to what the agent actually needs. |
27
- | **Self-improving fetch** | Cookie lifetimes adapt, walls that beat a real browser twice go into cooldown, searches pre-solve known walls. All local state, receipts in `status` and `doctor --improve`. |
27
+ | **Read only what you need** | `must_contain` answers "is X on this page?" per term with counts, sections and excerpts in a few hundred characters; `focus` returns the passages that answer, best first; `toc` then `section` reads one part. Every view searches the whole page, PDFs and site adapters included. |
28
+ | **Self-improving fetch** | Remembers which identity each site accepts, what each bot-wall vendor lets through, which route answered and how far each site's cache headers can be trusted. Hard-corpus first-try content: 13/28 cold, 26/28 learned. All local state, receipts in `status` and `doctor --improve`. |
28
29
  | **One command to verify** | `donsetch doctor` sweeps config, search health, egress, TLS, browser, DNS, secret permissions, and fixes what is mechanically fixable. |
29
30
 
30
31
  ## Install
@@ -47,7 +48,7 @@ Downloads the prebuilt binary for your platform from [GitHub Releases](https://g
47
48
  | Platform | Asset | OCR + rerank |
48
49
  |---|---|---|
49
50
  | Linux x86_64 (glibc >= 2.35) | `donsetch-linux-x64.tar.gz` | yes |
50
- | Linux ARM64 | `donsetch-linux-arm64.tar.gz` | no, and PDF is fragile (ONNX has no working aarch64 prebuilt) |
51
+ | Linux ARM64 (glibc >= 2.35) | `donsetch-linux-arm64.tar.gz` | no (core only: fetch, search, crawl, PDF; ONNX has no working aarch64 runtime) |
51
52
  | macOS Apple Silicon (>= 13.4) | `donsetch-darwin-arm64.tar.gz` | yes |
52
53
  | macOS Intel (>= 13.4) | `donsetch-darwin-x64.tar.gz` | yes |
53
54
  | Windows x86_64 | `donsetch-win32-x64.tar.gz` | yes |
@@ -87,7 +88,7 @@ Every client gets the `[meta]` fold by default: compact state and source evidenc
87
88
  ### 2. As a CLI (for humans and scripts)
88
89
 
89
90
  ```bash
90
- donsetch fetch https://example.com --focus "pricing"
91
+ donsetch fetch https://example.com/pricing --must-contain "free tier"
91
92
  donsetch search "rust async patterns" --intent code
92
93
  donsetch crawl https://docs.python.org --mode map --topic asyncio
93
94
  donsetch screenshot https://example.com --out page.png
@@ -99,10 +100,10 @@ donsetch update
99
100
 
100
101
  | Tool | What it does |
101
102
  |---|---|
102
- | `web_fetch` | Any URL as clean markdown. HTTP first, escalates to a headless browser on bot walls. PDFs with OCR and per-page confidence, `focus` / `toc` / `section`, pagination, `actions` for in-page control, `must_contain` probes. |
103
+ | `web_fetch` | Any URL as clean markdown, or only the part that answers: `must_contain` probes, `focus` passages, `toc` → `section`. Walls answered in-process, never by a browser. PDFs with OCR and per-page confidence, site adapters, page memory, batches of up to 12. |
103
104
  | `web_search` | Keyless multi-engine search: 10+ backends, consensus plus semantic reranking, query-aware official-source placement. Ranked URLs and snippets, never a scraped article dump. |
104
105
  | `web_crawl` | Best-first same-domain crawl. Sitemap plus frontier, `focus` ranking, elastic pacing, resume tokens, honest stop reasons. |
105
- | `web_screenshot` | Rendered PNG of any URL through the same browser tier, with the usual URL safety guards. |
106
+ | `web_screenshot` | Rendered PNG of any URL through DonSeTch's browser (used only for screenshots and login), with the usual URL safety guards. |
106
107
 
107
108
  MCP results default to self-contained text: leading `[meta]` JSON followed by source evidence, including for unknown client bridges. PNGs remain image blocks.
108
109
 
@@ -113,8 +114,8 @@ Every failure is structured: a stable `code` (`wall.challenge`, `guard.ssrf`, `d
113
114
  - **Search without keys.** Six keyless engines across four independent index families plus eight official verticals (GitHub, Wikipedia, HN, Semantic Scholar, arXiv, StackExchange, MDN, Google News), merged by consensus and re-ranked locally by an ONNX cross-encoder. 95.5% answer-in-snippet over 110 questions across 11 niches with no keys at all.
114
115
  - **PDFs done properly.** A custom PDFium FFI, no Python subprocess. Tables and borders come from the rendered bitmap, text from the glyph stream, and the trust audit flags exactly the regions that need OCR.
115
116
  - **Crawl with manners.** Per-host adaptive pacing that honors `Retry-After` and robots `Crawl-delay`, cross-process politeness so two crawls do not double a site's rate, near-duplicate detection, resume tokens that survive restarts, and honest stop reasons.
116
- - **Fetch that answers honestly.** `content_ok`, `thin`, `changed` with section diffs, `decoy suspected` instead of silently passing a cloaked page. No fake success.
117
- - **Login to walled sites.** `donsetch login x.com` opens YOUR browser, you sign in, and the session cookies land in the 0600 vault that tier-1 fetches and tier-2 renders already replay. Credentials never enter DonSeTch.
117
+ - **Fetch that answers honestly.** `content_ok`, `thin`, `changed` with section diffs, and `must_contain` says `unverified` instead of a false "not found" when part of a page was unread. No fake success.
118
+ - **Walls named, not guessed.** A bot wall names its vendor and what it asks for; a login wall is a declared `wall.auth` (fetch carries no session in v5; session support is planned).
118
119
  - **Works everywhere.** Linux, macOS, Windows. npm, Homebrew, the Pi agent as a native extension, and the DeepSeek Harness as a first-class plugin.
119
120
 
120
121
  ## Sponsors
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "donsetch",
3
- "version": "4.10.5",
3
+ "version": "5.0.0-beta.2",
4
4
  "description": "MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "author": "Bishesh Bhandari",