donsetch 4.2.4 → 4.2.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +103 -50
  2. package/package.json +25 -9
package/README.md CHANGED
@@ -1,12 +1,31 @@
1
1
  # DonSeTch
2
2
 
3
- > Web fetch, search, and crawl for AI agents. Zero API keys. Chrome-true TLS. Built in Rust.
3
+ **The web, for AI agents.** Fetch, search, crawl and screenshot from one local binary. Zero API keys, zero accounts.
4
4
 
5
5
  [![Release](https://img.shields.io/github/v/release/dondai44423/donsetch?color=00d4aa&style=flat-square)](https://github.com/dondai44423/donsetch/releases)
6
6
  [![CI](https://img.shields.io/github/actions/workflow/status/dondai44423/donsetch/ci.yml?label=CI&style=flat-square)](https://github.com/dondai44423/donsetch/actions/workflows/ci.yml)
7
+ [![npm downloads](https://img.shields.io/npm/dm/donsetch?color=cb3837&style=flat-square&label=downloads)](https://www.npmjs.com/package/donsetch)
8
+ [![GitHub stars](https://img.shields.io/github/stars/dondai44423/donsetch?style=flat-square&color=e3b341)](https://github.com/dondai44423/donsetch/stargazers)
7
9
  [![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-00d4aa?style=flat-square)](https://github.com/dondai44423/donsetch/blob/master/LICENSE)
8
10
 
9
- DonSeTch gives AI agents web research from a single local process : fetch any URL, search across 10+ engines, and crawl multi-page docs. Chrome-true TLS fingerprinting, bot-wall bypass, PDF extraction with OCR, semantic reranking. Zero API keys.
11
+ ![DonSeTch, the web, for AI agents](https://raw.githubusercontent.com/dondai44423/donsetch/master/assets/herobanner.png)
12
+
13
+ DonSeTch gives any AI agent full web research from a single local process: four tools, one Rust binary, no API keys, no accounts, nothing to configure. The transport is built from scratch (no hyper, no Playwright, no Selenium), which is why the fetch tier is fast *and* stealthy and the tool schema fits in ~2.4k tokens.
14
+
15
+ Works with every MCP client (Claude Code, Cursor, OpenCode, Pi, Hermes) and as a standalone CLI.
16
+
17
+ ## Why it's different
18
+
19
+ | | What it does |
20
+ |---|---|
21
+ | **Real Chrome TLS** | Drives Chrome's own BoringSSL natively: your ClientHello IS Chrome's, ML-DSA signature algorithms included. Emergent from the real engine, not a faked table that rots. |
22
+ | **Temporal stealth** | TLS session resumption, 304 revalidation, persistent cookies, connection pooling, TCP Fast Open. The loudest remaining bot tell, and nobody else fakes it. |
23
+ | **Solve-and-bounce** | The browser solves the challenge and hands cookies to tier 1, which then fetches at full speed. The browser almost never fetches content. |
24
+ | **Keyless search** | 10+ backends in parallel, fused by cross-engine consensus plus local semantic reranking. No keys, $0 forever. BYOK is optional. |
25
+ | **Pixel-fusion PDF** | Glyphs and rendered pixels come from the same stream and are fused deterministically, with a per-region trust audit. Scanned PDFs auto-OCR. |
26
+ | **Token control** | Links render as `[text](L12)` and results as `S1…Sn`, so `fetch S3` costs 3 tokens instead of 80. `focus`, `toc`, `section`, `must_contain` and `since_last` cut a page to what the agent actually needs. |
27
+ | **Self-improving fetch** | Cookie lifetimes adapt, walls that beat a real browser twice go into cooldown, searches pre-solve known walls. All local state, receipts in `status` and `doctor --improve`. |
28
+ | **One command to verify** | `donsetch doctor` sweeps config, search health, egress, TLS, browser, DNS, secret permissions, and fixes what is mechanically fixable. |
10
29
 
11
30
  ## Install
12
31
 
@@ -14,72 +33,106 @@ DonSeTch gives AI agents web research from a single local process : fetch any UR
14
33
  npm install -g donsetch
15
34
  ```
16
35
 
17
- Downloads the prebuilt binary for your platform from [GitHub Releases](https://github.com/dondai44423/donsetch/releases) with SHA256 verification.
36
+ Downloads the prebuilt binary for your platform from [GitHub Releases](https://github.com/dondai44423/donsetch/releases) with SHA256 verification. No build tools needed.
18
37
 
19
- | Platform | Binary |
20
- |---|---|
21
- | Linux x86_64 | `donsetch-linux-x64.tar.gz` |
22
- | Linux arm64 | `donsetch-linux-arm64.tar.gz` |
23
- | macOS arm64 | `donsetch-darwin-arm64.tar.gz` |
24
- | Windows x86_64 | `donsetch-win32-x64.tar.gz` |
25
-
26
- ## Troubleshooting install
27
-
28
- - **pnpm or bun:** approve the `donsetch` build script (`pnpm approve-builds`
29
- or the equivalent bun approval), then reinstall. If scripts were blocked,
30
- running `npx donsetch` invokes the self-healing shim.
31
- - **`--ignore-scripts`:** postinstall is intentionally skipped. Run
32
- `node node_modules/donsetch/install.js`, or invoke `npx donsetch` to
33
- download the binary when network access is available.
34
- - **Proxy:** set `HTTPS_PROXY` (or `https_proxy`, `HTTP_PROXY`, or
35
- `http_proxy`) to an HTTP CONNECT proxy.
36
- - **Release mirror:** set `DONSETCH_RELEASES_BASE` to a mirror containing
37
- `<tag>/<asset>` paths, for example
38
- `https://mirror.example/donsetch/releases`.
39
- - **Windows:** the installer requires `tar`; Windows 10 version 1803 and
40
- newer include it.
41
- - **musl/Alpine:** the published Linux binaries use glibc. Build from source
42
- with `cargo build --release` on musl systems.
43
- - **Windows ARM64:** the x64 build runs under Windows emulation; no native
44
- ARM64 asset is required.
45
-
46
- ## Two ways to use it
47
-
48
- ### MCP Server (for AI agents)
38
+ | Platform | Asset | OCR + rerank |
39
+ |---|---|---|
40
+ | Linux x86_64 (glibc >= 2.35) | `donsetch-linux-x64.tar.gz` | yes |
41
+ | Linux ARM64 | `donsetch-linux-arm64.tar.gz` | no, and PDF is fragile (ONNX has no working aarch64 prebuilt) |
42
+ | macOS Apple Silicon | `donsetch-darwin-arm64.tar.gz` | yes |
43
+ | macOS Intel | `donsetch-darwin-x64.tar.gz` | no (ONNX has no working x64 prebuilt) |
44
+ | Windows x86_64 | `donsetch-win32-x64.tar.gz` | yes |
45
+ | Windows ARM64 | same x64 asset, under emulation | yes |
49
46
 
50
- ```json
51
- {
52
- "mcpServers": {
53
- "donsetch": { "command": "donsetch", "args": ["mcp"] }
54
- }
55
- }
56
- ```
47
+ **Verify the install:** `donsetch doctor` (fast local sweep), `doctor --deep` (adds live browser and egress probes), `doctor --fix` (repairs mechanical problems), `doctor --json` (machine-readable, also prints ready-to-paste MCP registration blocks for your client).
57
48
 
58
- Or with `npx` (no global install):
49
+ **If something goes wrong:**
50
+
51
+ - **pnpm or bun:** the postinstall needs approval. `pnpm approve-builds` (or the bun equivalent), then reinstall. If scripts were blocked, `npx donsetch` runs the self-healing shim.
52
+ - **`--ignore-scripts`:** postinstall is intentionally skipped. Run `node node_modules/donsetch/install.js`, or use `npx donsetch` to download the binary when network access is available.
53
+ - **Proxy:** set `HTTPS_PROXY` (or `https_proxy`, `HTTP_PROXY`, `http_proxy`) to an HTTP CONNECT proxy.
54
+ - **Release mirror:** set `DONSETCH_RELEASES_BASE` to a mirror serving `<tag>/<asset>` paths, for example `https://mirror.example/donsetch/releases`.
55
+ - **Windows:** the installer needs `tar`, included since Windows 10 1803.
56
+ - **musl/Alpine:** published Linux binaries are glibc. Build from source on musl.
57
+ - **First OCR or search run:** it downloads the models (~24MB reranker, ~37MB OCR) and caches them forever.
58
+
59
+ ## Quickstart
60
+
61
+ ### 1. As an MCP server (for agents)
59
62
 
60
63
  ```json
61
64
  {
62
65
  "mcpServers": {
63
- "donsetch": { "command": "npx", "args": ["donsetch", "mcp"] }
66
+ "donsetch": { "command": "donsetch", "args": ["mcp", "--supervised"] }
64
67
  }
65
68
  }
66
69
  ```
67
70
 
68
- Three tools: `web_fetch`, `web_search`, `web_crawl`.
71
+ `--supervised` is the crash-only daemon: a panic becomes a blip, the daemon restarts, the session survives. Without a global install, use `"command": "npx", "args": ["donsetch", "mcp"]`.
72
+
73
+ Prefer HTTP over stdio? `donsetch mcp --http --port 8765`, clients connect to `http://localhost:8765/mcp`. Sessions, cancellation, `/health`, token auth (`DONSETCH_HTTP_TOKEN`) and per-request timeouts are documented in `donsetch mcp --help`.
69
74
 
70
- ### CLI (for humans and scripts)
75
+ If your client shows only half of each result (tool metadata but no page text, or text but no citable URLs), it is dropping one of the two MCP result surfaces. `DONSETCH_MCP_TEXT_ONLY=1 donsetch mcp` forces the `[meta]` fold that fixes it for every client.
76
+
77
+ ### 2. As a CLI (for humans and scripts)
71
78
 
72
79
  ```bash
73
- donsetch fetch https://example.com
74
- donsetch search "rust async patterns"
75
- donsetch crawl https://docs.example.com --topic "api reference"
76
- donsetch keys add tinyfish sk-tinyfish-...
80
+ donsetch fetch https://example.com --focus "pricing"
81
+ donsetch search "rust async patterns" --intent code
82
+ donsetch crawl https://docs.python.org --mode map --topic asyncio
83
+ donsetch screenshot https://example.com --out page.png
77
84
  donsetch doctor
78
85
  donsetch update
79
86
  ```
80
87
 
88
+ ## The 4 tools
89
+
90
+ | Tool | What it does |
91
+ |---|---|
92
+ | `web_fetch` | Any URL as clean markdown. HTTP first, escalates to a headless browser on bot walls. PDFs with OCR and per-page confidence, `focus` / `toc` / `section`, pagination, `actions` for in-page control, `must_contain` probes, `archive` resurrection. |
93
+ | `web_search` | Keyless multi-engine search: 10+ backends, consensus plus semantic reranking, query-aware official-source placement. Ranked URLs and snippets, never a scraped article dump. |
94
+ | `web_crawl` | Best-first same-domain crawl. Sitemap plus frontier, `focus` ranking, elastic pacing, resume tokens, honest stop reasons. |
95
+ | `web_screenshot` | Rendered PNG of any URL through the same browser tier, with the usual URL safety guards. |
96
+
97
+ Every failure is structured: a stable `code` (`wall.challenge`, `guard.ssrf`, `deadline.hit`, `network.dns`…), an `errorKind` (`permanent`, `transient`, `walled`) and a `next_action`, so agents branch on codes instead of parsing prose.
98
+
99
+ ## Highlights
100
+
101
+ - **Search without keys.** Six keyless engines across four independent index families plus eight official verticals (GitHub, Wikipedia, HN, Semantic Scholar, arXiv, StackExchange, MDN, Google News), merged by consensus and re-ranked locally by an ONNX cross-encoder. 95.5% answer-in-snippet over 110 questions across 11 niches with no keys at all.
102
+ - **PDFs done properly.** A custom PDFium FFI, no Python subprocess. Tables and borders come from the rendered bitmap, text from the glyph stream, and the trust audit flags exactly the regions that need OCR.
103
+ - **Crawl with manners.** Per-host adaptive pacing that honors `Retry-After` and robots `Crawl-delay`, cross-process politeness so two crawls do not double a site's rate, near-duplicate detection, resume tokens that survive restarts, and honest stop reasons.
104
+ - **Fetch that answers honestly.** `content_ok`, `thin`, `changed` with section diffs, `archive.stale` with the snapshot's age, `decoy suspected` instead of silently passing a cloaked page. No fake success.
105
+ - **Login to walled sites.** `donsetch login x.com` opens YOUR browser, you sign in, and the session cookies land in the 0600 vault that tier-1 fetches and tier-2 renders already replay. Credentials never enter DonSeTch.
106
+ - **Works everywhere.** Linux, macOS, Windows. npm, Homebrew, the Pi agent as a native extension, and the DeepSeek Harness as a first-class plugin.
107
+
108
+ ## Sponsors
109
+
110
+ DonSeTch is free and open source, and stays that way. Sponsorship pays for the time it takes to keep shipping.
111
+
112
+ | Tier | Price | What you get |
113
+ |---|---|---|
114
+ | Bronze | $10/mo | Name + link in the Sponsors section |
115
+ | Silver | $25/mo | Small logo + link in the Sponsors section |
116
+ | Gold | $49/mo | Large logo + link, pinned at the top of the Sponsors section |
117
+
118
+ Prepaid monthly, cancel anytime. One-time sponsorships are welcome at any amount.
119
+
120
+ Pricing goes up as the project grows. It is early now, so a Gold at $49/mo is near-zero investment for any company whose product touches agent web research. If your product is part of this space (proxy platforms, search infrastructure, BYO providers, anything a DonSeTch user would plug in), Gold goes one step further: fit natively inside DonSeTch and you get the placement plus an official integration shipped in the binary itself.
121
+
122
+ Email **bhandaribishesh879@gmail.com** to become a sponsor.
123
+
124
+ ## Docs
125
+
126
+ - **Full README, install matrix, configuration reference:** [github.com/dondai44423/donsetch](https://github.com/dondai44423/donsetch)
127
+ - **Every config knob with its origin:** `donsetch config show` (or `--markdown` for the whole table)
128
+ - **Changelog:** [CHANGELOG.md](https://github.com/dondai44423/donsetch/blob/master/CHANGELOG.md)
129
+ - **Issues and requests:** [github.com/dondai44423/donsetch/issues](https://github.com/dondai44423/donsetch/issues)
130
+ - **Contributing:** [CONTRIBUTING.md](https://github.com/dondai44423/donsetch/blob/master/CONTRIBUTING.md)
131
+ - **Pi agent:** `pi install npm:donsetch` registers the tools as native pi tools and self-updates.
132
+ - **DeepSeek Harness:** `dsh plugin --profile web add github:dondai44423/donsetch-dsh`
133
+
81
134
  ## License
82
135
 
83
- AGPL-3.0 : Copyright (c) 2026 Bishesh Bhandari
136
+ AGPL-3.0. Copyright (c) 2026 Bishesh Bhandari.
84
137
 
85
- Full documentation: [github.com/dondai44423/donsetch](https://github.com/dondai44423/donsetch)
138
+ If DonSeTch saves you time, [star the repo](https://github.com/dondai44423/donsetch) or [sponsor it](https://ko-fi.com/G5Y624N5RE).
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "donsetch",
3
- "version": "4.2.4",
4
- "description": "Web fetch, search and crawl for AI agents. Zero API keys. Chrome-true TLS.",
3
+ "version": "4.2.5",
4
+ "description": "MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.",
5
5
  "license": "AGPL-3.0-only",
6
6
  "author": "Bishesh Bhandari",
7
7
  "repository": {
@@ -9,22 +9,38 @@
9
9
  "url": "https://github.com/dondai44423/donsetch"
10
10
  },
11
11
  "homepage": "https://github.com/dondai44423/donsetch#readme",
12
+ "funding": {
13
+ "type": "individual",
14
+ "url": "https://ko-fi.com/G5Y624N5RE"
15
+ },
12
16
  "bugs": {
13
17
  "url": "https://github.com/dondai44423/donsetch/issues"
14
18
  },
15
19
  "keywords": [
16
20
  "pi-package",
17
21
  "mcp",
18
- "web-fetch",
19
- "search",
20
- "crawl",
22
+ "mcp-server",
23
+ "model-context-protocol",
21
24
  "ai-agent",
22
- "scraper",
25
+ "agent-tools",
26
+ "web-fetch",
27
+ "web-search",
28
+ "web-scraping",
29
+ "web-crawler",
30
+ "web-research",
31
+ "screenshot",
32
+ "markdown",
33
+ "pdf",
34
+ "ocr",
35
+ "extraction",
36
+ "anti-bot",
23
37
  "tls",
24
38
  "chrome",
25
- "anti-bot",
26
- "pdf",
27
- "extraction"
39
+ "stealth",
40
+ "no-api-key",
41
+ "claude",
42
+ "cursor",
43
+ "rust"
28
44
  ],
29
45
  "pi": {
30
46
  "extensions": ["./pi-extension.ts"]