openclaw-smart-fetch 0.2.22 → 0.2.24

package/README.md

@@ -1,52 +1,31 @@
 # openclaw-smart-fetch
 
-`openclaw-smart-fetch` adds smarter fetching tools for OpenClaw.
+`openclaw-smart-fetch` adds smarter web fetching tools to OpenClaw.
 
-It registers:
-- `smart_fetch`
-- `batch_smart_fetch`
-
-It combines:
-- `@thinkscape/wreq-js` for browser-like transport fingerprints
-- `Defuddle` for readable content extraction
-
-## Why use this instead of OpenClaw's built-in `web_fetch`
-
-Use this package when the built-in `web_fetch` is not enough.
+## Features
 
-Typical advantages:
-- **better resistance to bot detection** on sites that inspect TLS/HTTP client fingerprints
-- **more browser-like transport behavior** instead of a generic server-side HTTP client
-- **cleaner extracted content** instead of raw or noisy page output
-- **better article/document readability** for downstream agent analysis
-- **useful metadata** like title, author, published date, site, and language when available
-- **batch fan-out support** when you want to fetch multiple URLs in one tool call
-- **attachment and binary download support** when a server returns `Content-Disposition: attachment` or a non-text content type
-- **temp-file output** with sanitized filenames and file metadata instead of trying to render binary bytes as page text
+- 🔐 **Browser-like TLS/SSL + HTTP fingerprints** — better success on bot-defended pages
+- 🧹 **Defuddle extraction** — clean readable content instead of noisy HTML
+- 🧠 **Useful metadata** — title, author, site, language, published date when available
+- 📦 **Downloads + large file support** — stream attachments and binaries to temp files
+- ⚡ **Batch fetch** — fetch many URLs with bounded concurrency
+- 📝 **Multiple output formats** — `markdown`, `html`, `text`, `json`
 
-A good rule of thumb:
-- use built-in `web_fetch` for simple pages
-- use `smart_fetch` when pages are blocked, noisy, or extraction quality matters
-- use `batch_smart_fetch` when you need the same smarter fetch behavior over many URLs at once
+## Site optimisations
 
-## Bot-detection focus
+This package works on general web pages, but some site types benefit especially from Defuddle's extractors and cleanup:
 
-These tools are aimed at sites that detect bots through:
-- TLS/client fingerprinting
-- transport/header inconsistencies
-- non-browser HTTP behavior
+- YouTube pages and transcripts
+- Reddit posts and comment threads
+- X / Twitter posts
+- GitHub pages, issues, PRs, and discussions
+- Hacker News threads
+- Substack posts
+- Pages with code blocks, footnotes, math, and callouts
 
-They do **not** execute JavaScript or solve interactive anti-bot flows.
-
-If a page requires JS execution, login, scrolling, or clicking, use browser automation instead.
-
-## What tools it exposes
-
-This package registers:
-- `smart_fetch`
-- `batch_smart_fetch`
-
-OpenClaw keeps separate tool names because overriding/hoisting built-in `web_fetch` is not the desired path here.
+Notes:
+- Defuddle is the cleanup layer: it strips common page chrome like nav, sidebars, related links, share widgets, and footers
+- It does **not** execute JavaScript or solve interactive anti-bot/login flows
 
 ## Install
 
@@ -62,139 +41,58 @@ From a local checkout:
 openclaw plugins install -l /absolute/path/to/agent-smart-fetch/packages/openclaw-smart-fetch
 ```
 
-## Use cases
+## OpenClaw tools
 
-Use `smart_fetch` when you want to:
-- fetch pages that reject naive HTTP clients
-- extract the readable body from articles, docs, and blog posts
-- reduce noise before passing content to an agent
-- preserve page metadata for summarization or research
-- use browser-like fetching without paying the cost of full browser automation
-
-Use `batch_smart_fetch` when you want to:
-- fetch multiple URLs in one tool call
-- preserve a clear mapping between each input URL and its result
-- keep full content for successes while retaining per-item error strings for failures
-- run bounded-concurrency fetches instead of firing everything at once
+Registers:
+- `smart_fetch`
+- `batch_smart_fetch`
 
-## Tool synopsis
+Synopsis:
 
 ```text
-smart_fetch(url, browser?, os?, headers?, maxChars?, format?, removeImages?, includeReplies?, proxy?)
+smart_fetch(url, browser?, os?, headers?, maxChars?, timeoutMs?, format?, removeImages?, includeReplies?, proxy?)
 batch_smart_fetch(requests)
 ```
 
-For `batch_smart_fetch`, `requests` is an array of objects, and **each item accepts the same parameters as `smart_fetch`**.
+For `batch_smart_fetch`, each item in `requests` accepts the same parameters as `smart_fetch`.
 
-## Example output
+## Output formats
 
-### `smart_fetch`
+| Format | What you get |
+|---|---|
+| `markdown` | Best default for readable page content |
+| `html` | Cleaned HTML output |
+| `text` | Plain text with markdown stripped |
+| `json` | Structured JSON for metadata-heavy workflows |
 
-```text
-> URL: https://example.com/blog/some-article
-> Title: Some Article
-> Author: Jane Doe
-> Published: 2026-03-12
-> Site: Example Blog
-> Language: en
-> Words: 1284
-> Browser: chrome_145/windows
-
-# Some Article
-
-This is the cleaned readable content extracted from the page.
-```
+## Plugin defaults
 
-### `smart_fetch` attachment/binary output
+See `openclaw.plugin.json` for the schema. The effective defaults are:
 
-```text
-> URL: https://example.com/download/report
-> File size: 999999
-> Mime type: application/pdf
-> File path: /absolute/path/to/temp/report.pdf
+```json
+{
+  "maxChars": 50000,
+  "timeoutMs": 15000,
+  "browser": "chrome_145",
+  "os": "windows",
+  "removeImages": false,
+  "includeReplies": "extractors",
+  "batchConcurrency": 8,
+  "tempDir": "/tmp/openclaw-smart-fetch"
+}
 ```
 
-### `batch_smart_fetch`
-
-```text
-> Requests: 2
-> Succeeded: 1
-> Failed: 1
-> Concurrency: 8
-
-## [1/2] https://example.com/blog/some-article
-> URL: https://example.com/blog/some-article
-> Title: Some Article
-> Author: Jane Doe
-> Published: 2026-03-12
-> Site: Example Blog
-> Language: en
-> Words: 1284
-> Browser: chrome_145/windows
-
-# Some Article
-
-This is the cleaned readable content extracted from the page.
-
-## [2/2] https://blocked.example/post
-> URL: https://blocked.example/post
-> Status: error
-> Error: HTTP 403 Forbidden for https://blocked.example/post
-```
-
-## Parameters
-
-### `smart_fetch`
-
-| Parameter         | Type                          | Default         | Description                                               |
-|-------------------|-------------------------------|-----------------|-----------------------------------------------------------|
-| `url`             | string                        | required        | URL to fetch                                              |
-| `browser`         | string                        | `chrome_145`    | Browser profile used for transport fingerprinting         |
-| `os`              | string                        | `windows`       | OS profile: `windows`, `macos`, `linux`, `android`, `ios` |
-| `headers`         | object                        | auto            | Extra request headers                                     |
-| `maxChars`        | number                        | `50000`         | Maximum returned characters                               |
-| `format`          | `markdown` \| `html` \| `text` \| `json` | `markdown`      | Output format                                             |
-| `removeImages`    | boolean                       | `false`         | Strip image references from output                        |
-| `includeReplies`  | boolean \| `extractors`       | `extractors`    | Include replies/comments                                  |
-| `proxy`           | string                        | none            | Proxy URL                                                 |
-
-### `batch_smart_fetch`
-
-| Parameter   | Type             | Default   | Description |
-|-------------|------------------|-----------|-------------|
-| `requests`  | array of objects | required  | Array of fetch requests. Each item accepts the same parameters as `smart_fetch` |
-
-## OpenClaw config
-
-See `openclaw.plugin.json` for plugin config defaults and schema.
-
-Configurable defaults include:
-- `maxChars`
-- `timeoutMs`
-- `browser`
-- `os`
-- `removeImages`
-- `includeReplies`
-- `batchConcurrency`
-- `tempDir`
-
-`batchConcurrency` defaults to `8` and controls how many `batch_smart_fetch` requests run concurrently.
-
-`tempDir` lets the OpenClaw consumer choose where attachment/binary downloads are written before the tool returns their absolute file paths.
-
-## When not to use it
-
-Do not use these tools when:
-- the page requires JS rendering
-- you need login/session flows
-- you need clicks, scrolling, or form submission
-- a full browser session is required
-
-In those cases, use browser automation instead.
+| Setting | Default | Description |
+|---|---:|---|
+| `maxChars` | `50000` | Default maximum returned characters |
+| `timeoutMs` | `15000` | Default request timeout in milliseconds |
+| `browser` | `chrome_145` | Default browser fingerprint profile |
+| `os` | `windows` | Default OS fingerprint profile |
+| `removeImages` | `false` | Strip image references by default |
+| `includeReplies` | `extractors` | Include replies/comments only when site extractors support them |
+| `batchConcurrency` | `8` | Default bounded concurrency for `batch_smart_fetch` |
+| `tempDir` | OS temp dir | Directory for attachment and binary downloads |
 
-## Recent feature additions reflected here
+## Dev and publishing note
 
-Recent `feat:` work added:
-- publish-ready TS/test/build packaging workflow across the monorepo
-- richer animated batch progress behavior in pi-facing consumers
-- attachment and binary download streaming with sanitized temp-file output
+This repo uses Bun for local development, tests, and workspace scripts. Package publishing still goes through `npm publish` in CI so npm Trusted Publishing can be used.

package/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "smart-fetch",
   "name": "Smart Fetch",
-  "version": "0.2.22",
+  "version": "0.2.24",
   "description": "Clean web content extraction with browser-grade TLS fingerprinting. Uses wreq-js (Rust native bindings) for anti-bot bypass and Defuddle for superior content extraction.",
   "configSchema": {
     "type": "object",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openclaw-smart-fetch",
-  "version": "0.2.22",
+  "version": "0.2.24",
   "type": "module",
   "description": "OpenClaw smart fetch plugin with browser-grade TLS fingerprinting and Defuddle extraction.",
   "license": "MIT",