openclaw-smart-fetch 0.2.30 → 0.2.32

package/README.md

@@ -1,6 +1,6 @@
 # openclaw-smart-fetch
 
-`openclaw-smart-fetch` adds smarter web fetching tools to OpenClaw.
+`openclaw-smart-fetch` adds smarter web fetching tools to [OpenClaw](https://github.com/nicepkg/openclaw).
 
 ## Features
 
@@ -10,18 +10,26 @@
 - 📦 **Downloads + large file support** — stream attachments and binaries to temp files
 - ⚡ **Batch fetch** — fetch many URLs with bounded concurrency
 - 📝 **Multiple output formats** — `markdown`, `html`, `text`, `json`
+- 🔄 **Built-in `web_fetch` fallback** — automatically improves the core web_fetch tool
+- 📖 **Bundled skill** — agents get usage guidance injected into their system prompt
 
 ## Site optimisations
 
-This package works on general web pages, but some site types benefit especially from Defuddle's extractors and cleanup:
+This package works on general web pages, but some site types benefit especially
+from Defuddle's extractors and cleanup:
 
-- YouTube pages and transcripts
-- Reddit posts and comment threads
-- X / Twitter posts
-- GitHub pages, issues, PRs, and discussions
-- Hacker News threads
-- Substack posts
-- Pages with code blocks, footnotes, math, and callouts
+| Site / page type | What's improved |
+|---|---|
+| **X / Twitter posts** | oEmbed-based tweet extraction; deleted/protected tweet detection |
+| **Reddit posts & threads** | Comment thread extraction with `includeReplies` |
+| **YouTube** | Page metadata and transcript extraction |
+| **GitHub** | READMEs, issues, PRs, discussions — strips chrome, keeps code blocks |
+| **Hacker News** | Thread extraction with comment cleanup |
+| **Substack / Medium** | Article content with author, publish date, paywall bypass on open pages |
+| **Stack Overflow** | Q&A extraction with code blocks and accepted answers |
+| **Wikipedia** | Article content with infobox cleanup |
+| **Documentation sites** | Keeps code blocks, callouts, footnotes, math (MathML/KaTeX/MathJax) |
+| **Blog posts & articles** | Schema.org metadata, clean main-content extraction |
 
 Notes:
 - Defuddle is the cleanup layer: it strips common page chrome like nav, sidebars, related links, share widgets, and footers
@@ -44,8 +52,9 @@ openclaw plugins install -l /absolute/path/to/agent-smart-fetch/packages/opencla
 ## OpenClaw tools
 
 Registers:
-- `smart_fetch`
-- `batch_smart_fetch`
+
+- `smart_fetch` — single URL fetch with TLS fingerprinting and Defuddle extraction
+- `batch_smart_fetch` — multiple URLs with bounded concurrency and per-item results
 
 Synopsis:
 
@@ -56,6 +65,69 @@ batch_smart_fetch(requests)
 
 For `batch_smart_fetch`, each item in `requests` accepts the same parameters as `smart_fetch`.
 
+## Built-in `web_fetch` fallback provider
+
+When this plugin is installed and enabled, it **automatically registers as a
+WebFetch provider** for OpenClaw's built-in `web_fetch` tool. No extra
+configuration needed.
+
+### How it works
+
+When `web_fetch`'s built-in HTTP + Readability extraction fails (e.g. the page
+blocks plain HTTP clients or Readability can't find content), OpenClaw calls
+the smart_fetch provider as a fallback. The provider runs the full
+TLS-fingerprinted + Defuddle pipeline and returns clean content.
+
+This means you get smart_fetch's better extraction on bot-protected sites
+_without replacing `web_fetch` or changing any agent prompts_.
+
+### Provider priority
+
+| Provider        | `autoDetectOrder`   | Credential required |
+|-----------------|:-------------------:|:-------------------|
+| **smart-fetch** | **10** (highest)    | No                 |
+| firecrawl       | 50                  | Yes (API key)      |
+
+Because `smart-fetch` has the highest priority and requires no credentials, it
+is selected first during auto-detection. If the smart_fetch provider itself
+fails (e.g. the page needs full browser automation), OpenClaw falls through to
+the next configured provider.
+
+### Explicit provider selection
+
+You can force the built-in `web_fetch` to use smart_fetch when it needs a fallback:
+
+```json5
+{
+  "tools": {
+    "web": {
+      "fetch": {
+        "provider": "smart-fetch"
+      }
+    }
+  }
+}
+```
+
+Note: setting `provider` only affects which provider is selected as the fallback —
+the built-in HTTP fetch still runs first. The provider is only called when
+Readability extraction fails (or when `readability: false` is set for HTML
+responses). The provider then re-fetches the URL with its own TLS-fingerprinted
+client, so there is a double-fetch cost when the fallback kicks in.
+
+## Bundled skill
+
+The plugin ships a skill (`smart-fetch`) that OpenClaw injects into agent
+system prompts when the plugin is enabled. The skill documents:
+
+- When to prefer `smart_fetch` over `web_fetch` or the browser tool
+- Parameter reference for both tools
+- Workflow escalation pattern (smart_fetch → batch → web_fetch → browser)
+- The automatic fallback behavior
+
+Skills are declared in the manifest (`openclaw.plugin.json`) under `"skills":
+["./skills"]` and loaded from `skills/smart-fetch/SKILL.md`.
+
 ## Output formats
 
 | Format | What you get |
@@ -65,20 +137,29 @@ For `batch_smart_fetch`, each item in `requests` accepts the same parameters as
 | `text` | Plain text with markdown stripped |
 | `json` | Structured JSON for metadata-heavy workflows |
 
-## Plugin defaults
+## Plugin config
 
-See `openclaw.plugin.json` for the schema. The effective defaults are:
+See `openclaw.plugin.json` for the full schema. Configure under
+`plugins.entries.smart-fetch.config`:
 
-```json
+```json5
 {
-  "maxChars": 50000,
-  "timeoutMs": 15000,
-  "browser": "chrome_145",
-  "os": "windows",
-  "removeImages": false,
-  "includeReplies": "extractors",
-  "batchConcurrency": 8,
-  "tempDir": "/tmp/openclaw-smart-fetch"
+  "plugins": {
+    "entries": {
+      "smart-fetch": {
+        "enabled": true,
+        "config": {
+          "maxChars": 50000,
+          "timeoutMs": 15000,
+          "browser": "chrome_145",
+          "os": "windows",
+          "removeImages": false,
+          "includeReplies": "extractors",
+          "batchConcurrency": 8
+        }
+      }
+    }
+  }
 }
 ```
 
@@ -95,4 +176,6 @@ See `openclaw.plugin.json` for the schema. The effective defaults are:
 
 ## Dev and publishing note
 
-This repo uses Bun for local development, tests, and workspace scripts. Package publishing still goes through `npm publish` in CI so npm Trusted Publishing can be used.
+This repo uses Bun for local development, tests, and workspace scripts. Package
+publishing still goes through `npm publish` in CI so npm Trusted Publishing can
+be used.

package/dist/index.d.ts

@@ -21,6 +21,35 @@ interface FetchToolDefaults {
 }
 
 type PluginConfig = FetchToolConfig;
+/**
+ * WebFetch provider plugin shape — subset of OpenClaw's WebFetchProviderPlugin
+ * that we need for registration. Defined locally to avoid importing from the
+ * openclaw plugin SDK (which may not be installed in all environments).
+ */
+interface WebFetchProvider {
+    id: string;
+    label: string;
+    hint: string;
+    requiresCredential?: boolean;
+    envVars: string[];
+    placeholder: string;
+    signupUrl: string;
+    docsUrl?: string;
+    autoDetectOrder?: number;
+    credentialPath: string;
+    getCredentialValue: (fetchConfig?: Record<string, unknown>) => unknown;
+    setCredentialValue: (fetchConfigTarget: Record<string, unknown>, value: unknown) => void;
+    getConfiguredCredentialValue?: (config?: Record<string, unknown>) => unknown;
+    setConfiguredCredentialValue?: (configTarget: Record<string, unknown>, value: unknown) => void;
+    applySelectionConfig?: (config: Record<string, unknown>) => Record<string, unknown>;
+    createTool: (ctx: {
+        config?: Record<string, unknown>;
+    }) => {
+        description: string;
+        parameters: Record<string, unknown>;
+        execute: (args: Record<string, unknown>) => Promise<Record<string, unknown>>;
+    } | null;
+}
 interface ToolRegistrationApi {
     pluginConfig?: PluginConfig;
     registerTool(definition: {
@@ -35,12 +64,15 @@ interface ToolRegistrationApi {
             isError?: boolean;
         }>;
     }): void;
+    /** Register a WebFetch provider for the built-in web_fetch fallback pipeline. */
+    registerWebFetchProvider?: (provider: WebFetchProvider) => void;
     logger: {
         info(message: string): void;
     };
 }
 
 declare const resolvePluginDefaults: (pluginConfig?: PluginConfig) => FetchToolDefaults;
+
 declare const plugin: {
     id: string;
     name: string;

package/dist/index.js

@@ -1,10 +1,10 @@
-import { tmpdir } from 'os';
-import { join, parse } from 'path';
 import { Type } from '@sinclair/typebox';
 import { randomUUID } from 'crypto';
 import { once } from 'events';
 import { createWriteStream } from 'fs';
 import { mkdir, chmod, writeFile, unlink } from 'fs/promises';
+import { tmpdir } from 'os';
+import { join, parse } from 'path';
 import { pipeline } from 'stream/promises';
 import { Defuddle } from 'defuddle/node';
 import { getProfiles, fetch } from 'wreq-js';
@@ -10203,25 +10203,10 @@ function createDefuddleFetch(dependencies = runtimeDependencies) {
       });
       const fallbackDocument = parseLinkedomHTML(rawBody, finalUrl);
       const extractionDocument = parseLinkedomHTML(rawBody, finalUrl);
-      if (isTwitterJsDisabledPage(fallbackDocument, opts.url)) {
-        return {
-          error: `Server returned HTTP 404 Not Found for ${opts.url}.`,
-          code: "http_error",
-          phase: "loading",
-          retryable: false,
-          timeoutMs,
-          url: opts.url,
-          finalUrl,
-          statusCode: 404,
-          statusText: "Not Found",
-          mimeType: normalizeContentType(contentType) || void 0,
-          contentLength: errorContext.contentLength
-        };
-      }
       let extracted;
+      const suppressedErrors = [];
       try {
         const origConsoleError = console.error;
-        const suppressedErrors = [];
         console.error = (...args) => {
           suppressedErrors.push(args);
         };
@@ -10244,6 +10229,35 @@ function createDefuddleFetch(dependencies = runtimeDependencies) {
           wordCount: 0
         };
       }
+      const isXUrl = /^https?:\/\/(www\.)?(x\.com|twitter\.com)\//i.test(
+        opts.url
+      );
+      if (isXUrl) {
+        const hasOembed404 = suppressedErrors.some(
+          (args) => args.some(
+            (arg) => typeof arg === "string" && arg.includes("oEmbed request failed: 404")
+          )
+        );
+        const hasJsDisabledShell = isTwitterJsDisabledPage(
+          fallbackDocument,
+          opts.url
+        );
+        if (hasOembed404 || hasJsDisabledShell) {
+          return {
+            error: `Server returned HTTP 404 Not Found for ${opts.url}.`,
+            code: "http_error",
+            phase: "loading",
+            retryable: false,
+            timeoutMs,
+            url: opts.url,
+            finalUrl,
+            statusCode: 404,
+            statusText: "Not Found",
+            mimeType: normalizeContentType(contentType) || void 0,
+            contentLength: errorContext.contentLength
+          };
+        }
+      }
       let extractedContent = extracted.content;
       let wordCount = extracted.wordCount;
       if (!extractedContent || wordCount === 0) {
@@ -10549,12 +10563,80 @@ async function executeBatchFetchToolCall(params, defaults, options = {}) {
     batchConcurrency
   };
 }
-
-// src/index.ts
 var resolvePluginDefaults = (pluginConfig = {}) => resolveFetchToolDefaults({
   tempDir: join(tmpdir(), "smart-fetch-openclaw"),
   ...pluginConfig
 });
+function createSmartFetchWebFetchProvider() {
+  return {
+    id: "smart-fetch",
+    label: "Smart Fetch",
+    hint: "TLS-fingerprinted fetch with Defuddle extraction. Fallback for bot-protected sites.",
+    requiresCredential: false,
+    envVars: [],
+    placeholder: "No API key required",
+    signupUrl: "https://github.com/Thinkscape/agent-smart-fetch",
+    docsUrl: "https://github.com/Thinkscape/agent-smart-fetch#readme",
+    autoDetectOrder: 10,
+    // Lower = higher priority than firecrawl (50)
+    credentialPath: "",
+    getCredentialValue: () => void 0,
+    setCredentialValue: () => {
+    },
+    getConfiguredCredentialValue: () => void 0,
+    setConfiguredCredentialValue: () => {
+    },
+    applySelectionConfig: (config) => config,
+    createTool: (ctx) => ({
+      description: "Fetch a URL using Smart Fetch (TLS fingerprinting + Defuddle extraction).",
+      parameters: {},
+      execute: async (args) => {
+        const url = typeof args.url === "string" ? args.url : "";
+        const extractMode = args.extractMode === "text" ? "text" : "markdown";
+        const maxChars = typeof args.maxChars === "number" && Number.isFinite(args.maxChars) ? Math.floor(args.maxChars) : 5e4;
+        if (!url) {
+          return { text: "" };
+        }
+        const pluginConfig = extractPluginConfig(ctx.config);
+        const defaults = resolvePluginDefaults(pluginConfig);
+        try {
+          const result = await executeFetchToolCall(
+            {
+              url,
+              extractMode,
+              maxChars
+            },
+            defaults
+          );
+          if (isError(result)) {
+            return { text: "" };
+          }
+          return {
+            text: buildFetchResponseText(result, { verbose: true }),
+            title: result.title || void 0,
+            finalUrl: result.finalUrl || url,
+            extractor: "smart-fetch"
+          };
+        } catch {
+          return { text: "" };
+        }
+      }
+    })
+  };
+}
+function extractPluginConfig(config) {
+  if (!config) return void 0;
+  const plugins = config.plugins;
+  if (!plugins || typeof plugins !== "object") return void 0;
+  const entries = plugins.entries;
+  if (!entries || typeof entries !== "object") return void 0;
+  const smartFetch = entries["smart-fetch"];
+  if (!smartFetch || typeof smartFetch !== "object") return void 0;
+  const pluginConfig = smartFetch.config;
+  return pluginConfig;
+}
+
+// src/index.ts
 function renderToolResponse(result) {
   return {
     content: [
@@ -10570,6 +10652,9 @@ var plugin = {
   name: "Smart Fetch",
   description: "Clean web content extraction with TLS fingerprinting. Uses wreq-js (Rust native bindings) for browser-grade TLS and Defuddle for extraction.",
   register(api) {
+    if (api.registerWebFetchProvider) {
+      api.registerWebFetchProvider(createSmartFetchWebFetchProvider());
+    }
     const defaults = resolvePluginDefaults(api.pluginConfig);
     api.registerTool({
       name: "smart_fetch",