llama-index-tools-clawfetch 0.1.0__tar.gz

llama_index_tools_clawfetch-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.venv/

llama_index_tools_clawfetch-0.1.0/CHANGELOG.md

@@ -0,0 +1,6 @@
+# Changelog
+
+## [0.1.0] - 2026-10-18
+
+### Added
+- Initial release: `ClawFetchToolSpec` with `fetch`, `render`, `extract`, `research`, `parse`, `domains_check` and `domains_suggest`.

llama_index_tools_clawfetch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ClawFetch
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llama_index_tools_clawfetch-0.1.0/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.5
+Name: llama-index-tools-clawfetch
+Version: 0.1.0
+Summary: llama-index tools clawfetch integration — pay-per-call web intelligence for AI agents (x402, USDC on Base)
+Project-URL: Homepage, https://api.clawfetch.ai
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai-agents,clawfetch,llama-index,tools,web-scraping,x402
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Requires-Dist: clawfetch<0.3,>=0.2.2
+Requires-Dist: llama-index-core<0.15,>=0.12
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# llama-index-tools-clawfetch
+
+LlamaIndex tool spec for [ClawFetch](https://api.clawfetch.ai), a pay-per-call web intelligence API for AI agents. Each call is paid with an x402 micropayment in USDC on Base.
+
+## Install
+
+```bash
+pip install llama-index-tools-clawfetch
+```
+
+## Wallet
+
+You need a Base wallet private key that holds USDC. There is no API key and no signup. Payments are gasless EIP-3009 transfers signed locally by the [`clawfetch`](https://pypi.org/project/clawfetch/) SDK, so the wallet does not need ETH. Use a dedicated wallet with a small balance.
+
+```bash
+export CLAWFETCH_PRIVATE_KEY=0x...
+```
+
+## Quickstart
+
+```python
+from llama_index.tools.clawfetch import ClawFetchToolSpec
+
+spec = ClawFetchToolSpec()  # reads CLAWFETCH_PRIVATE_KEY
+# or: ClawFetchToolSpec(private_key="0x...", max_chars=8000)
+
+print(spec.fetch("https://example.com"))
+
+tools = spec.to_tool_list()
+```
+
+To use the tools with an agent:
+
+```python
+from llama_index.core.agent.workflow import FunctionAgent
+
+agent = FunctionAgent(tools=tools, llm=llm)  # llm: any LlamaIndex LLM with tool calling
+response = await agent.run("What's new on news.ycombinator.com?")
+```
+
+## Tools
+
+| Function | Price per call | What it does |
+|---|---|---|
+| `fetch(url)` | $0.001 | URL to readable text (no JavaScript) |
+| `render(url, wait_for=None)` | $0.005 | JavaScript-rendered page to text |
+| `extract(url, type=None)` | $0.008 | Typed JSON from 17 supported sites |
+| `research(query, max_results=None)` | $0.02 | Web search plus source texts for citations |
+| `parse(url, format=None)` | $0.005 | Office document or PDF at a URL to Markdown (no OCR) |
+| `domains_check(domains)` | $0.003 | Domain availability |
+| `domains_suggest(keywords, tlds=None, max_check=None)` | $0.003 | Generate and check domain names from keywords |
+
+Supported `extract` sites: twitter, coingecko, github, news, npm, pypi, crates, hackernews, youtube, wikipedia, producthunt, imdb, amazon, crunchbase, sec-edgar, zillow, redfin.
+
+## Output and errors
+
+Functions return a string. Fetch and render return the page title, URL and text. Parse returns Markdown. The other functions return the API response as JSON. Output is cut at `max_chars` (default 10,000).
+
+SDK errors (`PaymentError`, `RateLimitError`, `ApiError`, `NetworkError`, all subclasses of `clawfetch.ClawFetchError`) are raised unchanged.
+
+## License
+
+MIT

llama_index_tools_clawfetch-0.1.0/README.md

@@ -0,0 +1,63 @@
+# llama-index-tools-clawfetch
+
+LlamaIndex tool spec for [ClawFetch](https://api.clawfetch.ai), a pay-per-call web intelligence API for AI agents. Each call is paid with an x402 micropayment in USDC on Base.
+
+## Install
+
+```bash
+pip install llama-index-tools-clawfetch
+```
+
+## Wallet
+
+You need a Base wallet private key that holds USDC. There is no API key and no signup. Payments are gasless EIP-3009 transfers signed locally by the [`clawfetch`](https://pypi.org/project/clawfetch/) SDK, so the wallet does not need ETH. Use a dedicated wallet with a small balance.
+
+```bash
+export CLAWFETCH_PRIVATE_KEY=0x...
+```
+
+## Quickstart
+
+```python
+from llama_index.tools.clawfetch import ClawFetchToolSpec
+
+spec = ClawFetchToolSpec()  # reads CLAWFETCH_PRIVATE_KEY
+# or: ClawFetchToolSpec(private_key="0x...", max_chars=8000)
+
+print(spec.fetch("https://example.com"))
+
+tools = spec.to_tool_list()
+```
+
+To use the tools with an agent:
+
+```python
+from llama_index.core.agent.workflow import FunctionAgent
+
+agent = FunctionAgent(tools=tools, llm=llm)  # llm: any LlamaIndex LLM with tool calling
+response = await agent.run("What's new on news.ycombinator.com?")
+```
+
+## Tools
+
+| Function | Price per call | What it does |
+|---|---|---|
+| `fetch(url)` | $0.001 | URL to readable text (no JavaScript) |
+| `render(url, wait_for=None)` | $0.005 | JavaScript-rendered page to text |
+| `extract(url, type=None)` | $0.008 | Typed JSON from 17 supported sites |
+| `research(query, max_results=None)` | $0.02 | Web search plus source texts for citations |
+| `parse(url, format=None)` | $0.005 | Office document or PDF at a URL to Markdown (no OCR) |
+| `domains_check(domains)` | $0.003 | Domain availability |
+| `domains_suggest(keywords, tlds=None, max_check=None)` | $0.003 | Generate and check domain names from keywords |
+
+Supported `extract` sites: twitter, coingecko, github, news, npm, pypi, crates, hackernews, youtube, wikipedia, producthunt, imdb, amazon, crunchbase, sec-edgar, zillow, redfin.
+
+## Output and errors
+
+Functions return a string. Fetch and render return the page title, URL and text. Parse returns Markdown. The other functions return the API response as JSON. Output is cut at `max_chars` (default 10,000).
+
+SDK errors (`PaymentError`, `RateLimitError`, `ApiError`, `NetworkError`, all subclasses of `clawfetch.ClawFetchError`) are raised unchanged.
+
+## License
+
+MIT

llama_index_tools_clawfetch-0.1.0/llama_index/tools/clawfetch/__init__.py

@@ -0,0 +1,3 @@
+from llama_index.tools.clawfetch.base import ClawFetchToolSpec
+
+__all__ = ["ClawFetchToolSpec"]

llama_index_tools_clawfetch-0.1.0/llama_index/tools/clawfetch/base.py

@@ -0,0 +1,128 @@
+"""ClawFetch tool spec — pay-per-call web intelligence for AI agents (x402, USDC on Base)."""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any, List, Optional
+
+from clawfetch import ClawFetch
+from llama_index.core.tools.tool_spec.base import BaseToolSpec
+
+DEFAULT_MAX_CHARS = 10_000
+
+
+def _truncate(text: str, max_chars: int) -> str:
+    if len(text) <= max_chars:
+        return text
+    return text[:max_chars] + f"\n...[truncated {len(text) - max_chars} chars]"
+
+
+def _page(result: dict) -> str:
+    return f"# {result.get('title') or ''}\nURL: {result.get('url') or ''}\n\n{result.get('text') or ''}"
+
+
+class ClawFetchToolSpec(BaseToolSpec):
+    """ClawFetch web tools. Each call is paid in USDC on Base via x402.
+
+    Args:
+        private_key: Base wallet private key holding USDC. Defaults to the
+            ``CLAWFETCH_PRIVATE_KEY`` environment variable.
+        client: An existing ``clawfetch.ClawFetch`` client (overrides private_key).
+        max_chars: Maximum characters returned by each tool call.
+    """
+
+    spec_functions = [
+        "fetch",
+        "render",
+        "extract",
+        "research",
+        "parse",
+        "domains_check",
+        "domains_suggest",
+    ]
+
+    def __init__(
+        self,
+        private_key: Optional[str] = None,
+        client: Optional[ClawFetch] = None,
+        max_chars: int = DEFAULT_MAX_CHARS,
+    ) -> None:
+        if client is None:
+            key = private_key or os.environ.get("CLAWFETCH_PRIVATE_KEY")
+            if not key:
+                raise ValueError("Pass private_key=..., set CLAWFETCH_PRIVATE_KEY, or pass client=ClawFetch(...).")
+            client = ClawFetch(private_key=key)
+        self.client = client
+        self.max_chars = max_chars
+
+    def _json(self, result: Any) -> str:
+        return _truncate(json.dumps(result, ensure_ascii=False), self.max_chars)
+
+    def fetch(self, url: str) -> str:
+        """Fetch a web page and return its title, URL and readable text. No JavaScript execution; use render for JS-heavy pages. Costs $0.001 USDC per call.
+
+        Args:
+            url: Absolute http(s) URL of the page to fetch.
+        """
+        return _truncate(_page(self.client.fetch(url, max_chars=self.max_chars)), self.max_chars)
+
+    def render(self, url: str, wait_for: Optional[str] = None) -> str:
+        """Render a JavaScript-heavy page in a headless browser and return its visible text. Use when fetch returns empty or incomplete content. Costs $0.005 USDC per call.
+
+        Args:
+            url: Absolute http(s) URL of the page.
+            wait_for: Optional CSS selector to wait for before reading the page.
+        """
+        result = self.client.render(url, max_chars=self.max_chars, wait_for=wait_for)
+        return _truncate(_page(result), self.max_chars)
+
+    def extract(self, url: str, type: Optional[str] = None) -> str:
+        """Extract typed structured data (JSON) from a page on a supported site: twitter, coingecko, github, news, npm, pypi, crates, hackernews, youtube, wikipedia, producthunt, imdb, amazon, crunchbase, sec-edgar, zillow, redfin. Costs $0.008 USDC per call.
+
+        Args:
+            url: URL on a supported site.
+            type: Optional extractor name from the list above; auto-detected from the URL when omitted.
+        """
+        return self._json(self.client.extract(url, type=type))
+
+    def research(self, query: str, max_results: Optional[int] = None) -> str:
+        """Search the web for a query and return the text of the top sources with their URLs (for citations). Costs $0.02 USDC per call.
+
+        Args:
+            query: Research question or topic.
+            max_results: Number of sources to read (1-10, default 5).
+        """
+        return self._json(self.client.research(query, max_results=max_results))
+
+    def parse(self, url: str, format: Optional[str] = None) -> str:
+        """Convert an office document or PDF at a URL (docx, pptx, xlsx, pdf, odt, rtf, epub, csv, ...) to Markdown. No OCR: scanned PDFs fail. Costs $0.005 USDC per call.
+
+        Args:
+            url: Absolute http(s) URL of the document.
+            format: Optional format override (doc, docx, odt, pdf, ppt, pptx, rtf, epub, xlsx, ods, odp, csv).
+        """
+        return _truncate(self.client.parse(url, format=format).get("markdown") or "", self.max_chars)
+
+    def domains_check(self, domains: List[str]) -> str:
+        """Check whether specific domain names are available to register. Returns JSON with a result per domain (available true/false, or null with an error). Costs $0.003 USDC per call.
+
+        Args:
+            domains: Fully qualified domain names, e.g. ["acme.ai", "acme.com"] (max 20).
+        """
+        return self._json(self.client.domains_check(domains))
+
+    def domains_suggest(
+        self,
+        keywords: List[str],
+        tlds: Optional[List[str]] = None,
+        max_check: Optional[int] = None,
+    ) -> str:
+        """Generate domain names from keywords and check which are available. Returns JSON with the available domains and every checked result. Costs $0.003 USDC per call.
+
+        Args:
+            keywords: Keywords to combine into domain names, e.g. ["agent", "grid"].
+            tlds: TLDs to try without dots, e.g. ["com", "ai"]. Default: com, ai, io.
+            max_check: How many generated names to check (1-50, default 30).
+        """
+        return self._json(self.client.domains_suggest(keywords, tlds=tlds, max_check=max_check))

llama_index_tools_clawfetch-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "llama-index-tools-clawfetch"
+version = "0.1.0"
+description = "llama-index tools clawfetch integration — pay-per-call web intelligence for AI agents (x402, USDC on Base)"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+keywords = ["llama-index", "clawfetch", "x402", "web-scraping", "ai-agents", "tools"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "clawfetch>=0.2.2,<0.3",
+    "llama-index-core>=0.12,<0.15",
+]
+
+[project.optional-dependencies]
+test = ["pytest>=8"]
+
+[project.urls]
+Homepage = "https://api.clawfetch.ai"
+
+[tool.hatch.build.targets.sdist]
+include = ["llama_index/", "tests/", "README.md", "CHANGELOG.md", "LICENSE"]
+
+[tool.hatch.build.targets.wheel]
+include = ["llama_index/"]
+
+[tool.llamahub]
+contains_example = false
+import_path = "llama_index.tools.clawfetch"
+
+[tool.llamahub.class_authors]
+ClawFetchToolSpec = "clawfetch"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

llama_index_tools_clawfetch-0.1.0/tests/test_clawfetch_tool_spec.py

@@ -0,0 +1,85 @@
+"""Unit tests — the ClawFetch client is mocked: no network, no payments."""
+
+import json
+from unittest.mock import MagicMock
+
+import pytest
+from clawfetch import ClawFetch
+from llama_index.core.tools import FunctionTool
+from llama_index.core.tools.tool_spec.base import BaseToolSpec
+
+from llama_index.tools.clawfetch import ClawFetchToolSpec
+
+DUMMY_KEY = "0x0000000000000000000000000000000000000000000000000000000000000001"
+PAGE = {"url": "https://a.b", "title": "T", "text": "hello"}
+
+# (spec function, call kwargs, sdk method, expected args, expected kwargs, sdk result, expected output)
+CASES = [
+    ("fetch", {"url": "https://a.b"}, "fetch", ("https://a.b",), {"max_chars": 10_000},
+     PAGE, "# T\nURL: https://a.b\n\nhello"),
+    ("render", {"url": "https://a.b", "wait_for": "#app"}, "render", ("https://a.b",),
+     {"max_chars": 10_000, "wait_for": "#app"}, PAGE, "# T\nURL: https://a.b\n\nhello"),
+    ("extract", {"url": "https://github.com/a/b", "type": "github"}, "extract", ("https://github.com/a/b",),
+     {"type": "github"}, {"type": "github", "stars": 1}, json.dumps({"type": "github", "stars": 1})),
+    ("research", {"query": "x402"}, "research", ("x402",), {"max_results": None},
+     {"query": "x402", "sources": 0, "results": []}, json.dumps({"query": "x402", "sources": 0, "results": []})),
+    ("parse", {"url": "https://a.b/x.docx"}, "parse", ("https://a.b/x.docx",), {"format": None},
+     {"markdown": "# Doc", "format": "docx", "chars": 5}, "# Doc"),
+    ("domains_check", {"domains": ["a.ai"]}, "domains_check", (["a.ai"],), {},
+     {"checked": 1, "results": [{"domain": "a.ai", "available": None, "error": "timeout"}]},
+     json.dumps({"checked": 1, "results": [{"domain": "a.ai", "available": None, "error": "timeout"}]})),
+    ("domains_suggest", {"keywords": ["agent"], "tlds": ["ai"], "max_check": 5}, "domains_suggest",
+     (["agent"],), {"tlds": ["ai"], "max_check": 5},
+     {"keywords": ["agent"], "available": ["agent.ai"]}, json.dumps({"keywords": ["agent"], "available": ["agent.ai"]})),
+]
+
+
+@pytest.mark.parametrize("fn,call_kwargs,method,args,kwargs,result,expected", CASES)
+def test_spec_function_calls_sdk_and_formats(fn, call_kwargs, method, args, kwargs, result, expected):
+    client = MagicMock(spec=ClawFetch)
+    getattr(client, method).return_value = result
+    spec = ClawFetchToolSpec(client=client)
+    assert getattr(spec, fn)(**call_kwargs) == expected
+    getattr(client, method).assert_called_once_with(*args, **kwargs)
+
+
+@pytest.mark.parametrize("fn,call_kwargs,method,args,kwargs,result,expected", CASES)
+def test_function_tool_call_through_llama_index(fn, call_kwargs, method, args, kwargs, result, expected):
+    client = MagicMock(spec=ClawFetch)
+    getattr(client, method).return_value = result
+    tools = {t.metadata.name: t for t in ClawFetchToolSpec(client=client).to_tool_list()}
+    out = tools[fn].call(**call_kwargs)
+    assert out.content == expected
+    getattr(client, method).assert_called_once_with(*args, **kwargs)
+
+
+def test_tool_list_metadata():
+    tools = ClawFetchToolSpec(client=MagicMock(spec=ClawFetch)).to_tool_list()
+    assert issubclass(ClawFetchToolSpec, BaseToolSpec)
+    assert [t.metadata.name for t in tools] == [
+        "fetch", "render", "extract", "research", "parse", "domains_check", "domains_suggest",
+    ]
+    for t in tools:
+        assert isinstance(t, FunctionTool)
+        assert "$0." in t.metadata.description and "USDC" in t.metadata.description
+    schema = tools[6].metadata.get_parameters_dict()
+    assert schema["required"] == ["keywords"]
+    assert schema["properties"]["keywords"]["type"] == "array"
+
+
+def test_output_truncated_to_max_chars():
+    client = MagicMock(spec=ClawFetch)
+    client.parse.return_value = {"markdown": "x" * 50}
+    assert ClawFetchToolSpec(client=client, max_chars=10).parse("https://a.b/x.pdf") == "x" * 10 + "\n...[truncated 40 chars]"
+
+
+def test_from_private_key_and_env(monkeypatch):
+    assert isinstance(ClawFetchToolSpec(private_key=DUMMY_KEY).client, ClawFetch)
+    monkeypatch.setenv("CLAWFETCH_PRIVATE_KEY", DUMMY_KEY)
+    assert isinstance(ClawFetchToolSpec().client, ClawFetch)
+
+
+def test_missing_key_raises(monkeypatch):
+    monkeypatch.delenv("CLAWFETCH_PRIVATE_KEY", raising=False)
+    with pytest.raises(ValueError, match="CLAWFETCH_PRIVATE_KEY"):
+        ClawFetchToolSpec()