hermes-clawifi-web 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 ReARC Labs Inc.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,2 @@
1
+ include LICENSE README.md
2
+ recursive-include tests *.py *.json
@@ -0,0 +1,116 @@
1
+ Metadata-Version: 2.4
2
+ Name: hermes-clawifi-web
3
+ Version: 0.1.0
4
+ Summary: Clawifi web search and page reading as a web backend for Hermes Agent.
5
+ Author-email: "ReARC Labs Inc." <support@clawifi.dev>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://clawifi.dev/?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin
8
+ Project-URL: Documentation, https://clawifi.dev/docs
9
+ Project-URL: Repository, https://github.com/ReARCLabs/hermes-clawifi-web
10
+ Keywords: hermes-agent,hermes,plugin,web-search,web-extract,clawifi,ai-agents,search-api
11
+ Classifier: Development Status :: 4 - Beta
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Programming Language :: Python :: 3.14
18
+ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
19
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
20
+ Requires-Python: >=3.11
21
+ Description-Content-Type: text/markdown
22
+ License-File: LICENSE
23
+ Requires-Dist: httpx>=0.27
24
+ Provides-Extra: test
25
+ Requires-Dist: pytest>=8; extra == "test"
26
+ Requires-Dist: ruamel.yaml>=0.18; extra == "test"
27
+ Dynamic: license-file
28
+
29
+ # hermes-clawifi-web
30
+
31
+ [Clawifi](https://clawifi.dev/?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin) as a web backend for [Hermes Agent](https://hermes-agent.nousresearch.com). It serves both Hermes web tools:
32
+
33
+ | Hermes tool | Clawifi call | Cost |
34
+ |---|---|---|
35
+ | `web_search` | `POST /search/free`: ranked results (title, URL, snippet) | 1 credit; 0 when nothing is found or the answer is cached |
36
+ | `web_extract` | `POST /fetch` per URL: the page or PDF as Markdown | 1 credit per web page; 2 credits per started 50 pages of a PDF; 0 when the page cannot be read |
37
+
38
+ A free Clawifi account has 1,000 credits a month and needs no card. Without a key, the plugin uses Clawifi's keyless trial: 12 calls a minute and 50 a day per address, 3 results per search.
39
+
40
+ ## Install
41
+
42
+ 1. Install the plugin with Hermes. This works on every Hermes install:
43
+
44
+ ```bash
45
+ hermes plugins install ReARCLabs/hermes-clawifi-web/src/hermes_clawifi_web
46
+ ```
47
+
48
+ If you manage the Python environment that runs Hermes yourself, you can install the package from PyPI instead. Use the `pip` of that environment, not your system `pip`:
49
+
50
+ ```bash
51
+ ~/.hermes/hermes-agent/venv/bin/pip install hermes-clawifi-web
52
+ ```
53
+
54
+ 2. Enable the plugin. Hermes does not load third-party plugins until you do:
55
+
56
+ ```bash
57
+ hermes plugins enable web-clawifi
58
+ ```
59
+
60
+ 3. Set your API key (optional, see the keyless trial above). Create a free account at [clawifi.dev/signup](https://clawifi.dev/signup?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin), then add the key to `~/.hermes/.env`:
61
+
62
+ ```bash
63
+ CLAWIFI_API_KEY=your-key
64
+ ```
65
+
66
+ 4. Select the backend:
67
+
68
+ ```bash
69
+ hermes config set web.backend clawifi
70
+ ```
71
+
72
+ You can also pick **Clawifi** under **Web Search & Extract** in `hermes tools`.
73
+
74
+ 5. Restart Hermes.
75
+
76
+ ## Use Clawifi for one tool only
77
+
78
+ `web.backend` sets both tools. To mix backends, set them separately in `~/.hermes/config.yaml`:
79
+
80
+ ```yaml
81
+ web:
82
+ search_backend: "clawifi"
83
+ extract_backend: "firecrawl"
84
+ ```
85
+
86
+ ## Settings
87
+
88
+ | Variable | Required | Meaning |
89
+ |---|---|---|
90
+ | `CLAWIFI_API_KEY` | no | Your Clawifi API key. Without it the keyless trial is used. |
91
+ | `CLAWIFI_API_BASE` | no | Base URL of a self-hosted Clawifi. The default is `https://api.clawifi.dev`. |
92
+
93
+ ## Behavior to know
94
+
95
+ - **The keyless trial is used only when you select Clawifi and set no key.** Hermes never picks it on its own. Trial calls go to `https://api.clawifi.dev/mcp`, and each search result carries a `note` that says so. When the trial limit is reached, the error says how to get a key.
96
+ - **Hermes may answer from another service when Clawifi fails.** By default, Hermes retries a failed web call once through its own keyless free tier and adds a `backend_error` note with the Clawifi error.
97
+ - **`web.keyless_fallback: false` turns both off.** Clawifi then needs a key, and a failed call shows the Clawifi error.
98
+ - **Search returns at most 20 results**, the Hermes limit for every backend. Queries longer than 500 characters are cut to 500.
99
+ - **Extract returns Markdown.** Up to two pages are read at once (the limit of the Free plan), and results keep the order of the URLs you asked for. A page that fails has an `error` field; the other pages still return.
100
+ - **Pages are read without a browser.** A page that needs JavaScript, sits behind a sign-in or CAPTCHA, or reports "not found" returns an `error` and costs 0 credits.
101
+ - **Privacy.** Requests carry your API key and the User-Agent `hermes-clawifi-web/<version>`. They carry no other identifier.
102
+
103
+ ## Development
104
+
105
+ ```bash
106
+ pip install -e ".[test]"
107
+ pytest # unit tests, HTTP is mocked
108
+ ruff check .
109
+ CLAWIFI_LIVE=1 CLAWIFI_API_KEY=your-key pytest tests/test_live.py # real API, about 2 credits
110
+ ```
111
+
112
+ The tests in `tests/test_hermes_integration.py` run only when Hermes Agent is installed in the same environment. They use a temporary `HERMES_HOME`, so they never touch `~/.hermes`.
113
+
114
+ ## License
115
+
116
+ MIT. Copyright (c) 2026 ReARC Labs Inc.
@@ -0,0 +1,88 @@
1
+ # hermes-clawifi-web
2
+
3
+ [Clawifi](https://clawifi.dev/?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin) as a web backend for [Hermes Agent](https://hermes-agent.nousresearch.com). It serves both Hermes web tools:
4
+
5
+ | Hermes tool | Clawifi call | Cost |
6
+ |---|---|---|
7
+ | `web_search` | `POST /search/free`: ranked results (title, URL, snippet) | 1 credit; 0 when nothing is found or the answer is cached |
8
+ | `web_extract` | `POST /fetch` per URL: the page or PDF as Markdown | 1 credit per web page; 2 credits per started 50 pages of a PDF; 0 when the page cannot be read |
9
+
10
+ A free Clawifi account has 1,000 credits a month and needs no card. Without a key, the plugin uses Clawifi's keyless trial: 12 calls a minute and 50 a day per address, 3 results per search.
11
+
12
+ ## Install
13
+
14
+ 1. Install the plugin with Hermes. This works on every Hermes install:
15
+
16
+ ```bash
17
+ hermes plugins install ReARCLabs/hermes-clawifi-web/src/hermes_clawifi_web
18
+ ```
19
+
20
+ If you manage the Python environment that runs Hermes yourself, you can install the package from PyPI instead. Use the `pip` of that environment, not your system `pip`:
21
+
22
+ ```bash
23
+ ~/.hermes/hermes-agent/venv/bin/pip install hermes-clawifi-web
24
+ ```
25
+
26
+ 2. Enable the plugin. Hermes does not load third-party plugins until you do:
27
+
28
+ ```bash
29
+ hermes plugins enable web-clawifi
30
+ ```
31
+
32
+ 3. Set your API key (optional, see the keyless trial above). Create a free account at [clawifi.dev/signup](https://clawifi.dev/signup?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin), then add the key to `~/.hermes/.env`:
33
+
34
+ ```bash
35
+ CLAWIFI_API_KEY=your-key
36
+ ```
37
+
38
+ 4. Select the backend:
39
+
40
+ ```bash
41
+ hermes config set web.backend clawifi
42
+ ```
43
+
44
+ You can also pick **Clawifi** under **Web Search & Extract** in `hermes tools`.
45
+
46
+ 5. Restart Hermes.
47
+
48
+ ## Use Clawifi for one tool only
49
+
50
+ `web.backend` sets both tools. To mix backends, set them separately in `~/.hermes/config.yaml`:
51
+
52
+ ```yaml
53
+ web:
54
+ search_backend: "clawifi"
55
+ extract_backend: "firecrawl"
56
+ ```
57
+
58
+ ## Settings
59
+
60
+ | Variable | Required | Meaning |
61
+ |---|---|---|
62
+ | `CLAWIFI_API_KEY` | no | Your Clawifi API key. Without it the keyless trial is used. |
63
+ | `CLAWIFI_API_BASE` | no | Base URL of a self-hosted Clawifi. The default is `https://api.clawifi.dev`. |
64
+
65
+ ## Behavior to know
66
+
67
+ - **The keyless trial is used only when you select Clawifi and set no key.** Hermes never picks it on its own. Trial calls go to `https://api.clawifi.dev/mcp`, and each search result carries a `note` that says so. When the trial limit is reached, the error says how to get a key.
68
+ - **Hermes may answer from another service when Clawifi fails.** By default, Hermes retries a failed web call once through its own keyless free tier and adds a `backend_error` note with the Clawifi error.
69
+ - **`web.keyless_fallback: false` turns both off.** Clawifi then needs a key, and a failed call shows the Clawifi error.
70
+ - **Search returns at most 20 results**, the Hermes limit for every backend. Queries longer than 500 characters are cut to 500.
71
+ - **Extract returns Markdown.** Up to two pages are read at once (the limit of the Free plan), and results keep the order of the URLs you asked for. A page that fails has an `error` field; the other pages still return.
72
+ - **Pages are read without a browser.** A page that needs JavaScript, sits behind a sign-in or CAPTCHA, or reports "not found" returns an `error` and costs 0 credits.
73
+ - **Privacy.** Requests carry your API key and the User-Agent `hermes-clawifi-web/<version>`. They carry no other identifier.
74
+
75
+ ## Development
76
+
77
+ ```bash
78
+ pip install -e ".[test]"
79
+ pytest # unit tests, HTTP is mocked
80
+ ruff check .
81
+ CLAWIFI_LIVE=1 CLAWIFI_API_KEY=your-key pytest tests/test_live.py # real API, about 2 credits
82
+ ```
83
+
84
+ The tests in `tests/test_hermes_integration.py` run only when Hermes Agent is installed in the same environment. They use a temporary `HERMES_HOME`, so they never touch `~/.hermes`.
85
+
86
+ ## License
87
+
88
+ MIT. Copyright (c) 2026 ReARC Labs Inc.
@@ -0,0 +1,55 @@
1
+ [build-system]
2
+ requires = ["setuptools>=77"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "hermes-clawifi-web"
7
+ version = "0.1.0"
8
+ description = "Clawifi web search and page reading as a web backend for Hermes Agent."
9
+ readme = "README.md"
10
+ requires-python = ">=3.11"
11
+ license = "MIT"
12
+ license-files = ["LICENSE"]
13
+ authors = [{ name = "ReARC Labs Inc.", email = "support@clawifi.dev" }]
14
+ keywords = ["hermes-agent", "hermes", "plugin", "web-search", "web-extract", "clawifi", "ai-agents", "search-api"]
15
+ classifiers = [
16
+ "Development Status :: 4 - Beta",
17
+ "Intended Audience :: Developers",
18
+ "Programming Language :: Python :: 3",
19
+ "Programming Language :: Python :: 3.11",
20
+ "Programming Language :: Python :: 3.12",
21
+ "Programming Language :: Python :: 3.13",
22
+ "Programming Language :: Python :: 3.14",
23
+ "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
24
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
25
+ ]
26
+ # hermes-agent itself is not a dependency: the plugin is loaded by the Hermes you already run.
27
+ dependencies = ["httpx>=0.27"]
28
+
29
+ [project.optional-dependencies]
30
+ test = ["pytest>=8", "ruamel.yaml>=0.18"]
31
+
32
+ [project.urls]
33
+ Homepage = "https://clawifi.dev/?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin"
34
+ Documentation = "https://clawifi.dev/docs"
35
+ Repository = "https://github.com/ReARCLabs/hermes-clawifi-web"
36
+
37
+ [project.entry-points."hermes_agent.plugins"]
38
+ web-clawifi = "hermes_clawifi_web"
39
+
40
+ [tool.setuptools.packages.find]
41
+ where = ["src"]
42
+
43
+ [tool.setuptools.package-data]
44
+ hermes_clawifi_web = ["plugin.yaml", "py.typed"]
45
+
46
+ [tool.pytest.ini_options]
47
+ testpaths = ["tests"]
48
+ markers = ["live: calls the real Clawifi API (needs CLAWIFI_API_KEY and CLAWIFI_LIVE=1)"]
49
+
50
+ [tool.ruff]
51
+ line-length = 120
52
+ target-version = "py311"
53
+
54
+ [tool.ruff.lint]
55
+ select = ["E", "F", "I", "B", "UP", "SIM"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,17 @@
1
+ """Clawifi web search + page reading backend for Hermes Agent.
2
+
3
+ Loaded by Hermes through the ``hermes_agent.plugins`` entry point (pip install) or as a
4
+ directory plugin (``~/.hermes/plugins/web/clawifi/``). The import is relative so both work.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ from ._version import __version__
10
+ from .provider import ClawifiWebSearchProvider
11
+
12
+ __all__ = ["ClawifiWebSearchProvider", "__version__", "register"]
13
+
14
+
15
+ def register(ctx) -> None:
16
+ """Plugin entry point: called once by Hermes at load time."""
17
+ ctx.register_web_search_provider(ClawifiWebSearchProvider())
@@ -0,0 +1 @@
1
+ __version__ = "0.1.0"
@@ -0,0 +1,7 @@
1
+ name: web-clawifi
2
+ version: 0.1.0
3
+ description: "Clawifi web search + page reading (Markdown). Keyless trial when selected without a key; set CLAWIFI_API_KEY (free account, 1,000 credits a month) for full access: https://clawifi.dev/signup."
4
+ author: ReARC Labs Inc.
5
+ kind: backend
6
+ provides_web_providers:
7
+ - clawifi
@@ -0,0 +1,350 @@
1
+ """Clawifi (https://clawifi.dev) web search + page reading for Hermes Agent.
2
+
3
+ ``web_search`` -> ``POST /search/free`` (ranked results: url, title, snippet).
4
+ ``web_extract`` -> ``POST /fetch`` per URL (page or PDF as Markdown).
5
+
6
+ Env: ``CLAWIFI_API_KEY`` (a free account has 1,000 credits a month, https://clawifi.dev/signup)
7
+ and ``CLAWIFI_API_BASE`` (optional, for self-hosted Clawifi).
8
+
9
+ Without a key the REST API answers 401. Clawifi's keyless trial lives on its hosted MCP endpoint
10
+ (``POST /mcp``, one stateless JSON-RPC ``tools/call``): 12 calls a minute and 50 a day per address,
11
+ 3 search results per call. This provider uses it when Clawifi is the selected backend and no key is
12
+ set, the way Hermes' Tavily backend does opt-in keyless. ``is_available()`` stays key-only, so Hermes
13
+ never auto-selects the trial over another backend. ``web.keyless_fallback: false`` turns it off.
14
+
15
+ Only Hermes' public plugin surface is imported (``agent.web_search_provider``), so this
16
+ package does not depend on the layout of Hermes' bundled ``plugins.web`` helpers.
17
+ """
18
+
19
+ from __future__ import annotations
20
+
21
+ import json
22
+ import logging
23
+ import os
24
+ from concurrent.futures import ThreadPoolExecutor
25
+ from typing import Any
26
+
27
+ import httpx
28
+ from agent.web_search_provider import WebSearchProvider
29
+
30
+ from ._version import __version__
31
+
32
+ logger = logging.getLogger(__name__)
33
+
34
+ DEFAULT_API_BASE = "https://api.clawifi.dev"
35
+ KEY_ENV = "CLAWIFI_API_KEY"
36
+ BASE_ENV = "CLAWIFI_API_BASE"
37
+ SIGNUP_URL = "https://clawifi.dev/signup?utm_source=hermes&utm_medium=integration&utm_campaign=hermes-plugin"
38
+ USER_AGENT = f"hermes-clawifi-web/{__version__}"
39
+
40
+ SEARCH_LIMIT_CAP = 20 # Hermes caps every backend at 20 results; the API itself takes up to 50
41
+ QUERY_MAX_CHARS = 500 # POST /search/free rejects longer queries with 422
42
+ SEARCH_TIMEOUT_S = 30.0
43
+ FETCH_TIMEOUT_S = 30 # sent as timeout_seconds; past it the API answers 504 at 0 credits
44
+ FETCH_HTTP_TIMEOUT_S = 45.0
45
+ EXTRACT_WORKERS = 2 # the Free plan allows 2 calls in flight; a third answers 429
46
+ KEYLESS_SEARCH_CAP = 3 # the keyless trial returns at most 3 results per search
47
+ KEYLESS_NOTE = "Clawifi keyless trial (12 calls a minute, 50 a day, 3 results per search)."
48
+
49
+ MISSING_KEY_ERROR = (
50
+ f"{KEY_ENV} is not set. Create a free Clawifi account (1,000 credits a month) at "
51
+ "https://clawifi.dev/signup and add the key to ~/.hermes/.env."
52
+ )
53
+
54
+ # content_status values that mean the page itself gave nothing to read (each costs 0 credits).
55
+ _UNREADABLE = {
56
+ "blocked": "the page is behind a challenge, CAPTCHA or sign-in gate",
57
+ "needs_javascript": "the page needs a browser to render (JavaScript shell)",
58
+ "not_found": "the page says it was not found",
59
+ }
60
+
61
+
62
+ def _env(name: str) -> str:
63
+ """Config-aware lookup (os.environ, then ~/.hermes/.env) when Hermes offers it."""
64
+ try:
65
+ from agent.web_search_provider import get_provider_env
66
+ except ImportError: # older Hermes: plain environment only
67
+ return os.getenv(name, "").strip()
68
+ return get_provider_env(name)
69
+
70
+
71
+ def _interrupted() -> bool:
72
+ try:
73
+ from tools.interrupt import is_interrupted
74
+ except ImportError:
75
+ return False
76
+ try:
77
+ return bool(is_interrupted())
78
+ except Exception:
79
+ return False
80
+
81
+
82
+ def _keyless_allowed() -> bool:
83
+ """Hermes' ``web.keyless_fallback`` switch (default on). Never raises, no network."""
84
+ try:
85
+ from hermes_cli.config import load_config
86
+
87
+ web = load_config().get("web") or {}
88
+ return bool(web.get("keyless_fallback", True))
89
+ except Exception:
90
+ return True
91
+
92
+
93
+ def _api_base() -> str:
94
+ return (_env(BASE_ENV) or DEFAULT_API_BASE).rstrip("/")
95
+
96
+
97
+ def _headers(api_key: str) -> dict[str, str]:
98
+ # No user identifiers: the key authenticates, the User-Agent names this integration only.
99
+ if not api_key: # keyless trial: the MCP endpoint wants both media types in Accept
100
+ return {"Accept": "application/json, text/event-stream", "User-Agent": USER_AGENT}
101
+ return {"Authorization": f"Bearer {api_key}", "Accept": "application/json", "User-Agent": USER_AGENT}
102
+
103
+
104
+ def _error_detail(response: httpx.Response) -> str:
105
+ """Readable message for a >=400 answer. Clawifi sends ``{"detail", "error": {"message", ...}}``."""
106
+ message = ""
107
+ try:
108
+ body = response.json()
109
+ except ValueError:
110
+ body = None
111
+ if isinstance(body, dict):
112
+ error = body.get("error")
113
+ if isinstance(error, dict):
114
+ message = str(error.get("message") or "")
115
+ if not message and isinstance(body.get("detail"), str):
116
+ message = body["detail"]
117
+ message = message or (response.text or "").strip()[:300] or "no details"
118
+ status = response.status_code
119
+ if status == 401:
120
+ return f"Clawifi rejected the API key (HTTP 401: {message}). Check {KEY_ENV}; keys are at {SIGNUP_URL}"
121
+ if status == 402:
122
+ return f"Clawifi credits are used up (HTTP 402: {message}). See https://clawifi.dev/pricing"
123
+ if status == 429:
124
+ wait = response.headers.get("Retry-After")
125
+ suffix = f" Retry after {wait} s." if wait else ""
126
+ return f"Clawifi rate limit reached (HTTP 429: {message}).{suffix}"
127
+ return f"HTTP {status}: {message}"
128
+
129
+
130
+ def _first_error(payload: dict[str, Any]) -> str:
131
+ """Message of the first entry in a 200 envelope's ``errors`` list, or ""."""
132
+ errors = payload.get("errors")
133
+ if isinstance(errors, list) and errors:
134
+ first = errors[0]
135
+ if isinstance(first, dict):
136
+ return str(first.get("message") or first.get("code") or "")
137
+ return str(first)
138
+ return ""
139
+
140
+
141
+ def _search_fail(error: str) -> dict[str, Any]:
142
+ return {"success": False, "error": error}
143
+
144
+
145
+ def _page_error(url: str, error: str) -> dict[str, Any]:
146
+ return {"url": url, "title": "", "content": "", "error": error}
147
+
148
+
149
+ def _normalize_search(payload: dict[str, Any]) -> dict[str, Any]:
150
+ """Map a ``POST /search/free`` answer to Hermes' search envelope."""
151
+ rows = payload.get("results")
152
+ if not isinstance(rows, list):
153
+ rows = []
154
+ if payload.get("success") is False and not rows:
155
+ return _search_fail(f"Clawifi search failed: {_first_error(payload) or 'no results'}")
156
+ web = [
157
+ {
158
+ "title": str(row.get("title") or ""),
159
+ "url": str(row.get("url") or ""),
160
+ "description": str(row.get("snippet") or ""),
161
+ "position": index + 1,
162
+ }
163
+ for index, row in enumerate(r for r in rows if isinstance(r, dict) and r.get("url"))
164
+ ]
165
+ return {"success": True, "data": {"web": web}}
166
+
167
+
168
+ def _normalize_page(requested_url: str, payload: dict[str, Any]) -> dict[str, Any]:
169
+ """Map a ``POST /fetch`` envelope to one Hermes extract entry."""
170
+ data = payload.get("data")
171
+ if not isinstance(data, dict):
172
+ data = {}
173
+ content = str(data.get("content_markdown") or data.get("content_text") or "")
174
+ status = str(payload.get("content_status") or "ok")
175
+ if payload.get("success") is False and not content:
176
+ return _page_error(requested_url, f"Clawifi fetch failed: {_first_error(payload) or 'no content'}")
177
+ if status in _UNREADABLE: # any text here is the challenge or the 404 page, not the content asked for
178
+ return _page_error(requested_url, f"Clawifi could not read the page: {_UNREADABLE[status]}")
179
+ if not content:
180
+ return _page_error(requested_url, "Clawifi returned no text for this page")
181
+ title = str(data.get("title") or "")
182
+ metadata: dict[str, Any] = {"sourceURL": requested_url, "title": title, "content_status": status}
183
+ for key in ("published_at", "modified_at", "language"):
184
+ if data.get(key):
185
+ metadata[key] = data[key]
186
+ return {
187
+ "url": str(data.get("url") or requested_url),
188
+ "title": title,
189
+ "content": content,
190
+ "raw_content": content,
191
+ "metadata": metadata,
192
+ }
193
+
194
+
195
+ class ClawifiWebSearchProvider(WebSearchProvider):
196
+ """Clawifi search + extract provider (keyed, or the keyless trial when selected without a key)."""
197
+
198
+ def __init__(self, *, transport: httpx.BaseTransport | None = None) -> None:
199
+ self._transport = transport # tests pass httpx.MockTransport; None = real network
200
+
201
+ @property
202
+ def name(self) -> str:
203
+ return "clawifi"
204
+
205
+ @property
206
+ def display_name(self) -> str:
207
+ return "Clawifi"
208
+
209
+ def is_available(self) -> bool:
210
+ return bool(_env(KEY_ENV))
211
+
212
+ def is_keyless_available(self) -> bool:
213
+ return _keyless_allowed()
214
+
215
+ def supports_search(self) -> bool:
216
+ return True
217
+
218
+ def supports_extract(self) -> bool:
219
+ return True
220
+
221
+ def _post(self, client: httpx.Client, path: str, payload: dict[str, Any], timeout: float) -> dict[str, Any]:
222
+ """POST JSON and return the parsed body. ValueError carries a ready-to-show message."""
223
+ response = client.post(f"{_api_base()}{path}", json=payload, timeout=timeout)
224
+ if response.status_code >= 400:
225
+ raise ValueError(_error_detail(response))
226
+ try:
227
+ body = response.json()
228
+ except ValueError as exc:
229
+ raise ValueError("Clawifi answered with a body that is not JSON") from exc
230
+ if not isinstance(body, dict):
231
+ raise ValueError("Clawifi answered with an unexpected JSON shape")
232
+ return body
233
+
234
+ def _call_tool(self, client: httpx.Client, tool: str, arguments: dict[str, Any], timeout: float) -> dict[str, Any]:
235
+ """Keyless trial: one stateless JSON-RPC ``tools/call`` on the hosted MCP endpoint. Returns the
236
+ tool's JSON answer (the same envelope the REST route sends); ValueError carries the message."""
237
+ rpc = {"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": tool, "arguments": arguments}}
238
+ body = self._post(client, "/mcp", rpc, timeout)
239
+ if isinstance(body.get("error"), dict):
240
+ raise ValueError(str(body["error"].get("message") or "MCP error"))
241
+ result = body.get("result")
242
+ if not isinstance(result, dict):
243
+ raise ValueError("Clawifi answered with an unexpected JSON shape")
244
+ text = ""
245
+ content = result.get("content")
246
+ if isinstance(content, list) and content and isinstance(content[0], dict):
247
+ text = str(content[0].get("text") or "")
248
+ if result.get("isError"): # limit reached, or the tool needs a key; the text says which
249
+ raise ValueError(
250
+ f"{text or 'the keyless trial refused the call'} Set {KEY_ENV} for full access: {SIGNUP_URL}"
251
+ )
252
+ payload = result.get("structuredContent")
253
+ if not isinstance(payload, dict):
254
+ try:
255
+ payload = json.loads(text)
256
+ except ValueError as exc:
257
+ raise ValueError("Clawifi answered with a body that is not JSON") from exc
258
+ if not isinstance(payload, dict):
259
+ raise ValueError("Clawifi answered with an unexpected JSON shape")
260
+ return payload
261
+
262
+ def _client(self, api_key: str) -> httpx.Client:
263
+ return httpx.Client(headers=_headers(api_key), transport=self._transport)
264
+
265
+ def search(self, query: str, limit: int = 5) -> dict[str, Any]:
266
+ try:
267
+ if _interrupted():
268
+ return _search_fail("Interrupted")
269
+ api_key = _env(KEY_ENV)
270
+ if not api_key and not _keyless_allowed():
271
+ return _search_fail(MISSING_KEY_ERROR)
272
+ query = (query or "").strip()
273
+ if not query:
274
+ return _search_fail("Clawifi search needs a non-empty query")
275
+ try:
276
+ max_results = min(max(1, int(limit)), SEARCH_LIMIT_CAP)
277
+ except (TypeError, ValueError):
278
+ max_results = 5
279
+ logger.info("Clawifi search: '%s' (limit=%d)", query, max_results)
280
+ payload = {"query": query[:QUERY_MAX_CHARS], "max_results": max_results}
281
+ with self._client(api_key) as client:
282
+ if api_key:
283
+ return _normalize_search(self._post(client, "/search/free", payload, SEARCH_TIMEOUT_S))
284
+ payload["max_results"] = min(max_results, KEYLESS_SEARCH_CAP)
285
+ result = _normalize_search(self._call_tool(client, "search_free", payload, SEARCH_TIMEOUT_S))
286
+ if result.get("success"):
287
+ result["data"]["note"] = f"{KEYLESS_NOTE} Set {KEY_ENV} for more: {SIGNUP_URL}"
288
+ return result
289
+ except ValueError as exc:
290
+ return _search_fail(f"Clawifi search failed: {exc}")
291
+ except httpx.HTTPError as exc:
292
+ logger.warning("Clawifi search error: %s", exc)
293
+ return _search_fail(f"Could not reach Clawifi: {exc}")
294
+ except Exception as exc: # the contract is "never raise": Hermes shows the envelope to the model
295
+ logger.warning("Clawifi search error: %s", exc)
296
+ return _search_fail(f"Clawifi search failed: {exc}")
297
+
298
+ def _fetch_one(self, client: httpx.Client, url: str, keyed: bool = True) -> dict[str, Any]:
299
+ if _interrupted():
300
+ return {"url": url, "title": "", "error": "Interrupted"}
301
+ try:
302
+ if keyed:
303
+ payload = {"url": url, "timeout_seconds": FETCH_TIMEOUT_S}
304
+ body = self._post(client, "/fetch", payload, FETCH_HTTP_TIMEOUT_S)
305
+ else:
306
+ body = self._call_tool(client, "fetch_url", {"url": url}, FETCH_HTTP_TIMEOUT_S)
307
+ return _normalize_page(url, body)
308
+ except ValueError as exc:
309
+ return _page_error(url, f"Clawifi fetch failed: {exc}")
310
+ except httpx.HTTPError as exc:
311
+ return _page_error(url, f"Could not reach Clawifi: {exc}")
312
+ except Exception as exc:
313
+ logger.warning("Clawifi extract error for %s: %s", url, exc)
314
+ return _page_error(url, f"Clawifi fetch failed: {exc}")
315
+
316
+ def extract(self, urls: list[str], **kwargs: Any) -> list[dict[str, Any]]:
317
+ """One entry per requested URL, in request order. Per-URL failures carry ``error``; nothing is raised.
318
+ ``kwargs`` (``format``, ...) are accepted and ignored: Clawifi returns Markdown."""
319
+ urls = [str(u) for u in (urls or [])]
320
+ if not urls:
321
+ return []
322
+ if _interrupted():
323
+ return [{"url": u, "title": "", "error": "Interrupted"} for u in urls]
324
+ try:
325
+ api_key = _env(KEY_ENV)
326
+ except Exception as exc:
327
+ return [_page_error(u, f"Clawifi fetch failed: {exc}") for u in urls]
328
+ if not api_key and not _keyless_allowed():
329
+ return [_page_error(u, MISSING_KEY_ERROR) for u in urls]
330
+ keyed = bool(api_key)
331
+ logger.info("Clawifi %sextract: %d URL(s)", "" if keyed else "keyless ", len(urls))
332
+ with self._client(api_key) as client:
333
+ if len(urls) == 1:
334
+ return [self._fetch_one(client, urls[0], keyed)]
335
+ with ThreadPoolExecutor(max_workers=min(EXTRACT_WORKERS, len(urls))) as pool:
336
+ return list(pool.map(lambda u: self._fetch_one(client, u, keyed), urls))
337
+
338
+ def get_setup_schema(self) -> dict[str, Any]:
339
+ return {
340
+ "name": "Clawifi",
341
+ "badge": "free · key optional",
342
+ "tag": "Web search + page and PDF reading as Markdown. Keyless trial; a free key has 1,000 credits/month.",
343
+ "env_vars": [
344
+ {
345
+ "key": KEY_ENV,
346
+ "prompt": "Clawifi API key (optional: a keyless trial works without)",
347
+ "url": SIGNUP_URL,
348
+ }
349
+ ],
350
+ }