hermes-oxylabs-web-api 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Oxylabs
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,133 @@
1
+ Metadata-Version: 2.4
2
+ Name: hermes-oxylabs-web-api
3
+ Version: 0.1.0
4
+ Summary: Oxylabs Web API search and page extraction provider for Hermes Agent
5
+ Author-email: Oxylabs <giedrius.stankauskas@oxylabs.io>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/oxylabs/hermes-oxylabs-web-api
8
+ Project-URL: Repository, https://github.com/oxylabs/hermes-oxylabs-web-api
9
+ Project-URL: Issues, https://github.com/oxylabs/hermes-oxylabs-web-api/issues
10
+ Keywords: hermes-agent,plugin,oxylabs,web-search,web-scraping,markdown
11
+ Classifier: Development Status :: 4 - Beta
12
+ Classifier: Environment :: Plugins
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Programming Language :: Python :: 3.14
18
+ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
19
+ Requires-Python: <3.16,>=3.11
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: httpx<1,>=0.27
23
+ Dynamic: license-file
24
+
25
+ # Oxylabs Web API for Hermes Agent
26
+
27
+ A [Hermes Agent](https://hermes-agent.nousresearch.com/) plugin that adds the [Oxylabs Web API](https://oxylabs.io/products/web-api) as a provider for Hermes's built-in `web_search` and `web_extract` tools.
28
+
29
+ - **web_search**: Google results with titles, URLs and a short outline of each page. Set a country to see results as users in that country do.
30
+ - **web_extract**: any page as Markdown, including JavaScript-heavy pages (turn on JavaScript rendering) and pages that block plain HTTP clients.
31
+
32
+ The agent keeps using the tools Hermes already has; you choose Oxylabs as the backend.
33
+
34
+ ## Requirements
35
+
36
+ - Hermes Agent 0.23.1 or later.
37
+ - An Oxylabs Web API key. Get one in the [Oxylabs dashboard](https://dashboard.oxylabs.io/).
38
+
39
+ ## Set up
40
+
41
+ 1. Install and enable the plugin:
42
+
43
+ ```bash
44
+ hermes plugins install oxylabs/hermes-oxylabs-web-api --enable
45
+ ```
46
+
47
+ Hermes warns that the plugin is not from its reviewed catalog, and asks before installing its one dependency (`httpx`, which Hermes already uses). In scripts, add `--yes-deps`.
48
+
49
+ 2. Add your key. The install asks for it; paste it there, or skip and add it later. To add it later, run `hermes tools`, open **Web Search & Extract**, select **Oxylabs Web API** and paste the key. Or add it to your profile's `.env` file (`~/.hermes/.env` for the default profile):
50
+
51
+ ```bash
52
+ OXYLABS_WEB_API_KEY=your-web-api-key
53
+ ```
54
+
55
+ 3. Select Oxylabs for search, extraction, or both:
56
+
57
+ ```bash
58
+ hermes config set web.search_backend oxylabs
59
+ hermes config set web.extract_backend oxylabs
60
+ ```
61
+
62
+ Start a new session after these steps.
63
+
64
+ ## Settings
65
+
66
+ Both are optional:
67
+
68
+ ```bash
69
+ hermes config set plugins.entries.oxylabs-web-api.settings.location US
70
+ hermes config set plugins.entries.oxylabs-web-api.settings.render_js true
71
+ ```
72
+
73
+ | Setting | Default | What it does |
74
+ |---|---|---|
75
+ | `location` | none | Two-letter country code for searches, such as `US` or `DE`. |
76
+ | `render_js` | `false` | Load pages in a browser before extracting them. Slower and uses more credits; turn it on if pages come back nearly empty. |
77
+
78
+ The Hermes desktop app shows the same settings on the plugin's page under **Settings → Plugins**.
79
+
80
+ Slow pages are queued on the asynchronous Web API and polled, and each page gets up to 110 seconds, just inside Hermes's 120-second extract timeout.
81
+
82
+ ## Your data
83
+
84
+ - The plugin sends your search queries and the URLs you ask Hermes to read to `https://webapi.oxylabs.io`, together with your API key. It contacts no other service. Requests are billed to your Oxylabs account.
85
+ - Oxylabs runs each search and loads each page, so the sites you read receive Oxylabs's request. Before a page is fetched, Hermes checks the address on your machine and blocks private and internal ones.
86
+ - Requests carry an `x-oxylabs-sdk: oxylabs-web-api-hermes/<version>` header so Oxylabs can count usage per integration. It holds the plugin and Python versions only.
87
+ - The plugin reads the key through Hermes and never includes it in results or error messages. It collects no telemetry, runs no shell commands or background processes, and reads none of your files.
88
+
89
+ ## Update or remove
90
+
91
+ ```bash
92
+ hermes plugins update oxylabs-web-api
93
+ hermes plugins remove oxylabs-web-api
94
+ hermes config unset web.search_backend
95
+ hermes config unset web.extract_backend
96
+ ```
97
+
98
+ ## Troubleshooting
99
+
100
+ | Problem | Fix |
101
+ |---|---|
102
+ | `OXYLABS_WEB_API_KEY is not set` | Add the key as in step 2, then start a new session. |
103
+ | `rejected the API key (401)` | The key is wrong or revoked. Check it in the Oxylabs dashboard. |
104
+ | `quota is used up (429 QUOTA_EXCEEDED)` | Top up the plan in the Oxylabs dashboard. |
105
+ | `web_search` uses another provider | Run the `hermes config set` commands from step 3. Check `hermes plugins list` shows `oxylabs-web-api` as enabled. |
106
+ | A page comes back nearly empty | Turn on `render_js`. |
107
+
108
+ ## Install from PyPI
109
+
110
+ If you run Hermes from a Python environment you manage yourself, install the package into the interpreter that runs Hermes and enable it:
111
+
112
+ ```bash
113
+ uv pip install --python /path/to/hermes/python hermes-oxylabs-web-api
114
+ hermes plugins enable oxylabs-web-api
115
+ ```
116
+
117
+ `hermes plugins update` and `remove` don't manage a PyPI install; upgrade or uninstall it with the same tool.
118
+
119
+ ## Development
120
+
121
+ ```bash
122
+ uv sync --dev
123
+ uv run ruff check . && uv run ruff format --check .
124
+ uv run pytest # unit tests
125
+ OXYLABS_WEB_API_KEY=... uv run pytest -m live # calls the real API
126
+ hermes plugins validate . # the checks the Hermes catalog runs
127
+ ```
128
+
129
+ Releases: bump the version in `pyproject.toml`, both `plugin.yaml` files and `src/hermes_oxylabs_web_api/__init__.py`, add a `CHANGELOG.md` entry, then push a `v<version>` tag. The `Publish` workflow publishes to PyPI with trusted publishing.
130
+
131
+ ## License
132
+
133
+ MIT
@@ -0,0 +1,109 @@
1
+ # Oxylabs Web API for Hermes Agent
2
+
3
+ A [Hermes Agent](https://hermes-agent.nousresearch.com/) plugin that adds the [Oxylabs Web API](https://oxylabs.io/products/web-api) as a provider for Hermes's built-in `web_search` and `web_extract` tools.
4
+
5
+ - **web_search**: Google results with titles, URLs and a short outline of each page. Set a country to see results as users in that country do.
6
+ - **web_extract**: any page as Markdown, including JavaScript-heavy pages (turn on JavaScript rendering) and pages that block plain HTTP clients.
7
+
8
+ The agent keeps using the tools Hermes already has; you choose Oxylabs as the backend.
9
+
10
+ ## Requirements
11
+
12
+ - Hermes Agent 0.23.1 or later.
13
+ - An Oxylabs Web API key. Get one in the [Oxylabs dashboard](https://dashboard.oxylabs.io/).
14
+
15
+ ## Set up
16
+
17
+ 1. Install and enable the plugin:
18
+
19
+ ```bash
20
+ hermes plugins install oxylabs/hermes-oxylabs-web-api --enable
21
+ ```
22
+
23
+ Hermes warns that the plugin is not from its reviewed catalog, and asks before installing its one dependency (`httpx`, which Hermes already uses). In scripts, add `--yes-deps`.
24
+
25
+ 2. Add your key. The install asks for it; paste it there, or skip and add it later. To add it later, run `hermes tools`, open **Web Search & Extract**, select **Oxylabs Web API** and paste the key. Or add it to your profile's `.env` file (`~/.hermes/.env` for the default profile):
26
+
27
+ ```bash
28
+ OXYLABS_WEB_API_KEY=your-web-api-key
29
+ ```
30
+
31
+ 3. Select Oxylabs for search, extraction, or both:
32
+
33
+ ```bash
34
+ hermes config set web.search_backend oxylabs
35
+ hermes config set web.extract_backend oxylabs
36
+ ```
37
+
38
+ Start a new session after these steps.
39
+
40
+ ## Settings
41
+
42
+ Both are optional:
43
+
44
+ ```bash
45
+ hermes config set plugins.entries.oxylabs-web-api.settings.location US
46
+ hermes config set plugins.entries.oxylabs-web-api.settings.render_js true
47
+ ```
48
+
49
+ | Setting | Default | What it does |
50
+ |---|---|---|
51
+ | `location` | none | Two-letter country code for searches, such as `US` or `DE`. |
52
+ | `render_js` | `false` | Load pages in a browser before extracting them. Slower and uses more credits; turn it on if pages come back nearly empty. |
53
+
54
+ The Hermes desktop app shows the same settings on the plugin's page under **Settings → Plugins**.
55
+
56
+ Slow pages are queued on the asynchronous Web API and polled, and each page gets up to 110 seconds, just inside Hermes's 120-second extract timeout.
57
+
58
+ ## Your data
59
+
60
+ - The plugin sends your search queries and the URLs you ask Hermes to read to `https://webapi.oxylabs.io`, together with your API key. It contacts no other service. Requests are billed to your Oxylabs account.
61
+ - Oxylabs runs each search and loads each page, so the sites you read receive Oxylabs's request. Before a page is fetched, Hermes checks the address on your machine and blocks private and internal ones.
62
+ - Requests carry an `x-oxylabs-sdk: oxylabs-web-api-hermes/<version>` header so Oxylabs can count usage per integration. It holds the plugin and Python versions only.
63
+ - The plugin reads the key through Hermes and never includes it in results or error messages. It collects no telemetry, runs no shell commands or background processes, and reads none of your files.
64
+
65
+ ## Update or remove
66
+
67
+ ```bash
68
+ hermes plugins update oxylabs-web-api
69
+ hermes plugins remove oxylabs-web-api
70
+ hermes config unset web.search_backend
71
+ hermes config unset web.extract_backend
72
+ ```
73
+
74
+ ## Troubleshooting
75
+
76
+ | Problem | Fix |
77
+ |---|---|
78
+ | `OXYLABS_WEB_API_KEY is not set` | Add the key as in step 2, then start a new session. |
79
+ | `rejected the API key (401)` | The key is wrong or revoked. Check it in the Oxylabs dashboard. |
80
+ | `quota is used up (429 QUOTA_EXCEEDED)` | Top up the plan in the Oxylabs dashboard. |
81
+ | `web_search` uses another provider | Run the `hermes config set` commands from step 3. Check `hermes plugins list` shows `oxylabs-web-api` as enabled. |
82
+ | A page comes back nearly empty | Turn on `render_js`. |
83
+
84
+ ## Install from PyPI
85
+
86
+ If you run Hermes from a Python environment you manage yourself, install the package into the interpreter that runs Hermes and enable it:
87
+
88
+ ```bash
89
+ uv pip install --python /path/to/hermes/python hermes-oxylabs-web-api
90
+ hermes plugins enable oxylabs-web-api
91
+ ```
92
+
93
+ `hermes plugins update` and `remove` don't manage a PyPI install; upgrade or uninstall it with the same tool.
94
+
95
+ ## Development
96
+
97
+ ```bash
98
+ uv sync --dev
99
+ uv run ruff check . && uv run ruff format --check .
100
+ uv run pytest # unit tests
101
+ OXYLABS_WEB_API_KEY=... uv run pytest -m live # calls the real API
102
+ hermes plugins validate . # the checks the Hermes catalog runs
103
+ ```
104
+
105
+ Releases: bump the version in `pyproject.toml`, both `plugin.yaml` files and `src/hermes_oxylabs_web_api/__init__.py`, add a `CHANGELOG.md` entry, then push a `v<version>` tag. The `Publish` workflow publishes to PyPI with trusted publishing.
106
+
107
+ ## License
108
+
109
+ MIT
@@ -0,0 +1,56 @@
1
+ [build-system]
2
+ requires = ["setuptools>=77,<83"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "hermes-oxylabs-web-api"
7
+ version = "0.1.0"
8
+ description = "Oxylabs Web API search and page extraction provider for Hermes Agent"
9
+ readme = "README.md"
10
+ requires-python = ">=3.11,<3.16"
11
+ license = "MIT"
12
+ license-files = ["LICENSE"]
13
+ authors = [{ name = "Oxylabs", email = "giedrius.stankauskas@oxylabs.io" }]
14
+ keywords = ["hermes-agent", "plugin", "oxylabs", "web-search", "web-scraping", "markdown"]
15
+ classifiers = [
16
+ "Development Status :: 4 - Beta",
17
+ "Environment :: Plugins",
18
+ "Programming Language :: Python :: 3",
19
+ "Programming Language :: Python :: 3.11",
20
+ "Programming Language :: Python :: 3.12",
21
+ "Programming Language :: Python :: 3.13",
22
+ "Programming Language :: Python :: 3.14",
23
+ "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
24
+ ]
25
+ dependencies = ["httpx>=0.27,<1"]
26
+
27
+ [project.entry-points."hermes_agent.plugins"]
28
+ oxylabs-web-api = "hermes_oxylabs_web_api"
29
+
30
+ [project.urls]
31
+ Homepage = "https://github.com/oxylabs/hermes-oxylabs-web-api"
32
+ Repository = "https://github.com/oxylabs/hermes-oxylabs-web-api"
33
+ Issues = "https://github.com/oxylabs/hermes-oxylabs-web-api/issues"
34
+
35
+ [dependency-groups]
36
+ dev = ["pytest>=8.3,<10", "pytest-asyncio>=0.24,<2", "ruff>=0.12,<1", "twine>=6.1,<7"]
37
+
38
+ [tool.setuptools.packages.find]
39
+ where = ["src"]
40
+
41
+ [tool.setuptools.package-data]
42
+ hermes_oxylabs_web_api = ["plugin.yaml", "py.typed"]
43
+
44
+ [tool.pytest.ini_options]
45
+ addopts = "-q -m 'not live'"
46
+ asyncio_mode = "auto"
47
+ asyncio_default_fixture_loop_scope = "function"
48
+ testpaths = ["tests"]
49
+ markers = ["live: calls the real Oxylabs Web API and needs OXYLABS_WEB_API_KEY"]
50
+
51
+ [tool.ruff]
52
+ line-length = 120
53
+ target-version = "py311"
54
+
55
+ [tool.ruff.lint]
56
+ select = ["B", "E", "F", "I", "UP"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,14 @@
1
+ """Oxylabs Web API search and page extraction for Hermes Agent."""
2
+
3
+ from __future__ import annotations
4
+
5
+ __version__ = "0.1.0"
6
+
7
+ __all__ = ["__version__", "register"]
8
+
9
+
10
+ def register(ctx) -> None:
11
+ """Register the Oxylabs web search and extract provider."""
12
+ from .provider import OxylabsWebSearchProvider
13
+
14
+ ctx.register_web_search_provider(OxylabsWebSearchProvider(settings=ctx.get_config))
@@ -0,0 +1,196 @@
1
+ """HTTP client for the Oxylabs Web API (`/v1/search`, `/v1/scrape`, `/v1/async/scrape`)."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import asyncio
6
+ import platform
7
+ import time
8
+ from typing import Any
9
+
10
+ import httpx
11
+
12
+ from . import __version__
13
+
14
+ API_BASE_URL = "https://webapi.oxylabs.io"
15
+ API_KEY_ENV = "OXYLABS_WEB_API_KEY"
16
+ MAX_SEARCH_RESULTS = 20
17
+ SEARCH_TIMEOUT_SECONDS = 30.0
18
+ SYNC_SCRAPE_TIMEOUT_SECONDS = 60.0
19
+ POLL_INTERVAL_SECONDS = 3.0
20
+
21
+ _ERROR_DETAIL_MAX_CHARS = 300
22
+ _SCRAPE_TIMEOUT_TITLE = "REQUEST_FAILED_SCRAPE_TIMEOUT"
23
+
24
+
25
+ class OxylabsApiError(RuntimeError):
26
+ """A Web API error with a message safe to show the agent. It never contains the API key."""
27
+
28
+ def __init__(self, message: str, status: int, request_id: str | None = None) -> None:
29
+ super().__init__(message)
30
+ self.status = status
31
+ self.request_id = request_id
32
+
33
+
34
+ def sdk_header() -> str:
35
+ """The `x-oxylabs-sdk` value Oxylabs uses to count usage per integration."""
36
+ return f"oxylabs-web-api-hermes/{__version__} (hermes; python {platform.python_version()})"
37
+
38
+
39
+ def _headers(api_key: str) -> dict[str, str]:
40
+ return {
41
+ "Accept": "application/json",
42
+ "Authorization": f"Bearer {api_key}",
43
+ "x-oxylabs-sdk": sdk_header(),
44
+ }
45
+
46
+
47
+ def _json_body(response: httpx.Response) -> dict[str, Any] | None:
48
+ try:
49
+ body = response.json()
50
+ except ValueError:
51
+ return None
52
+ return body if isinstance(body, dict) else None
53
+
54
+
55
+ def _request_id(body: dict[str, Any] | None) -> str | None:
56
+ if not body:
57
+ return None
58
+ metadata = body.get("metadata")
59
+ value = metadata.get("request_id") if isinstance(metadata, dict) else None
60
+ value = value if value is not None else body.get("request_id")
61
+ return str(value) if isinstance(value, (str, int)) else None
62
+
63
+
64
+ def describe_api_error(status: int, body: dict[str, Any] | None) -> str:
65
+ """Turn a failed response into a message the agent (and the user) can act on."""
66
+ request_id = _request_id(body)
67
+ suffix = f" (request_id {request_id})" if request_id else ""
68
+ if status == 401:
69
+ return f"Oxylabs Web API rejected the API key (401). Check {API_KEY_ENV}."
70
+ if status == 403:
71
+ return f"Oxylabs Web API refused the request (403): the account has no access to this feature{suffix}."
72
+ title = body.get("title") if body else None
73
+ if status == 429 and title == "QUOTA_EXCEEDED":
74
+ return "Oxylabs Web API quota is used up (429 QUOTA_EXCEEDED). Top up the plan in the Oxylabs dashboard."
75
+ errors = body.get("errors") if body else None
76
+ field_errors = "; ".join(
77
+ f"{str(entry.get('pointer', '')).removeprefix('#/')}: {entry.get('detail', '')}"
78
+ for entry in (errors if isinstance(errors, list) else [])
79
+ if isinstance(entry, dict)
80
+ )
81
+ detail = field_errors or (body.get("detail") if body and isinstance(body.get("detail"), str) else "")
82
+ message = f"Oxylabs Web API error {status} {title if isinstance(title, str) else 'request failed'}"
83
+ if detail:
84
+ message += f": {detail}"
85
+ if len(message) > _ERROR_DETAIL_MAX_CHARS:
86
+ message = message[:_ERROR_DETAIL_MAX_CHARS] + "…"
87
+ return message + suffix
88
+
89
+
90
+ def _ensure_ok(response: httpx.Response) -> dict[str, Any]:
91
+ body = _json_body(response)
92
+ if 200 <= response.status_code < 300 and body is not None:
93
+ return body
94
+ raise OxylabsApiError(describe_api_error(response.status_code, body), response.status_code, _request_id(body))
95
+
96
+
97
+ def search_web(api_key: str, query: str, limit: int, location: str | None = None) -> list[dict[str, Any]]:
98
+ """`POST /v1/search`: organic results, each with a short outline of the page."""
99
+ payload: dict[str, Any] = {"query": query, "max_results": limit}
100
+ if location:
101
+ payload["location"] = location
102
+ try:
103
+ response = httpx.post(
104
+ f"{API_BASE_URL}/v1/search", json=payload, headers=_headers(api_key), timeout=SEARCH_TIMEOUT_SECONDS
105
+ )
106
+ except httpx.RequestError as exc:
107
+ raise OxylabsApiError(f"Could not reach the Oxylabs Web API ({type(exc).__name__}).", 0) from None
108
+ data = _ensure_ok(response)
109
+ results = data.get("results")
110
+ hits = []
111
+ for entry in results if isinstance(results, list) else []:
112
+ if not isinstance(entry, dict):
113
+ continue
114
+ outline = entry.get("outline")
115
+ metadata = entry.get("metadata")
116
+ position = metadata.get("position") if isinstance(metadata, dict) else None
117
+ hits.append(
118
+ {
119
+ "title": str(entry.get("title") or ""),
120
+ "url": str(entry.get("url") or ""),
121
+ "description": " … ".join(line for line in outline if isinstance(line, str))
122
+ if isinstance(outline, list)
123
+ else "",
124
+ "position": position if isinstance(position, int) else len(hits) + 1,
125
+ }
126
+ )
127
+ return hits
128
+
129
+
130
+ def _read_page(data: dict[str, Any]) -> tuple[str, int | None]:
131
+ results = data.get("results")
132
+ first = results[0] if isinstance(results, list) and results and isinstance(results[0], dict) else {}
133
+ markdown = first.get("markdown")
134
+ metadata = first.get("metadata")
135
+ statuses = metadata.get("statuses") if isinstance(metadata, dict) else None
136
+ http_request = statuses.get("http_request") if isinstance(statuses, dict) else None
137
+ code = http_request.get("code") if isinstance(http_request, dict) else None
138
+ return (markdown if isinstance(markdown, str) else ""), (code if isinstance(code, int) else None)
139
+
140
+
141
+ async def scrape_page(
142
+ client: httpx.AsyncClient, api_key: str, url: str, *, render_js: bool, deadline: float
143
+ ) -> tuple[str, int | None]:
144
+ """`POST /v1/scrape` for one page as Markdown; returns ``(markdown, target HTTP status)``.
145
+
146
+ Most pages finish inside the synchronous call. When it times out, the same request is queued on
147
+ `/v1/async/scrape` and polled until ``deadline`` (a ``time.monotonic()`` value), instead of being
148
+ resent synchronously.
149
+ """
150
+ payload: dict[str, Any] = {"url": url, "output": ["markdown"]}
151
+ if render_js:
152
+ payload["run_js"] = True
153
+ headers = _headers(api_key)
154
+
155
+ def remaining() -> float:
156
+ return max(1.0, deadline - time.monotonic())
157
+
158
+ try:
159
+ response = await client.post(
160
+ f"{API_BASE_URL}/v1/scrape",
161
+ json=payload,
162
+ headers=headers,
163
+ timeout=min(SYNC_SCRAPE_TIMEOUT_SECONDS, remaining()),
164
+ )
165
+ body = _json_body(response)
166
+ timed_out = response.status_code == 504 or (body or {}).get("title") == _SCRAPE_TIMEOUT_TITLE
167
+ if not timed_out:
168
+ return _read_page(_ensure_ok(response))
169
+ except httpx.TimeoutException:
170
+ pass
171
+ except httpx.RequestError as exc:
172
+ raise OxylabsApiError(f"Could not reach the Oxylabs Web API ({type(exc).__name__}).", 0) from None
173
+
174
+ try:
175
+ queued = _ensure_ok(
176
+ await client.post(f"{API_BASE_URL}/v1/async/scrape", json=payload, headers=headers, timeout=remaining())
177
+ )
178
+ request_id = _request_id(queued)
179
+ if not request_id:
180
+ raise OxylabsApiError("Oxylabs Web API queued the scrape but returned no request_id.", 502)
181
+ while True:
182
+ if time.monotonic() + POLL_INTERVAL_SECONDS > deadline:
183
+ raise OxylabsApiError(
184
+ f"Oxylabs Web API did not finish scraping {url} in time (request_id {request_id}). Try again.",
185
+ 504,
186
+ request_id,
187
+ )
188
+ await asyncio.sleep(POLL_INTERVAL_SECONDS)
189
+ polled = await client.get(
190
+ f"{API_BASE_URL}/v1/async/scrape/{request_id}", headers=headers, timeout=remaining()
191
+ )
192
+ if (_json_body(polled) or {}).get("state") == "pending":
193
+ continue
194
+ return _read_page(_ensure_ok(polled))
195
+ except httpx.RequestError as exc:
196
+ raise OxylabsApiError(f"Could not reach the Oxylabs Web API ({type(exc).__name__}).", 0) from None
@@ -0,0 +1,26 @@
1
+ name: oxylabs-web-api
2
+ version: 0.1.0
3
+ requires_hermes: ">=0.23.1"
4
+ description: "Oxylabs Web API search and page extraction for Hermes Agent: Google results with page outlines, any page as Markdown, including JavaScript-heavy and bot-protected pages."
5
+ author: Oxylabs
6
+ license: MIT
7
+ homepage: https://github.com/oxylabs/hermes-oxylabs-web-api
8
+ kind: backend
9
+ provides_web_providers:
10
+ - oxylabs
11
+ requires_env:
12
+ - name: OXYLABS_WEB_API_KEY
13
+ description: "Oxylabs Web API key"
14
+ url: "https://dashboard.oxylabs.io/"
15
+ secret: true
16
+ config_schema:
17
+ location:
18
+ type: str
19
+ default: ""
20
+ label: "Search country"
21
+ description: "Two-letter country code (for example US or DE) to see search results as users in that country do. Empty means no targeting."
22
+ render_js:
23
+ type: bool
24
+ default: false
25
+ label: "Render JavaScript"
26
+ description: "Load pages in a browser before extracting them. Slower and uses more credits; turn it on if pages come back nearly empty."
@@ -0,0 +1,143 @@
1
+ """Hermes web search and extraction backed by the Oxylabs Web API."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import asyncio
6
+ import logging
7
+ import re
8
+ import time
9
+ from collections.abc import Callable
10
+ from typing import Any
11
+
12
+ import httpx
13
+ from agent.web_search_provider import WebSearchProvider, get_provider_env
14
+
15
+ from .client import API_KEY_ENV, MAX_SEARCH_RESULTS, OxylabsApiError, scrape_page, search_web
16
+
17
+ logger = logging.getLogger(__name__)
18
+
19
+ PROVIDER_NAME = "oxylabs"
20
+ SIGNUP_URL = "https://dashboard.oxylabs.io/"
21
+ # Hermes cancels a whole extract() dispatch after `web.extract_timeout` (120 s by default);
22
+ # finish a little earlier so each page reports its own error instead of a blanket timeout.
23
+ EXTRACT_DEADLINE_SECONDS = 110.0
24
+ _COUNTRY_CODE = re.compile(r"^[A-Za-z]{2}$")
25
+
26
+ # Reads one of this plugin's settings (`plugins.entries.oxylabs-web-api.settings.<key>`).
27
+ SettingReader = Callable[[str, Any], Any]
28
+
29
+
30
+ def _no_settings(key: str, default: Any = None) -> Any:
31
+ return default
32
+
33
+
34
+ def _missing_key_error() -> str:
35
+ return (
36
+ f"{API_KEY_ENV} is not set. Get a Web API key at {SIGNUP_URL}, then run `hermes tools` "
37
+ f"(Web Search & Extract → Oxylabs Web API) or add {API_KEY_ENV} to ~/.hermes/.env."
38
+ )
39
+
40
+
41
+ def _title_from_markdown(markdown: str) -> str:
42
+ for line in markdown.splitlines():
43
+ if line.startswith("# "):
44
+ return line[2:].strip()
45
+ return ""
46
+
47
+
48
+ class OxylabsWebSearchProvider(WebSearchProvider):
49
+ """Search Google and fetch pages as Markdown through the Oxylabs Web API."""
50
+
51
+ def __init__(self, settings: SettingReader = _no_settings) -> None:
52
+ self._settings = settings
53
+
54
+ @property
55
+ def name(self) -> str:
56
+ return PROVIDER_NAME
57
+
58
+ @property
59
+ def display_name(self) -> str:
60
+ return "Oxylabs Web API"
61
+
62
+ def is_available(self) -> bool:
63
+ """True when a key is configured. No network call."""
64
+ return bool(get_provider_env(API_KEY_ENV))
65
+
66
+ def supports_search(self) -> bool:
67
+ return True
68
+
69
+ def supports_extract(self) -> bool:
70
+ return True
71
+
72
+ def _location(self) -> tuple[str | None, str | None]:
73
+ """The configured country as ``(code, None)``, or ``(None, error)`` when it is not a country code."""
74
+ value = self._settings("location", "")
75
+ if not value:
76
+ return None, None
77
+ if not isinstance(value, str) or not _COUNTRY_CODE.match(value.strip()):
78
+ return None, f"Oxylabs location setting must be a two-letter country code such as US, not {value!r}."
79
+ return value.strip().upper(), None
80
+
81
+ def search(self, query: str, limit: int = 5) -> dict[str, Any]:
82
+ query = str(query or "").strip()
83
+ if not query:
84
+ return {"success": False, "error": "Search query must not be empty"}
85
+ api_key = get_provider_env(API_KEY_ENV)
86
+ if not api_key:
87
+ return {"success": False, "error": _missing_key_error()}
88
+ location, location_error = self._location()
89
+ if location_error:
90
+ return {"success": False, "error": location_error}
91
+ try:
92
+ count = max(1, min(int(limit), MAX_SEARCH_RESULTS))
93
+ except (TypeError, ValueError):
94
+ count = 5
95
+ try:
96
+ hits = search_web(api_key, query, count, location)
97
+ except OxylabsApiError as exc:
98
+ logger.warning("Oxylabs search failed (HTTP %s)", exc.status)
99
+ return {"success": False, "error": str(exc)}
100
+ logger.info("Oxylabs search returned %d result(s)", len(hits))
101
+ return {"success": True, "data": {"web": hits}}
102
+
103
+ async def extract(self, urls: list[str], **kwargs: Any) -> list[dict[str, Any]]:
104
+ """Fetch each URL as Markdown, in parallel, keeping the input order and per-URL errors.
105
+
106
+ Hermes has already refused private and internal addresses before calling this.
107
+ """
108
+ if not urls:
109
+ return []
110
+ api_key = get_provider_env(API_KEY_ENV)
111
+ if not api_key:
112
+ return [{"url": url, "title": "", "content": "", "error": _missing_key_error()} for url in urls]
113
+ render_js = self._settings("render_js", False) is True
114
+ deadline = time.monotonic() + EXTRACT_DEADLINE_SECONDS
115
+
116
+ async def extract_one(client: httpx.AsyncClient, url: str) -> dict[str, Any]:
117
+ try:
118
+ markdown, status = await scrape_page(client, api_key, url, render_js=render_js, deadline=deadline)
119
+ except OxylabsApiError as exc:
120
+ return {"url": url, "title": "", "content": "", "error": str(exc)}
121
+ if status is not None and not 200 <= status < 300:
122
+ return {"url": url, "title": "", "content": "", "error": f"{url} returned HTTP {status}."}
123
+ if not markdown:
124
+ return {"url": url, "title": "", "content": "", "error": f"No content returned for {url}."}
125
+ title = _title_from_markdown(markdown)
126
+ return {
127
+ "url": url,
128
+ "title": title,
129
+ "content": markdown,
130
+ "raw_content": markdown,
131
+ "metadata": {"sourceURL": url, "title": title, "statusCode": status},
132
+ }
133
+
134
+ async with httpx.AsyncClient() as client:
135
+ return list(await asyncio.gather(*(extract_one(client, url) for url in urls)))
136
+
137
+ def get_setup_schema(self) -> dict[str, Any]:
138
+ return {
139
+ "name": "Oxylabs Web API",
140
+ "badge": "paid",
141
+ "tag": "Google search and page extraction, including JavaScript-heavy and bot-protected pages.",
142
+ "env_vars": [{"key": API_KEY_ENV, "prompt": "Oxylabs Web API key", "url": SIGNUP_URL}],
143
+ }
@@ -0,0 +1,133 @@
1
+ Metadata-Version: 2.4
2
+ Name: hermes-oxylabs-web-api
3
+ Version: 0.1.0
4
+ Summary: Oxylabs Web API search and page extraction provider for Hermes Agent
5
+ Author-email: Oxylabs <giedrius.stankauskas@oxylabs.io>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/oxylabs/hermes-oxylabs-web-api
8
+ Project-URL: Repository, https://github.com/oxylabs/hermes-oxylabs-web-api
9
+ Project-URL: Issues, https://github.com/oxylabs/hermes-oxylabs-web-api/issues
10
+ Keywords: hermes-agent,plugin,oxylabs,web-search,web-scraping,markdown
11
+ Classifier: Development Status :: 4 - Beta
12
+ Classifier: Environment :: Plugins
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Programming Language :: Python :: 3.14
18
+ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
19
+ Requires-Python: <3.16,>=3.11
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: httpx<1,>=0.27
23
+ Dynamic: license-file
24
+
25
+ # Oxylabs Web API for Hermes Agent
26
+
27
+ A [Hermes Agent](https://hermes-agent.nousresearch.com/) plugin that adds the [Oxylabs Web API](https://oxylabs.io/products/web-api) as a provider for Hermes's built-in `web_search` and `web_extract` tools.
28
+
29
+ - **web_search**: Google results with titles, URLs and a short outline of each page. Set a country to see results as users in that country do.
30
+ - **web_extract**: any page as Markdown, including JavaScript-heavy pages (turn on JavaScript rendering) and pages that block plain HTTP clients.
31
+
32
+ The agent keeps using the tools Hermes already has; you choose Oxylabs as the backend.
33
+
34
+ ## Requirements
35
+
36
+ - Hermes Agent 0.23.1 or later.
37
+ - An Oxylabs Web API key. Get one in the [Oxylabs dashboard](https://dashboard.oxylabs.io/).
38
+
39
+ ## Set up
40
+
41
+ 1. Install and enable the plugin:
42
+
43
+ ```bash
44
+ hermes plugins install oxylabs/hermes-oxylabs-web-api --enable
45
+ ```
46
+
47
+ Hermes warns that the plugin is not from its reviewed catalog, and asks before installing its one dependency (`httpx`, which Hermes already uses). In scripts, add `--yes-deps`.
48
+
49
+ 2. Add your key. The install asks for it; paste it there, or skip and add it later. To add it later, run `hermes tools`, open **Web Search & Extract**, select **Oxylabs Web API** and paste the key. Or add it to your profile's `.env` file (`~/.hermes/.env` for the default profile):
50
+
51
+ ```bash
52
+ OXYLABS_WEB_API_KEY=your-web-api-key
53
+ ```
54
+
55
+ 3. Select Oxylabs for search, extraction, or both:
56
+
57
+ ```bash
58
+ hermes config set web.search_backend oxylabs
59
+ hermes config set web.extract_backend oxylabs
60
+ ```
61
+
62
+ Start a new session after these steps.
63
+
64
+ ## Settings
65
+
66
+ Both are optional:
67
+
68
+ ```bash
69
+ hermes config set plugins.entries.oxylabs-web-api.settings.location US
70
+ hermes config set plugins.entries.oxylabs-web-api.settings.render_js true
71
+ ```
72
+
73
+ | Setting | Default | What it does |
74
+ |---|---|---|
75
+ | `location` | none | Two-letter country code for searches, such as `US` or `DE`. |
76
+ | `render_js` | `false` | Load pages in a browser before extracting them. Slower and uses more credits; turn it on if pages come back nearly empty. |
77
+
78
+ The Hermes desktop app shows the same settings on the plugin's page under **Settings → Plugins**.
79
+
80
+ Slow pages are queued on the asynchronous Web API and polled, and each page gets up to 110 seconds, just inside Hermes's 120-second extract timeout.
81
+
82
+ ## Your data
83
+
84
+ - The plugin sends your search queries and the URLs you ask Hermes to read to `https://webapi.oxylabs.io`, together with your API key. It contacts no other service. Requests are billed to your Oxylabs account.
85
+ - Oxylabs runs each search and loads each page, so the sites you read receive Oxylabs's request. Before a page is fetched, Hermes checks the address on your machine and blocks private and internal ones.
86
+ - Requests carry an `x-oxylabs-sdk: oxylabs-web-api-hermes/<version>` header so Oxylabs can count usage per integration. It holds the plugin and Python versions only.
87
+ - The plugin reads the key through Hermes and never includes it in results or error messages. It collects no telemetry, runs no shell commands or background processes, and reads none of your files.
88
+
89
+ ## Update or remove
90
+
91
+ ```bash
92
+ hermes plugins update oxylabs-web-api
93
+ hermes plugins remove oxylabs-web-api
94
+ hermes config unset web.search_backend
95
+ hermes config unset web.extract_backend
96
+ ```
97
+
98
+ ## Troubleshooting
99
+
100
+ | Problem | Fix |
101
+ |---|---|
102
+ | `OXYLABS_WEB_API_KEY is not set` | Add the key as in step 2, then start a new session. |
103
+ | `rejected the API key (401)` | The key is wrong or revoked. Check it in the Oxylabs dashboard. |
104
+ | `quota is used up (429 QUOTA_EXCEEDED)` | Top up the plan in the Oxylabs dashboard. |
105
+ | `web_search` uses another provider | Run the `hermes config set` commands from step 3. Check `hermes plugins list` shows `oxylabs-web-api` as enabled. |
106
+ | A page comes back nearly empty | Turn on `render_js`. |
107
+
108
+ ## Install from PyPI
109
+
110
+ If you run Hermes from a Python environment you manage yourself, install the package into the interpreter that runs Hermes and enable it:
111
+
112
+ ```bash
113
+ uv pip install --python /path/to/hermes/python hermes-oxylabs-web-api
114
+ hermes plugins enable oxylabs-web-api
115
+ ```
116
+
117
+ `hermes plugins update` and `remove` don't manage a PyPI install; upgrade or uninstall it with the same tool.
118
+
119
+ ## Development
120
+
121
+ ```bash
122
+ uv sync --dev
123
+ uv run ruff check . && uv run ruff format --check .
124
+ uv run pytest # unit tests
125
+ OXYLABS_WEB_API_KEY=... uv run pytest -m live # calls the real API
126
+ hermes plugins validate . # the checks the Hermes catalog runs
127
+ ```
128
+
129
+ Releases: bump the version in `pyproject.toml`, both `plugin.yaml` files and `src/hermes_oxylabs_web_api/__init__.py`, add a `CHANGELOG.md` entry, then push a `v<version>` tag. The `Publish` workflow publishes to PyPI with trusted publishing.
130
+
131
+ ## License
132
+
133
+ MIT
@@ -0,0 +1,18 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ src/hermes_oxylabs_web_api/__init__.py
5
+ src/hermes_oxylabs_web_api/client.py
6
+ src/hermes_oxylabs_web_api/plugin.yaml
7
+ src/hermes_oxylabs_web_api/provider.py
8
+ src/hermes_oxylabs_web_api/py.typed
9
+ src/hermes_oxylabs_web_api.egg-info/PKG-INFO
10
+ src/hermes_oxylabs_web_api.egg-info/SOURCES.txt
11
+ src/hermes_oxylabs_web_api.egg-info/dependency_links.txt
12
+ src/hermes_oxylabs_web_api.egg-info/entry_points.txt
13
+ src/hermes_oxylabs_web_api.egg-info/requires.txt
14
+ src/hermes_oxylabs_web_api.egg-info/top_level.txt
15
+ tests/test_client.py
16
+ tests/test_installation.py
17
+ tests/test_live.py
18
+ tests/test_provider.py
@@ -0,0 +1,2 @@
1
+ [hermes_agent.plugins]
2
+ oxylabs-web-api = hermes_oxylabs_web_api
@@ -0,0 +1,50 @@
1
+ from __future__ import annotations
2
+
3
+ import re
4
+ import tomllib
5
+ from pathlib import Path
6
+
7
+ ROOT = Path(__file__).resolve().parents[1]
8
+ MANIFEST = ROOT / "src" / "hermes_oxylabs_web_api" / "plugin.yaml"
9
+
10
+
11
+ def test_versions_match(client_module) -> None:
12
+ from hermes_oxylabs_web_api import __version__
13
+
14
+ pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["version"]
15
+ manifest = re.search(r"^version: (\S+)$", MANIFEST.read_text(), re.M)
16
+ assert manifest is not None
17
+ assert __version__ == pyproject == manifest.group(1)
18
+
19
+
20
+ def test_root_manifest_matches_package_manifest() -> None:
21
+ assert (ROOT / "plugin.yaml").read_text() == MANIFEST.read_text()
22
+
23
+
24
+ def test_sdk_header_matches_the_oxylabs_allowlist(client_module) -> None:
25
+ pattern = r"oxylabs-web-api-hermes/\d+\.\d+\.\d+ \(hermes; python [\d.]+\S*\)"
26
+ assert re.fullmatch(pattern, client_module.sdk_header())
27
+
28
+
29
+ def test_describe_rejected_key(client_module) -> None:
30
+ assert "rejected the API key" in client_module.describe_api_error(401, None)
31
+
32
+
33
+ def test_describe_field_errors_and_request_id(client_module) -> None:
34
+ message = client_module.describe_api_error(
35
+ 400,
36
+ {
37
+ "title": "VALIDATION_ERROR",
38
+ "detail": "1 request field is invalid",
39
+ "errors": [{"pointer": "#/max_results", "detail": "Input should be less than or equal to 20"}],
40
+ "metadata": {"request_id": "123"},
41
+ },
42
+ )
43
+ assert message == (
44
+ "Oxylabs Web API error 400 VALIDATION_ERROR: max_results: Input should be less than or equal to 20"
45
+ " (request_id 123)"
46
+ )
47
+
48
+
49
+ def test_describe_spent_quota(client_module) -> None:
50
+ assert "quota is used up" in client_module.describe_api_error(429, {"title": "QUOTA_EXCEEDED"})
@@ -0,0 +1,68 @@
1
+ from __future__ import annotations
2
+
3
+ import importlib.util
4
+ import shutil
5
+ import sys
6
+ import tomllib
7
+ import types
8
+ from pathlib import Path
9
+
10
+ import pytest
11
+
12
+ ROOT = Path(__file__).resolve().parents[1]
13
+
14
+
15
+ class Context:
16
+ def __init__(self) -> None:
17
+ self.providers = []
18
+
19
+ def register_web_search_provider(self, provider) -> None:
20
+ self.providers.append(provider)
21
+
22
+ def get_config(self, key, default=None):
23
+ return {"location": "US"}.get(key, default)
24
+
25
+
26
+ def test_pip_entry_point_registers_the_provider(fake_hermes) -> None:
27
+ del fake_hermes
28
+ entry_points = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"]["entry-points"]
29
+ assert entry_points["hermes_agent.plugins"] == {"oxylabs-web-api": "hermes_oxylabs_web_api"}
30
+
31
+ import hermes_oxylabs_web_api
32
+
33
+ context = Context()
34
+ hermes_oxylabs_web_api.register(context)
35
+ assert [provider.name for provider in context.providers] == ["oxylabs"]
36
+ assert context.providers[0]._location() == ("US", None)
37
+
38
+
39
+ def test_git_checkout_loads_without_the_pypi_package(
40
+ fake_hermes, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
41
+ ) -> None:
42
+ del fake_hermes
43
+ plugin_dir = tmp_path / "plugins" / "oxylabs-web-api"
44
+ plugin_dir.mkdir(parents=True)
45
+ shutil.copy2(ROOT / "__init__.py", plugin_dir / "__init__.py")
46
+ shutil.copytree(
47
+ ROOT / "src" / "hermes_oxylabs_web_api",
48
+ plugin_dir / "src" / "hermes_oxylabs_web_api",
49
+ ignore=shutil.ignore_patterns("__pycache__"),
50
+ )
51
+ # A Git install must work even when the PyPI module is not importable.
52
+ monkeypatch.setitem(sys.modules, "hermes_oxylabs_web_api", None)
53
+ namespace = types.ModuleType("hermes_plugins")
54
+ namespace.__path__ = []
55
+ monkeypatch.setitem(sys.modules, "hermes_plugins", namespace)
56
+ module_name = "hermes_plugins.oxylabs_web_api"
57
+ spec = importlib.util.spec_from_file_location(
58
+ module_name, plugin_dir / "__init__.py", submodule_search_locations=[str(plugin_dir)]
59
+ )
60
+ assert spec is not None and spec.loader is not None
61
+ module = importlib.util.module_from_spec(spec)
62
+ monkeypatch.setitem(sys.modules, module_name, module)
63
+ spec.loader.exec_module(module)
64
+
65
+ context = Context()
66
+ module.register(context)
67
+ assert [provider.name for provider in context.providers] == ["oxylabs"]
68
+ assert type(context.providers[0]).__module__ == f"{module_name}.src.hermes_oxylabs_web_api.provider"
@@ -0,0 +1,36 @@
1
+ """Calls the real Oxylabs Web API. Run with `pytest -m live`; needs OXYLABS_WEB_API_KEY."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import pytest
6
+ from conftest import LIVE_API_KEY
7
+
8
+ pytestmark = [pytest.mark.live, pytest.mark.skipif(not LIVE_API_KEY, reason="OXYLABS_WEB_API_KEY is not set")]
9
+
10
+
11
+ @pytest.fixture
12
+ def live_provider(fake_hermes, monkeypatch: pytest.MonkeyPatch):
13
+ del fake_hermes
14
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", LIVE_API_KEY)
15
+ from hermes_oxylabs_web_api.provider import OxylabsWebSearchProvider
16
+
17
+ return OxylabsWebSearchProvider(lambda key, default=None: {"location": "DE"}.get(key, default))
18
+
19
+
20
+ def test_live_search(live_provider) -> None:
21
+ result = live_provider.search("oxylabs web api", limit=3)
22
+ assert result["success"] is True, result.get("error")
23
+ assert 0 < len(result["data"]["web"]) <= 3
24
+
25
+
26
+ async def test_live_extract(live_provider) -> None:
27
+ docs = await live_provider.extract(["https://example.com"])
28
+ assert "error" not in docs[0], docs[0].get("error")
29
+ assert "documentation examples" in docs[0]["content"]
30
+
31
+
32
+ def test_live_bad_key(live_provider, monkeypatch: pytest.MonkeyPatch) -> None:
33
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "not-a-real-key")
34
+ result = live_provider.search("rejected key check")
35
+ assert result["success"] is False
36
+ assert "rejected the API key (401)" in result["error"]
@@ -0,0 +1,132 @@
1
+ from __future__ import annotations
2
+
3
+ import json
4
+
5
+ import httpx
6
+ import pytest
7
+ from conftest import json_response, scrape_result
8
+
9
+
10
+ def settings(values: dict):
11
+ return lambda key, default=None: values.get(key, default)
12
+
13
+
14
+ def test_available_only_with_a_key(provider_module, monkeypatch: pytest.MonkeyPatch) -> None:
15
+ provider = provider_module.OxylabsWebSearchProvider()
16
+ assert provider.is_available() is False
17
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "test-key")
18
+ assert provider.is_available() is True
19
+ assert provider.name == "oxylabs"
20
+ assert provider.supports_search() and provider.supports_extract()
21
+
22
+
23
+ def test_search_without_a_key_explains_setup(provider_module) -> None:
24
+ result = provider_module.OxylabsWebSearchProvider().search("q")
25
+ assert result["success"] is False
26
+ assert "OXYLABS_WEB_API_KEY is not set" in result["error"]
27
+
28
+
29
+ def test_search_maps_results_and_sends_tracking_header(
30
+ provider_module, fake_api, monkeypatch: pytest.MonkeyPatch
31
+ ) -> None:
32
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "test-key")
33
+ requests = fake_api(
34
+ lambda request: json_response(
35
+ 200,
36
+ {
37
+ "results": [
38
+ {
39
+ "title": "Example",
40
+ "url": "https://example.com/a",
41
+ "outline": ["one", "two"],
42
+ "metadata": {"position": 1},
43
+ }
44
+ ]
45
+ },
46
+ )
47
+ )
48
+ provider = provider_module.OxylabsWebSearchProvider(settings({"location": "de"}))
49
+ result = provider.search("espresso", limit=50)
50
+
51
+ assert result == {
52
+ "success": True,
53
+ "data": {
54
+ "web": [{"title": "Example", "url": "https://example.com/a", "description": "one … two", "position": 1}]
55
+ },
56
+ }
57
+ request = requests[0]
58
+ assert request.url == "https://webapi.oxylabs.io/v1/search"
59
+ assert json.loads(request.content) == {"query": "espresso", "max_results": 20, "location": "DE"}
60
+ assert request.headers["authorization"] == "Bearer test-key"
61
+ assert request.headers["x-oxylabs-sdk"].startswith("oxylabs-web-api-hermes/")
62
+
63
+
64
+ def test_search_rejects_a_location_that_is_not_a_country_code(
65
+ provider_module, fake_api, monkeypatch: pytest.MonkeyPatch
66
+ ) -> None:
67
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "test-key")
68
+ requests = fake_api(lambda request: json_response(200, {"results": []}))
69
+ result = provider_module.OxylabsWebSearchProvider(settings({"location": "Germany"})).search("q")
70
+ assert result["success"] is False
71
+ assert "two-letter country code" in result["error"]
72
+ assert requests == []
73
+
74
+
75
+ def test_search_reports_api_errors(provider_module, fake_api, monkeypatch: pytest.MonkeyPatch) -> None:
76
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "wrong-key")
77
+ fake_api(lambda request: httpx.Response(401))
78
+ result = provider_module.OxylabsWebSearchProvider().search("q")
79
+ assert result["success"] is False
80
+ assert "rejected the API key (401)" in result["error"]
81
+ assert "wrong-key" not in result["error"]
82
+
83
+
84
+ async def test_extract_returns_documents_in_order(provider_module, fake_api, monkeypatch: pytest.MonkeyPatch) -> None:
85
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "test-key")
86
+
87
+ def handler(request: httpx.Request) -> httpx.Response:
88
+ url = json.loads(request.content)["url"]
89
+ if url.endswith("/missing"):
90
+ return json_response(200, scrape_result("Not found", code=404))
91
+ return json_response(200, scrape_result(f"# Title of {url}\n\nBody"))
92
+
93
+ requests = fake_api(handler)
94
+ provider = provider_module.OxylabsWebSearchProvider(settings({"render_js": True}))
95
+ docs = await provider.extract(["https://example.com/a", "https://example.com/missing"])
96
+
97
+ assert docs[0]["url"] == "https://example.com/a"
98
+ assert docs[0]["title"] == "Title of https://example.com/a"
99
+ assert docs[0]["content"] == docs[0]["raw_content"] == "# Title of https://example.com/a\n\nBody"
100
+ assert docs[0]["metadata"]["statusCode"] == 200
101
+ assert docs[1]["error"] == "https://example.com/missing returned HTTP 404."
102
+ assert all(json.loads(r.content)["run_js"] is True for r in requests)
103
+ assert all(json.loads(r.content)["output"] == ["markdown"] for r in requests)
104
+
105
+
106
+ async def test_extract_falls_back_to_async_when_the_page_is_slow(
107
+ provider_module, client_module, fake_api, monkeypatch: pytest.MonkeyPatch
108
+ ) -> None:
109
+ monkeypatch.setenv("OXYLABS_WEB_API_KEY", "test-key")
110
+ monkeypatch.setattr(client_module, "POLL_INTERVAL_SECONDS", 0)
111
+ polls = []
112
+
113
+ def handler(request: httpx.Request) -> httpx.Response:
114
+ if request.url.path == "/v1/scrape":
115
+ return json_response(504, {"title": "REQUEST_FAILED_SCRAPE_TIMEOUT"})
116
+ if request.url.path == "/v1/async/scrape":
117
+ return json_response(202, {"metadata": {"request_id": "abc"}})
118
+ polls.append(request.url.path)
119
+ if len(polls) == 1:
120
+ return json_response(200, {"state": "pending"})
121
+ return json_response(200, scrape_result("# Slow page"))
122
+
123
+ fake_api(handler)
124
+ docs = await provider_module.OxylabsWebSearchProvider().extract(["https://example.com/slow"])
125
+ assert docs[0]["content"] == "# Slow page"
126
+ assert polls == ["/v1/async/scrape/abc", "/v1/async/scrape/abc"]
127
+
128
+
129
+ async def test_extract_without_a_key_fails_each_url(provider_module) -> None:
130
+ docs = await provider_module.OxylabsWebSearchProvider().extract(["https://example.com"])
131
+ assert docs[0]["url"] == "https://example.com"
132
+ assert "OXYLABS_WEB_API_KEY is not set" in docs[0]["error"]