webget-cli 0.12.1__tar.gz → 0.15.0__tar.gz

{webget_cli-0.12.1 → webget_cli-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webget-cli
-Version: 0.12.1
+Version: 0.15.0
 Summary: Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys.
 Author-email: David Tarigan <tarigansdavid@gmail.com>
 License: Apache-2.0
@@ -27,10 +27,12 @@ Requires-Dist: pypdf
 Provides-Extra: browser
 Requires-Dist: crawl4ai>=0.9; extra == "browser"
 Provides-Extra: mcp
-Requires-Dist: fastmcp>=2; extra == "mcp"
+Requires-Dist: fastmcp>=4; extra == "mcp"
 Provides-Extra: dev
 Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: pytest-timeout>=2; extra == "dev"
+Requires-Dist: pytest-socket>=0.7; extra == "dev"
 Dynamic: license-file
 
 <div align="center">
@@ -101,7 +103,7 @@ webget --help
 
 ### Browser runtime (optional)
 
-Crawl4AI drives a Playwright Chromium. `pip install "webget-cli[browser]"`
+Crawl4AI drives a Playwright browser. `pip install "webget-cli[browser]"`
 installs the Python packages; the browser binary itself is downloaded
 separately:
 
@@ -109,6 +111,41 @@ separately:
 python -m playwright install chromium
 ```
 
+**You may not need that download.** Before reaching for Playwright's bundled
+Chromium, webget looks for a Chromium-family browser already installed on the
+machine and uses it when it can actually be driven. Run `webget doctor` to see
+what was found and what will be used:
+
+```bash
+webget doctor
+```
+
+Resolution order (first match wins):
+
+| Step | Source | Notes |
+|---|---|---|
+| 1 | `WEBGET_BROWSER_CDP` | Attach to a running browser over CDP. The only route that works for browsers with no Playwright channel. |
+| 2 | `WEBGET_BROWSER_CHANNEL` | A Playwright channel: `chrome`, `msedge`, or `chromium`. |
+| 3 | `WEBGET_BROWSER_PATH` | An explicit binary. Accepted, but see the limitation below. |
+| 4 | Auto-detected browser | Chrome, Edge, Chromium, Brave, Vivaldi or Opera found on the machine. |
+| 5 | Playwright bundled Chromium | The fallback when nothing usable is found. |
+
+Two limitations worth knowing, both reported honestly by `webget doctor`
+instead of failing quietly:
+
+- **Brave, Vivaldi and Opera cannot be launched by crawl4ai.** Playwright has no
+  channel for them, and crawl4ai 0.9.2 does not forward `executable_path` to
+  Playwright, so there is no path to the binary. They are detected and reported
+  as NOT USABLE. The workaround is CDP: start the browser with
+  `--remote-debugging-port=9222`, then set
+  `WEBGET_BROWSER_CDP=http://127.0.0.1:9222`.
+- **CDP is opt-in on purpose.** Attaching to a browser you are already logged
+  into mixes your personal session cookies into crawl output, so webget never
+  auto-detects an open debugging port.
+
+System Firefox is never selected: Playwright drives Firefox over WebDriver BiDi
+with its own patched build.
+
 Without the browser extra, `webget` still works for search and plain HTTP
 fetches. A fetch that needs the browser (JS rendering, `--profile` sessions,
 `login`) prints a clear warning telling you how to install it.
@@ -124,7 +161,9 @@ uv pip install -e ".[dev,browser]"
 ## Usage
 
 ```bash
-webget s "rust async runtime"             # search DuckDuckGo (top 5)
+webget s "rust async runtime"             # search (ddgs metasearch, top 5)
+webget s "rust async runtime" --engine brave,duckduckgo   # subset: faster than auto
+webget s "rust async runtime" --json      # machine-readable, includes engine provenance
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
@@ -133,6 +172,86 @@ webget fetch https://example.com --json   # machine-readable result
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+## Search engines
+
+webget searches through `ddgs`, a metasearch library that aggregates several
+keyless engines. The default `auto` queries all of them; naming a subset is
+faster and skips engines that are having a bad hour.
+
+```bash
+webget s "query" --engine brave,duckduckgo
+```
+
+**Do not hardcode engine names.** The set ddgs offers changes between its
+releases, and webget validates against the registry at runtime rather than
+a built-in list. Pass a deliberately bogus name to see what your installed
+version supports:
+
+```bash
+$ webget s "query" --engine bogus
+warning: unknown search engine(s): bogus - using auto
+  (known: brave, duckduckgo, google, grokipedia, mojeek, startpage, wikipedia, yahoo)
+```
+
+Unknown names degrade to `auto` with a warning instead of failing, so a
+typo never kills a search.
+
+**Failover is automatic.** If the engine you named fails, or returns zero
+results, webget tries the remaining engines until a time budget is spent
+(15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
+first non-empty set. The budget replaced a fixed "3 engines" cap: a count is
+either too many (3 x 20s timeout = a minute of dead air) or too few (a live
+engine sitting fifth in line never got reached). At least two alternates are
+always tried, however fast the budget expires. This is not paranoia: engine
+reachability depends on where you are, not just whether a service is up. A
+benchmark from one residential connection found only 2 of 9 engines
+reachable, and `auto` survived purely because they did.
+
+Every substitution is announced, never silent:
+
+```bash
+$ webget s "rust programming language book" --engine google
+warning: engine 'google' failed (No results found.); fell back to 'brave'
+1. The Rust Programming Language - doc.rust-lang.org/book/
+```
+
+**Results carry provenance.** Because failover can answer from an engine you
+did not ask for, `--json` reports which one actually did:
+
+```bash
+$ webget s "linux kernel" --engine google --json
+{
+  "results": [ ... ],
+  "engine": "grokipedia",         # who actually answered
+  "requested_engine": "google",
+  "failed_over": true
+}
+```
+
+This matters most over MCP, where a stderr warning is invisible to the
+agent: the payload is the only signal that the results came from elsewhere.
+
+**Failover order is learned, not hardcoded.** webget keeps a small health
+ledger per install (`~/.local/state/webget/engine_health.json`) recording
+whether each engine answered and how fast. When failover kicks in, engines
+are tried best-first by that record instead of in registry order. The ledger
+is advisory - it reorders candidates, it never removes one - and it decays:
+an engine marked dead last week is retried with no penalty this week, because
+the blocking that made it dead is exactly what changes. There is no baked-in
+ranking anywhere in the source; the same webget learns a different order on
+your machine than on the author's, which is the point.
+
+Diagnostics: `python -c "from webget import health; print(health.health_snapshot())"`
+
+> **MCP note (breaking in 0.13.0):** the `search` and `search_fetch` tools
+> now return an object `{results, engine, requested_engine, failed_over}`
+> instead of a bare list, so provenance has somewhere to live. Errors are
+> `{error, results: []}`.
+
+Provenance is per-call, not per-result: `ddgs` merges every engine's hits
+into one list and discards which engine supplied each hit, so per-hit
+attribution is not obtainable through its public API.
+
 ### Options
 
 | Flag | Meaning |
@@ -260,6 +379,51 @@ Register as a local MCP server in opencode:
 Then prompt with `use webget` for search and scrape tasks. Run the server
 standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 
+### Breaking change in 0.13.0: search returns an object, not a list
+
+`search` and `search_fetch` used to return a bare JSON array of results.
+They now return an object, because a list has nowhere to carry provenance:
+
+```jsonc
+// <= 0.12.1  ->  a list
+[ { "title": "...", "href": "...", "body": "..." }, ... ]
+
+// >= 0.13.0  ->  an object
+{
+  "results": [ { "title": "...", "href": "...", "body": "..." }, ... ],
+  "engine": "brave",            // the engine that actually answered
+  "requested_engine": "google", // what was asked for
+  "failed_over": true           // an alternative was attempted
+}
+```
+
+Error responses changed shape too:
+
+```jsonc
+// <= 0.12.1  ->  a plain error string / empty list
+// >= 0.13.0
+{
+  "error": "SearchError: RequestError(...)",
+  "results": []
+}
+```
+
+**How to migrate.** If you read the tool result as an array, read
+`.results` instead. In JavaScript that is `result.results` rather than
+`result`; in Python `data["results"]` rather than `data`. If you only
+iterate the hits, the change is mechanical.
+
+**Why it had to break.** Failover means the engine that answers is not
+always the engine that was asked for, and over MCP there is no stderr for a
+warning to land on, so the payload is the only place that fact can live.
+Keeping the list shape would have meant silently returning results from an
+unexpected engine with no way for a client to notice. An additive field was
+not possible: a JSON array cannot carry sibling keys.
+
+`fetch` is unchanged and still returns its string payload. The Python API is
+unchanged: `search()` still returns a list, and `search_with_provenance()`
+is the new opt-in that returns `(results, provenance)`.
+
 ### Authenticated sessions (profiles)
 
 MCP tools can use locally stored login sessions. Create one first with the

{webget_cli-0.12.1 → webget_cli-0.15.0}/README.md

@@ -66,7 +66,7 @@ webget --help
 
 ### Browser runtime (optional)
 
-Crawl4AI drives a Playwright Chromium. `pip install "webget-cli[browser]"`
+Crawl4AI drives a Playwright browser. `pip install "webget-cli[browser]"`
 installs the Python packages; the browser binary itself is downloaded
 separately:
 
@@ -74,6 +74,41 @@ separately:
 python -m playwright install chromium
 ```
 
+**You may not need that download.** Before reaching for Playwright's bundled
+Chromium, webget looks for a Chromium-family browser already installed on the
+machine and uses it when it can actually be driven. Run `webget doctor` to see
+what was found and what will be used:
+
+```bash
+webget doctor
+```
+
+Resolution order (first match wins):
+
+| Step | Source | Notes |
+|---|---|---|
+| 1 | `WEBGET_BROWSER_CDP` | Attach to a running browser over CDP. The only route that works for browsers with no Playwright channel. |
+| 2 | `WEBGET_BROWSER_CHANNEL` | A Playwright channel: `chrome`, `msedge`, or `chromium`. |
+| 3 | `WEBGET_BROWSER_PATH` | An explicit binary. Accepted, but see the limitation below. |
+| 4 | Auto-detected browser | Chrome, Edge, Chromium, Brave, Vivaldi or Opera found on the machine. |
+| 5 | Playwright bundled Chromium | The fallback when nothing usable is found. |
+
+Two limitations worth knowing, both reported honestly by `webget doctor`
+instead of failing quietly:
+
+- **Brave, Vivaldi and Opera cannot be launched by crawl4ai.** Playwright has no
+  channel for them, and crawl4ai 0.9.2 does not forward `executable_path` to
+  Playwright, so there is no path to the binary. They are detected and reported
+  as NOT USABLE. The workaround is CDP: start the browser with
+  `--remote-debugging-port=9222`, then set
+  `WEBGET_BROWSER_CDP=http://127.0.0.1:9222`.
+- **CDP is opt-in on purpose.** Attaching to a browser you are already logged
+  into mixes your personal session cookies into crawl output, so webget never
+  auto-detects an open debugging port.
+
+System Firefox is never selected: Playwright drives Firefox over WebDriver BiDi
+with its own patched build.
+
 Without the browser extra, `webget` still works for search and plain HTTP
 fetches. A fetch that needs the browser (JS rendering, `--profile` sessions,
 `login`) prints a clear warning telling you how to install it.
@@ -89,7 +124,9 @@ uv pip install -e ".[dev,browser]"
 ## Usage
 
 ```bash
-webget s "rust async runtime"             # search DuckDuckGo (top 5)
+webget s "rust async runtime"             # search (ddgs metasearch, top 5)
+webget s "rust async runtime" --engine brave,duckduckgo   # subset: faster than auto
+webget s "rust async runtime" --json      # machine-readable, includes engine provenance
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
@@ -98,6 +135,86 @@ webget fetch https://example.com --json   # machine-readable result
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+## Search engines
+
+webget searches through `ddgs`, a metasearch library that aggregates several
+keyless engines. The default `auto` queries all of them; naming a subset is
+faster and skips engines that are having a bad hour.
+
+```bash
+webget s "query" --engine brave,duckduckgo
+```
+
+**Do not hardcode engine names.** The set ddgs offers changes between its
+releases, and webget validates against the registry at runtime rather than
+a built-in list. Pass a deliberately bogus name to see what your installed
+version supports:
+
+```bash
+$ webget s "query" --engine bogus
+warning: unknown search engine(s): bogus - using auto
+  (known: brave, duckduckgo, google, grokipedia, mojeek, startpage, wikipedia, yahoo)
+```
+
+Unknown names degrade to `auto` with a warning instead of failing, so a
+typo never kills a search.
+
+**Failover is automatic.** If the engine you named fails, or returns zero
+results, webget tries the remaining engines until a time budget is spent
+(15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
+first non-empty set. The budget replaced a fixed "3 engines" cap: a count is
+either too many (3 x 20s timeout = a minute of dead air) or too few (a live
+engine sitting fifth in line never got reached). At least two alternates are
+always tried, however fast the budget expires. This is not paranoia: engine
+reachability depends on where you are, not just whether a service is up. A
+benchmark from one residential connection found only 2 of 9 engines
+reachable, and `auto` survived purely because they did.
+
+Every substitution is announced, never silent:
+
+```bash
+$ webget s "rust programming language book" --engine google
+warning: engine 'google' failed (No results found.); fell back to 'brave'
+1. The Rust Programming Language - doc.rust-lang.org/book/
+```
+
+**Results carry provenance.** Because failover can answer from an engine you
+did not ask for, `--json` reports which one actually did:
+
+```bash
+$ webget s "linux kernel" --engine google --json
+{
+  "results": [ ... ],
+  "engine": "grokipedia",         # who actually answered
+  "requested_engine": "google",
+  "failed_over": true
+}
+```
+
+This matters most over MCP, where a stderr warning is invisible to the
+agent: the payload is the only signal that the results came from elsewhere.
+
+**Failover order is learned, not hardcoded.** webget keeps a small health
+ledger per install (`~/.local/state/webget/engine_health.json`) recording
+whether each engine answered and how fast. When failover kicks in, engines
+are tried best-first by that record instead of in registry order. The ledger
+is advisory - it reorders candidates, it never removes one - and it decays:
+an engine marked dead last week is retried with no penalty this week, because
+the blocking that made it dead is exactly what changes. There is no baked-in
+ranking anywhere in the source; the same webget learns a different order on
+your machine than on the author's, which is the point.
+
+Diagnostics: `python -c "from webget import health; print(health.health_snapshot())"`
+
+> **MCP note (breaking in 0.13.0):** the `search` and `search_fetch` tools
+> now return an object `{results, engine, requested_engine, failed_over}`
+> instead of a bare list, so provenance has somewhere to live. Errors are
+> `{error, results: []}`.
+
+Provenance is per-call, not per-result: `ddgs` merges every engine's hits
+into one list and discards which engine supplied each hit, so per-hit
+attribution is not obtainable through its public API.
+
 ### Options
 
 | Flag | Meaning |
@@ -225,6 +342,51 @@ Register as a local MCP server in opencode:
 Then prompt with `use webget` for search and scrape tasks. Run the server
 standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 
+### Breaking change in 0.13.0: search returns an object, not a list
+
+`search` and `search_fetch` used to return a bare JSON array of results.
+They now return an object, because a list has nowhere to carry provenance:
+
+```jsonc
+// <= 0.12.1  ->  a list
+[ { "title": "...", "href": "...", "body": "..." }, ... ]
+
+// >= 0.13.0  ->  an object
+{
+  "results": [ { "title": "...", "href": "...", "body": "..." }, ... ],
+  "engine": "brave",            // the engine that actually answered
+  "requested_engine": "google", // what was asked for
+  "failed_over": true           // an alternative was attempted
+}
+```
+
+Error responses changed shape too:
+
+```jsonc
+// <= 0.12.1  ->  a plain error string / empty list
+// >= 0.13.0
+{
+  "error": "SearchError: RequestError(...)",
+  "results": []
+}
+```
+
+**How to migrate.** If you read the tool result as an array, read
+`.results` instead. In JavaScript that is `result.results` rather than
+`result`; in Python `data["results"]` rather than `data`. If you only
+iterate the hits, the change is mechanical.
+
+**Why it had to break.** Failover means the engine that answers is not
+always the engine that was asked for, and over MCP there is no stderr for a
+warning to land on, so the payload is the only place that fact can live.
+Keeping the list shape would have meant silently returning results from an
+unexpected engine with no way for a client to notice. An additive field was
+not possible: a JSON array cannot carry sibling keys.
+
+`fetch` is unchanged and still returns its string payload. The Python API is
+unchanged: `search()` still returns a list, and `search_with_provenance()`
+is the new opt-in that returns `(results, provenance)`.
+
 ### Authenticated sessions (profiles)
 
 MCP tools can use locally stored login sessions. Create one first with the

{webget_cli-0.12.1 → webget_cli-0.15.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "webget-cli"
-version = "0.12.1"
+version = "0.15.0"
 description = "Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -36,8 +36,14 @@ Changelog = "https://github.com/DavidPandleton/webget/blob/main/CHANGELOG.md"
 
 [project.optional-dependencies]
 browser = ["crawl4ai>=0.9"]
-mcp = ["fastmcp>=2"]
-dev = ["pytest>=8", "ruff>=0.6"]
+# fastmcp pulls in the `mcp` SDK. The MCP SDK renamed CallToolResult.isError to
+# is_error in the 2.x line, and fastmcp 4.x ships that newer SDK. The code and
+# tests here use is_error, so the floor is the version that has it.
+# Without a floor, an install could resolve to a fastmcp whose result objects
+# expose isError only, and the MCP surface would look broken for a reason that
+# has nothing to do with webget.
+mcp = ["fastmcp>=4"]
+dev = ["pytest>=8", "ruff>=0.6", "pytest-timeout>=2", "pytest-socket>=0.7"]
 
 [project.scripts]
 webget = "webget.cli:main"
@@ -50,7 +56,22 @@ packages = ["webget"]
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 addopts = "-q"
+markers = [
+  # Tests that reach the PUBLIC internet (not the local test server). They are
+  # excluded from offline jobs with -m "not live_network" and run separately,
+  # because engine reachability is a property of the runner's network: during
+  # 0.13.0, DuckDuckGo was unreachable from the author's ISP while Google,
+  # Brave and Yandex answered in under a second. A red test here does not mean
+  # the release is broken.
+  "live_network: test reaches the public internet; excluded from offline CI",
+]
 
 [tool.ruff]
 line-length = 100
 target-version = "py311"
+
+[tool.ruff.lint.per-file-ignores]
+# Ops scripts drive subprocesses and inspect returncode manually; requiring
+# check=True everywhere would force try/except noise where a plain if already
+# handles the failure.
+"scripts/*" = ["PLW1510"]

{webget_cli-0.12.1 → webget_cli-0.15.0}/tests/test_adversarial_http.py

@@ -166,9 +166,14 @@ class TestResponseBodies:
         assert _one(res)["status"] == "success"  # httpx auto-decompresses
 
     def test_huge_response_is_bounded(self, fresh_cache):
+        from webget.truncate import ELLIPSIS_MARKER
+
         server = fresh_cache
         # 5MB body; scrape_many must truncate, not blow memory or hang.
         res = asyncio.run(_fetch(server.url("/huge"), max_chars=1000))
         out = _one(res)
         assert out["status"] == "success"
-        assert len(out["markdown"]) <= 1000
+        # smart_truncate cuts at the limit and appends the ellipsis marker
+        # (this body has no clean boundary, so it is a hard cut at the limit).
+        assert len(out["markdown"]) <= 1000 + len(ELLIPSIS_MARKER)
+        assert out["markdown"].endswith(ELLIPSIS_MARKER)

{webget_cli-0.12.1 → webget_cli-0.15.0}/tests/test_adversarial_mcp.py

@@ -10,6 +10,8 @@ import pytest
 from mcp import ClientSession, StdioServerParameters
 from mcp.client.stdio import stdio_client
 
+from tests.conftest import tool_failed
+
 ROOT = Path(__file__).resolve().parent.parent
 MCP = ROOT / "webget_mcp.py"
 
@@ -58,7 +60,7 @@ class TestMalformedArguments:
                 return res
 
         res = _run(run())
-        assert res.isError
+        assert tool_failed(res)
 
 
 class TestInvalidURLs:
@@ -72,7 +74,7 @@ class TestInvalidURLs:
                 return res
 
         res = _run(run())
-        assert "error" in res.content[0].text or res.isError
+        assert "error" in res.content[0].text or tool_failed(res)
 
 
 class TestRepeatedCalls:
@@ -86,7 +88,7 @@ class TestRepeatedCalls:
                         "fetch",
                         {"url": "https://example.com", "strategy": "http", "no_cache": True},
                     )
-                    if res.isError:
+                    if tool_failed(res):
                         return "ERROR"
                 return "OK"
 
@@ -106,7 +108,7 @@ class TestRepeatedCalls:
                         for _ in range(5)
                     ]
                 )
-                return [r.isError for r in results]
+                return [tool_failed(r) for r in results]
 
         assert _run(run()) == [False] * 5
 
@@ -133,7 +135,7 @@ class TestInputCaps:
                 return res
 
         res = _run(run())
-        assert "must be between" in res.content[0].text or res.isError
+        assert "must be between" in res.content[0].text or tool_failed(res)
 
 
 class TestToolFailureIsolation:
@@ -171,8 +173,18 @@ class TestSSRFViaMCP:
         res = _run(run())
         assert "private" in res.content[0].text.lower()
 
+    @pytest.mark.live_network
     def test_search_output_shape(self):
-        """search must return list of dicts with title/url/snippet."""
+        """search must return {results: [...], engine: ...} with provenance.
+
+        Marked live_network deliberately: this is the ONLY test in the suite
+        that talks to the public internet (verified by running the whole MCP
+        suite with sockets blocked except localhost - everything passes except
+        this one, which fails exactly because the network is unavailable). It
+        exercises the real engine path end-to-end through a real server
+        process, which no mock can replace, so it stays but is excluded from
+        the offline jobs.
+        """
 
         async def run():
             params = StdioServerParameters(command=sys.executable, args=[str(MCP)])
@@ -183,6 +195,11 @@ class TestSSRFViaMCP:
 
         res = _run(run())
         payload = json.loads(res.content[0].text)
-        assert isinstance(payload, list)
-        if payload:
-            assert {"title", "url", "snippet"} <= set(payload[0])
+        assert isinstance(payload, dict)
+        # Provenance keys are part of the contract: `engine` is the one that
+        # actually answered, which can differ from `requested_engine`.
+        assert {"results", "engine", "requested_engine", "failed_over"} <= set(payload)
+        results = payload["results"]
+        assert isinstance(results, list)
+        if results:
+            assert {"title", "url", "snippet"} <= set(results[0])