webget-cli 0.12.1__tar.gz → 0.14.0__tar.gz

{webget_cli-0.12.1 → webget_cli-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webget-cli
-Version: 0.12.1
+Version: 0.14.0
 Summary: Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys.
 Author-email: David Tarigan <tarigansdavid@gmail.com>
 License: Apache-2.0
@@ -27,10 +27,12 @@ Requires-Dist: pypdf
 Provides-Extra: browser
 Requires-Dist: crawl4ai>=0.9; extra == "browser"
 Provides-Extra: mcp
-Requires-Dist: fastmcp>=2; extra == "mcp"
+Requires-Dist: fastmcp>=4; extra == "mcp"
 Provides-Extra: dev
 Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: pytest-timeout>=2; extra == "dev"
+Requires-Dist: pytest-socket>=0.7; extra == "dev"
 Dynamic: license-file
 
 <div align="center">
@@ -124,7 +126,9 @@ uv pip install -e ".[dev,browser]"
 ## Usage
 
 ```bash
-webget s "rust async runtime"             # search DuckDuckGo (top 5)
+webget s "rust async runtime"             # search (ddgs metasearch, top 5)
+webget s "rust async runtime" --engine brave,duckduckgo   # subset: faster than auto
+webget s "rust async runtime" --json      # machine-readable, includes engine provenance
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
@@ -133,6 +137,86 @@ webget fetch https://example.com --json   # machine-readable result
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+## Search engines
+
+webget searches through `ddgs`, a metasearch library that aggregates several
+keyless engines. The default `auto` queries all of them; naming a subset is
+faster and skips engines that are having a bad hour.
+
+```bash
+webget s "query" --engine brave,duckduckgo
+```
+
+**Do not hardcode engine names.** The set ddgs offers changes between its
+releases, and webget validates against the registry at runtime rather than
+a built-in list. Pass a deliberately bogus name to see what your installed
+version supports:
+
+```bash
+$ webget s "query" --engine bogus
+warning: unknown search engine(s): bogus - using auto
+  (known: brave, duckduckgo, google, grokipedia, mojeek, startpage, wikipedia, yahoo)
+```
+
+Unknown names degrade to `auto` with a warning instead of failing, so a
+typo never kills a search.
+
+**Failover is automatic.** If the engine you named fails, or returns zero
+results, webget tries the remaining engines until a time budget is spent
+(15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
+first non-empty set. The budget replaced a fixed "3 engines" cap: a count is
+either too many (3 x 20s timeout = a minute of dead air) or too few (a live
+engine sitting fifth in line never got reached). At least two alternates are
+always tried, however fast the budget expires. This is not paranoia: engine
+reachability depends on where you are, not just whether a service is up. A
+benchmark from one residential connection found only 2 of 9 engines
+reachable, and `auto` survived purely because they did.
+
+Every substitution is announced, never silent:
+
+```bash
+$ webget s "rust programming language book" --engine google
+warning: engine 'google' failed (No results found.); fell back to 'brave'
+1. The Rust Programming Language - doc.rust-lang.org/book/
+```
+
+**Results carry provenance.** Because failover can answer from an engine you
+did not ask for, `--json` reports which one actually did:
+
+```bash
+$ webget s "linux kernel" --engine google --json
+{
+  "results": [ ... ],
+  "engine": "grokipedia",         # who actually answered
+  "requested_engine": "google",
+  "failed_over": true
+}
+```
+
+This matters most over MCP, where a stderr warning is invisible to the
+agent: the payload is the only signal that the results came from elsewhere.
+
+**Failover order is learned, not hardcoded.** webget keeps a small health
+ledger per install (`~/.local/state/webget/engine_health.json`) recording
+whether each engine answered and how fast. When failover kicks in, engines
+are tried best-first by that record instead of in registry order. The ledger
+is advisory - it reorders candidates, it never removes one - and it decays:
+an engine marked dead last week is retried with no penalty this week, because
+the blocking that made it dead is exactly what changes. There is no baked-in
+ranking anywhere in the source; the same webget learns a different order on
+your machine than on the author's, which is the point.
+
+Diagnostics: `python -c "from webget import health; print(health.health_snapshot())"`
+
+> **MCP note (breaking in 0.13.0):** the `search` and `search_fetch` tools
+> now return an object `{results, engine, requested_engine, failed_over}`
+> instead of a bare list, so provenance has somewhere to live. Errors are
+> `{error, results: []}`.
+
+Provenance is per-call, not per-result: `ddgs` merges every engine's hits
+into one list and discards which engine supplied each hit, so per-hit
+attribution is not obtainable through its public API.
+
 ### Options
 
 | Flag | Meaning |
@@ -260,6 +344,51 @@ Register as a local MCP server in opencode:
 Then prompt with `use webget` for search and scrape tasks. Run the server
 standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 
+### Breaking change in 0.13.0: search returns an object, not a list
+
+`search` and `search_fetch` used to return a bare JSON array of results.
+They now return an object, because a list has nowhere to carry provenance:
+
+```jsonc
+// <= 0.12.1  ->  a list
+[ { "title": "...", "href": "...", "body": "..." }, ... ]
+
+// >= 0.13.0  ->  an object
+{
+  "results": [ { "title": "...", "href": "...", "body": "..." }, ... ],
+  "engine": "brave",            // the engine that actually answered
+  "requested_engine": "google", // what was asked for
+  "failed_over": true           // an alternative was attempted
+}
+```
+
+Error responses changed shape too:
+
+```jsonc
+// <= 0.12.1  ->  a plain error string / empty list
+// >= 0.13.0
+{
+  "error": "SearchError: RequestError(...)",
+  "results": []
+}
+```
+
+**How to migrate.** If you read the tool result as an array, read
+`.results` instead. In JavaScript that is `result.results` rather than
+`result`; in Python `data["results"]` rather than `data`. If you only
+iterate the hits, the change is mechanical.
+
+**Why it had to break.** Failover means the engine that answers is not
+always the engine that was asked for, and over MCP there is no stderr for a
+warning to land on, so the payload is the only place that fact can live.
+Keeping the list shape would have meant silently returning results from an
+unexpected engine with no way for a client to notice. An additive field was
+not possible: a JSON array cannot carry sibling keys.
+
+`fetch` is unchanged and still returns its string payload. The Python API is
+unchanged: `search()` still returns a list, and `search_with_provenance()`
+is the new opt-in that returns `(results, provenance)`.
+
 ### Authenticated sessions (profiles)
 
 MCP tools can use locally stored login sessions. Create one first with the

{webget_cli-0.12.1 → webget_cli-0.14.0}/README.md

@@ -89,7 +89,9 @@ uv pip install -e ".[dev,browser]"
 ## Usage
 
 ```bash
-webget s "rust async runtime"             # search DuckDuckGo (top 5)
+webget s "rust async runtime"             # search (ddgs metasearch, top 5)
+webget s "rust async runtime" --engine brave,duckduckgo   # subset: faster than auto
+webget s "rust async runtime" --json      # machine-readable, includes engine provenance
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
@@ -98,6 +100,86 @@ webget fetch https://example.com --json   # machine-readable result
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+## Search engines
+
+webget searches through `ddgs`, a metasearch library that aggregates several
+keyless engines. The default `auto` queries all of them; naming a subset is
+faster and skips engines that are having a bad hour.
+
+```bash
+webget s "query" --engine brave,duckduckgo
+```
+
+**Do not hardcode engine names.** The set ddgs offers changes between its
+releases, and webget validates against the registry at runtime rather than
+a built-in list. Pass a deliberately bogus name to see what your installed
+version supports:
+
+```bash
+$ webget s "query" --engine bogus
+warning: unknown search engine(s): bogus - using auto
+  (known: brave, duckduckgo, google, grokipedia, mojeek, startpage, wikipedia, yahoo)
+```
+
+Unknown names degrade to `auto` with a warning instead of failing, so a
+typo never kills a search.
+
+**Failover is automatic.** If the engine you named fails, or returns zero
+results, webget tries the remaining engines until a time budget is spent
+(15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
+first non-empty set. The budget replaced a fixed "3 engines" cap: a count is
+either too many (3 x 20s timeout = a minute of dead air) or too few (a live
+engine sitting fifth in line never got reached). At least two alternates are
+always tried, however fast the budget expires. This is not paranoia: engine
+reachability depends on where you are, not just whether a service is up. A
+benchmark from one residential connection found only 2 of 9 engines
+reachable, and `auto` survived purely because they did.
+
+Every substitution is announced, never silent:
+
+```bash
+$ webget s "rust programming language book" --engine google
+warning: engine 'google' failed (No results found.); fell back to 'brave'
+1. The Rust Programming Language - doc.rust-lang.org/book/
+```
+
+**Results carry provenance.** Because failover can answer from an engine you
+did not ask for, `--json` reports which one actually did:
+
+```bash
+$ webget s "linux kernel" --engine google --json
+{
+  "results": [ ... ],
+  "engine": "grokipedia",         # who actually answered
+  "requested_engine": "google",
+  "failed_over": true
+}
+```
+
+This matters most over MCP, where a stderr warning is invisible to the
+agent: the payload is the only signal that the results came from elsewhere.
+
+**Failover order is learned, not hardcoded.** webget keeps a small health
+ledger per install (`~/.local/state/webget/engine_health.json`) recording
+whether each engine answered and how fast. When failover kicks in, engines
+are tried best-first by that record instead of in registry order. The ledger
+is advisory - it reorders candidates, it never removes one - and it decays:
+an engine marked dead last week is retried with no penalty this week, because
+the blocking that made it dead is exactly what changes. There is no baked-in
+ranking anywhere in the source; the same webget learns a different order on
+your machine than on the author's, which is the point.
+
+Diagnostics: `python -c "from webget import health; print(health.health_snapshot())"`
+
+> **MCP note (breaking in 0.13.0):** the `search` and `search_fetch` tools
+> now return an object `{results, engine, requested_engine, failed_over}`
+> instead of a bare list, so provenance has somewhere to live. Errors are
+> `{error, results: []}`.
+
+Provenance is per-call, not per-result: `ddgs` merges every engine's hits
+into one list and discards which engine supplied each hit, so per-hit
+attribution is not obtainable through its public API.
+
 ### Options
 
 | Flag | Meaning |
@@ -225,6 +307,51 @@ Register as a local MCP server in opencode:
 Then prompt with `use webget` for search and scrape tasks. Run the server
 standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 
+### Breaking change in 0.13.0: search returns an object, not a list
+
+`search` and `search_fetch` used to return a bare JSON array of results.
+They now return an object, because a list has nowhere to carry provenance:
+
+```jsonc
+// <= 0.12.1  ->  a list
+[ { "title": "...", "href": "...", "body": "..." }, ... ]
+
+// >= 0.13.0  ->  an object
+{
+  "results": [ { "title": "...", "href": "...", "body": "..." }, ... ],
+  "engine": "brave",            // the engine that actually answered
+  "requested_engine": "google", // what was asked for
+  "failed_over": true           // an alternative was attempted
+}
+```
+
+Error responses changed shape too:
+
+```jsonc
+// <= 0.12.1  ->  a plain error string / empty list
+// >= 0.13.0
+{
+  "error": "SearchError: RequestError(...)",
+  "results": []
+}
+```
+
+**How to migrate.** If you read the tool result as an array, read
+`.results` instead. In JavaScript that is `result.results` rather than
+`result`; in Python `data["results"]` rather than `data`. If you only
+iterate the hits, the change is mechanical.
+
+**Why it had to break.** Failover means the engine that answers is not
+always the engine that was asked for, and over MCP there is no stderr for a
+warning to land on, so the payload is the only place that fact can live.
+Keeping the list shape would have meant silently returning results from an
+unexpected engine with no way for a client to notice. An additive field was
+not possible: a JSON array cannot carry sibling keys.
+
+`fetch` is unchanged and still returns its string payload. The Python API is
+unchanged: `search()` still returns a list, and `search_with_provenance()`
+is the new opt-in that returns `(results, provenance)`.
+
 ### Authenticated sessions (profiles)
 
 MCP tools can use locally stored login sessions. Create one first with the

{webget_cli-0.12.1 → webget_cli-0.14.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "webget-cli"
-version = "0.12.1"
+version = "0.14.0"
 description = "Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -36,8 +36,14 @@ Changelog = "https://github.com/DavidPandleton/webget/blob/main/CHANGELOG.md"
 
 [project.optional-dependencies]
 browser = ["crawl4ai>=0.9"]
-mcp = ["fastmcp>=2"]
-dev = ["pytest>=8", "ruff>=0.6"]
+# fastmcp pulls in the `mcp` SDK. The MCP SDK renamed CallToolResult.isError to
+# is_error in the 2.x line, and fastmcp 4.x ships that newer SDK. The code and
+# tests here use is_error, so the floor is the version that has it.
+# Without a floor, an install could resolve to a fastmcp whose result objects
+# expose isError only, and the MCP surface would look broken for a reason that
+# has nothing to do with webget.
+mcp = ["fastmcp>=4"]
+dev = ["pytest>=8", "ruff>=0.6", "pytest-timeout>=2", "pytest-socket>=0.7"]
 
 [project.scripts]
 webget = "webget.cli:main"
@@ -50,7 +56,22 @@ packages = ["webget"]
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 addopts = "-q"
+markers = [
+  # Tests that reach the PUBLIC internet (not the local test server). They are
+  # excluded from offline jobs with -m "not live_network" and run separately,
+  # because engine reachability is a property of the runner's network: during
+  # 0.13.0, DuckDuckGo was unreachable from the author's ISP while Google,
+  # Brave and Yandex answered in under a second. A red test here does not mean
+  # the release is broken.
+  "live_network: test reaches the public internet; excluded from offline CI",
+]
 
 [tool.ruff]
 line-length = 100
 target-version = "py311"
+
+[tool.ruff.lint.per-file-ignores]
+# Ops scripts drive subprocesses and inspect returncode manually; requiring
+# check=True everywhere would force try/except noise where a plain if already
+# handles the failure.
+"scripts/*" = ["PLW1510"]

{webget_cli-0.12.1 → webget_cli-0.14.0}/tests/test_adversarial_mcp.py

@@ -10,6 +10,8 @@ import pytest
 from mcp import ClientSession, StdioServerParameters
 from mcp.client.stdio import stdio_client
 
+from tests.conftest import tool_failed
+
 ROOT = Path(__file__).resolve().parent.parent
 MCP = ROOT / "webget_mcp.py"
 
@@ -58,7 +60,7 @@ class TestMalformedArguments:
                 return res
 
         res = _run(run())
-        assert res.isError
+        assert tool_failed(res)
 
 
 class TestInvalidURLs:
@@ -72,7 +74,7 @@ class TestInvalidURLs:
                 return res
 
         res = _run(run())
-        assert "error" in res.content[0].text or res.isError
+        assert "error" in res.content[0].text or tool_failed(res)
 
 
 class TestRepeatedCalls:
@@ -86,7 +88,7 @@ class TestRepeatedCalls:
                         "fetch",
                         {"url": "https://example.com", "strategy": "http", "no_cache": True},
                     )
-                    if res.isError:
+                    if tool_failed(res):
                         return "ERROR"
                 return "OK"
 
@@ -106,7 +108,7 @@ class TestRepeatedCalls:
                         for _ in range(5)
                     ]
                 )
-                return [r.isError for r in results]
+                return [tool_failed(r) for r in results]
 
         assert _run(run()) == [False] * 5
 
@@ -133,7 +135,7 @@ class TestInputCaps:
                 return res
 
         res = _run(run())
-        assert "must be between" in res.content[0].text or res.isError
+        assert "must be between" in res.content[0].text or tool_failed(res)
 
 
 class TestToolFailureIsolation:
@@ -171,8 +173,18 @@ class TestSSRFViaMCP:
         res = _run(run())
         assert "private" in res.content[0].text.lower()
 
+    @pytest.mark.live_network
     def test_search_output_shape(self):
-        """search must return list of dicts with title/url/snippet."""
+        """search must return {results: [...], engine: ...} with provenance.
+
+        Marked live_network deliberately: this is the ONLY test in the suite
+        that talks to the public internet (verified by running the whole MCP
+        suite with sockets blocked except localhost - everything passes except
+        this one, which fails exactly because the network is unavailable). It
+        exercises the real engine path end-to-end through a real server
+        process, which no mock can replace, so it stays but is excluded from
+        the offline jobs.
+        """
 
         async def run():
             params = StdioServerParameters(command=sys.executable, args=[str(MCP)])
@@ -183,6 +195,11 @@ class TestSSRFViaMCP:
 
         res = _run(run())
         payload = json.loads(res.content[0].text)
-        assert isinstance(payload, list)
-        if payload:
-            assert {"title", "url", "snippet"} <= set(payload[0])
+        assert isinstance(payload, dict)
+        # Provenance keys are part of the contract: `engine` is the one that
+        # actually answered, which can differ from `requested_engine`.
+        assert {"results", "engine", "requested_engine", "failed_over"} <= set(payload)
+        results = payload["results"]
+        assert isinstance(results, list)
+        if results:
+            assert {"title", "url", "snippet"} <= set(results[0])

webget_cli-0.14.0/tests/test_cli_engine.py

@@ -0,0 +1,62 @@
+"""Tests for the CLI surface of multi-engine search."""
+
+from webget import cli
+
+
+def _opts(*args):
+    return cli.parse_opts(list(args))
+
+
+class TestParseOptsEngine:
+    def test_engine_flag_long(self):
+        r = _opts("s", "q", "--engine", "brave")
+        assert r[-1] == "brave"
+
+    def test_engine_flag_short(self):
+        r = _opts("s", "q", "-e", "brave,duckduckgo")
+        assert r[-1] == "brave,duckduckgo"
+
+    def test_engine_defaults_to_none(self):
+        r = _opts("s", "q")
+        assert r[-1] is None
+
+
+class TestCmdSearchEngine:
+    def test_engine_reaches_search_call(self, monkeypatch, capsys):
+        captured = {}
+
+        def fake_prov(query, n=5, engine=None):
+            captured["query"] = query
+            captured["n"] = n
+            captured["engine"] = engine
+            return (
+                [{"title": "T", "url": "https://x.example", "snippet": "s"}],
+                {"requested": engine or "auto", "engine": engine or "auto", "failed_over": False},
+            )
+
+        # patch where it is USED (webget.cli namespace), not where defined.
+        # The CLI prefers search_with_provenance, so patching only `search`
+        # would leave the real network path live.
+        monkeypatch.setattr(cli, "search_with_provenance", fake_prov)
+        monkeypatch.setattr("sys.argv", ["webget", "s", "query", "--engine", "brave"])
+        try:
+            cli.main()
+        except SystemExit as e:
+            assert e.code in (0, None)
+        assert captured["engine"] == "brave"
+        assert captured["query"] == "query"
+
+    def test_invalid_engine_does_not_crash_cli(self, monkeypatch, capsys):
+        monkeypatch.setattr(
+            cli,
+            "search_with_provenance",
+            lambda query, n=5, engine=None: (
+                [{"title": "T", "url": "https://y.example", "snippet": "s"}],
+                {"requested": engine or "auto", "engine": engine or "auto", "failed_over": False},
+            ),
+        )
+        monkeypatch.setattr("sys.argv", ["webget", "s", "q", "--engine", "totally-bogus"])
+        try:
+            cli.main()
+        except SystemExit as e:
+            assert e.code in (0, None)

webget_cli-0.14.0/tests/test_cli_provenance.py

@@ -0,0 +1,98 @@
+"""Tests for engine provenance surfacing through the CLI.
+
+The point of provenance is that it must be REACHABLE. A stderr warning is
+lost when output is piped, so `webget s q --json` has to carry it in the
+payload. Note the bare `webget s q --json` path did not previously exist at
+all (cmd == "s" had no json branch), so this adds it.
+"""
+
+from __future__ import annotations
+
+import json
+
+from webget import cli
+
+
+def _run(argv, monkeypatch, capsys, search_impl=None, prov_impl=None):
+    """Run the CLI with search stubbed.
+
+    Both hooks are patched: the CLI prefers `search_with_provenance`, so
+    patching only `search` would leave the real network path live. Pass
+    `prov_impl` to control provenance explicitly.
+    """
+    impl = search_impl or (lambda *a, **k: [])
+
+    if prov_impl is None:
+
+        def _default_prov(query, n=5, engine=None):
+            return impl(query, n, engine), {
+                "requested": engine or "auto",
+                "engine": engine or "auto",
+                "failed_over": False,
+            }
+
+        prov_impl = _default_prov
+
+    monkeypatch.setattr(cli, "search", lambda *a, **k: impl(*a, **k))
+    monkeypatch.setattr(cli, "search_with_provenance", prov_impl)
+    monkeypatch.setattr("sys.argv", ["webget", *argv])
+    try:
+        cli.main()
+    except SystemExit as e:
+        assert e.code in (0, None)
+    return capsys.readouterr()
+
+
+class TestProvenanceIsWiredUp:
+    """Guard against the fallback silently swallowing real provenance.
+
+    `_search_with_prov` falls back to a best-effort dict when
+    `search_with_provenance` is missing from the module namespace. That
+    fallback exists for older tests, but if the real import is ever
+    dropped the CLI would report `failed_over: False` forever while
+    looking perfectly green. This asserts the real function is reachable.
+    """
+
+    def test_cli_namespace_exposes_the_real_function(self):
+        assert hasattr(cli, "search_with_provenance")
+
+
+class TestSearchJsonOutput:
+    def test_json_flag_emits_valid_json(self, monkeypatch, capsys):
+        out = _run(
+            ["s", "q", "--json"],
+            monkeypatch,
+            capsys,
+            lambda *a, **k: [{"title": "T", "url": "https://x.example/", "snippet": "s"}],
+        )
+        payload = json.loads(out.out)
+        assert payload["results"][0]["url"] == "https://x.example/"
+
+    def test_json_includes_engine_provenance_when_available(self, monkeypatch, capsys):
+        def fake_prov(query, n=5, engine=None):
+            return (
+                [{"title": "T", "url": "https://x.example/", "snippet": "s"}],
+                {
+                    "requested": "google",
+                    "engine": "brave",
+                    "failed_over": True,
+                    "tried": ["google", "brave"],
+                    "first_error": "No results found.",
+                },
+            )
+
+        out = _run(["s", "q", "--json"], monkeypatch, capsys, prov_impl=fake_prov)
+        payload = json.loads(out.out)
+        assert payload["engine"] == "brave"
+        assert payload["requested_engine"] == "google"
+        assert payload["failed_over"] is True
+
+    def test_plain_output_still_prints_results(self, monkeypatch, capsys):
+        out = _run(
+            ["s", "q"],
+            monkeypatch,
+            capsys,
+            lambda *a, **k: [{"title": "T", "url": "https://x.example/", "snippet": "snip"}],
+        )
+        assert "https://x.example/" in out.out
+        assert "1." in out.out