browser-goat 0.1.0__tar.gz

browser_goat-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 browser-goat
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

browser_goat-0.1.0/PKG-INFO

@@ -0,0 +1,24 @@
+Metadata-Version: 2.4
+Name: browser-goat
+Version: 0.1.0
+Summary: Meta-layer search intelligence wrapping SearXNG with pre/post processing, extraction, strategy, and verification from SearchWala, local-deep-research, Marco-DeepResearch, Tongyi-DeepResearch, and Scrapling.
+License-Expression: MIT
+Requires-Python: >=3.13
+License-File: LICENSE
+Requires-Dist: httpx[http2]>=0.28
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: lxml>=5.3
+Requires-Dist: pydantic>=2.10
+Requires-Dist: tenacity>=9.0
+Requires-Dist: tiktoken>=0.8
+Requires-Dist: playwright>=1.49
+Requires-Dist: scrapling>=0.2
+Requires-Dist: mcp>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.3; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.25; extra == "dev"
+Requires-Dist: pytest-cov>=6.0; extra == "dev"
+Requires-Dist: ruff>=0.8; extra == "dev"
+Requires-Dist: mypy>=1.13; extra == "dev"
+Requires-Dist: bandit>=1.8; extra == "dev"
+Dynamic: license-file

browser_goat-0.1.0/README.md

@@ -0,0 +1,177 @@
+# browser-goat
+
+[![Tests](https://img.shields.io/badge/tests-304%20passed-brightgreen)](https://github.com/Im-Busy/browser-goat)
+[![Python](https://img.shields.io/badge/python-3.13%2B-blue)](https://python.org)
+[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+> Meta-layer search intelligence wrapping SearXNG — Tavily-quality results on your own infrastructure.
+
+browser-goat adds six processing layers to [SearXNG](https://docs.searxng.org/): intent detection, hybrid ranking, anti-bot content extraction, reliability gating, adaptive strategy, and multi-rollout verification. The result is agent-ready search output competitive with commercial APIs — running entirely on your own infrastructure.
+
+```mermaid
+flowchart TD
+    Q["Query"] --> L1
+
+    subgraph L1["1. Pre-Search"]
+        A["Intent detection<br/>Browser profiles<br/>Language detection"]
+    end
+
+    L1 --> SX["SearXNG Engine"]
+
+    SX --> L2
+    subgraph L2["2. Post-Search"]
+        B["URL normalization<br/>RRF + BM25 + MMR"]
+    end
+
+    L2 --> L3
+    subgraph L3["3. Extraction"]
+        C["7-tier cascading<br/>Anti-bot bypass<br/>Goal-oriented summary"]
+    end
+
+    L3 --> L4
+    subgraph L4["4. Reliability"]
+        D["Give-up detection<br/>Quality-gated retry<br/>Force synthesis"]
+    end
+
+    L4 --> L5
+    subgraph L5["5. Strategy"]
+        E["Query classification<br/>Adaptive exploration<br/>Recursive decomposition"]
+    end
+
+    L5 --> L6
+    subgraph L6["6. Verification"]
+        F["Multi-rollout voting<br/>Consensus verification<br/>LLM tie-breaking"]
+    end
+
+    L6 --> A["Answer"]
+```
+
+---
+
+## Quick Start
+
+### MCP (AI Agents)
+
+```json
+{
+  "mcpServers": {
+    "browser-goat": {
+      "command": "npx",
+      "args": ["browser-goat"],
+      "env": { "SEARXNG_URL": "http://localhost:8080" }
+    }
+  }
+}
+```
+
+Requires Python 3.13+ and a running SearXNG instance.
+
+### CLI
+
+```bash
+uvx browser-goat search "latest AI research"
+uvx browser-goat search "Python vs Rust" --strategy explore
+uvx browser-goat extract "https://example.com/article"
+```
+
+### Library
+
+```bash
+pip install browser-goat
+```
+
+```python
+from browser_goat import BrowserGoat
+
+meta = BrowserGoat(searxng_url="http://localhost:8080")
+result = await meta.search("quantum computing")
+print(result.answer)
+```
+
+---
+
+## MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `search` | Full pipeline: intent analysis → SearXNG → ranking → extraction → reliability. Supports `time_range` (day/week/month/year), `max_sources`, and `strategy` (default/auto/explore/decompose). |
+| `extract` | Fetch and extract a single URL with anti-bot bypass (Cloudflare Turnstile). Returns title, clean text, and extraction tier. |
+
+---
+
+## Client Configuration
+
+### Claude Desktop
+
+```json
+{
+  "mcpServers": {
+    "browser-goat": {
+      "command": "uvx",
+      "args": ["browser-goat-mcp", "--searxng-url", "http://localhost:8080"]
+    }
+  }
+}
+```
+
+### Cursor / VS Code
+
+```json
+{
+  "mcpServers": {
+    "browser-goat": {
+      "command": "npx",
+      "args": ["browser-goat"],
+      "env": { "SEARXNG_URL": "http://localhost:8080" }
+    }
+  }
+}
+```
+
+---
+
+## Docker
+
+Bundled SearXNG + Redis sidecar deployment:
+
+```bash
+docker compose up
+```
+
+SearXNG starts at `localhost:8080`, browser-goat API at `localhost:8000`.
+
+```bash
+docker exec browser-goat uv run browser-goat search "your query"
+```
+
+---
+
+## How It Works
+
+Each search passes through six layers before returning an answer. The diagram above shows the full pipeline. Layers 1-4 run on every query; Layers 5-6 activate when `--strategy` or `--reliability` are set.
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/Im-Busy/browser-goat.git
+cd browser-goat
+uv sync
+
+uv run pytest                  # 304 tests (287 unit + 17 integration)
+uv run ruff check src/ tests/  # zero violations
+uv run mypy src/               # zero errors
+```
+
+Tests require SearXNG at `localhost:8080`. Skip integration tests:
+
+```bash
+uv run pytest -m "not integration"
+```
+
+---
+
+## License
+
+MIT

browser_goat-0.1.0/pyproject.toml

@@ -0,0 +1,65 @@
+[project]
+name = "browser-goat"
+version = "0.1.0"
+description = "Meta-layer search intelligence wrapping SearXNG with pre/post processing, extraction, strategy, and verification from SearchWala, local-deep-research, Marco-DeepResearch, Tongyi-DeepResearch, and Scrapling."
+requires-python = ">=3.13"
+license = "MIT"
+license-files = ["LICENSE"]
+dependencies = [
+    "httpx[http2]>=0.28",
+    "beautifulsoup4>=4.12",
+    "lxml>=5.3",
+    "pydantic>=2.10",
+    "tenacity>=9.0",
+    "tiktoken>=0.8",
+    "playwright>=1.49",
+    "scrapling>=0.2",
+    "mcp>=1.0",
+]
+
+[project.scripts]
+browser-goat = "browser_goat.cli:main"
+browser-goat-mcp = "browser_goat.mcp_server:main"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.3",
+    "pytest-asyncio>=0.25",
+    "pytest-cov>=6.0",
+    "ruff>=0.8",
+    "mypy>=1.13",
+    "bandit>=1.8",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py313"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "W", "UP", "B", "C4", "SIM"]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.13"
+strict = true
+warn_unreachable = true
+warn_unused_ignores = true
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+markers = [
+    "integration: end-to-end tests requiring a running SearXNG instance",
+]
+
+[tool.uv]
+package = true
+
+[dependency-groups]
+dev = [
+    "mypy>=2.1.0",
+    "pytest>=9.1.0",
+    "pytest-asyncio>=1.4.0",
+    "pytest-cov>=7.1.0",
+    "ruff>=0.15.17",
+]

browser_goat-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

browser_goat-0.1.0/src/browser_goat/__init__.py

@@ -0,0 +1,15 @@
+"""browser-goat — Meta-layer search intelligence wrapping SearXNG.
+
+Layers:
+    pre_search   — Query intelligence, language detection, browser profiles
+    post_search  — URL pipeline, RRF+BM25+MMR ranking, dedup
+    extraction   — Content extraction, goal-oriented, Scrapling anti-bot
+    reliability  — Give-up detection, quality gating, force answer
+    strategy     — Query classification, adaptive exploration (Phase 2)
+    verification — Multi-rollout voting (Phase 3)
+"""
+
+from browser_goat.router import BrowserGoat
+
+__version__ = "0.1.0"
+__all__ = ["BrowserGoat"]

browser_goat-0.1.0/src/browser_goat/cli.py

@@ -0,0 +1,277 @@
+"""CLI entry point for browser-goat.
+
+Usage:
+    browser-goat search "What is Python?" --searxng-url http://localhost:8080
+    browser-goat search "latest AI news" --time-range week --strategy explore
+    browser-goat search "Python vs Rust" --reliability high
+    uvx browser-goat search "quantum computing research"
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sys
+from typing import Any
+
+from browser_goat.router import BrowserGoat
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="browser-goat",
+        description="Meta-layer search intelligence wrapping SearXNG",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    # ── search ──
+    search = sub.add_parser("search", help="Run a full search pipeline")
+    search.add_argument("query", help="Search query string")
+    search.add_argument(
+        "--searxng-url",
+        default="http://localhost:8080",
+        help="SearXNG instance URL (default: http://localhost:8080)",
+    )
+    search.add_argument(
+        "--engines",
+        nargs="*",
+        default=None,
+        help="SearXNG engines to use (e.g. google bing scholar)",
+    )
+    search.add_argument(
+        "--time-range",
+        choices=["day", "week", "month", "year"],
+        default=None,
+        help="Time filter for results",
+    )
+    search.add_argument(
+        "--language",
+        default="en",
+        help="Language code for results (default: en)",
+    )
+    search.add_argument(
+        "--max-sources",
+        type=int,
+        default=15,
+        help="Maximum sources to extract (default: 15)",
+    )
+    search.add_argument(
+        "--strategy",
+        choices=["default", "auto", "explore", "decompose"],
+        default="default",
+        help="Search strategy (default: default)",
+    )
+    search.add_argument(
+        "--reliability",
+        choices=["standard", "high", "maximum"],
+        default="standard",
+        help="Reliability mode (default: standard)",
+    )
+    search.add_argument(
+        "--format",
+        choices=["json", "pretty"],
+        default="json",
+        help="Output format (default: json)",
+    )
+
+    # ── extract ──
+    extract = sub.add_parser("extract", help="Extract content from a URL")
+    extract.add_argument("url", help="URL to extract content from")
+    extract.add_argument(
+        "--searxng-url",
+        default="http://localhost:8080",
+        help="SearXNG instance URL",
+    )
+    extract.add_argument(
+        "--format",
+        choices=["json", "pretty"],
+        default="json",
+        help="Output format (default: json)",
+    )
+
+    # ── verify ──
+    verify = sub.add_parser("verify", help="Verify an answer via multi-rollout voting")
+    verify.add_argument("query", help="The query to verify")
+    verify.add_argument(
+        "--searxng-url",
+        default="http://localhost:8080",
+        help="SearXNG instance URL",
+    )
+    verify.add_argument(
+        "--rollouts",
+        type=int,
+        default=5,
+        help="Number of parallel rollouts (default: 5)",
+    )
+    verify.add_argument(
+        "--format",
+        choices=["json", "pretty"],
+        default="json",
+        help="Output format (default: json)",
+    )
+
+    # ── serve ──
+    serve = sub.add_parser("serve", help="Run browser-goat as an HTTP JSON API server")
+    serve.add_argument(
+        "--host",
+        default="0.0.0.0",
+        help="Host to bind to (default: 0.0.0.0)",
+    )
+    serve.add_argument(
+        "--port",
+        type=int,
+        default=8000,
+        help="Port to listen on (default: 8000)",
+    )
+    serve.add_argument(
+        "--searxng-url",
+        default="http://localhost:8080",
+        help="SearXNG instance URL (default: http://localhost:8080)",
+    )
+
+    return parser
+
+
+def format_output(data: Any, fmt: str) -> str:
+    """Format output as JSON or pretty-printed."""
+    if fmt == "pretty":
+        if hasattr(data, "model_dump"):
+            return json.dumps(data.model_dump(), indent=2, ensure_ascii=False)
+        return json.dumps(data, indent=2, ensure_ascii=False)
+    if hasattr(data, "model_dump"):
+        return str(data.model_dump_json())
+    return json.dumps(data, ensure_ascii=False)
+
+
+async def cmd_search(args: argparse.Namespace) -> None:
+    meta = BrowserGoat(searxng_url=args.searxng_url)
+    result = await meta.search(
+        query=args.query,
+        engines=args.engines,
+        time_range=args.time_range,
+        language=args.language,
+        max_sources=args.max_sources,
+        strategy=args.strategy,
+        reliability_mode=args.reliability,
+    )
+    output = format_output(result, args.format)
+    print(output)
+
+
+async def cmd_extract(args: argparse.Namespace) -> None:
+    meta = BrowserGoat(searxng_url=args.searxng_url)
+    fetcher = meta.scrapling
+    extractor = meta.content_extractor
+    profile = meta.browser_profiles.get_random_profile()
+
+    fetch_result = await fetcher.fetch(args.url, profile)
+    if not fetch_result.success:
+        print(json.dumps({"error": fetch_result.error or "fetch failed"}), file=sys.stderr)
+        sys.exit(1)
+
+    content = extractor.extract(fetch_result.html, args.url)
+
+    output = format_output(
+        {
+            "url": args.url,
+            "title": content.title,
+            "text": content.text[:1000] + "..." if len(content.text) > 1000 else content.text,
+            "extraction_tier": content.extraction_tier,
+            "text_length": len(content.text),
+        },
+        args.format,
+    )
+    print(output)
+
+
+async def cmd_verify(args: argparse.Namespace) -> None:
+    meta = BrowserGoat(searxng_url=args.searxng_url)
+    result = await meta.search(
+        query=args.query,
+        reliability_mode="high" if args.rollouts <= 5 else "maximum",
+    )
+    output = format_output(result, args.format)
+    print(output)
+
+
+async def cmd_serve(args: argparse.Namespace) -> None:
+    """Run browser-goat as a minimal HTTP JSON API server (zero extra deps)."""
+    meta = BrowserGoat(searxng_url=args.searxng_url)
+
+    async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
+        try:
+            raw = await asyncio.wait_for(reader.readuntil(b"\r\n\r\n"), timeout=30)
+            request_line, *_ = raw.decode("utf-8", errors="replace").split("\r\n")
+            method, path, *_ = request_line.split(" ") + ["", ""]
+
+            if method == "GET" and path in ("/health", "/"):
+                body = b'{"status":"ok"}'
+                writer.write(
+                    b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n"
+                    b"Content-Length: " + str(len(body)).encode() + b"\r\n\r\n" + body
+                )
+                await writer.drain()
+                return
+
+            if method == "POST" and path == "/search":
+                content_length = 0
+                for line in raw.decode("utf-8", errors="replace").split("\r\n"):
+                    if line.lower().startswith("content-length:"):
+                        content_length = int(line.split(":")[1].strip())
+                body_raw = await asyncio.wait_for(reader.readexactly(content_length), timeout=5)
+                params = json.loads(body_raw)
+
+                result = await meta.search(
+                    query=params.get("query", ""),
+                    time_range=params.get("time_range"),
+                    max_sources=params.get("max_sources", 15),
+                    strategy=params.get("strategy", "default"),
+                    reliability_mode=params.get("reliability", "standard"),
+                )
+                body = result.model_dump_json().encode()
+                writer.write(
+                    b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n"
+                    b"Content-Length: " + str(len(body)).encode() + b"\r\n\r\n" + body
+                )
+                await writer.drain()
+                return
+
+            body = b'{"error":"not found"}'
+            writer.write(
+                b"HTTP/1.1 404 Not Found\r\nContent-Type: application/json\r\n"
+                b"Content-Length: " + str(len(body)).encode() + b"\r\n\r\n" + body
+            )
+            await writer.drain()
+        except Exception:
+            pass
+        finally:
+            writer.close()
+            await writer.wait_closed()
+
+    server = await asyncio.start_server(handle, host=args.host, port=args.port)
+    print(f"browser-goat API listening on http://{args.host}:{args.port}", file=sys.stderr)
+    async with server:
+        await server.serve_forever()
+
+
+def main() -> None:
+    parser = build_parser()
+    args = parser.parse_args()
+
+    try:
+        if args.command == "search":
+            asyncio.run(cmd_search(args))
+        elif args.command == "extract":
+            asyncio.run(cmd_extract(args))
+        elif args.command == "verify":
+            asyncio.run(cmd_verify(args))
+        elif args.command == "serve":
+            asyncio.run(cmd_serve(args))
+    except Exception as e:
+        print(json.dumps({"error": str(e)}), file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

browser_goat-0.1.0/src/browser_goat/extraction/__init__.py

@@ -0,0 +1,9 @@
+"""Extraction layer: 7-tier content extraction, goal-oriented extraction, Scrapling anti-bot."""
+
+from browser_goat.extraction.content_extractor import ContentExtractor
+from browser_goat.extraction.goal_oriented import GoalOrientedExtractor
+from browser_goat.extraction.scrapling_fetcher import ScraplingFetcher
+
+__all__ = ["ContentExtractor", "GoalOrientedExtractor", "ScraplingFetcher"]
+
+__all__ = ["ContentExtractor", "GoalOrientedExtractor", "ScraplingFetcher"]