lionscraper 1.0.2__tar.gz

lionscraper-1.0.2/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.4
+Name: lionscraper
+Version: 1.0.2
+Summary: LionScraper bridge daemon, thin MCP stdio, and CLI — local HTTP + WebSocket to Chrome extension (Python)
+Author: LionScraper
+License: MIT
+Keywords: lionscraper,mcp,cli,web-scraping,chrome-extension
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pydantic>=2.6.0
+Requires-Dist: mcp>=1.8.0
+Requires-Dist: websockets>=12.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
+
+# LionScraper (Python)
+
+Python edition of **LionScraper**: a local **HTTP + WebSocket** daemon that talks to the **Chrome/Edge LionScraper extension**, plus a **thin MCP server** over **stdio** that forwards tool calls to the daemon. Behavior is intended to match the npm/`packages/node` implementation in this repository.
+
+## Requirements
+
+- Python **3.10+**
+- The LionScraper browser extension installed and configured to use the same **PORT** as the daemon (default **13808**).
+
+## Install
+
+From PyPI (when published):
+
+```bash
+pip install lionscraper
+```
+
+From a checkout of this repo:
+
+```bash
+cd packages/python
+pip install -e ".[dev]"
+```
+
+## Commands
+
+| Command | Role |
+|--------|------|
+| `lionscraper` | Full CLI (`daemon`, `stop`, `scrape`, `ping`, …). |
+| `lionscraper-mcp` | If **no** extra arguments: thin MCP over stdio. **Any** extra argument delegates to the same CLI as `lionscraper`. |
+
+Equivalent module entry:
+
+```bash
+python -m lionscraper --help
+python -m lionscraper daemon
+```
+
+## MCP client configuration
+
+Use **`lionscraper-mcp`** as the MCP server command (no arguments) so the host spawns thin MCP stdio:
+
+```json
+{
+  "mcpServers": {
+    "lionscraper": {
+      "command": "lionscraper-mcp",
+      "env": {
+        "PORT": "13808",
+        "LANG": "en-US"
+      }
+    }
+  }
+}
+```
+
+To run the same routing as `lionscraper-mcp` via Python (stdio MCP when there are **no** extra arguments after the module name):
+
+```json
+"command": "python",
+"args": ["-m", "lionscraper"]
+```
+
+Thin stdio mode is selected only when `sys.argv` after the program entry has **length 0** (same as Node). Any extra token (including `--debug`) selects the CLI path instead.
+
+## Environment variables
+
+| Variable | Meaning |
+|----------|---------|
+| `PORT` | HTTP + WebSocket listen port (default **13808**). Must match the extension bridge port. |
+| `TOKEN` | Optional bearer token for `Authorization` on loopback HTTP. |
+| `DAEMON` | Set to `0` to disable auto-start of the daemon from CLI / thin MCP. |
+| `TIMEOUT` | Default timeout hints (see root repo docs). |
+| `LANG` | Locale for log and tool metadata (`en-US` / `zh-CN`). |
+
+## PyPI release (maintainers)
+
+```bash
+cd packages/python
+python -m pip install build twine
+python -m build
+python -m twine upload dist/*
+```
+
+Project name on PyPI: **`lionscraper`** (single package providing both console scripts).
+
+## Parity with Node
+
+This package mirrors `packages/node`: bridge protocol, daemon HTTP API (`/v1/health`, `/v1/daemon/shutdown`, `/v1/tools/call` with optional NDJSON progress), port probing, and tool input validation (Pydantic vs Zod). Locale JSON is copied under `src/locale/` so the Python wheel does not depend on the Node tree.
+
+## Development tests
+
+```bash
+cd packages/python
+pytest
+```
+
+## Manual smoke test
+
+1. Start the daemon: `lionscraper daemon` (or let CLI / `lionscraper-mcp` auto-spawn with default `DAEMON`).
+2. Connect the extension to `ws://127.0.0.1:<PORT>` (same as HTTP).
+3. `curl -s http://127.0.0.1:13808/v1/health` — expect JSON with `"ok": true` and `identity` **lionscraper** when ready.
+4. Run `lionscraper-mcp` with no args under an MCP host and list tools — expect `ping`, `scrape`, `scrape_article`, etc.
+
+Chinese documentation: [README_cn.md](README_cn.md).

lionscraper-1.0.2/README.md

@@ -0,0 +1,105 @@
+# LionScraper (Python)
+
+Python edition of **LionScraper**: a local **HTTP + WebSocket** daemon that talks to the **Chrome/Edge LionScraper extension**, plus a **thin MCP server** over **stdio** that forwards tool calls to the daemon. Behavior is intended to match the npm/`packages/node` implementation in this repository.
+
+## Requirements
+
+- Python **3.10+**
+- The LionScraper browser extension installed and configured to use the same **PORT** as the daemon (default **13808**).
+
+## Install
+
+From PyPI (when published):
+
+```bash
+pip install lionscraper
+```
+
+From a checkout of this repo:
+
+```bash
+cd packages/python
+pip install -e ".[dev]"
+```
+
+## Commands
+
+| Command | Role |
+|--------|------|
+| `lionscraper` | Full CLI (`daemon`, `stop`, `scrape`, `ping`, …). |
+| `lionscraper-mcp` | If **no** extra arguments: thin MCP over stdio. **Any** extra argument delegates to the same CLI as `lionscraper`. |
+
+Equivalent module entry:
+
+```bash
+python -m lionscraper --help
+python -m lionscraper daemon
+```
+
+## MCP client configuration
+
+Use **`lionscraper-mcp`** as the MCP server command (no arguments) so the host spawns thin MCP stdio:
+
+```json
+{
+  "mcpServers": {
+    "lionscraper": {
+      "command": "lionscraper-mcp",
+      "env": {
+        "PORT": "13808",
+        "LANG": "en-US"
+      }
+    }
+  }
+}
+```
+
+To run the same routing as `lionscraper-mcp` via Python (stdio MCP when there are **no** extra arguments after the module name):
+
+```json
+"command": "python",
+"args": ["-m", "lionscraper"]
+```
+
+Thin stdio mode is selected only when `sys.argv` after the program entry has **length 0** (same as Node). Any extra token (including `--debug`) selects the CLI path instead.
+
+## Environment variables
+
+| Variable | Meaning |
+|----------|---------|
+| `PORT` | HTTP + WebSocket listen port (default **13808**). Must match the extension bridge port. |
+| `TOKEN` | Optional bearer token for `Authorization` on loopback HTTP. |
+| `DAEMON` | Set to `0` to disable auto-start of the daemon from CLI / thin MCP. |
+| `TIMEOUT` | Default timeout hints (see root repo docs). |
+| `LANG` | Locale for log and tool metadata (`en-US` / `zh-CN`). |
+
+## PyPI release (maintainers)
+
+```bash
+cd packages/python
+python -m pip install build twine
+python -m build
+python -m twine upload dist/*
+```
+
+Project name on PyPI: **`lionscraper`** (single package providing both console scripts).
+
+## Parity with Node
+
+This package mirrors `packages/node`: bridge protocol, daemon HTTP API (`/v1/health`, `/v1/daemon/shutdown`, `/v1/tools/call` with optional NDJSON progress), port probing, and tool input validation (Pydantic vs Zod). Locale JSON is copied under `src/locale/` so the Python wheel does not depend on the Node tree.
+
+## Development tests
+
+```bash
+cd packages/python
+pytest
+```
+
+## Manual smoke test
+
+1. Start the daemon: `lionscraper daemon` (or let CLI / `lionscraper-mcp` auto-spawn with default `DAEMON`).
+2. Connect the extension to `ws://127.0.0.1:<PORT>` (same as HTTP).
+3. `curl -s http://127.0.0.1:13808/v1/health` — expect JSON with `"ok": true` and `identity` **lionscraper** when ready.
+4. Run `lionscraper-mcp` with no args under an MCP host and list tools — expect `ping`, `scrape`, `scrape_article`, etc.
+
+Chinese documentation: [README_cn.md](README_cn.md).

lionscraper-1.0.2/lionscraper.egg-info/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.4
+Name: lionscraper
+Version: 1.0.2
+Summary: LionScraper bridge daemon, thin MCP stdio, and CLI — local HTTP + WebSocket to Chrome extension (Python)
+Author: LionScraper
+License: MIT
+Keywords: lionscraper,mcp,cli,web-scraping,chrome-extension
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pydantic>=2.6.0
+Requires-Dist: mcp>=1.8.0
+Requires-Dist: websockets>=12.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
+
+# LionScraper (Python)
+
+Python edition of **LionScraper**: a local **HTTP + WebSocket** daemon that talks to the **Chrome/Edge LionScraper extension**, plus a **thin MCP server** over **stdio** that forwards tool calls to the daemon. Behavior is intended to match the npm/`packages/node` implementation in this repository.
+
+## Requirements
+
+- Python **3.10+**
+- The LionScraper browser extension installed and configured to use the same **PORT** as the daemon (default **13808**).
+
+## Install
+
+From PyPI (when published):
+
+```bash
+pip install lionscraper
+```
+
+From a checkout of this repo:
+
+```bash
+cd packages/python
+pip install -e ".[dev]"
+```
+
+## Commands
+
+| Command | Role |
+|--------|------|
+| `lionscraper` | Full CLI (`daemon`, `stop`, `scrape`, `ping`, …). |
+| `lionscraper-mcp` | If **no** extra arguments: thin MCP over stdio. **Any** extra argument delegates to the same CLI as `lionscraper`. |
+
+Equivalent module entry:
+
+```bash
+python -m lionscraper --help
+python -m lionscraper daemon
+```
+
+## MCP client configuration
+
+Use **`lionscraper-mcp`** as the MCP server command (no arguments) so the host spawns thin MCP stdio:
+
+```json
+{
+  "mcpServers": {
+    "lionscraper": {
+      "command": "lionscraper-mcp",
+      "env": {
+        "PORT": "13808",
+        "LANG": "en-US"
+      }
+    }
+  }
+}
+```
+
+To run the same routing as `lionscraper-mcp` via Python (stdio MCP when there are **no** extra arguments after the module name):
+
+```json
+"command": "python",
+"args": ["-m", "lionscraper"]
+```
+
+Thin stdio mode is selected only when `sys.argv` after the program entry has **length 0** (same as Node). Any extra token (including `--debug`) selects the CLI path instead.
+
+## Environment variables
+
+| Variable | Meaning |
+|----------|---------|
+| `PORT` | HTTP + WebSocket listen port (default **13808**). Must match the extension bridge port. |
+| `TOKEN` | Optional bearer token for `Authorization` on loopback HTTP. |
+| `DAEMON` | Set to `0` to disable auto-start of the daemon from CLI / thin MCP. |
+| `TIMEOUT` | Default timeout hints (see root repo docs). |
+| `LANG` | Locale for log and tool metadata (`en-US` / `zh-CN`). |
+
+## PyPI release (maintainers)
+
+```bash
+cd packages/python
+python -m pip install build twine
+python -m build
+python -m twine upload dist/*
+```
+
+Project name on PyPI: **`lionscraper`** (single package providing both console scripts).
+
+## Parity with Node
+
+This package mirrors `packages/node`: bridge protocol, daemon HTTP API (`/v1/health`, `/v1/daemon/shutdown`, `/v1/tools/call` with optional NDJSON progress), port probing, and tool input validation (Pydantic vs Zod). Locale JSON is copied under `src/locale/` so the Python wheel does not depend on the Node tree.
+
+## Development tests
+
+```bash
+cd packages/python
+pytest
+```
+
+## Manual smoke test
+
+1. Start the daemon: `lionscraper daemon` (or let CLI / `lionscraper-mcp` auto-spawn with default `DAEMON`).
+2. Connect the extension to `ws://127.0.0.1:<PORT>` (same as HTTP).
+3. `curl -s http://127.0.0.1:13808/v1/health` — expect JSON with `"ok": true` and `identity` **lionscraper** when ready.
+4. Run `lionscraper-mcp` with no args under an MCP host and list tools — expect `ping`, `scrape`, `scrape_article`, etc.
+
+Chinese documentation: [README_cn.md](README_cn.md).

lionscraper-1.0.2/lionscraper.egg-info/SOURCES.txt

@@ -0,0 +1,56 @@
+README.md
+pyproject.toml
+lionscraper.egg-info/PKG-INFO
+lionscraper.egg-info/SOURCES.txt
+lionscraper.egg-info/dependency_links.txt
+lionscraper.egg-info/entry_points.txt
+lionscraper.egg-info/requires.txt
+lionscraper.egg-info/top_level.txt
+src/__init__.py
+src/__main__.py
+src/version.py
+src/bridge/__init__.py
+src/bridge/protocol.py
+src/bridge/session.py
+src/bridge/timeout.py
+src/bridge/websocket.py
+src/cli/__init__.py
+src/cli/build_tool_args.py
+src/cli/entry.py
+src/cli/mcp_bin_argv.py
+src/cli/router.py
+src/client/__init__.py
+src/client/daemon_client.py
+src/client/daemon_lifecycle.py
+src/core/__init__.py
+src/core/bridge_service.py
+src/daemon/__init__.py
+src/daemon/daemon_main.py
+src/daemon/http_api.py
+src/i18n/__init__.py
+src/i18n/lang.py
+src/locale/en-US.json
+src/locale/zh-CN.json
+src/mcp/__init__.py
+src/mcp/handler.py
+src/mcp/mcp_prompts.py
+src/mcp/mcp_resources.py
+src/mcp/mcp_stdio_app.py
+src/mcp/thin_mcp_server.py
+src/mcp/tools.py
+src/mcp/validate_tool_input.py
+src/types/__init__.py
+src/types/bridge.py
+src/types/errors.py
+src/utils/__init__.py
+src/utils/browser_env.py
+src/utils/config.py
+src/utils/daemon_config.py
+src/utils/logger.py
+src/utils/port.py
+tests/test_cli_build_tool_args.py
+tests/test_mcp_bin_argv.py
+tests/test_port.py
+tests/test_protocol.py
+tests/test_timeout.py
+tests/test_validate_tool_input.py

lionscraper-1.0.2/lionscraper.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

lionscraper-1.0.2/lionscraper.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+lionscraper = lionscraper.cli.entry:main_cli
+lionscraper-mcp = lionscraper.cli.entry:main_mcp_or_cli

lionscraper-1.0.2/lionscraper.egg-info/requires.txt

@@ -0,0 +1,9 @@
+aiohttp>=3.9.0
+httpx>=0.27.0
+pydantic>=2.6.0
+mcp>=1.8.0
+websockets>=12.0
+
+[dev]
+pytest>=8.0
+pytest-asyncio>=0.24.0

lionscraper-1.0.2/lionscraper.egg-info/top_level.txt

@@ -0,0 +1 @@
+lionscraper

lionscraper-1.0.2/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "lionscraper"
+version = "1.0.2"
+description = "LionScraper bridge daemon, thin MCP stdio, and CLI — local HTTP + WebSocket to Chrome extension (Python)"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.10"
+authors = [{ name = "LionScraper" }]
+keywords = ["lionscraper", "mcp", "cli", "web-scraping", "chrome-extension"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+  "aiohttp>=3.9.0",
+  "httpx>=0.27.0",
+  "pydantic>=2.6.0",
+  "mcp>=1.8.0",
+  "websockets>=12.0",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0", "pytest-asyncio>=0.24.0"]
+
+[project.scripts]
+lionscraper = "lionscraper.cli.entry:main_cli"
+"lionscraper-mcp" = "lionscraper.cli.entry:main_mcp_or_cli"
+
+[tool.setuptools]
+package-dir = { "lionscraper" = "src" }
+packages = [
+  "lionscraper",
+  "lionscraper.i18n",
+  "lionscraper.utils",
+  "lionscraper.types",
+  "lionscraper.bridge",
+  "lionscraper.core",
+  "lionscraper.daemon",
+  "lionscraper.client",
+  "lionscraper.cli",
+  "lionscraper.mcp",
+]
+
+[tool.setuptools.package-data]
+lionscraper = ["locale/*.json"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

lionscraper-1.0.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

lionscraper-1.0.2/src/__init__.py

@@ -0,0 +1,6 @@
+try:
+    from importlib.metadata import version
+
+    __version__ = version("lionscraper")
+except Exception:  # pragma: no cover
+    __version__ = "1.0.2"

lionscraper-1.0.2/src/__main__.py

@@ -0,0 +1,4 @@
+from lionscraper.cli.entry import main_mcp_or_cli
+
+if __name__ == "__main__":
+    main_mcp_or_cli()

lionscraper-1.0.2/src/bridge/protocol.py

@@ -0,0 +1,148 @@
+from __future__ import annotations
+
+import asyncio
+import uuid
+from typing import Any, Callable
+
+from lionscraper.i18n.lang import SupportedLang, t
+from lionscraper.types.bridge import BridgeMethod, BridgeProgressParams, BridgeRequest, BridgeResponse
+from lionscraper.types.errors import BridgeErrorCode, LionScraperError, create_error
+from lionscraper.utils.logger import logger
+
+DEFAULT_TIMEOUT_MS = 60_000
+
+BridgeProgressHandler = Callable[[BridgeProgressParams], None]
+
+
+def create_bridge_request(method: BridgeMethod, params: dict[str, Any] | None = None) -> BridgeRequest:
+    req: BridgeRequest = {
+        "jsonrpc": "2.0",
+        "id": str(uuid.uuid4()),
+        "method": method,
+    }
+    if params is not None:
+        req["params"] = params
+    return req
+
+
+def create_bridge_response(req_id: str, result: Any) -> BridgeResponse:
+    return {"jsonrpc": "2.0", "id": req_id, "result": result}
+
+
+def create_bridge_error_response(
+    req_id: str,
+    code: int,
+    message: str,
+    data: dict[str, Any] | None = None,
+) -> BridgeResponse:
+    err: dict[str, Any] = {"code": code, "message": message}
+    if data:
+        err["data"] = data
+    return {"jsonrpc": "2.0", "id": req_id, "error": err}
+
+
+DisconnectRejectId = str
+
+
+class PendingRequest:
+    __slots__ = ("id", "method", "fut", "timer_handle", "created_at", "lang", "on_progress")
+
+    def __init__(
+        self,
+        id: str,
+        method: str,
+        fut: asyncio.Future[Any],
+        timer_handle: asyncio.TimerHandle,
+        lang: SupportedLang,
+        on_progress: BridgeProgressHandler | None,
+    ):
+        self.id = id
+        self.method = method
+        self.fut = fut
+        self.timer_handle = timer_handle
+        self.lang = lang
+        self.on_progress = on_progress
+
+
+class PendingRequestManager:
+    def __init__(self) -> None:
+        self._pending: dict[str, PendingRequest] = {}
+
+    def begin_request(
+        self,
+        req_id: str,
+        method: BridgeMethod,
+        timeout_ms: int = DEFAULT_TIMEOUT_MS,
+        lang: SupportedLang = "en-US",
+        on_progress: BridgeProgressHandler | None = None,
+    ) -> asyncio.Future[Any]:
+        loop = asyncio.get_running_loop()
+        fut: asyncio.Future[Any] = loop.create_future()
+
+        def on_timeout() -> None:
+            self._pending.pop(req_id, None)
+            if not fut.done():
+                logger.warn(f"Bridge request timed out: {req_id} ({method})")
+                fut.set_exception(
+                    create_error(
+                        BridgeErrorCode.BRIDGE_TIMEOUT,
+                        t(lang, "bridge_timeout", {"ms": timeout_ms}),
+                    )
+                )
+
+        handle = loop.call_later(timeout_ms / 1000.0, on_timeout)
+        self._pending[req_id] = PendingRequest(req_id, method, fut, handle, lang, on_progress)
+        return fut
+
+    def dispatch_progress(self, request_id: str, params: BridgeProgressParams) -> bool:
+        pending = self._pending.get(request_id)
+        if not pending:
+            return False
+        if pending.on_progress:
+            try:
+                pending.on_progress(params)
+            except Exception:
+                pass
+        return True
+
+    def resolve(self, req_id: str, result: Any) -> bool:
+        pending = self._pending.pop(req_id, None)
+        if not pending:
+            return False
+        pending.timer_handle.cancel()
+        if not pending.fut.done():
+            pending.fut.set_result(result)
+        return True
+
+    def reject(self, req_id: str, error: LionScraperError) -> bool:
+        pending = self._pending.pop(req_id, None)
+        if not pending:
+            return False
+        pending.timer_handle.cancel()
+        if not pending.fut.done():
+            pending.fut.set_exception(error)
+        return True
+
+    def reject_all_disconnected(self, reason_key: str) -> None:
+        for req_id, pending in list(self._pending.items()):
+            pending.timer_handle.cancel()
+            if not pending.fut.done():
+                pending.fut.set_exception(
+                    create_error(
+                        BridgeErrorCode.BRIDGE_DISCONNECTED,
+                        t(pending.lang, reason_key),
+                    )
+                )
+            self._pending.pop(req_id, None)
+
+    def has(self, req_id: str) -> bool:
+        return req_id in self._pending
+
+    @property
+    def size(self) -> int:
+        return len(self._pending)
+
+    def clear(self) -> None:
+        for p in self._pending.values():
+            p.timer_handle.cancel()
+        self._pending.clear()