spidermanager-sdk 0.1.3.dev0__tar.gz

spidermanager_sdk-0.1.3.dev0/.github/workflows/publish.yml

@@ -0,0 +1,46 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - 'v*' # 当你推送以 v 开头的标签时触发，如 v0.1.0
+
+jobs:
+  build-n-publish:
+    runs-on: ubuntu-latest
+    # 这一步是为了安全，建议使用 PyPI 官方推荐的受信任发布者（Trusted Publishers）
+    permissions:
+      id-token: write
+      contents: write
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # 必须设为 0，setuptools-scm 需要完整的 Git 历史来计算版本
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        # 推荐使用 Trusted Publishers，不再需要手动在 Secrets 里存 Token
+        # 只需要在 PyPI 后台关联这个 GitHub 仓库即可
+
+      - name: Create GitHub Release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          generate_release_notes: true # 自动根据 commit 记录生成更新日志
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

spidermanager_sdk-0.1.3.dev0/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.env
+.venv/
+venv/
+.idea/
+.vscode/
+*.swp
+*.swo

spidermanager_sdk-0.1.3.dev0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rosia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spidermanager_sdk-0.1.3.dev0/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.4
+Name: spidermanager-sdk
+Version: 0.1.3.dev0
+Summary: 极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端
+Author: SpiderManager Team
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.25.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
+Dynamic: license-file
+
+# SpiderManager SDK
+
+极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端，实现爬虫逻辑与数据存储的完全解耦。
+
+## 特性
+
+- **零配置启动**：自动从环境变量 `TASK_ID` / `SPIDER_API_URL` 读取配置
+- **异步缓冲**：内存 buffer 按阈值（20条）或时间窗口（3秒）批量上报，减少网络 IO
+- **容器安全**：通过 `atexit` + `SIGTERM` 双保险，确保 Docker 销毁前 flush 全部数据
+- **最小依赖**：仅依赖 `httpx`，不侵入爬虫业务代码
+
+## 安装
+
+```bash
+pip install spidermanager-sdk
+```
+
+## 快速开始
+
+```python
+from spidermanager_sdk import sdk
+
+# 初始化（容器中自动读取环境变量，本地开发可手动指定）
+sdk.init(api_url="http://localhost:8000", task_id="task-001")
+
+# 上报数据（会自动缓冲、批量上报）
+sdk.insert("articles", {"title": "Hello", "url": "https://example.com"})
+
+# 批量上报
+sdk.insert("articles", [
+    {"title": "A", "url": "https://a.com"},
+    {"title": "B", "url": "https://b.com"},
+])
+
+# 程序结束时自动 flush，也可手动触发
+sdk.flush()
+```
+
+## 配置选项
+
+| 参数 | 环境变量 | 默认值 | 说明 |
+|------|----------|--------|------|
+| `api_url` | `SPIDER_API_URL` | — | 后端地址 |
+| `task_id` | `TASK_ID` | — | 任务 ID |
+| `buffer_size` | — | `20` | 缓冲条数阈值 |
+| `flush_interval` | — | `3.0` | 时间窗口（秒） |
+
+## 异步 API (Asyncio)
+
+由于爬虫开发经常使用 `httpx`, `aiohttp`, `Playwright` 等异步工具，SDK 也提供了原生的 Async 接口。
+推荐使用 `async with` 上下文管理器，离开上下文时将自动触发 flush。
+
+```python
+import asyncio
+from spidermanager_sdk.aio import async_sdk
+
+async def main():
+    # 自动读取环境变量并初始化，退出时自动 flush
+    async with async_sdk:
+        await async_sdk.insert("articles", {"title": "Async Data", "url": "https://a.com"})
+        
+        # 批量插入
+        await async_sdk.insert("articles", [
+            {"title": "B", "url": "https://b.com"},
+            {"title": "C", "url": "https://c.com"},
+        ])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## 架构
+
+```
+sdk.insert()
+    ↓
+FlushBuffer (内存缓冲, 线程安全)
+    ↓  条数阈值 / 定时器触发
+HttpTransport.send_batch()
+    ↓
+POST /api/v1/tasks/data/ingest?task_id=xxx
+```

spidermanager_sdk-0.1.3.dev0/README.md

@@ -0,0 +1,82 @@
+# SpiderManager SDK
+
+极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端，实现爬虫逻辑与数据存储的完全解耦。
+
+## 特性
+
+- **零配置启动**：自动从环境变量 `TASK_ID` / `SPIDER_API_URL` 读取配置
+- **异步缓冲**：内存 buffer 按阈值（20条）或时间窗口（3秒）批量上报，减少网络 IO
+- **容器安全**：通过 `atexit` + `SIGTERM` 双保险，确保 Docker 销毁前 flush 全部数据
+- **最小依赖**：仅依赖 `httpx`，不侵入爬虫业务代码
+
+## 安装
+
+```bash
+pip install spidermanager-sdk
+```
+
+## 快速开始
+
+```python
+from spidermanager_sdk import sdk
+
+# 初始化（容器中自动读取环境变量，本地开发可手动指定）
+sdk.init(api_url="http://localhost:8000", task_id="task-001")
+
+# 上报数据（会自动缓冲、批量上报）
+sdk.insert("articles", {"title": "Hello", "url": "https://example.com"})
+
+# 批量上报
+sdk.insert("articles", [
+    {"title": "A", "url": "https://a.com"},
+    {"title": "B", "url": "https://b.com"},
+])
+
+# 程序结束时自动 flush，也可手动触发
+sdk.flush()
+```
+
+## 配置选项
+
+| 参数 | 环境变量 | 默认值 | 说明 |
+|------|----------|--------|------|
+| `api_url` | `SPIDER_API_URL` | — | 后端地址 |
+| `task_id` | `TASK_ID` | — | 任务 ID |
+| `buffer_size` | — | `20` | 缓冲条数阈值 |
+| `flush_interval` | — | `3.0` | 时间窗口（秒） |
+
+## 异步 API (Asyncio)
+
+由于爬虫开发经常使用 `httpx`, `aiohttp`, `Playwright` 等异步工具，SDK 也提供了原生的 Async 接口。
+推荐使用 `async with` 上下文管理器，离开上下文时将自动触发 flush。
+
+```python
+import asyncio
+from spidermanager_sdk.aio import async_sdk
+
+async def main():
+    # 自动读取环境变量并初始化，退出时自动 flush
+    async with async_sdk:
+        await async_sdk.insert("articles", {"title": "Async Data", "url": "https://a.com"})
+        
+        # 批量插入
+        await async_sdk.insert("articles", [
+            {"title": "B", "url": "https://b.com"},
+            {"title": "C", "url": "https://c.com"},
+        ])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## 架构
+
+```
+sdk.insert()
+    ↓
+FlushBuffer (内存缓冲, 线程安全)
+    ↓  条数阈值 / 定时器触发
+HttpTransport.send_batch()
+    ↓
+POST /api/v1/tasks/data/ingest?task_id=xxx
+```

spidermanager_sdk-0.1.3.dev0/pyproject.toml

@@ -0,0 +1,31 @@
+[build-system]
+requires = ["setuptools>=68.0", "setuptools_scm>=8.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "spidermanager-sdk"
+dynamic = ["version"]
+description = "极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [
+    {name = "SpiderManager Team"},
+]
+dependencies = [
+    "httpx>=0.25.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "pytest-asyncio>=0.21",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools_scm]
+version_file = "src/spidermanager_sdk/_version.py"
+# 强制不显示 +xxx 这种本地版本后缀，确保符合 PyPI 标准
+local_scheme = "no-local-version"

spidermanager_sdk-0.1.3.dev0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/__init__.py

@@ -0,0 +1,24 @@
+"""
+SpiderManager SDK — 极简爬虫数据上报库
+
+典型用法::
+
+    from spidermanager_sdk import sdk
+
+    sdk.init()                               # 自动读取环境变量
+    sdk.insert("articles", {"title": "..."}) # 内部缓冲, 达到阈值后异步上报
+    sdk.flush()                              # 手动强制上报（一般不需要）
+"""
+
+from spidermanager_sdk.client import SpiderManagerClient
+from spidermanager_sdk.aio import async_sdk, AsyncSpiderManagerClient
+
+# ── 全局单例，用户直接操作此对象 ──
+sdk: SpiderManagerClient = SpiderManagerClient()
+
+__all__ = ["sdk", "SpiderManagerClient", "async_sdk", "AsyncSpiderManagerClient"]
+
+try:
+    from spidermanager_sdk._version import version as __version__
+except ImportError:
+    __version__ = "0.0.0"

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/_version.py

@@ -0,0 +1,34 @@
+# file generated by setuptools-scm
+# don't change, don't track in version control
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple
+    from typing import Union
+
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+    COMMIT_ID = Union[str, None]
+else:
+    VERSION_TUPLE = object
+    COMMIT_ID = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+commit_id: COMMIT_ID
+__commit_id__: COMMIT_ID
+
+__version__ = version = '0.1.3.dev0'
+__version_tuple__ = version_tuple = (0, 1, 3, 'dev0')
+
+__commit_id__ = commit_id = 'g2385ac666'

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/aio.py

@@ -0,0 +1,213 @@
+"""
+SpiderManager SDK 异步客户端 (asyncio)
+
+提供原生的 async/await 支持，适用于各类异步爬虫框架 (如 httpx, aiohttp, playwright)。
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+from collections import defaultdict
+from dataclasses import dataclass, field
+from typing import Any, Callable
+
+import httpx
+
+from spidermanager_sdk.buffer import BufferEntry
+from spidermanager_sdk.client import _DEFAULT_BUFFER_SIZE, _DEFAULT_FLUSH_INTERVAL
+from spidermanager_sdk.transport import _INGEST_PATH, _DEFAULT_TIMEOUT
+
+logger = logging.getLogger("spidermanager_sdk.aio")
+
+@dataclass
+class AsyncHttpTransport:
+    api_url: str = ""
+    task_id: str = ""
+    _client: httpx.AsyncClient | None = field(default=None, init=False, repr=False)
+
+    async def open(self) -> None:
+        if self._client is not None:
+            return
+        self._client = httpx.AsyncClient(
+            base_url=self.api_url,
+            timeout=_DEFAULT_TIMEOUT,
+            headers={"Content-Type": "application/json"},
+        )
+
+    async def close(self) -> None:
+        if self._client is not None:
+            try:
+                await self._client.aclose()
+            except Exception:
+                pass
+            finally:
+                self._client = None
+
+    async def send_batch(self, table_name: str, records: list[dict[str, Any]]) -> bool:
+        if not self._client:
+            await self.open()
+            assert self._client is not None
+
+        url = _INGEST_PATH
+        payload: dict[str, Any] = {"table_name": table_name, "data": records}
+        params: dict[str, str] = {"task_id": self.task_id}
+
+        try:
+            response = await self._client.post(url, json=payload, params=params)
+            if response.status_code == 200:
+                return True
+            logger.warning("上报失败 HTTP %d", response.status_code)
+            return False
+        except Exception as exc:
+            logger.error("上报异常: %s", exc)
+            return False
+
+@dataclass
+class AsyncFlushBuffer:
+    max_size: int = 20
+    flush_interval: float = 3.0
+    on_flush: Callable[[list[BufferEntry]], Any] | None = None
+
+    _entries: list[BufferEntry] = field(default_factory=list, init=False, repr=False)
+    _lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False, repr=False)
+    _task: asyncio.Task[None] | None = field(default=None, init=False, repr=False)
+    _started: bool = field(default=False, init=False, repr=False)
+
+    async def start(self) -> None:
+        if self._started:
+            return
+        self._started = True
+        self._task = asyncio.create_task(self._loop())
+
+    async def stop(self) -> None:
+        self._started = False
+        if self._task:
+            self._task.cancel()
+            try:
+                await self._task
+            except asyncio.CancelledError:
+                pass
+            self._task = None
+        await self.flush()
+
+    async def add(self, entry: BufferEntry) -> None:
+        async with self._lock:
+            self._entries.append(entry)
+            current_size = len(self._entries)
+
+        if current_size >= self.max_size:
+            await self.flush()
+
+    async def flush(self) -> None:
+        async with self._lock:
+            if not self._entries:
+                return
+            batch = self._entries.copy()
+            self._entries.clear()
+
+        if self.on_flush:
+            try:
+                res = self.on_flush(batch)
+                if asyncio.iscoroutine(res):
+                    await res
+            except Exception:
+                logger.exception("flush 处理失败")
+
+    async def _loop(self) -> None:
+        while self._started:
+            try:
+                await asyncio.sleep(self.flush_interval)
+                await self.flush()
+            except asyncio.CancelledError:
+                break
+            except Exception:
+                logger.exception("异步缓冲区定时器异常")
+
+class AsyncSpiderManagerClient:
+    def __init__(self) -> None:
+        self._api_url: str = ""
+        self._task_id: str = ""
+        self._initialized: bool = False
+        self._transport: AsyncHttpTransport | None = None
+        self._buffer: AsyncFlushBuffer | None = None
+
+    async def init(
+        self,
+        api_url: str | None = None,
+        task_id: str | None = None,
+        *,
+        buffer_size: int = _DEFAULT_BUFFER_SIZE,
+        flush_interval: float = _DEFAULT_FLUSH_INTERVAL,
+    ) -> None:
+        self._api_url = api_url or os.environ.get("SPIDER_API_URL", "")
+        self._task_id = task_id or os.environ.get("TASK_ID", "")
+        if not self._api_url or not self._task_id:
+            raise ValueError("api_url 或 task_id 未配置 (或环境变量缺失)")
+        self._api_url = self._api_url.rstrip("/")
+
+        self._transport = AsyncHttpTransport(api_url=self._api_url, task_id=self._task_id)
+        await self._transport.open()
+
+        self._buffer = AsyncFlushBuffer(
+            max_size=buffer_size,
+            flush_interval=flush_interval,
+            on_flush=self._handle_flush,
+        )
+        await self._buffer.start()
+        self._initialized = True
+        logger.info("Async SDK 初始化: url=%s task=%s", self._api_url, self._task_id)
+
+    async def insert(self, table_name: str, data: dict[str, Any] | list[dict[str, Any]]) -> None:
+        self._ensure_initialized()
+        if isinstance(data, dict):
+            data = [data]
+        if not data:
+            return
+        assert self._buffer is not None
+        for record in data:
+            await self._buffer.add(BufferEntry(table_name=table_name, data=record))
+
+    async def flush(self) -> None:
+        if self._buffer:
+            await self._buffer.flush()
+
+    async def shutdown(self) -> None:
+        if self._buffer:
+            await self._buffer.stop()
+        if self._transport:
+            await self._transport.close()
+        self._initialized = False
+
+    async def __aenter__(self):
+        # 兼容自动初始化
+        if not self._initialized:
+            await self.init()
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        if self._initialized:
+            await self.shutdown()
+
+    def _ensure_initialized(self) -> None:
+        if not self._initialized:
+            raise RuntimeError("SDK 尚未初始化")
+
+    async def _handle_flush(self, entries: list[BufferEntry]) -> None:
+        if not self._transport:
+            return
+        grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+        for entry in entries:
+            grouped[entry.table_name].append(entry.data)
+        
+        # 内部并发上报多表
+        tasks = []
+        for table_name, records in grouped.items():
+            tasks.append(self._transport.send_batch(table_name, records))
+        
+        if tasks:
+            await asyncio.gather(*tasks)
+
+# 默认全局异步单例
+async_sdk = AsyncSpiderManagerClient()

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/buffer.py

@@ -0,0 +1,126 @@
+"""
+异步缓冲区模块
+
+维护内存 buffer，在满足条件（数据量阈值 或 时间窗口）时
+触发一次批量 HTTP POST，减少爬虫端网络 IO 开销。
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+from dataclasses import dataclass, field
+from typing import Callable
+
+logger = logging.getLogger("spidermanager_sdk.buffer")
+
+
+@dataclass
+class BufferEntry:
+    """
+    单条缓冲记录，对应一次 sdk.insert() 调用。
+    """
+    table_name: str
+    data: dict[str, object]
+
+
+@dataclass
+class FlushBuffer:
+    """
+    线程安全的内存缓冲区。
+
+    Parameters
+    ----------
+    max_size : int
+        缓冲数据条数阈值，达到后触发 flush。
+    flush_interval : float
+        时间窗口（秒），即使未达到 max_size 也触发 flush。
+    on_flush : Callable[[list[BufferEntry]], None]
+        实际的 flush 回调，由 Client 层注入。
+    """
+    max_size: int = 20
+    flush_interval: float = 3.0
+    on_flush: Callable[[list[BufferEntry]], None] | None = None
+
+    # ── 内部状态 ──
+    _entries: list[BufferEntry] = field(default_factory=list, init=False, repr=False)
+    _lock: threading.Lock = field(default_factory=threading.Lock, init=False, repr=False)
+    _timer: threading.Timer | None = field(default=None, init=False, repr=False)
+    _started: bool = field(default=False, init=False, repr=False)
+
+    # ──────────────────────────────────────────────
+    # 公开 API
+    # ──────────────────────────────────────────────
+
+    def start(self) -> None:
+        """启动定时 flush 循环。"""
+        if self._started:
+            return
+        self._started = True
+        self._schedule_timer()
+
+    def stop(self) -> None:
+        """停止定时器并强制 flush 剩余数据。"""
+        self._started = False
+        self._cancel_timer()
+        self.flush()
+
+    def add(self, entry: BufferEntry) -> None:
+        """
+        向缓冲区追加一条记录。
+        如积累到阈值，立即触发 flush。
+        """
+        with self._lock:
+            self._entries.append(entry)
+            current_size = len(self._entries)
+
+        if current_size >= self.max_size:
+            self.flush()
+
+    def flush(self) -> None:
+        """
+        将缓冲区中所有数据取出，交给 on_flush 回调处理。
+        本方法线程安全，可被定时器 / 阈值触发 / atexit 多次调用。
+        """
+        with self._lock:
+            if not self._entries:
+                return
+            batch = self._entries.copy()
+            self._entries.clear()
+
+        if not self.on_flush:
+            logger.warning("on_flush 回调未设置，%d 条数据被丢弃", len(batch))
+            return
+
+        try:
+            self.on_flush(batch)
+        except Exception:
+            logger.exception("flush 回调执行失败，%d 条数据可能丢失", len(batch))
+
+    @property
+    def pending_count(self) -> int:
+        """当前缓冲区中等待 flush 的数据条数。"""
+        with self._lock:
+            return len(self._entries)
+
+    # ──────────────────────────────────────────────
+    # 内部辅助
+    # ──────────────────────────────────────────────
+
+    def _schedule_timer(self) -> None:
+        """注册下一次定时 flush。"""
+        if not self._started:
+            return
+        self._timer = threading.Timer(self.flush_interval, self._on_timer_tick)
+        self._timer.daemon = True
+        self._timer.start()
+
+    def _cancel_timer(self) -> None:
+        if self._timer is not None:
+            self._timer.cancel()
+            self._timer = None
+
+    def _on_timer_tick(self) -> None:
+        """定时器触发：flush 后重新调度。"""
+        self.flush()
+        self._schedule_timer()

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/client.py

@@ -0,0 +1,238 @@
+"""
+SpiderManager SDK 客户端
+
+对外暴露三个核心方法::
+
+    sdk.init(api_url=None, task_id=None)
+    sdk.insert(table_name, data)
+    sdk.flush()
+
+内部协调 Buffer ↔ Transport 之间的数据流转，
+并通过 atexit + signal 保证容器销毁前数据完整性。
+"""
+
+from __future__ import annotations
+
+import atexit
+import logging
+import os
+import signal
+import sys
+from collections import defaultdict
+from typing import Any
+
+from spidermanager_sdk.buffer import BufferEntry, FlushBuffer
+from spidermanager_sdk.transport import HttpTransport
+
+logger = logging.getLogger("spidermanager_sdk.client")
+
+# ── 默认缓冲参数 ──
+_DEFAULT_BUFFER_SIZE: int = 20
+_DEFAULT_FLUSH_INTERVAL: float = 3.0
+
+
+class SpiderManagerClient:
+    """
+    SDK 主客户端。
+
+    职责：
+    1. 管理初始化配置（api_url / task_id）
+    2. 维护内存 flush buffer
+    3. 注册退出钩子确保数据不丢失
+    4. 将用户的 ``insert()`` 调用转化为最终的 HTTP POST
+    """
+
+    def __init__(self) -> None:
+        self._api_url: str = ""
+        self._task_id: str = ""
+        self._initialized: bool = False
+        self._transport: HttpTransport | None = None
+        self._buffer: FlushBuffer | None = None
+        self._atexit_registered: bool = False
+
+    # ──────────────────────────────────────────────
+    # 公开 API
+    # ──────────────────────────────────────────────
+
+    def init(
+        self,
+        api_url: str | None = None,
+        task_id: str | None = None,
+        *,
+        buffer_size: int = _DEFAULT_BUFFER_SIZE,
+        flush_interval: float = _DEFAULT_FLUSH_INTERVAL,
+    ) -> None:
+        """
+        初始化 SDK。
+
+        Parameters
+        ----------
+        api_url : str | None
+            SpiderManager 后端地址。若不传，自动读取环境变量 ``SPIDER_API_URL``。
+        task_id : str | None
+            当前任务 ID。若不传，自动读取环境变量 ``TASK_ID``。
+        buffer_size : int
+            缓冲数据条数阈值，默认 20。
+        flush_interval : float
+            时间窗口（秒），默认 3.0。
+        """
+        # ── 1. 解析配置 ──
+        self._api_url = api_url or os.environ.get("SPIDER_API_URL", "")
+        self._task_id = task_id or os.environ.get("TASK_ID", "")
+
+        if not self._api_url:
+            raise ValueError(
+                "api_url 未指定且环境变量 SPIDER_API_URL 未设置，"
+                "请通过 sdk.init(api_url='...') 或设置环境变量来配置后端地址。"
+            )
+        if not self._task_id:
+            raise ValueError(
+                "task_id 未指定且环境变量 TASK_ID 未设置，"
+                "请通过 sdk.init(task_id='...') 或设置环境变量来配置任务 ID。"
+            )
+
+        # 去除尾部斜杠
+        self._api_url = self._api_url.rstrip("/")
+
+        # ── 2. 初始化传输层 ──
+        self._transport = HttpTransport(api_url=self._api_url, task_id=self._task_id)
+        self._transport.open()
+
+        # ── 3. 初始化缓冲区 ──
+        self._buffer = FlushBuffer(
+            max_size=buffer_size,
+            flush_interval=flush_interval,
+            on_flush=self._handle_flush,
+        )
+        self._buffer.start()
+
+        # ── 4. 注册退出钩子 ──
+        self._register_exit_hooks()
+
+        self._initialized = True
+        logger.info(
+            "SDK 初始化完成: api_url=%s, task_id=%s, buffer_size=%d, interval=%.1fs",
+            self._api_url, self._task_id, buffer_size, flush_interval,
+        )
+
+    def insert(self, table_name: str, data: dict[str, Any] | list[dict[str, Any]]) -> None:
+        """
+        将采集数据提交到缓冲区。
+
+        Parameters
+        ----------
+        table_name : str
+            目标数据表名。
+        data : dict | list[dict]
+            单条或多条数据记录。
+        """
+        self._ensure_initialized()
+
+        if isinstance(data, dict):
+            data = [data]
+
+        if not data:
+            return
+
+        assert self._buffer is not None
+        for record in data:
+            self._buffer.add(BufferEntry(table_name=table_name, data=record))
+
+    def flush(self) -> None:
+        """
+        手动强制将缓冲区中所有数据立即上报到后端。
+        通常在程序结束前调用，或在需要即时可见性时使用。
+        """
+        if self._buffer:
+            self._buffer.flush()
+
+    def shutdown(self) -> None:
+        """
+        优雅关闭 SDK：停止定时器 → flush 剩余数据 → 关闭 HTTP 连接。
+        """
+        logger.info("SDK 正在关闭...")
+        if self._buffer:
+            self._buffer.stop()
+        if self._transport:
+            self._transport.close()
+        self._initialized = False
+        logger.info("SDK 已关闭")
+
+    @property
+    def is_initialized(self) -> bool:
+        return self._initialized
+
+    @property
+    def pending_count(self) -> int:
+        """当前缓冲区中等待上报的数据条数。"""
+        if self._buffer:
+            return self._buffer.pending_count
+        return 0
+
+    # ──────────────────────────────────────────────
+    # 内部方法
+    # ──────────────────────────────────────────────
+
+    def _ensure_initialized(self) -> None:
+        if not self._initialized:
+            raise RuntimeError(
+                "SDK 尚未初始化，请先调用 sdk.init() 进行配置。"
+            )
+
+    def _handle_flush(self, entries: list[BufferEntry]) -> None:
+        """
+        FlushBuffer 的回调。
+        将 entries 按 table_name 分组，逐组上报。
+        """
+        if not self._transport:
+            logger.warning("Transport 未初始化，%d 条数据被丢弃", len(entries))
+            return
+
+        # 按表名分组
+        grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+        for entry in entries:
+            grouped[entry.table_name].append(entry.data)
+
+        for table_name, records in grouped.items():
+            success = self._transport.send_batch(table_name, records)
+            if not success:
+                logger.error(
+                    "表 '%s' 的 %d 条数据上报失败", table_name, len(records),
+                )
+
+    def _register_exit_hooks(self) -> None:
+        """
+        注册 atexit 和 SIGTERM 信号处理，确保容器销毁前 flush 数据。
+        """
+        if self._atexit_registered:
+            return
+
+        # atexit: 正常退出时 flush
+        atexit.register(self.shutdown)
+
+        # SIGTERM: Docker 在 stop 容器时发送此信号
+        # 仅在主线程中注册，且仅支持 POSIX 系统
+        if _can_register_signal():
+            _original_sigterm = signal.getsignal(signal.SIGTERM)
+
+            def _sigterm_handler(signum: int, frame: object) -> None:
+                logger.info("收到 SIGTERM 信号，正在 flush 缓冲数据...")
+                self.shutdown()
+                # 调用原始 handler（如果有）
+                if callable(_original_sigterm):
+                    _original_sigterm(signum, frame)
+                sys.exit(0)
+
+            signal.signal(signal.SIGTERM, _sigterm_handler)
+
+        self._atexit_registered = True
+        logger.debug("退出钩子已注册")
+
+
+def _can_register_signal() -> bool:
+    """
+    判断当前环境是否允许注册信号处理。
+    只有主线程才能注册 signal handler。
+    """
+    import threading
+    return threading.current_thread() is threading.main_thread()

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk/transport.py

@@ -0,0 +1,119 @@
+"""
+HTTP 传输层
+
+负责将序列化后的数据批量 POST 到 SpiderManager 后端。
+使用 httpx 同步客户端，确保在 atexit / 信号处理等
+非 async 上下文中也能可靠发送。
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass, field
+from typing import Any
+
+import httpx
+
+logger = logging.getLogger("spidermanager_sdk.transport")
+
+# 上报接口固定路径
+_INGEST_PATH: str = "/api/v1/tasks/data/ingest"
+
+# 默认超时配置（连接 / 读取 / 写入 / 总计）
+_DEFAULT_TIMEOUT = httpx.Timeout(connect=5.0, read=10.0, write=10.0, pool=30.0)
+
+
+@dataclass
+class HttpTransport:
+    """
+    同步 HTTP 传输客户端。
+
+    Parameters
+    ----------
+    api_url : str
+        SpiderManager 后端根地址，例如 ``http://backend:8000``。
+    task_id : str
+        当前任务 ID，每次请求作为 query 参数传递。
+    """
+    api_url: str = ""
+    task_id: str = ""
+
+    _client: httpx.Client | None = field(default=None, init=False, repr=False)
+
+    # ──────────────────────────────────────────────
+    # 生命周期
+    # ──────────────────────────────────────────────
+
+    def open(self) -> None:
+        """初始化底层 httpx 连接池。"""
+        if self._client is not None:
+            return
+        self._client = httpx.Client(
+            base_url=self.api_url,
+            timeout=_DEFAULT_TIMEOUT,
+            headers={"Content-Type": "application/json"},
+        )
+        logger.debug("HTTP transport opened → %s", self.api_url)
+
+    def close(self) -> None:
+        """关闭连接池，释放资源。"""
+        if self._client is not None:
+            try:
+                self._client.close()
+            except Exception:
+                logger.debug("关闭 HTTP 客户端时出现异常", exc_info=True)
+            finally:
+                self._client = None
+
+    # ──────────────────────────────────────────────
+    # 数据上报
+    # ──────────────────────────────────────────────
+
+    def send_batch(self, table_name: str, records: list[dict[str, Any]]) -> bool:
+        """
+        向后端发送一批数据。
+
+        Parameters
+        ----------
+        table_name : str
+            目标表名。
+        records : list[dict[str, Any]]
+            数据记录列表。
+
+        Returns
+        -------
+        bool
+            上报成功返回 True，否则 False（不抛异常，由调用方决策）。
+        """
+        if not self._client:
+            self.open()
+            assert self._client is not None
+
+        url = _INGEST_PATH
+        payload: dict[str, Any] = {
+            "table_name": table_name,
+            "data": records,
+        }
+        params: dict[str, str] = {"task_id": self.task_id}
+
+        try:
+            response = self._client.post(url, json=payload, params=params)
+            if response.status_code == 200:
+                logger.debug(
+                    "上报成功: table=%s, count=%d", table_name, len(records),
+                )
+                return True
+            else:
+                logger.warning(
+                    "上报失败 HTTP %d: %s", response.status_code, response.text[:200],
+                )
+                return False
+        except httpx.TimeoutException:
+            logger.error("上报超时: table=%s, count=%d", table_name, len(records))
+            return False
+        except httpx.ConnectError as exc:
+            logger.error("连接后端失败 (%s): %s", self.api_url, exc)
+            return False
+        except Exception:
+            logger.exception("上报数据时发生未知异常")
+            return False

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk.egg-info/PKG-INFO

@@ -0,0 +1,97 @@
+Metadata-Version: 2.4
+Name: spidermanager-sdk
+Version: 0.1.3.dev0
+Summary: 极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端
+Author: SpiderManager Team
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: httpx>=0.25.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
+Dynamic: license-file
+
+# SpiderManager SDK
+
+极简 Python SDK，将爬虫采集数据通过 HTTP 异步中转至 SpiderManager 后端，实现爬虫逻辑与数据存储的完全解耦。
+
+## 特性
+
+- **零配置启动**：自动从环境变量 `TASK_ID` / `SPIDER_API_URL` 读取配置
+- **异步缓冲**：内存 buffer 按阈值（20条）或时间窗口（3秒）批量上报，减少网络 IO
+- **容器安全**：通过 `atexit` + `SIGTERM` 双保险，确保 Docker 销毁前 flush 全部数据
+- **最小依赖**：仅依赖 `httpx`，不侵入爬虫业务代码
+
+## 安装
+
+```bash
+pip install spidermanager-sdk
+```
+
+## 快速开始
+
+```python
+from spidermanager_sdk import sdk
+
+# 初始化（容器中自动读取环境变量，本地开发可手动指定）
+sdk.init(api_url="http://localhost:8000", task_id="task-001")
+
+# 上报数据（会自动缓冲、批量上报）
+sdk.insert("articles", {"title": "Hello", "url": "https://example.com"})
+
+# 批量上报
+sdk.insert("articles", [
+    {"title": "A", "url": "https://a.com"},
+    {"title": "B", "url": "https://b.com"},
+])
+
+# 程序结束时自动 flush，也可手动触发
+sdk.flush()
+```
+
+## 配置选项
+
+| 参数 | 环境变量 | 默认值 | 说明 |
+|------|----------|--------|------|
+| `api_url` | `SPIDER_API_URL` | — | 后端地址 |
+| `task_id` | `TASK_ID` | — | 任务 ID |
+| `buffer_size` | — | `20` | 缓冲条数阈值 |
+| `flush_interval` | — | `3.0` | 时间窗口（秒） |
+
+## 异步 API (Asyncio)
+
+由于爬虫开发经常使用 `httpx`, `aiohttp`, `Playwright` 等异步工具，SDK 也提供了原生的 Async 接口。
+推荐使用 `async with` 上下文管理器，离开上下文时将自动触发 flush。
+
+```python
+import asyncio
+from spidermanager_sdk.aio import async_sdk
+
+async def main():
+    # 自动读取环境变量并初始化，退出时自动 flush
+    async with async_sdk:
+        await async_sdk.insert("articles", {"title": "Async Data", "url": "https://a.com"})
+        
+        # 批量插入
+        await async_sdk.insert("articles", [
+            {"title": "B", "url": "https://b.com"},
+            {"title": "C", "url": "https://c.com"},
+        ])
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## 架构
+
+```
+sdk.insert()
+    ↓
+FlushBuffer (内存缓冲, 线程安全)
+    ↓  条数阈值 / 定时器触发
+HttpTransport.send_batch()
+    ↓
+POST /api/v1/tasks/data/ingest?task_id=xxx
+```

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+.gitignore
+LICENSE
+README.md
+pyproject.toml
+.github/workflows/publish.yml
+src/spidermanager_sdk/__init__.py
+src/spidermanager_sdk/_version.py
+src/spidermanager_sdk/aio.py
+src/spidermanager_sdk/buffer.py
+src/spidermanager_sdk/client.py
+src/spidermanager_sdk/transport.py
+src/spidermanager_sdk.egg-info/PKG-INFO
+src/spidermanager_sdk.egg-info/SOURCES.txt
+src/spidermanager_sdk.egg-info/dependency_links.txt
+src/spidermanager_sdk.egg-info/requires.txt
+src/spidermanager_sdk.egg-info/top_level.txt

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk.egg-info/requires.txt

@@ -0,0 +1,5 @@
+httpx>=0.25.0
+
+[dev]
+pytest>=7.0
+pytest-asyncio>=0.21

spidermanager_sdk-0.1.3.dev0/src/spidermanager_sdk.egg-info/top_level.txt

@@ -0,0 +1 @@
+spidermanager_sdk