redis-http-client 0.1.1__tar.gz

redis_http_client-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 victor
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

redis_http_client-0.1.1/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: redis-http-client
+Version: 0.1.1
+Summary: A Redis client over HTTP supporting both sync (requests) and async (aiohttp) modes.
+Author-email: victor <xianyu.wu@innodealing.com>
+License-Expression: MIT
+Project-URL: Homepage, http://git.innodealing.cn/dm_spider_service/redis-http-client.git
+Project-URL: Source, http://git.innodealing.cn/dm_spider_service/redis-http-client.git
+Project-URL: Issues, http://git.innodealing.cn/dm_spider_service/redis-http-client.git/issues
+Keywords: redis,http,client,asyncio,requests,aiohttp
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: urllib3>=2.0.0
+Requires-Dist: loguru>=0.7.0
+Provides-Extra: dev
+Requires-Dist: build>=1.2.1; extra == "dev"
+Requires-Dist: twine>=5.1.0; extra == "dev"
+Dynamic: license-file
+
+# redis-http-client
+
+A lightweight Redis client over HTTP, supporting both async (`aiohttp`) and sync (`requests`) execution modes.
+
+## Features
+
+- Async mode based on `aiohttp`
+- Sync mode based on `requests`
+- Dynamic Redis command proxy via `__getattr__`
+- Pipeline support (`redis_pipeline_async` / `redis_pipeline_sync`)
+- Built-in retry and connection-pool handling
+
+## Installation
+
+```bash
+pip install redis-http-client
+```
+
+## Quick Start
+
+### Async
+
+```python
+import asyncio
+from redis_http_client import RedisHttpClient
+
+
+async def main():
+    client = RedisHttpClient(
+        spider_redis_manager_service_url="http://127.0.0.1:8000",
+        mode="async",
+    )
+
+    result = await client.get("my_key")
+    print(result)
+
+    await client.close()
+
+
+asyncio.run(main())
+```
+
+### Sync
+
+```python
+from redis_http_client import RedisHttpClient
+
+client = RedisHttpClient(
+    spider_redis_manager_service_url="http://127.0.0.1:8000",
+    mode="sync",
+)
+
+result = client.get("my_key")
+print(result)
+
+# sync mode reuses internal requests session; close when done
+import asyncio
+asyncio.run(client.close())
+```
+
+### Pipeline
+
+```python
+commands = [
+    {"command": "GET", "args": ["k1"], "kwargs": {}},
+    {"command": "GET", "args": ["k2"], "kwargs": {}},
+]
+
+# async
+# results = await client.redis_pipeline_async(commands)
+
+# sync
+# results = client.redis_pipeline_sync(commands)
+```
+
+## Build
+
+```bash
+python -m pip install --upgrade build twine
+python -m build
+```
+
+Build outputs are generated in `dist/`.
+
+## Upload To PyPI
+
+### 1. TestPyPI (recommended first)
+
+```bash
+twine check dist/*
+twine upload -r testpypi dist/*
+```
+
+### 2. PyPI
+
+```bash
+twine upload dist/*
+```
+
+You can use API token authentication:
+
+- username: `__token__`
+- password: `pypi-xxxx...`
+
+## Versioning
+
+Current version: `0.1.1`
+
+When releasing a new version:
+
+1. Update `version` in `pyproject.toml`
+2. Update `__version__` in `redis_http_client/__init__.py`
+3. Rebuild and upload
+
+## License
+
+MIT

redis_http_client-0.1.1/README.md

@@ -0,0 +1,117 @@
+# redis-http-client
+
+A lightweight Redis client over HTTP, supporting both async (`aiohttp`) and sync (`requests`) execution modes.
+
+## Features
+
+- Async mode based on `aiohttp`
+- Sync mode based on `requests`
+- Dynamic Redis command proxy via `__getattr__`
+- Pipeline support (`redis_pipeline_async` / `redis_pipeline_sync`)
+- Built-in retry and connection-pool handling
+
+## Installation
+
+```bash
+pip install redis-http-client
+```
+
+## Quick Start
+
+### Async
+
+```python
+import asyncio
+from redis_http_client import RedisHttpClient
+
+
+async def main():
+    client = RedisHttpClient(
+        spider_redis_manager_service_url="http://127.0.0.1:8000",
+        mode="async",
+    )
+
+    result = await client.get("my_key")
+    print(result)
+
+    await client.close()
+
+
+asyncio.run(main())
+```
+
+### Sync
+
+```python
+from redis_http_client import RedisHttpClient
+
+client = RedisHttpClient(
+    spider_redis_manager_service_url="http://127.0.0.1:8000",
+    mode="sync",
+)
+
+result = client.get("my_key")
+print(result)
+
+# sync mode reuses internal requests session; close when done
+import asyncio
+asyncio.run(client.close())
+```
+
+### Pipeline
+
+```python
+commands = [
+    {"command": "GET", "args": ["k1"], "kwargs": {}},
+    {"command": "GET", "args": ["k2"], "kwargs": {}},
+]
+
+# async
+# results = await client.redis_pipeline_async(commands)
+
+# sync
+# results = client.redis_pipeline_sync(commands)
+```
+
+## Build
+
+```bash
+python -m pip install --upgrade build twine
+python -m build
+```
+
+Build outputs are generated in `dist/`.
+
+## Upload To PyPI
+
+### 1. TestPyPI (recommended first)
+
+```bash
+twine check dist/*
+twine upload -r testpypi dist/*
+```
+
+### 2. PyPI
+
+```bash
+twine upload dist/*
+```
+
+You can use API token authentication:
+
+- username: `__token__`
+- password: `pypi-xxxx...`
+
+## Versioning
+
+Current version: `0.1.1`
+
+When releasing a new version:
+
+1. Update `version` in `pyproject.toml`
+2. Update `__version__` in `redis_http_client/__init__.py`
+3. Rebuild and upload
+
+## License
+
+MIT

redis_http_client-0.1.1/pyproject.toml

@@ -0,0 +1,53 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "redis-http-client"
+version = "0.1.1"
+description = "A Redis client over HTTP supporting both sync (requests) and async (aiohttp) modes."
+authors = [
+  { name="victor", email="xianyu.wu@innodealing.com" }
+]
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+keywords = ["redis", "http", "client", "asyncio", "requests", "aiohttp"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Operating System :: OS Independent",
+  "Topic :: Database",
+  "Topic :: Software Development :: Libraries :: Python Modules"
+]
+
+dependencies = [
+  "aiohttp>=3.9.0",
+  "requests>=2.31.0",
+  "urllib3>=2.0.0",
+  "loguru>=0.7.0"
+]
+
+[project.urls]
+Homepage = "http://git.innodealing.cn/dm_spider_service/redis-http-client.git"
+Source = "http://git.innodealing.cn/dm_spider_service/redis-http-client.git"
+Issues = "http://git.innodealing.cn/dm_spider_service/redis-http-client.git/issues"
+
+[project.optional-dependencies]
+dev = [
+  "build>=1.2.1",
+  "twine>=5.1.0"
+]
+
+[tool.setuptools]
+include-package-data = true
+license-files = ["LICENSE"]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["redis_http_client*"]

redis_http_client-0.1.1/redis_http_client/__init__.py

@@ -0,0 +1,4 @@
+from .client import RedisHttpClient
+
+__all__ = ["RedisHttpClient"]
+__version__ = "0.1.1"

redis_http_client-0.1.1/redis_http_client/client.py

@@ -0,0 +1,443 @@
+import asyncio
+import json
+import time
+import traceback
+import aiohttp
+import requests
+from urllib3.util import Retry
+from requests.adapters import HTTPAdapter
+from redis_http_client.other_logger import other_logger
+from aiohttp.client_exceptions import (
+    ServerDisconnectedError, ClientOSError, ClientPayloadError, ClientConnectionError
+)
+
+
+class RedisHttpClient:
+    def __init__(self,
+                 spider_redis_manager_service_url,
+                 base_url: str = None,
+                 mode: str = "async",
+                 local=False,
+                 force_close: bool = False,
+                 limit: int = 1000
+                 ):
+        """
+        mode: "async" 使用 aiohttp, "sync" 使用 requests
+        decode_responses: True → 返回 JSON 结果 (字符串)
+                          False → 返回原始 bytes
+        """
+        if local:
+            self.base_url = base_url or f"{spider_redis_manager_service_url}/redis/exec/local"
+            self.pipeline_url = f"{spider_redis_manager_service_url}/redis/pipeline/local"
+        else:
+            self.base_url = base_url or f"{spider_redis_manager_service_url}/redis/exec"
+            self.pipeline_url = f"{spider_redis_manager_service_url}/redis/pipeline"
+        if mode not in ("async", "sync"):
+            raise ValueError("mode must be 'async' or 'sync'")
+        self.mode = mode
+        self._fail_count = 0
+        # 异步 session 延迟初始化
+        self.session = None
+        self._lock = asyncio.Lock()
+        self._session_created_at = 0
+        # 同步 session 立即初始化 + 连接池
+        if self.mode == "sync":
+            self.sync_session = requests.Session()
+            retry_strategy = Retry(
+                total=3,
+                backoff_factor=0.3,
+                status_forcelist=[500, 502, 503, 504],
+                allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
+            )
+            adapter = HTTPAdapter(
+                pool_connections=50,  # 连接池大小
+                pool_maxsize=1000,  # 单个 host 最大连接数
+                max_retries=retry_strategy,
+            )
+            self.sync_session.mount("http://", adapter)
+            self.sync_session.mount("https://", adapter)
+        else:
+            self.sync_session = None
+        # 记录session的创建时间
+        self._session_created_at = time.time()
+        # 计数 每隔xx秒输出一次日志
+        self._request_count = 0
+        self.max_tries = 5
+        self.force_close = force_close
+        self.limit = limit
+
+    async def _init_session(self, force=False):
+        """延迟初始化 aiohttp session，带锁"""
+        # 保证只有一个协程能创建/关闭 session
+        if force or self.session is None or self.session.closed:
+            async with self._lock:
+                if force or self.session is None or self.session.closed:
+                    connector = aiohttp.TCPConnector(
+                        limit=self.limit,
+                        ttl_dns_cache=60,
+                        keepalive_timeout=50,
+                        enable_cleanup_closed=True,
+                        force_close=self.force_close,
+                        ssl=False,
+                    )
+                    timeout = aiohttp.ClientTimeout(
+                        total=120,  # 容忍整体请求更久
+                        connect=10,  # 连接最长 10s
+                        sock_connect=10,
+                        sock_read=100  # Redis pipeline 大响应时给足时间
+                    )
+                    self.session = aiohttp.ClientSession(connector=connector, timeout=timeout)
+                    self._session_created_at = time.time()
+                    other_logger.info("[RedisHttpClient] 新建 aiohttp session")
+
+    async def _recreate_session(self):
+        # 冷却防抖（建议 >=10s）
+        if time.time() - self._session_created_at < 10:
+            other_logger.warning("[RedisHttpClient] recreate skipped (cooldown)")
+            return
+        try:
+            # 锁内：原子“新建并替换”
+            async with self._lock:
+                old = self.session
+                connector = aiohttp.TCPConnector(
+                    limit=self.limit,
+                    ttl_dns_cache=60,
+                    keepalive_timeout=50,
+                    enable_cleanup_closed=True,
+                    force_close=self.force_close,
+                    ssl=False,
+                )
+                timeout = aiohttp.ClientTimeout(
+                    total=120,  # 容忍整体请求更久
+                    connect=10,  # 连接最长 10s
+                    sock_connect=10,
+                    sock_read=100  # Redis pipeline 大响应时给足时间
+                )
+                new_session = aiohttp.ClientSession(connector=connector, timeout=timeout)
+                self.session = new_session
+                self._session_created_at = time.time()
+                other_logger.warning("[RedisHttpClient] session swapped")
+        except Exception:
+            other_logger.error(traceback.format_exc())
+            return
+        # 🔕 锁外：**延迟**关闭旧 session，给在途请求留出时间
+
+        if old and not old.closed:
+            asyncio.create_task(self._delayed_close(old))
+
+    async def _delayed_close(self, session, delay=1.0):
+        """延迟关闭 session，给正在使用的请求留出时间"""
+        try:
+            other_logger.info("[RedisHttpClient] start close old session ")
+            # 防止多次异步触发 _delayed_close() 重复关闭。
+            if not session or getattr(session, "_closing", False):
+                return
+            setattr(session, "_closing", True)
+            await asyncio.sleep(delay)
+            if session and not session.closed:
+                await session.close()
+                other_logger.info("[RedisHttpClient]  old session closed !")
+        except Exception:
+            other_logger.warning(traceback.format_exc())
+
+    async def _check_pool_health(self):
+        other_logger.info("[RedisHttpClient] 检查 redis pool health")
+        if not self.session or not self.session.connector:
+            return
+        dead_conns = 0
+        self.session.connector._cleanup_closed()
+        for key, conns in list(self.session.connector._conns.items()):
+            for conn, _ in list(conns):
+                if not conn:
+                    continue
+                transport = getattr(conn, "transport", None)
+                if not transport:
+                    # 没 transport -> 真死连接
+                    continue
+                if transport.is_closing():
+                    # socket 已关闭
+                    conn.close()
+                    dead_conns += 1
+        if dead_conns > 0:
+            other_logger.warning(f"[RedisHttpClient] 清理 {dead_conns} 个死连接")
+
+    async def close(self):
+        """关闭 aiohttp/requests session"""
+        if self.session and not self.session.closed:
+            await self.session.close()
+        if self.sync_session:
+            self.sync_session.close()
+
+    # ------------------ async 部分 ------------------
+    async def _fetch_async(self, url, method="GET", data=None, headers=None, params=None, resp_type="json"):
+        await self._init_session()
+
+        start_time = time.time()
+
+        try:
+            for attempt in range(self.max_tries):
+                session = self.session
+                if session is None or session.closed:
+                    other_logger.warning("[RedisHttpClient] session closed before request, re-init...")
+                    await self._recreate_session()
+                    session = self.session
+                self._request_count += 1
+                if self._request_count % 500 == 0:  # 每500次请求输出一次
+                    other_logger.info(
+                        f"[RedisHttpClient]链接池: 当前正在被使用的连接数量: {len(session.connector._acquired)} "
+                        f"连接池中当前可用的空闲连接数量: {sum(len(conn_list) for conn_list in session.connector._conns.values())} "
+                        f" 连接池允许的最大连接总数: {session.connector.limit} "
+                    )
+                if self._request_count % 200 == 0:
+                    await self._check_pool_health()
+                try:
+                    async with session.request(method, url, headers=headers, params=params, json=data,
+                                               ssl=False) as response:
+                        if response.status != 200:
+                            text = await response.text()
+                            # 服务端400但包含Redis错误 → 说明是可恢复的
+                            if response.status == 400 and any(
+                                    k in text.lower() for k in
+                                    ["redis", "connection", "timeout", "broken", "db", "error"]
+                            ):
+                                other_logger.warning(
+                                    f"[RedisHttpClient] retryable 400: {text[:200]} ({attempt + 1}/{self.max_tries})"
+                                )
+                                await asyncio.sleep(0.3 * (attempt + 1))
+                                continue
+                            # 其他400、403、404不重试
+                            raise Exception(f"[HTTP {response.status}] {text}")
+                        if resp_type == "json":
+                            result = await response.json()
+                        elif resp_type == "text":
+                            result = await response.text()
+                        elif resp_type == "bytes":
+                            result = await response.read()
+                        duration = (time.time() - start_time) * 1000
+                        # other_logger.info(f"[RedisHttpClient] {url} ok, cost={duration:.2f}ms")
+                        return result
+                except RuntimeError as e:
+                    if "Session is closed" in str(e):
+                        other_logger.warning(
+                            f"[RedisHttpClient] session closed mid-flight ({attempt + 1}/3); recreating")
+                        await self._recreate_session()
+                        await asyncio.sleep(0.2)
+                        continue
+                    raise
+
+                except asyncio.TimeoutError:
+                    other_logger.warning(f"[RedisHttpClient] TimeoutError ({attempt + 1}/{self.max_tries}): {url}")
+                    await asyncio.sleep(0.3 * (attempt + 1))
+                    continue
+                except asyncio.CancelledError:
+                    other_logger.warning(f"[RedisHttpClient] CancelledError  ({attempt + 1}/{self.max_tries}): {url}")
+                    raise  # 不要吞掉取消异常
+                except (ServerDisconnectedError, ClientOSError, ClientConnectionError, ClientPayloadError) as e:
+                    other_logger.warning(
+                        f"[RedisHttpClient] 'ServerDisconnectedError, ClientOSError, ClientConnectionError, ClientPayloadError'"
+                        f" ({attempt + 1}/{self.max_tries}): {url}")
+                    await asyncio.sleep(0.3 * (attempt + 1))
+                    if attempt == self.max_tries - 2:
+                        await self._recreate_session()
+                    else:
+                        await self._check_pool_health()
+                    continue
+
+            raise Exception(f"[aiohttp] Failed after {self.max_tries} retries, url={url}")
+        except Exception as e:
+            other_logger.error(f'[RedisHttpClient] Unhandled error: {traceback.format_exc()}')
+            return None
+
+    # ------------------ sync 部分 ------------------
+    def _fetch_sync(self, url: str, method: str = "GET", data=None, headers=None,
+                    params=None, resp_type: str = "json"):
+        if not self.sync_session:
+            raise RuntimeError("Sync session not initialized")
+        try:
+            for i in range(3):
+                try:
+                    resp = self.sync_session.request(
+                        method, url,
+                        headers=headers,
+                        params=params,
+                        json=data,
+                        timeout=(10, 60),  # 连接10s，读取60s（你原来60,60也行）
+                        verify=False
+                    )
+
+                    # 关键：先把非 200 的情况打印出来
+                    if resp.status_code != 200:
+                        ct = resp.headers.get("Content-Type", "")
+                        other_logger.error(
+                            f"[RedisHttpClient][sync] HTTP {resp.status_code} url={url} "
+                            f"Content-Type={ct} body={resp.text[:500]!r}"
+                        )
+                        # 让 Retry/上层逻辑决定是否重试
+                        time.sleep(0.3 * (i + 1))
+                        continue
+
+                    if resp_type == "json":
+                        # 关键：body 为空直接返回 None（避免 JSONDecodeError）
+                        if not resp.content:
+                            other_logger.error(
+                                f"[RedisHttpClient][sync] empty body url={url} headers={dict(resp.headers)}"
+                            )
+                            return None
+
+                        ct = resp.headers.get("Content-Type", "")
+                        if "application/json" not in ct.lower():
+                            other_logger.error(
+                                f"[RedisHttpClient][sync] non-json content url={url} "
+                                f"Content-Type={ct} body={resp.text[:500]!r}"
+                            )
+                            return None
+
+                        return resp.json()
+
+                    elif resp_type == "text":
+                        return resp.text
+                    elif resp_type == "bytes":
+                        return resp.content
+                    else:
+                        raise ValueError("Invalid resp_type")
+
+                except Exception as e:
+                    other_logger.warning(f"[RedisHttpClient][sync] retry {i + 1}/3 {type(e).__name__}: {e}")
+                    other_logger.warning(f"[RedisHttpClient][sync] error {traceback.format_exc()}")
+                    try:
+                        other_logger.info(
+                            f"status={resp.status_code} "
+                            f"ct={resp.headers.get('Content-Type')} "
+                            f"ce={resp.headers.get('Content-Encoding')} "
+                            f"len={len(resp.content)} "
+                            f"first16={resp.content[:16]!r}"
+                            f"data={data}"
+                        )
+                    except Exception as e:
+                        pass
+                    # 关键：清理可能的“半死连接”
+                    time.sleep(0.3 * (i + 1))
+                    continue
+
+        except Exception as e:
+            other_logger.info('[RedisHttpClient] unhandled error: {}'.format(traceback.format_exc()))
+            return {}
+        return None
+
+    # ------------------ 动态方法代理 ------------------
+    def __getattr__(self, command: str):
+        if self.mode == "async":
+            async def wrapper(*args, **kwargs):
+                try:
+                    return await self._request_redis_exec_async(command, list(args), kwargs)
+                except Exception as e:
+                    other_logger.error(f"Redis async command error [{command}]: {traceback.format_exc()}")
+                    return None  # 返回 None 或自定义默认值，防止抛出异常
+
+            return wrapper
+        else:
+            def wrapper(*args, **kwargs):
+                try:
+                    return self._request_redis_exec_sync(command, list(args), kwargs)
+                except Exception as e:
+                    other_logger.error(f"Redis sync command error [{command}]: {traceback.format_exc()}")
+                    return None
+
+            return wrapper
+
+    def _maybe_json_decode(self, obj):
+        """安全地对嵌套的 JSON 字符串进行一次解包"""
+        if isinstance(obj, str):
+            s = obj.strip()
+            if s and s[0] in '[{' and s[-1] in ']}':
+                try:
+                    return json.loads(s)
+                except Exception:
+                    return obj
+        return obj
+
+    # ------------------ redis 执行方法 ------------------
+    async def _request_redis_exec_async(self, command: str, args_list: list, kwargs: dict = None):
+        headers = {"accept": "application/json", "Content-Type": "application/json"}
+        data = {
+            "command": command,
+            "args": args_list,
+            "kwargs": kwargs or {}
+        }
+        result = await self._real_exec(data, headers)
+        return result
+
+    def _request_redis_exec_sync(self, command: str, args_list: list, kwargs: dict = None):
+        headers = {"accept": "application/json", "Content-Type": "application/json", "Connection": "close"}
+        data = {
+            "command": command,
+            "args": args_list,
+            "kwargs": kwargs or {}
+        }
+
+        result = self._fetch_sync(
+            self.base_url, "POST", data=data, headers=headers,
+            resp_type="json"
+        )
+        return result
+
+    async def redis_pipeline_async(self, commands: list):
+        """
+        commands: [
+          {"command": "GETBIT", "args": ["my_bloom_key", 1234], "kwargs": {}},
+          {"command": "GETBIT", "args": ["my_bloom_key", 4567], "kwargs": {}}
+        ]
+        """
+        headers = {"accept": "application/json", "Content-Type": "application/json"}
+        data = {"commands": []}
+        for cmd in commands:
+            data["commands"].append({
+                "command": cmd["command"],
+                "args": cmd.get("args", []),
+                "kwargs": cmd.get("kwargs", {})
+            })
+        result = await  self._real_pipeline(data, headers)
+        if len(commands) > 1:
+            converted = []
+            for c, r in zip(commands, result):
+                converted.append(self._maybe_json_decode(r))
+            return converted
+        else:
+            maybe = self._maybe_json_decode(result[0])
+            if isinstance(maybe, (list, dict)):
+                result = maybe
+            return result
+
+    def redis_pipeline_sync(self, commands: list):
+        headers = {"accept": "application/json", "Content-Type": "application/json", "Connection": "close"}
+        data = {"commands": []}
+        for cmd in commands:
+            data["commands"].append({
+                "command": cmd["command"],
+                "args": cmd.get("args", []),
+                "kwargs": cmd.get("kwargs", {})
+            })
+
+        result = self._fetch_sync(
+            self.pipeline_url, "POST", data=data, headers=headers, resp_type="json"
+        )
+        if len(commands) > 1:
+            converted = []
+            for c, r in zip(commands, result):
+                converted.append(self._maybe_json_decode(r))
+            return converted
+        else:
+            maybe = self._maybe_json_decode(result[0])
+            if isinstance(maybe, (list, dict)):
+                result = maybe
+            return result
+
+    async def _real_pipeline(self, data, headers):
+        return await self._fetch_async(
+            self.pipeline_url, "POST", data=data, headers=headers, resp_type="json"
+        )
+
+    async def _real_exec(self, data, headers):
+        return await self._fetch_async(
+            self.base_url, "POST", data=data, headers=headers, resp_type="json"
+        )

redis_http_client-0.1.1/redis_http_client/other_logger.py

@@ -0,0 +1,22 @@
+class OtherLogger(object):
+    def __init__(self, types='[other_logger]'):
+        import sys
+        from loguru import logger
+        logger.remove()
+        logger.add(sys.stderr, format="{time:YYYY-MM-DD HH:mm:ss} {extra[types]} {level}: {message}", enqueue=True)
+        self.logger = logger.bind(types=types)
+
+    def info(self, message):
+        self.logger.info(message)
+
+    def error(self, message):
+        self.logger.error(message)
+
+    def warning(self, message):
+        self.logger.warning(message)
+
+    def debug(self, message):
+        self.logger.debug(message)
+
+
+other_logger = OtherLogger()

redis_http_client-0.1.1/redis_http_client/sub_run.py

@@ -0,0 +1,55 @@
+import asyncio, threading, os, time
+
+'''额外起一个子线程，用于跑其他的loop
+为了解决scrapy线程多层loop嵌套的情况。，loop用的过多，没有剩余的loop去处理redis-http-client的请求，导致超时
+'''
+
+
+class SubLoopRunner:
+    def __init__(self):
+        self.loop = None
+
+    def _build(self):
+        self.pid = os.getpid()
+        self.loop = asyncio.new_event_loop()
+        self.thread = threading.Thread(target=self._run, daemon=True, name=f"SubLoop-{self.pid}")
+        self.thread.start()
+        # 等到 loop 真跑起来
+        for _ in range(100):
+            """避免loop未被创建"""
+            if self.loop.is_running():
+                break
+            time.sleep(0.1)
+        if not self.loop.is_running():
+            raise RuntimeError("SubLoop failed to start")
+
+    def _ensure_alive(self):
+        """不再init时初始化loop，避免 在主进程创建 loop，否则爬虫启动后，主进程fork后的子进程无法拿到fork后loop线程，只能拿到subloop_runner对象，
+
+        """
+        if self.loop is None or os.getpid() != self.pid or self.loop.is_closed():
+            try:
+                # 尝试优雅停止旧 loop（若在子进程调用，这里多半是 no-op）
+                self.loop.call_soon_threadsafe(self.loop.stop)
+            except Exception:
+                pass
+            self._build()
+
+    def _run(self):
+        """
+        在子线程中运行事件循环的主循环（loop.run_forever()），让 loop 真正处理异步任务
+        就相当于是「子线程中的 asyncio 事件调度器」
+
+        """
+        asyncio.set_event_loop_policy(asyncio.DefaultEventLoopPolicy())
+
+        asyncio.set_event_loop(self.loop)
+        self.loop.run_forever()
+
+    def run_coroutine(self, coro):
+        """run_coroutine_threadsafe 此方法线程安全"""
+        self._ensure_alive()
+        return asyncio.run_coroutine_threadsafe(coro, self.loop)
+
+
+subloop_runner = SubLoopRunner()

redis_http_client-0.1.1/redis_http_client/to_method.py

@@ -0,0 +1,242 @@
+import json
+import os
+import platform
+import aiohttp
+
+
+def to_str(value):
+    if value is None:
+        return None
+    return str(value)
+
+
+def to_int(value):
+    if value is None:
+        return None
+    return int(value)
+
+
+def to_bool(value):
+    if value is None:
+        return None
+    if isinstance(value, bool):
+        return value
+    if isinstance(value, (int, str)):
+        return str(value).lower() in ("1", "true", "ok")
+    return False
+
+
+def to_set(value):
+    if value is None:
+        return set()
+    return set(value)
+
+
+def to_list(value):
+    if value is None:
+        return []
+    return list(value)
+
+
+def _unpack_cursor_items(value, default_items):
+    if not value:
+        return 0, default_items
+    cursor = value[0]
+    try:
+        cursor = int(cursor)
+    except Exception:
+        # 有的实现返回的是字符串 '0'
+        cursor = int(str(cursor))
+    items = value[1] if len(value) > 1 else default_items
+    return cursor, items
+
+
+def to_scan(value):
+    # SCAN -> (cursor, [keys...])
+    cursor, items = _unpack_cursor_items(value, [])
+    # 确保是 list
+    return cursor, list(items) if items is not None else []
+
+
+def to_sscan(value):
+    # SSCAN -> (cursor, [members...])
+    cursor, items = _unpack_cursor_items(value, [])
+    return cursor, list(items) if items is not None else []
+
+
+def to_hscan(value):
+    # HSCAN -> (cursor, {field: value}) 或 (cursor, [f1, v1, f2, v2, ...])
+    cursor, items = _unpack_cursor_items(value, {})
+    if isinstance(items, dict):
+        return cursor, items
+    # 兼容交替列表形式
+    items = list(items) if items is not None else []
+    if len(items) % 2 != 0:
+        # 长度异常时，尽力配对，多出来的最后一个丢弃或给 None（这里选择丢弃）
+        items = items[:len(items) // 2 * 2]
+    return cursor, dict(zip(items[0::2], items[1::2]))
+
+
+def to_zscan(value):
+    # ZSCAN -> (cursor, [(member, score), ...]) 或 (cursor, [member, score, ...])
+    cursor, items = _unpack_cursor_items(value, [])
+    # 常见两种格式：list-of-tuples 或 扁平交替列表
+    # 统一成 list[(member, float(score))]
+    if isinstance(items, dict):
+        # 极少数实现可能给 dict；统一转 list
+        pairs = [(m, float(s)) for m, s in items.items()]
+        return cursor, pairs
+
+    items = list(items) if items is not None else []
+    if items and not isinstance(items[0], (list, tuple)):
+        # 扁平交替：[member, score, member, score, ...]
+        if len(items) % 2 != 0:
+            items = items[:len(items) // 2 * 2]
+        it = iter(items)
+        pairs = []
+        for m, s in zip(it, it):
+            try:
+                pairs.append((m, float(s)))
+            except Exception:
+                # 分数解析失败时保持原样
+                pairs.append((m, s))
+        return cursor, pairs
+    else:
+        # 已经是 [(member, score), ...] 形式
+        pairs = []
+        for m, s in items:
+            try:
+                pairs.append((m, float(s)))
+            except Exception:
+                pairs.append((m, s))
+        return cursor, pairs
+
+
+def to_eval(value):
+    if isinstance(value, str):
+        try:
+            return json.loads(value)
+        except Exception:
+            return value
+    return value
+
+
+def to_zrange(value):
+    # 可能是 list[str] / list[ [member, score], ... ] / 扁平 ["m1", 1.0, "m2", 2.0] / 或者单个字符串
+    if value is None:
+        return []
+
+    # 已是字符串（且没被 _maybe_json_decode 解开），按单元素处理，避免被 list() 拆字符
+    if isinstance(value, str):
+        return [value]
+
+    if isinstance(value, list):
+        # 形如 [["a", 1.0], ["b", 2.0]]
+        if all(isinstance(i, list) and len(i) == 2 for i in value):
+            return [(str(k), float(v)) for k, v in value]
+        # 形如 ["a", 1.0, "b", 2.0]
+        if len(value) % 2 == 0 and any(isinstance(v, (int, float)) for v in value[1::2]):
+            return list(zip(map(str, value[0::2]), map(float, value[1::2])))
+        # 纯成员列表 ["a","b","c"]
+        return [str(x) for x in value]
+
+    # 其它情况兜底成列表
+    return [value]
+
+
+REDIS_TYPE_MAP = {
+    # === String 相关 ===
+    "get": to_str,  # key 不存在 -> None
+    "set": to_bool,  # "OK" -> True
+    "incr": to_int,  # 自增后的值
+    "incrby": to_int,  # 自增后的值
+    "getbit": to_int,  # 返回 0 / 1
+    "setbit": to_int,
+    "strlen": to_int,  # 字符串长度
+    "bitcount": to_int,  # bit count
+
+    # === Key 通用 ===
+    "delete": to_int,  # 删除 key 个数
+    "exists": to_int,  # 存在的 key 个数
+    "expire": to_bool,  # True/False
+    "keys": to_list,  # 列表
+    "ttl": to_int,  # 剩余过期秒数
+
+    # === Hash ===
+    "hset": to_int,  # 新增/修改数量
+    "hget": to_str,  # 不存在 -> None
+    "hdel": to_int,  # 删除字段数
+    "hincrby": to_int,
+    "hexists": to_bool,
+    "hscan": to_hscan,
+    # "hgetall": to_dict,  # dict
+    "hlen": to_int,
+    "hkeys": to_list,
+    "hvals": to_list,
+
+    # === Set ===
+    "sadd": to_int,
+    "smembers": to_set,
+    "sismember": to_int,
+    "srem": to_int,
+    "spop": to_str,  # 单个值（批量 -> list）
+    "srandmember": to_str,  # 单个值（批量 -> list）
+    "scard": to_int,
+    "sscan": to_sscan,
+    "sdiff": to_set,
+
+    # === List ===
+    "lpush": to_int,
+    "rpush": to_int,
+    "lpop": to_str,
+    "rpop": to_str,
+    "lrange": to_list,
+    "llen": to_int,
+    "lindex": to_str,
+    "rpoplpush": to_str,
+    "lrem": to_int,
+
+    # === ZSet ===
+    "zadd": to_int,
+    "zremrangebyrank": to_int,
+    "zscore": to_str,  # (可转 float)
+    "zremrangebyscore": to_int,
+    "zincrby": to_str,  # (可转 float)
+    "zcount": to_int,
+    "zcard": to_int,
+    "zrem": to_int,
+    "zscan": to_zscan,
+    "zrange": to_zrange,
+    "zrangebyscore": to_zrange,
+
+    # === 脚本 / 特殊 ===
+    "eval": to_eval,
+    "scan": to_scan,
+}
+
+
+def create_async_session(self):
+    """创建同步的session"""
+    connector = aiohttp.TCPConnector(
+        limit=self.limit,
+        ttl_dns_cache=60,
+        enable_cleanup_closed=True,
+        force_close=False,
+        ssl=False,
+    )
+    timeout = aiohttp.ClientTimeout(
+        total=120,  # 容忍整体请求更久
+        connect=10,  # 连接最长 10s
+        sock_connect=10,
+        sock_read=100  # Redis pipeline 大响应时给足时间
+    )
+    return aiohttp.ClientSession(connector=connector, timeout=timeout)
+
+
+SYSTEM_NAME = platform.system().lower()
+# 等待删除
+CUR_DIR = os.path.abspath(os.path.realpath(os.path.dirname(__file__)))
+if SYSTEM_NAME in ["windows", "darwin"]:
+    local_env = True
+else:
+    local_env = False

redis_http_client-0.1.1/redis_http_client.egg-info/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: redis-http-client
+Version: 0.1.1
+Summary: A Redis client over HTTP supporting both sync (requests) and async (aiohttp) modes.
+Author-email: victor <xianyu.wu@innodealing.com>
+License-Expression: MIT
+Project-URL: Homepage, http://git.innodealing.cn/dm_spider_service/redis-http-client.git
+Project-URL: Source, http://git.innodealing.cn/dm_spider_service/redis-http-client.git
+Project-URL: Issues, http://git.innodealing.cn/dm_spider_service/redis-http-client.git/issues
+Keywords: redis,http,client,asyncio,requests,aiohttp
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: urllib3>=2.0.0
+Requires-Dist: loguru>=0.7.0
+Provides-Extra: dev
+Requires-Dist: build>=1.2.1; extra == "dev"
+Requires-Dist: twine>=5.1.0; extra == "dev"
+Dynamic: license-file
+
+# redis-http-client
+
+A lightweight Redis client over HTTP, supporting both async (`aiohttp`) and sync (`requests`) execution modes.
+
+## Features
+
+- Async mode based on `aiohttp`
+- Sync mode based on `requests`
+- Dynamic Redis command proxy via `__getattr__`
+- Pipeline support (`redis_pipeline_async` / `redis_pipeline_sync`)
+- Built-in retry and connection-pool handling
+
+## Installation
+
+```bash
+pip install redis-http-client
+```
+
+## Quick Start
+
+### Async
+
+```python
+import asyncio
+from redis_http_client import RedisHttpClient
+
+
+async def main():
+    client = RedisHttpClient(
+        spider_redis_manager_service_url="http://127.0.0.1:8000",
+        mode="async",
+    )
+
+    result = await client.get("my_key")
+    print(result)
+
+    await client.close()
+
+
+asyncio.run(main())
+```
+
+### Sync
+
+```python
+from redis_http_client import RedisHttpClient
+
+client = RedisHttpClient(
+    spider_redis_manager_service_url="http://127.0.0.1:8000",
+    mode="sync",
+)
+
+result = client.get("my_key")
+print(result)
+
+# sync mode reuses internal requests session; close when done
+import asyncio
+asyncio.run(client.close())
+```
+
+### Pipeline
+
+```python
+commands = [
+    {"command": "GET", "args": ["k1"], "kwargs": {}},
+    {"command": "GET", "args": ["k2"], "kwargs": {}},
+]
+
+# async
+# results = await client.redis_pipeline_async(commands)
+
+# sync
+# results = client.redis_pipeline_sync(commands)
+```
+
+## Build
+
+```bash
+python -m pip install --upgrade build twine
+python -m build
+```
+
+Build outputs are generated in `dist/`.
+
+## Upload To PyPI
+
+### 1. TestPyPI (recommended first)
+
+```bash
+twine check dist/*
+twine upload -r testpypi dist/*
+```
+
+### 2. PyPI
+
+```bash
+twine upload dist/*
+```
+
+You can use API token authentication:
+
+- username: `__token__`
+- password: `pypi-xxxx...`
+
+## Versioning
+
+Current version: `0.1.1`
+
+When releasing a new version:
+
+1. Update `version` in `pyproject.toml`
+2. Update `__version__` in `redis_http_client/__init__.py`
+3. Rebuild and upload
+
+## License
+
+MIT

redis_http_client-0.1.1/redis_http_client.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+LICENSE
+README.md
+pyproject.toml
+redis_http_client/__init__.py
+redis_http_client/client.py
+redis_http_client/other_logger.py
+redis_http_client/sub_run.py
+redis_http_client/to_method.py
+redis_http_client.egg-info/PKG-INFO
+redis_http_client.egg-info/SOURCES.txt
+redis_http_client.egg-info/dependency_links.txt
+redis_http_client.egg-info/requires.txt
+redis_http_client.egg-info/top_level.txt

redis_http_client-0.1.1/redis_http_client.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

redis_http_client-0.1.1/redis_http_client.egg-info/requires.txt

@@ -0,0 +1,8 @@
+aiohttp>=3.9.0
+requests>=2.31.0
+urllib3>=2.0.0
+loguru>=0.7.0
+
+[dev]
+build>=1.2.1
+twine>=5.1.0

redis_http_client-0.1.1/redis_http_client.egg-info/top_level.txt

@@ -0,0 +1 @@
+redis_http_client

redis_http_client-0.1.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+