esmd-pysdk 0.1.0__tar.gz

esmd_pysdk-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Liu Zhenhao
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

esmd_pysdk-0.1.0/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.4
+Name: esmd-pysdk
+Version: 0.1.0
+Summary: Asyncio-based Python driver for esmdMQ
+Project-URL: Homepage, https://pypi.org/project/esmd-pysdk/
+Project-URL: Repository, https://pypi.org/project/esmd-pysdk/
+Project-URL: Documentation, https://pypi.org/project/esmd-pysdk/
+Author: Liu Zhenhao
+License-Expression: MIT
+License-File: LICENSE
+Keywords: asyncio,esmd,message-queue,mq,sdk
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: System :: Networking
+Requires-Python: >=3.10
+Requires-Dist: lz4
+Description-Content-Type: text/markdown
+
+# esmd-pysdk
+
+Asyncio-based Python driver for esmdMQ. This project lives at the repo root and is parallel to `drivers/`.
+
+## Requirements
+
+- Python 3.10+
+- `lz4` (see `requirements.txt` or `pyproject.toml`)
+
+## Quick Start
+
+```bash
+pip install -r requirements.txt
+```
+
+Install from PyPI:
+
+```bash
+pip install esmd-pysdk
+```
+
+Check installed SDK version:
+
+```bash
+python -c "import esmd_pysdk; print(esmd_pysdk.__version__)"
+```
+
+Producer example:
+
+```bash
+python examples/producer.py
+```
+
+Consumer example:
+
+```bash
+python examples/consumer.py
+```
+
+Query example:
+
+```bash
+python examples/query.py
+```
+
+Matrix example:
+
+```bash
+python examples/matrix.py
+```
+
+Delete example:
+
+```bash
+python examples/delete.py
+```
+
+Schedule example (DelayTime + Punctual):
+
+```bash
+python examples/schedule.py
+```
+
+## Notes
+
+- JSON serialization is canonicalized (sorted keys, compact separators) to keep hash routing consistent with Go.
+- Driver uses asyncio; call APIs inside an event loop.
+- Producer schedule:
+  - `ScheduleMode.SERVER_TIME`: 服务端当前时间（默认）
+  - `ScheduleMode.DELAY_TIME`: 延迟投递，参数单位为毫秒（例如 `2000`）
+  - `ScheduleMode.PUNCTUAL`: 绝对时间投递，参数为 Unix 毫秒时间戳（例如 `int(time.time() * 1000) + 5000`）
+- Consumer delay:
+  - `DelayMode.BASE_SERVER`: 基于服务端时间消费（推荐默认）
+  - `DelayMode.IGNORE_DELAY`: 忽略延迟立即消费
+  - `DelayMode.TIME_DELAY`: 指定延迟窗口（毫秒）
+
+## Recommended Demo Flow
+
+```bash
+# 1) 先写入一条固定索引数据
+python examples/producer.py
+
+# 2) 查询和矩阵查询（使用与 producer 一致的 filter）
+python examples/query.py
+python examples/matrix.py
+
+# 3) 删除同一条件下的数据
+python examples/delete.py
+```
+
+## Release Notes
+
+- Version is managed in `pyproject.toml` (`[project].version`).
+- Release guide: `RELEASING.md` (TestPyPI -> PyPI).
+
+Pre-release checklist:
+
+```bash
+# 1) version check
+python - <<'PY'
+import re
+print(re.search(r'^version\\s*=\\s*\"([^\"]+)\"', open('pyproject.toml').read(), re.M).group(1))
+PY
+
+# 2) tests
+python -m pytest tests
+
+# 3) build + metadata check
+python -m build
+python -m twine check dist/*
+
+# 4) install/import smoke
+python -m venv .venv-release
+source .venv-release/bin/activate
+python -m pip install --upgrade pip
+python -m pip install dist/*.whl
+python -c "import esmd_pysdk; print('import ok')"
+```

esmd_pysdk-0.1.0/README.md

@@ -0,0 +1,118 @@
+# esmd-pysdk
+
+Asyncio-based Python driver for esmdMQ. This project lives at the repo root and is parallel to `drivers/`.
+
+## Requirements
+
+- Python 3.10+
+- `lz4` (see `requirements.txt` or `pyproject.toml`)
+
+## Quick Start
+
+```bash
+pip install -r requirements.txt
+```
+
+Install from PyPI:
+
+```bash
+pip install esmd-pysdk
+```
+
+Check installed SDK version:
+
+```bash
+python -c "import esmd_pysdk; print(esmd_pysdk.__version__)"
+```
+
+Producer example:
+
+```bash
+python examples/producer.py
+```
+
+Consumer example:
+
+```bash
+python examples/consumer.py
+```
+
+Query example:
+
+```bash
+python examples/query.py
+```
+
+Matrix example:
+
+```bash
+python examples/matrix.py
+```
+
+Delete example:
+
+```bash
+python examples/delete.py
+```
+
+Schedule example (DelayTime + Punctual):
+
+```bash
+python examples/schedule.py
+```
+
+## Notes
+
+- JSON serialization is canonicalized (sorted keys, compact separators) to keep hash routing consistent with Go.
+- Driver uses asyncio; call APIs inside an event loop.
+- Producer schedule:
+  - `ScheduleMode.SERVER_TIME`: 服务端当前时间（默认）
+  - `ScheduleMode.DELAY_TIME`: 延迟投递，参数单位为毫秒（例如 `2000`）
+  - `ScheduleMode.PUNCTUAL`: 绝对时间投递，参数为 Unix 毫秒时间戳（例如 `int(time.time() * 1000) + 5000`）
+- Consumer delay:
+  - `DelayMode.BASE_SERVER`: 基于服务端时间消费（推荐默认）
+  - `DelayMode.IGNORE_DELAY`: 忽略延迟立即消费
+  - `DelayMode.TIME_DELAY`: 指定延迟窗口（毫秒）
+
+## Recommended Demo Flow
+
+```bash
+# 1) 先写入一条固定索引数据
+python examples/producer.py
+
+# 2) 查询和矩阵查询（使用与 producer 一致的 filter）
+python examples/query.py
+python examples/matrix.py
+
+# 3) 删除同一条件下的数据
+python examples/delete.py
+```
+
+## Release Notes
+
+- Version is managed in `pyproject.toml` (`[project].version`).
+- Release guide: `RELEASING.md` (TestPyPI -> PyPI).
+
+Pre-release checklist:
+
+```bash
+# 1) version check
+python - <<'PY'
+import re
+print(re.search(r'^version\\s*=\\s*\"([^\"]+)\"', open('pyproject.toml').read(), re.M).group(1))
+PY
+
+# 2) tests
+python -m pytest tests
+
+# 3) build + metadata check
+python -m build
+python -m twine check dist/*
+
+# 4) install/import smoke
+python -m venv .venv-release
+source .venv-release/bin/activate
+python -m pip install --upgrade pip
+python -m pip install dist/*.whl
+python -c "import esmd_pysdk; print('import ok')"
+```

esmd_pysdk-0.1.0/esmd_pysdk/__init__.py

@@ -0,0 +1,54 @@
+from importlib.metadata import PackageNotFoundError, version
+
+from .claim import Claim, new_claim, new_opinion
+from .config import Config, Endpoint, new_config
+from .constants import DelayMode, ScheduleMode
+from .errors import (
+    Error,
+    ErrHashSlotIsFull,
+    ErrInvalidTopicName,
+    ErrMessageTooLarge,
+    ErrNoDataFound,
+    ErrSessionClosed,
+    ErrUnAckSizeExceeded,
+)
+from .message import ConsumerMessage, Messages, ProducerMessage, new_producer_message
+from .driver import new_async_producer, new_consumer, new_producer
+
+try:
+    __version__ = version("esmd-pysdk")
+except PackageNotFoundError:
+    # Fallback for source checkout before package installation.
+    __version__ = "0.1.0"
+
+
+def get_version() -> str:
+    return __version__
+
+
+__all__ = [
+    "__version__",
+    "get_version",
+    "Claim",
+    "new_claim",
+    "new_opinion",
+    "Config",
+    "Endpoint",
+    "new_config",
+    "DelayMode",
+    "ScheduleMode",
+    "Error",
+    "ErrHashSlotIsFull",
+    "ErrInvalidTopicName",
+    "ErrMessageTooLarge",
+    "ErrNoDataFound",
+    "ErrSessionClosed",
+    "ErrUnAckSizeExceeded",
+    "ConsumerMessage",
+    "Messages",
+    "ProducerMessage",
+    "new_producer_message",
+    "new_producer",
+    "new_async_producer",
+    "new_consumer",
+]

esmd_pysdk-0.1.0/esmd_pysdk/claim.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+import asyncio
+import uuid
+from dataclasses import dataclass, field
+from typing import Any, Awaitable, Callable, Dict, List, Optional
+
+from .constants import DelayMode
+from .limiter import AsyncLimiter
+
+
+ConsumeHandler = Callable[[asyncio.Queue], Awaitable[None]]
+
+
+@dataclass
+class Claim:
+    topics: List[str]
+    handle: Optional[ConsumeHandler] = None
+    delay: int = 0
+    filter: Optional[Dict[str, Any]] = None
+    consume_group: str = ""
+    max_load: int = 1
+    cid: str = field(default_factory=lambda: uuid.uuid4().hex)
+    msg: asyncio.Queue = field(default_factory=lambda: asyncio.Queue(maxsize=20000))
+    token_bucket: Optional[AsyncLimiter] = None
+    best_pos: int = 0
+
+    def set_max_load(self, max_load: int) -> None:
+        self.max_load = max_load
+
+    def ignore_data(self) -> None:
+        self.max_load = 0
+
+    def set_pop_limiter(self, size: int) -> None:
+        self.token_bucket = AsyncLimiter(1.0, size)
+
+    def set_custom_limiter(self, limiter: AsyncLimiter) -> None:
+        self.token_bucket = limiter
+
+    def set_delay(self, mode: DelayMode, delay_ms: Optional[int] = None) -> None:
+        if mode == DelayMode.IGNORE_DELAY:
+            self.delay = -1
+        elif mode == DelayMode.BASE_SERVER:
+            self.delay = 0
+        elif mode == DelayMode.TIME_DELAY:
+            if delay_ms is None or delay_ms <= 1:
+                raise ValueError("time value > 1ms required")
+            self.delay = int(delay_ms)
+
+    def try_filter(self, filter_data: Dict[str, Any]) -> None:
+        self.filter = filter_data
+
+    def set_consume_group(self, group: str) -> None:
+        self.consume_group = group
+
+    async def add_message(self, msg) -> None:
+        if self.token_bucket:
+            await self.token_bucket.wait()
+        await self.msg.put(msg)
+
+
+def new_claim(topics: List[str], handle: Optional[ConsumeHandler]) -> Claim:
+    return Claim(topics=topics, handle=handle)
+
+
+def new_opinion() -> Claim:
+    return Claim(topics=[], max_load=1)

esmd_pysdk-0.1.0/esmd_pysdk/client.py

@@ -0,0 +1,102 @@
+from __future__ import annotations
+
+import asyncio
+import json
+from typing import List, Optional, Tuple
+
+from .cluster import partitions_from_cluster, watch_cluster
+from .config import Config, Endpoint
+from .constants import MessageType, VERSION_V1_2_0
+from .errors import Error, new_error
+from .partition import Partition
+from .session import Session
+from .slot import PeerSlot
+
+
+class Client:
+    def __init__(self, conf: Config, slot: PeerSlot):
+        self.conf = conf
+        self.slot = slot
+        self.closed = False
+        self.stop_event = asyncio.Event()
+        self._watch_tasks: List[asyncio.Task] = []
+
+    async def close(self) -> None:
+        if self.closed:
+            return
+        self.closed = True
+        self.stop_event.set()
+        for task in self._watch_tasks:
+            task.cancel()
+        if self._watch_tasks:
+            await asyncio.gather(*self._watch_tasks, return_exceptions=True)
+        for part in self.slot.get_node_list():
+            while not part.sessions.empty():
+                sess = await part.sessions.get()
+                await sess.close()
+
+    def keep_retry(self, duration_sec: Optional[float] = None) -> None:
+        if duration_sec is None or duration_sec <= 0:
+            self.conf.retry_times = -1
+            return
+        interval = int(duration_sec / 0.5)
+        self.conf.retry_times = interval if duration_sec % 0.5 == 0 else interval + 1
+
+    def get_partition(self, slot: int) -> Optional[Partition]:
+        return self.slot.get_partition(slot)
+
+    def get_partitions(self) -> List[Partition]:
+        return self.slot.get_node_list()
+
+
+async def new_client(raw_url: str, conf: Optional[Config] = None) -> Tuple[Optional[Client], Optional[Error]]:
+    conf = conf or Config()
+    conf.fix_config()
+    endpoints = []
+    for raw in raw_url.split(","):
+        endpoints.append(conf.parse_url(raw.strip()))
+    parts, err = await get_part_list(endpoints, conf)
+    if err:
+        return None, err
+    if not parts:
+        return None, new_error(Exception("no partitions available"))
+    slot = PeerSlot(conf)
+    err = await slot.initial(parts)
+    if err:
+        return None, err
+    client = Client(conf, slot)
+    for part in parts:
+        task = asyncio.create_task(
+            watch_cluster(part, slot.rebalance, stop_event=client.stop_event)
+        )
+        client._watch_tasks.append(task)
+    return client, None
+
+
+async def get_part_list(
+    endpoints: List[Endpoint], conf: Config
+) -> Tuple[List[Partition], Optional[Error]]:
+    for ep in endpoints:
+        part = Partition(endpoint=ep, address=ep.address, no=0, conf=conf)
+        sess = Session(part, ep.version)
+        err = await sess.connect()
+        if err:
+            await sess.close()
+            continue
+        req = conf.new_protocol(ep.version)
+        resp, err = await sess.request(MessageType.PARTITION, req)
+        await sess.close()
+        if err or resp is None:
+            continue
+        try:
+            block = resp.block
+            if ep.version == VERSION_V1_2_0:
+                data = json.loads(block)
+            else:
+                data = json.loads(block)
+            parts = partitions_from_cluster(data, ep.version, conf, fallback_endpoint=ep)
+            if parts:
+                return parts, None
+        except Exception as exc:
+            return [], new_error(exc)
+    return [], new_error(Exception("no partitions available"))

esmd_pysdk-0.1.0/esmd_pysdk/cluster.py

@@ -0,0 +1,111 @@
+from __future__ import annotations
+
+import asyncio
+from typing import Any, List, Optional
+
+from .constants import MessageType, VERSION_V1_2_0
+from .config import Endpoint
+from .partition import Partition
+from .session import Session
+
+
+def _endpoint_from_node(node: dict, version: str, fallback: Optional[Endpoint]) -> Endpoint:
+    ep = node.get("endpoint") or {}
+    address = (ep.get("address") or node.get("address") or (fallback.address if fallback else "")).strip()
+    scheme = (ep.get("scheme") or (fallback.scheme if fallback else "esmd")).strip()
+    username = ep.get("username") or (fallback.username if fallback else "")
+    password = ep.get("password") or (fallback.password if fallback else "")
+    v = ep.get("version") or version or (fallback.version if fallback else "")
+    return Endpoint(address=address, scheme=scheme, username=username, password=password, version=v)
+
+
+def partitions_from_cluster(
+    data: Any, version: str, conf, fallback_endpoint: Optional[Endpoint] = None
+) -> List[Partition]:
+    nodes = []
+    if version == VERSION_V1_2_0:
+        raw_nodes = data or []
+    else:
+        raw_nodes = (data or {}).get("nodes", [])
+    for node in raw_nodes:
+        endpoint = _endpoint_from_node(node, version, fallback_endpoint)
+        if not endpoint.address:
+            continue
+        part = Partition(
+            endpoint=endpoint,
+            address=node.get("address") or endpoint.address,
+            no=node.get("partition", 0),
+            master=node.get("master", False),
+            stat=node.get("stat", node.get("status", 0)),
+            conf=conf,
+        )
+        nodes.append(part)
+    return nodes
+
+
+async def _sleep_or_stop(stop_event: Optional[asyncio.Event], seconds: float) -> None:
+    if stop_event is None:
+        await asyncio.sleep(seconds)
+        return
+    try:
+        await asyncio.wait_for(stop_event.wait(), timeout=seconds)
+    except asyncio.TimeoutError:
+        return
+
+
+async def watch_cluster(
+    partition: Partition, rebalance_cb, stop_event: Optional[asyncio.Event] = None
+) -> None:
+    try:
+        while stop_event is None or not stop_event.is_set():
+            sess = Session(partition, partition.get_version())
+            err = await sess.connect()
+            if err:
+                await _sleep_or_stop(stop_event, 1)
+                continue
+
+            try:
+                req = partition.conf.new_protocol(partition.get_version())
+                _, req_err = await sess.request(MessageType.CLUSTER, req)
+                if req_err:
+                    await _sleep_or_stop(stop_event, 1)
+                    continue
+
+                while stop_event is None or not stop_event.is_set():
+                    data, err = await sess.read_json()
+                    if err:
+                        break
+                    if not data or data.get("code") != 0:
+                        continue
+                    block = data.get("block")
+                    try:
+                        import base64
+
+                        if isinstance(block, str):
+                            block = base64.b64decode(block)
+                    except Exception:
+                        pass
+                    try:
+                        import json
+
+                        decoded = (
+                            json.loads(block)
+                            if isinstance(block, (bytes, bytearray))
+                            else block
+                        )
+                    except Exception:
+                        decoded = block
+                    parts = partitions_from_cluster(
+                        decoded,
+                        partition.get_version(),
+                        partition.conf,
+                        fallback_endpoint=partition.endpoint,
+                    )
+                    if parts:
+                        rebalance_cb(parts)
+            finally:
+                await sess.close()
+
+            await _sleep_or_stop(stop_event, 1)
+    except asyncio.CancelledError:
+        return

esmd_pysdk-0.1.0/esmd_pysdk/compress.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+import gzip
+import zlib
+from dataclasses import dataclass
+from typing import Optional
+
+try:
+    import lz4.frame as _lz4_frame
+except Exception:  # pragma: no cover - optional dependency
+    _lz4_frame = None
+
+
+class CompressType:
+    GZIP = 1
+    ZLIB = 2
+    LZ4 = 3
+
+
+class CompressLevel:
+    NORMAL = 0
+    LEVEL1 = 1
+    LEVEL2 = 2
+    LEVEL3 = 3
+    LEVEL4 = 4
+    LEVEL5 = 5
+    LEVEL6 = 6
+    LEVEL7 = 7
+    LEVEL8 = 8
+    LEVEL9 = 9
+
+
+@dataclass
+class Compression:
+    zip_type: int
+    level: int
+
+    def compress(self, data: bytes) -> bytes:
+        if self.zip_type == CompressType.GZIP:
+            return gzip.compress(data, compresslevel=self.level or 1)
+        if self.zip_type == CompressType.ZLIB:
+            return zlib.compress(data, level=self.level or 1)
+        if self.zip_type == CompressType.LZ4:
+            if _lz4_frame is None:
+                raise RuntimeError("lz4 is not installed")
+            return _lz4_frame.compress(data)
+        return data
+
+    def decompress(self, data: bytes) -> bytes:
+        if self.zip_type == CompressType.GZIP:
+            return gzip.decompress(data)
+        if self.zip_type == CompressType.ZLIB:
+            return zlib.decompress(data)
+        if self.zip_type == CompressType.LZ4:
+            if _lz4_frame is None:
+                raise RuntimeError("lz4 is not installed")
+            return _lz4_frame.decompress(data)
+        return data
+
+
+def new_compress(zip_type: int, level: Optional[int] = None) -> Optional[Compression]:
+    if zip_type not in (CompressType.GZIP, CompressType.ZLIB, CompressType.LZ4):
+        return None
+    return Compression(zip_type=zip_type, level=level or 0)