queue-func 0.1__tar.gz → 0.1.2__tar.gz

queue_func-0.1.2/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.4
+Name: queue-func
+Version: 0.1.2
+Summary: Async coroutine functions as queue-backed tasks (QueueFunc).
+License: Apache-2.0
+License-File: LICENSE
+Author: liuseki
+Author-email: liuseki@163.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Requires-Dist: orjson
+Requires-Dist: pydantic (>=2.14.1,<3.0.0)
+Project-URL: Changelog, https://github.com/sekil9529/queue-func/blob/master/Changelog.rst
+Project-URL: Documentation, https://github.com/sekil9529/queue-func#readme
+Project-URL: Homepage, https://github.com/sekil9529/queue-func
+Project-URL: Issues, https://github.com/sekil9529/queue-func/issues
+Project-URL: Repository, https://github.com/sekil9529/queue-func
+Description-Content-Type: text/markdown
+
+# queue-func
+
+把 **`async def` 协程** 变成可 **`push`** 的队列任务：在 asyncio 里投递、消费，并按通道控制并发与背压。
+
+需要 **Python 3.12+**；任务入口必须是协程，不支持普通 `def`。
+
+仓库：[github.com/sekil9529/queue-func](https://github.com/sekil9529/queue-func)
+
+## 特点
+
+- **asyncio 原生**：consumer 在同一事件循环里池化 `await` 任务，与 FastAPI / Starlette 等同进程组合很自然。
+- **运行时聚焦队列链路**：`pack` → 投递 → `unpack` → 执行；生命周期由 `broker` / `QueueFuncRuntime` 显式管理。
+- **环节可扩展**：通过 `register_backend` 接入新队列；`Serializer.pack` / `unpack` 决定队列里的数据结构。
+- **每个 `QueueFunc` 独立消费并发**：各自 `consumer_pool_maxsize`。一个进程里可用 `QueueFuncRuntime` 同时跑多段流水线（例如 count → list → detail），**各段并发互不影响**。
+
+## 架构设计
+
+### Func
+
+业务协程：一个 `async def`，用 `@queue` 装饰。
+
+### Queue
+
+装饰之后，这个 Func 就是一条队列，`push` 投到它自己这条队列上。
+
+### Broker
+
+组装 Producer、Consumer 的公有对象，提供对外接口。
+
+### Serializer
+
+控制队列中的数据结构：`pack` 写入，`unpack` 读出后还原成 `(args, kwargs)`。
+
+- `raw.Serializer`：队列中保持 `(args, kwargs)`。
+- `orjson_ser.Serializer`：队列中是 `bytes`。
+
+### Producer
+
+调用 `Serializer.pack` 后把数据写入队列。
+
+### Consumer
+
+从队列得到数据，`Serializer.unpack` 后交给 PoolExecutor 执行。
+
+### PoolExecutor
+
+按并发上限执行业务协程。
+
+
+
+### 组件关系
+
+```mermaid
+flowchart TB
+  PUSH["push"] --> B["Broker"]
+  B --> PRO["Producer"]
+  B -.->|启停| CON["Consumer"]
+
+  PRO -->|Serializer.pack| Q[("QUEUE")]
+  Q -->|Serializer.unpack| CON
+  CON --> POOL["PoolExecutor"]
+  POOL --> FN["async def"]
+```
+
+## 当前实现
+
+| Backend | 说明 |
+|---------|------|
+| **`Backend.QUEUE`** | 进程内 `asyncio.Queue`（已可用） |
+| RabbitMQ、Kafka 等 | 通过同一套 Broker / Producer / Consumer 扩展点接入（见 [Changelog.rst](Changelog.rst)） |
+
+接入 MQ 后端后，producer / consumer 可部署在不同进程或机器；**现阶段发行版仅包含内存队列实现**。
+
+## 快速开始
+
+安装（开发）：
+
+```bash
+git clone https://github.com/sekil9529/queue-func.git
+cd queue-func
+poetry install
+```
+
+单个 `QueueFunc`：
+
+```python
+import asyncio
+
+from queue_func.factory import Backend, queue
+
+
+@queue(Backend.QUEUE, consume_timeout=0.5, maxsize=1000)
+async def process_order(order_id: int):
+    ...
+
+
+async def main():
+    await process_order.broker.start_consumer()
+    await process_order.push(args=(42,))
+    await process_order.broker.stop_consumer()
+    await process_order.broker.join_consumer()
+
+
+asyncio.run(main())
+```
+
+- `push` 等价于 `broker.publish`。不传 `timeout` 时用 `publish_timeout`；传入数字只改这一次。`None` 表示不限时。超时抛出 `queue_func.PublishTimeoutError`
+- 仍可直接 `await process_order(1)`，不经过队列
+
+多段流水线（各段独立队列与并发上限）：
+
+```python
+from queue_func.factory import Backend, QueueFuncRuntime, queue
+
+# task_count 内 await task_list.push(...)，task_list 内 await task_detail.push(...)
+
+
+async def main():
+    runtime = QueueFuncRuntime([task_count, task_list, task_detail])
+    await runtime.start()  # 按列表反序启动 consumer，下游先就绪
+    await task_count.push(args=(...,))
+    await runtime.stop()
+```
+
+链式示例（含日志）：
+
+```bash
+poetry run python examples/demo_queue.py
+```
+
+常用配置：`maxsize` / `publish_timeout`（队列背压与投递等待）、`consumer_pool_maxsize`（同时执行的协程数）、`serializer_cls`（队列中的数据结构）。低级组装见 `queue_func.config.queue.Config` 与 `queue_func.broker.queue.Broker`。
+
+## 项目结构
+
+```
+queue_func/
+├── broker/
+├── producer/
+├── consumer/
+├── serializer/       # raw、orjson
+├── config/
+├── factory/          # @queue、QueueFunc、QueueFuncRuntime
+└── pool_executor/
+examples/
+└── demo_queue.py
+```
+
+## 开发与测试
+
+版本与路线图：[Changelog.rst](Changelog.rst)
+
+```bash
+pytest tests/ -q
+```
+
+## 许可证
+
+Apache-2.0
+

queue_func-0.1.2/README.md

@@ -0,0 +1,158 @@
+# queue-func
+
+把 **`async def` 协程** 变成可 **`push`** 的队列任务：在 asyncio 里投递、消费，并按通道控制并发与背压。
+
+需要 **Python 3.12+**；任务入口必须是协程，不支持普通 `def`。
+
+仓库：[github.com/sekil9529/queue-func](https://github.com/sekil9529/queue-func)
+
+## 特点
+
+- **asyncio 原生**：consumer 在同一事件循环里池化 `await` 任务，与 FastAPI / Starlette 等同进程组合很自然。
+- **运行时聚焦队列链路**：`pack` → 投递 → `unpack` → 执行；生命周期由 `broker` / `QueueFuncRuntime` 显式管理。
+- **环节可扩展**：通过 `register_backend` 接入新队列；`Serializer.pack` / `unpack` 决定队列里的数据结构。
+- **每个 `QueueFunc` 独立消费并发**：各自 `consumer_pool_maxsize`。一个进程里可用 `QueueFuncRuntime` 同时跑多段流水线（例如 count → list → detail），**各段并发互不影响**。
+
+## 架构设计
+
+### Func
+
+业务协程：一个 `async def`，用 `@queue` 装饰。
+
+### Queue
+
+装饰之后，这个 Func 就是一条队列，`push` 投到它自己这条队列上。
+
+### Broker
+
+组装 Producer、Consumer 的公有对象，提供对外接口。
+
+### Serializer
+
+控制队列中的数据结构：`pack` 写入，`unpack` 读出后还原成 `(args, kwargs)`。
+
+- `raw.Serializer`：队列中保持 `(args, kwargs)`。
+- `orjson_ser.Serializer`：队列中是 `bytes`。
+
+### Producer
+
+调用 `Serializer.pack` 后把数据写入队列。
+
+### Consumer
+
+从队列得到数据，`Serializer.unpack` 后交给 PoolExecutor 执行。
+
+### PoolExecutor
+
+按并发上限执行业务协程。
+
+
+
+### 组件关系
+
+```mermaid
+flowchart TB
+  PUSH["push"] --> B["Broker"]
+  B --> PRO["Producer"]
+  B -.->|启停| CON["Consumer"]
+
+  PRO -->|Serializer.pack| Q[("QUEUE")]
+  Q -->|Serializer.unpack| CON
+  CON --> POOL["PoolExecutor"]
+  POOL --> FN["async def"]
+```
+
+## 当前实现
+
+| Backend | 说明 |
+|---------|------|
+| **`Backend.QUEUE`** | 进程内 `asyncio.Queue`（已可用） |
+| RabbitMQ、Kafka 等 | 通过同一套 Broker / Producer / Consumer 扩展点接入（见 [Changelog.rst](Changelog.rst)） |
+
+接入 MQ 后端后，producer / consumer 可部署在不同进程或机器；**现阶段发行版仅包含内存队列实现**。
+
+## 快速开始
+
+安装（开发）：
+
+```bash
+git clone https://github.com/sekil9529/queue-func.git
+cd queue-func
+poetry install
+```
+
+单个 `QueueFunc`：
+
+```python
+import asyncio
+
+from queue_func.factory import Backend, queue
+
+
+@queue(Backend.QUEUE, consume_timeout=0.5, maxsize=1000)
+async def process_order(order_id: int):
+    ...
+
+
+async def main():
+    await process_order.broker.start_consumer()
+    await process_order.push(args=(42,))
+    await process_order.broker.stop_consumer()
+    await process_order.broker.join_consumer()
+
+
+asyncio.run(main())
+```
+
+- `push` 等价于 `broker.publish`。不传 `timeout` 时用 `publish_timeout`；传入数字只改这一次。`None` 表示不限时。超时抛出 `queue_func.PublishTimeoutError`
+- 仍可直接 `await process_order(1)`，不经过队列
+
+多段流水线（各段独立队列与并发上限）：
+
+```python
+from queue_func.factory import Backend, QueueFuncRuntime, queue
+
+# task_count 内 await task_list.push(...)，task_list 内 await task_detail.push(...)
+
+
+async def main():
+    runtime = QueueFuncRuntime([task_count, task_list, task_detail])
+    await runtime.start()  # 按列表反序启动 consumer，下游先就绪
+    await task_count.push(args=(...,))
+    await runtime.stop()
+```
+
+链式示例（含日志）：
+
+```bash
+poetry run python examples/demo_queue.py
+```
+
+常用配置：`maxsize` / `publish_timeout`（队列背压与投递等待）、`consumer_pool_maxsize`（同时执行的协程数）、`serializer_cls`（队列中的数据结构）。低级组装见 `queue_func.config.queue.Config` 与 `queue_func.broker.queue.Broker`。
+
+## 项目结构
+
+```
+queue_func/
+├── broker/
+├── producer/
+├── consumer/
+├── serializer/       # raw、orjson
+├── config/
+├── factory/          # @queue、QueueFunc、QueueFuncRuntime
+└── pool_executor/
+examples/
+└── demo_queue.py
+```
+
+## 开发与测试
+
+版本与路线图：[Changelog.rst](Changelog.rst)
+
+```bash
+pytest tests/ -q
+```
+
+## 许可证
+
+Apache-2.0

{queue_func-0.1 → queue_func-0.1.2}/pyproject.toml

@@ -1,12 +1,19 @@
 [tool.poetry]
 name = "queue-func"
-version = "0.1"
+version = "0.1.2"
 description = "Async coroutine functions as queue-backed tasks (QueueFunc)."
 authors = ["liuseki <liuseki@163.com>"]
 license = "Apache-2.0"
 readme = "README.md"
+homepage = "https://github.com/sekil9529/queue-func"
+repository = "https://github.com/sekil9529/queue-func"
+documentation = "https://github.com/sekil9529/queue-func#readme"
 packages = [{ include = "queue_func" }]
 
+[tool.poetry.urls]
+Changelog = "https://github.com/sekil9529/queue-func/blob/master/Changelog.rst"
+Issues = "https://github.com/sekil9529/queue-func/issues"
+
 [tool.poetry.dependencies]
 python = "^3.12"
 orjson = "*"

queue_func-0.1.2/queue_func/__init__.py

@@ -0,0 +1,7 @@
+# coding: utf-8
+
+from queue_func.errors import PublishTimeoutError
+
+__all__ = (
+    "PublishTimeoutError",
+)

{queue_func-0.1 → queue_func-0.1.2}/queue_func/broker/base.py

@@ -3,7 +3,7 @@
 import abc
 import typing as t
 
-from queue_func.message.base import BaseMessage, Packed
+from queue_func.serializer.base import BaseSerializer, Message
 
 if t.TYPE_CHECKING:
     from queue_func.config.general import Config
@@ -14,16 +14,16 @@ if t.TYPE_CHECKING:
 ConfigType = t.TypeVar("ConfigType", bound="Config")
 
 
-class BaseBroker(t.Generic[ConfigType, Packed], metaclass=abc.ABCMeta):
+class BaseBroker(t.Generic[ConfigType, Message], metaclass=abc.ABCMeta):
     """一条通道的装配者。
 
-    创建时准备好双方共用的 Message，并持有配置与 Producer/Consumer。
+    创建时准备好双方共用的 Serializer，并持有配置与 Producer/Consumer。
     对外通过 publish、start_consumer 等入口操作；运行期业务逻辑仍在 Producer/Consumer 内完成。
     """
 
     __slots__ = (
         "_config",
-        "_message",
+        "_serializer",
         "_producer",
         "_consumer",
     )
@@ -31,35 +31,33 @@ class BaseBroker(t.Generic[ConfigType, Packed], metaclass=abc.ABCMeta):
     def __init__(self, config: ConfigType):
 
         self._config = config
-        self._message: BaseMessage[Packed] = self._config.message_cls(
-            message_serializer_cls=self._config.message_serializer_cls,
-            enable_serializer=self._config.enable_message_serializer,
-        )
+        self._serializer: type[BaseSerializer[Message]] = self._config.serializer_cls
         self._producer = self._create_producer()
         self._consumer = self._create_consumer()
 
     @abc.abstractmethod
-    def _create_producer(self) -> "BaseProducer[Packed]":
+    def _create_producer(self) -> "BaseProducer[Message]":
 
         pass
 
     @abc.abstractmethod
-    def _create_consumer(self) -> "BaseConsumer[Packed]":
+    def _create_consumer(self) -> "BaseConsumer[Message]":
 
         pass
 
     @property
-    def message(self) -> BaseMessage[Packed]:
+    def serializer(self) -> type[BaseSerializer[Message]]:
 
-        return self._message
+        return self._serializer
 
     async def publish(
             self,
             coro_args: tuple[t.Any, ...],
             coro_kwargs: dict[str, t.Any] | None = None,
-            **kwargs: t.Any) -> None:
+            *,
+            timeout: float | None = None) -> None:
 
-        await self._producer.publish(coro_args, coro_kwargs, **kwargs)
+        await self._producer.publish(coro_args, coro_kwargs, timeout=timeout)
 
     async def start_consumer(self) -> None:
 

{queue_func-0.1 → queue_func-0.1.2}/queue_func/broker/queue.py

@@ -3,7 +3,7 @@
 import asyncio
 import typing as t
 
-from queue_func.message.base import Packed
+from queue_func.serializer.base import Message
 from queue_func.broker.base import BaseBroker as _BaseBroker
 from queue_func.producer.queue import Producer
 from queue_func.consumer.queue import Consumer
@@ -12,7 +12,7 @@ if t.TYPE_CHECKING:
     from queue_func.config.queue import Config
 
 
-class Broker(_BaseBroker["Config", Packed]):
+class Broker(_BaseBroker["Config", Message]):
     """内存队列通道。队列由这里创建，再交给 Producer 和 Consumer。"""
 
     __slots__ = (
@@ -20,23 +20,23 @@ class Broker(_BaseBroker["Config", Packed]):
     )
 
     def __init__(self, config: "Config"):
-        self._queue: asyncio.Queue[Packed] = asyncio.Queue(maxsize=config.maxsize)
+        self._queue: asyncio.Queue[Message] = asyncio.Queue(maxsize=config.maxsize)
         super().__init__(config)
 
-    def _create_producer(self) -> Producer[Packed]:
+    def _create_producer(self) -> Producer[Message]:
 
         return Producer(
             queue=self._queue,
             publish_timeout=self._config.publish_timeout,
-            message=self._message,
+            serializer=self._serializer,
         )
 
-    def _create_consumer(self) -> Consumer[Packed]:
+    def _create_consumer(self) -> Consumer[Message]:
 
         return Consumer(
             queue=self._queue,
             consume_timeout=self._config.consume_timeout,
-            message=self._message,
+            serializer=self._serializer,
             coro=self._config.coro,
             enable_consumer=self._config.enable_consumer,
             pool_executor_cls=self._config.consumer_pool_executor_cls,

{queue_func-0.1 → queue_func-0.1.2}/queue_func/config/general.py

@@ -5,10 +5,8 @@ import typing as t
 from pydantic.dataclasses import dataclass as pyd_dataclass
 
 from queue_func.config.utils import ExtraIgnoreConfig
-from queue_func.message.base import BaseMessage
-from queue_func.message.general import Message
 from queue_func.serializer.base import BaseSerializer
-from queue_func.serializer.orjson_ser import OrJSONSerializer
+from queue_func.serializer.orjson_ser import Serializer
 from queue_func.pool_executor.base import BasePoolExecutor
 from queue_func.pool_executor.general import PoolExecutor
 
@@ -24,9 +22,7 @@ class Config:
     consumer_pool_maxsize: int = 200
 
     # broker
-    message_cls: type[BaseMessage] = Message
-    message_serializer_cls: type[BaseSerializer] = OrJSONSerializer
-    enable_message_serializer: bool = True
+    serializer_cls: type[BaseSerializer] = Serializer
 
     # producer
     publish_timeout: float | None = None

{queue_func-0.1 → queue_func-0.1.2}/queue_func/config/queue.py

@@ -5,6 +5,8 @@ from pydantic.dataclasses import dataclass as pyd_dataclass
 
 from queue_func.config.utils import ExtraIgnoreConfig
 from queue_func.config.general import Config as _Config
+from queue_func.serializer.base import BaseSerializer
+from queue_func.serializer.raw import Serializer
 
 
 @pyd_dataclass(frozen=True, config=ExtraIgnoreConfig)
@@ -12,3 +14,4 @@ class Config(_Config):
 
     # broker
     maxsize: int = 200
+    serializer_cls: type[BaseSerializer] = Serializer

{queue_func-0.1 → queue_func-0.1.2}/queue_func/consumer/base.py

@@ -7,14 +7,14 @@ import contextlib
 import typing as t
 from functools import wraps
 
-from queue_func.message.base import BaseMessage, Packed
+from queue_func.serializer.base import BaseSerializer, Message
 from queue_func.pool_executor.base import BasePoolExecutor
 
 
-class BaseConsumer(t.Generic[Packed], metaclass=abc.ABCMeta):
+class BaseConsumer(t.Generic[Message], metaclass=abc.ABCMeta):
 
     __slots__ = (
-        "_message",
+        "_serializer",
         "_coro",
         "_enable_consumer",
         "_pool_executor",
@@ -25,13 +25,13 @@ class BaseConsumer(t.Generic[Packed], metaclass=abc.ABCMeta):
 
     def __init__(
             self,
-            message: BaseMessage[Packed],
+            serializer: type[BaseSerializer[Message]],
             coro: t.Callable[..., t.Coroutine],
             enable_consumer: bool,
             pool_executor_cls: type[BasePoolExecutor],
             pool_maxsize: int):
 
-        self._message = message
+        self._serializer = serializer
         self._coro = coro
         self._enable_consumer = enable_consumer
         self._pool_executor = self._init_pool_executor(pool_executor_cls, pool_maxsize)

{queue_func-0.1 → queue_func-0.1.2}/queue_func/consumer/queue.py

@@ -3,12 +3,12 @@
 import asyncio
 import typing as t
 
-from queue_func.message.base import BaseMessage, Packed
+from queue_func.serializer.base import BaseSerializer, Message
 from queue_func.pool_executor.base import BasePoolExecutor
 from queue_func.consumer.base import BaseConsumer as _BaseConsumer
 
 
-class Consumer(_BaseConsumer[Packed]):
+class Consumer(_BaseConsumer[Message]):
 
     __slots__ = (
         "_queue",
@@ -17,16 +17,16 @@ class Consumer(_BaseConsumer[Packed]):
 
     def __init__(
             self,
-            queue: asyncio.Queue[Packed],
+            queue: asyncio.Queue[Message],
             consume_timeout: float,
-            message: BaseMessage[Packed],
+            serializer: type[BaseSerializer[Message]],
             coro: t.Callable[..., t.Coroutine],
             enable_consumer: bool,
             pool_executor_cls: type[BasePoolExecutor],
             pool_maxsize: int):
 
         super().__init__(
-            message,
+            serializer,
             coro,
             enable_consumer,
             pool_executor_cls,
@@ -39,13 +39,13 @@ class Consumer(_BaseConsumer[Packed]):
 
         while (not self._stop_flag) or (not self._queue.empty()):
             try:
-                message: Packed = await asyncio.wait_for(
+                packed: Message = await asyncio.wait_for(
                     self._queue.get(),
                     timeout=self._consume_timeout,
                 )
             except asyncio.TimeoutError:
                 continue
-            coro_args, coro_kwargs = self._message.unpack(message)
+            coro_args, coro_kwargs = self._serializer.unpack(packed)
             await self._dispatch(coro_args, coro_kwargs)
             self._queue.task_done()
 

queue_func-0.1.2/queue_func/errors.py

@@ -0,0 +1,7 @@
+# coding: utf-8
+
+
+class PublishTimeoutError(TimeoutError):
+    """入队在给定时间内未完成"""
+
+    pass

{queue_func-0.1 → queue_func-0.1.2}/queue_func/factory/deco.py

@@ -9,7 +9,6 @@ from queue_func.factory.registry import build_broker
 if t.TYPE_CHECKING:
     from queue_func.broker.base import BaseBroker
     from queue_func.config.general import Config as GeneralConfig
-    from queue_func.message.base import BaseMessage
     from queue_func.pool_executor.base import BasePoolExecutor
     from queue_func.serializer.base import BaseSerializer
 
@@ -46,9 +45,10 @@ class QueueFunc(t.Generic[_P, _R]):
             self,
             *,
             args: tuple[t.Any, ...] = (),
-            kwargs: dict[str, t.Any] | None = None) -> None:
+            kwargs: dict[str, t.Any] | None = None,
+            timeout: float | None = None) -> None:
 
-        await self._broker.publish(coro_args=args, coro_kwargs=kwargs)
+        await self._broker.publish(coro_args=args, coro_kwargs=kwargs, timeout=timeout)
 
     @property
     def broker(self) -> "BaseBroker[t.Any, t.Any]":
@@ -76,9 +76,7 @@ def queue(
         consume_timeout: float = ...,
         consumer_pool_executor_cls: type["BasePoolExecutor"] = ...,
         consumer_pool_maxsize: int = ...,
-        message_cls: type["BaseMessage"] = ...,
-        message_serializer_cls: type["BaseSerializer"] = ...,
-        enable_message_serializer: bool = ...,
+        serializer_cls: type["BaseSerializer"] = ...,
         publish_timeout: float | None = ...,
         maxsize: int = ...,
 ) -> t.Callable[[t.Callable[_P, t.Coroutine[t.Any, t.Any, _R]]], QueueFunc[_P, _R]]: