py-messor 0.2.4__tar.gz

py_messor-0.2.4/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

py_messor-0.2.4/PKG-INFO

@@ -0,0 +1,311 @@
+Metadata-Version: 2.4
+Name: py-messor
+Version: 0.2.4
+Summary: Async coroutine functions as queue-backed tasks (QueueFunc).
+License: Apache-2.0
+License-File: LICENSE
+Author: liuseki
+Author-email: liuseki@163.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Provides-Extra: rabbit
+Requires-Dist: aio-pika (>=9.6,<10.0) ; extra == "rabbit"
+Requires-Dist: orjson
+Requires-Dist: pydantic (>=2.14.1,<3.0.0)
+Project-URL: Changelog, https://github.com/sekil9529/py-messor/blob/master/CHANGELOG.md
+Project-URL: Documentation, https://github.com/sekil9529/py-messor#readme
+Project-URL: Homepage, https://github.com/sekil9529/py-messor
+Project-URL: Issues, https://github.com/sekil9529/py-messor/issues
+Project-URL: Repository, https://github.com/sekil9529/py-messor
+Description-Content-Type: text/markdown
+
+# messor
+
+[![Coverage Status](https://coveralls.io/repos/github/sekil9529/py-messor/badge.svg?branch=master)](https://coveralls.io/github/sekil9529/py-messor)
+[![tests](https://github.com/sekil9529/py-messor/actions/workflows/tests.yml/badge.svg)](https://github.com/sekil9529/py-messor/actions?query=workflow%3Atests)
+[![Latest Version](https://img.shields.io/pypi/v/py-messor.svg)](https://pypi.org/project/py-messor/)
+[![Wheel](https://img.shields.io/pypi/wheel/py-messor.svg)](https://pypi.org/project/py-messor/)
+[![Python Versions](https://img.shields.io/pypi/pyversions/py-messor.svg)](https://pypi.org/project/py-messor/)
+[![License](https://img.shields.io/pypi/l/py-messor.svg)](https://pypi.org/project/py-messor/)
+
+把一个 `async def` 收成队列任务：`push` 投递，消费者在同一个事件循环里执行。
+
+仓库示例在 [examples/](examples/)。版本记录见 [CHANGELOG.md](CHANGELOG.md)。
+
+> 需要 Python 3.12 及以上。任务必须是协程函数。
+>
+> RabbitMQ 后端是可选依赖，底层用 [aio-pika](https://github.com/mosquito/aio-pika) `^9.6`。
+
+## 特点
+
+- 进程内 `asyncio.Queue` 和 RabbitMQ 两种后端，统一用 `App` 的 `@app.queue` 装饰。
+- 每个任务有自己的队列、序列化和消费并发上限。
+- `App.start()` 逆序启动消费者（下游先就绪），`App.stop()` 按关闭编排回收。
+- `start(queues=[...])` 可以只启动部分消费者；绑定的内存队列必须全部包含，否则报错。
+- `start` 会等到消费者开始收消息再返回。
+- 装饰之后仍可以直接 `await` 调用，不经过队列。
+
+## 启动和停止
+
+`app.start()` 启动消费者：内存队列 consumer 先按逆序启动，其余队列随后也按逆序启动（下游先就绪）。`start` 会等到消费者开始收消息再返回——内存队列已经堵在 `get()` 上，RabbitMQ 是队列已经订阅上。
+
+`app.stop()` 按编排关闭：
+
+1. 非内存队列 consumer 逐个 stop/join——先断掉往内存队列投递的外部源；
+2. `wait_drained` 等全部在途消息处理完成（此时内存 consumer 仍在消费）；
+3. 内存队列 consumer 逐个 stop/join；
+4. 生产者全部 stop/join；绑定但未启动的 producer 也按绑定顺序一并收尾。
+
+单个 consumer 的 stop 先标上停止，再等手头这次收消息结束；空闲时最多再等一个 `consume_timeout`。还没取走的 RabbitMQ 消息留在代理上。
+
+RabbitMQ 在消息交给消费池之后确认。处理函数出错不会把消息退回队列。`push` 会等发布确认，超时抛出 `messor.PublishTimeoutError`。
+
+## 安装
+
+```shell
+pip install py-messor
+```
+
+使用 RabbitMQ：
+
+```shell
+pip install "py-messor[rabbit]"
+```
+
+从仓库安装：
+
+```shell
+git clone https://github.com/sekil9529/py-messor.git
+cd py-messor
+poetry install -E rabbit
+```
+
+## 用法
+
+从 `messor.factory` 引入 `App`、`Backend`。`name` 必填且唯一，用来在 `start(queues=[...])` 里点名。
+
+### 进程内队列
+
+```python
+import asyncio
+
+from messor.factory import App, Backend
+
+app = App()
+
+
+@app.queue(Backend.QUEUE, name="process_order", consume_timeout=0.5, maxsize=1000)
+async def process_order(order_id: int) -> None:
+    print(order_id)
+
+
+async def main() -> None:
+    await app.start()
+    await process_order.push(args=(42,))
+    await app.stop()
+
+
+asyncio.run(main())
+```
+
+`push` 就是 `broker.publish`。不传 `timeout` 时用配置里的 `publish_timeout`；传入数字只对这一次有效。`None` 表示不限时。
+
+直接调用：
+
+```python
+await process_order(1)
+```
+
+### 多段流水线
+
+`App.start()` 逆序启动消费者：内存队列 consumer 先按逆序启动，其余队列随后也按逆序启动，下游先就绪。`stop()` 复用关闭编排：非内存 consumer 先停 -> wait_drained -> 内存 consumer -> producer。
+
+```python
+import asyncio
+
+from messor.factory import App, Backend
+
+app = App()
+
+
+@app.queue(Backend.QUEUE, name="task_b", consumer_pool_maxsize=4)
+async def task_b(job: str) -> None:
+    print(job)
+
+
+@app.queue(Backend.QUEUE, name="task_a", consumer_pool_maxsize=2)
+async def task_a(job: str) -> None:
+    await task_b.push(args=(job,))
+
+
+async def main() -> None:
+    await app.start()
+    await task_a.push(args=("job-1",))
+    await app.stop()
+
+
+asyncio.run(main())
+```
+
+三段链式示例：
+
+```shell
+poetry run python examples/demo_queue.py
+```
+
+### 只启动部分队列
+
+`app.start(queues=[name, ...])` 只启动点名的那几个消费者，其余队列的 consumer 不启动。不传 `queues` 则全部启动。
+
+- 绑定的内存队列必须全部包含在 `queues` 里，否则报错（内存队列没有独立持久化，漏启会导致投递无处消费）。
+- 未启动队列的 producer 依然可用，`app.stop()` 收尾时会按绑定顺序把它一并 stop/join。
+
+示例（内存 `task_a`/`task_b` 必须全选，rabbit 只启动 `task_r1`）：
+
+```python
+import asyncio
+
+from messor.factory import App, Backend
+
+app = App()
+
+
+@app.queue(Backend.QUEUE, name="task_a", consume_timeout=0.5)
+async def task_a(job: str) -> None:
+    print("a", job)
+    await task_b.push(args=(job,))
+
+
+@app.queue(Backend.QUEUE, name="task_b", consume_timeout=0.5)
+async def task_b(job: str) -> None:
+    print("b", job)
+
+
+@app.queue(
+    Backend.RABBIT_AIO_PIKA,
+    name="task_r1",
+    url="amqp://guest:guest@127.0.0.1:5672/",
+    queue_name="demo_partial_r1",
+    enable_declare_queue=True,
+    consume_timeout=0.5,
+)
+async def task_r1(job: str) -> None:
+    print("r1", job)
+
+
+async def main() -> None:
+    await app.start(queues=["task_a", "task_b", "task_r1"])
+    await task_a.push(args=("订单-001",))
+    await asyncio.sleep(1)
+    await app.stop()
+
+
+asyncio.run(main())
+```
+
+完整示例：
+
+```shell
+poetry run python examples/demo_partial_start.py
+```
+
+### RabbitMQ
+
+`url` 和 `queue_name` 必填。`enable_declare_queue=True` 时，队列不存在会创建。默认交换机名是 `""`，路由键默认用队列名。已经存在的队列再声明时，持久化等参数必须和代理上的一致。
+
+```python
+import asyncio
+
+from messor.factory import App, Backend
+
+app = App()
+
+
+@app.queue(
+    Backend.RABBIT_AIO_PIKA,
+    name="process_order",
+    url="amqp://guest:guest@127.0.0.1/",
+    queue_name="process_order",
+    enable_declare_queue=True,
+    consume_timeout=0.5,
+)
+async def process_order(order_id: int) -> None:
+    print(order_id)
+
+
+async def main() -> None:
+    await app.start()
+    await process_order.push(args=(42,))
+    await app.stop()
+
+
+asyncio.run(main())
+```
+
+本地 RabbitMQ（`127.0.0.1:5672`，`guest` / `guest`）上的链式示例：
+
+```shell
+poetry run python examples/demo_rabbit_aio_pika.py
+```
+
+常用参数：
+
+| 参数 | 作用 |
+|------|------|
+| `consumer_pool_maxsize` | 同时执行的协程数，默认 200 |
+| `consume_timeout` | 一次等消息的时间，默认 2 秒 |
+| `publish_timeout` | 投递等待上限，默认不限 |
+| `maxsize` | 仅内存队列，`asyncio.Queue` 容量，默认 200 |
+| `serializer_cls` | 队列中的数据结构。内存队列默认 `raw`，RabbitMQ 默认 orjson |
+| `prefetch_count` | 仅 RabbitMQ，未确认消息窗口 |
+| `delivery_mode` | 仅 RabbitMQ，默认持久化 |
+| `exchange_name` / `routing_key` | 仅 RabbitMQ |
+| `queue_durable` | 仅 RabbitMQ，声明队列时是否持久化，默认否 |
+
+其余字段见 `messor.backends.queue.config.Config` 和 `messor.backends.rabbit_aio_pika.config.Config`。
+
+## 一条任务里有什么
+
+`@app.queue` 装饰的协程就是一条队列。`push` 投到它自己这条队列上。
+
+- **Broker** 组装生产者、消费者和双方共用的资源，并提供启停与投递。
+- **Producer** 序列化后把数据写入队列。
+- **Consumer** 取出数据，反序列化后交给消费池。
+- **PoolExecutor** 按并发上限执行业务协程。
+
+```mermaid
+flowchart TB
+  PUSH["投递"] --> B["通道"]
+  B --> PRO["生产者"]
+  B -.->|启停| CON["消费者"]
+
+  PRO -->|序列化| Q[("队列")]
+  Q -->|反序列化| CON
+  CON --> POOL["消费池"]
+  POOL --> FN["业务协程"]
+```
+
+## 测试
+
+```shell
+poetry install -E rabbit
+poetry run pytest
+```
+
+单元测试使用假的 RabbitMQ 连接，不要求本机有代理。`examples/demo_rabbit_aio_pika.py` 需要本机 `guest` / `guest`、端口 `5672`。
+
+`pytest.ini` 会写出覆盖率页面 `htmlcov/` 和 `report.html`。
+
+## 版本
+
+遵循[语义化版本](https://semver.org/lang/zh-CN/)。变更记录在 [CHANGELOG.md](CHANGELOG.md)。
+
+## 许可证
+
+Apache-2.0
+