queueio 0.1.2__tar.gz → 0.2.0__tar.gz

queueio-0.2.0/.devcontainer/devcontainer.json

@@ -0,0 +1,7 @@
+{
+  "dockerComposeFile": ["docker-compose.yml"],
+  "service": "dev",
+  "features": {
+    "ghcr.io/va-h/devcontainers-features/uv:1": {}
+  }
+}

queueio-0.2.0/.devcontainer/docker-compose.yml

@@ -0,0 +1,10 @@
+services:
+  dev:
+    image: mcr.microsoft.com/devcontainers/python
+    volumes:
+      - ../..:/workspaces
+    command: sleep infinity
+  rabbitmq:
+    image: rabbitmq:management
+    ports:
+      - 15672:15672

queueio-0.2.0/.github/workflows/build.yml

@@ -0,0 +1,38 @@
+name: Build
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    services:
+      rabbitmq:
+        image: rabbitmq
+        env:
+          RABBITMQ_DEFAULT_USER: guest
+          RABBITMQ_DEFAULT_PASS: guest
+        ports:
+          - 5672:5672
+        options: >-
+          --health-cmd "rabbitmq-diagnostics -q ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+      - name: Set up Python
+        run: uv python install 3.14t
+      - name: Install dependencies
+        run: uv sync --all-groups
+      - name: Check formatting
+        run: uv run ruff format --check
+      - name: Run ruff linting
+        run: uv run ruff check
+      - name: Run basedpyright
+        run: uv run basedpyright
+      - name: Run tests
+        run: uv run pytest
+        timeout-minutes: 2

queueio-0.2.0/.github/workflows/publish.yml

@@ -0,0 +1,17 @@
+name: Publish to PyPI
+on:
+  release:
+    types:
+      - published
+
+jobs:
+  publish:
+    permissions:
+      id-token: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv build
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

queueio-0.2.0/.gitignore

@@ -0,0 +1,17 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+.coverage*
+
+# Virtual environments
+.venv
+
+# uv
+uv.lock
+.python-version
+.ruff_cache/
+.ropeproject/

queueio-0.2.0/CHANGELOG.md

@@ -0,0 +1,32 @@
+Changelog
+=========
+
+All notable changes to this project will be documented in this file.
+
+The format is loosely based on [Keep a Changelog](https://keepachangelog.com).
+
+[0.2.0] - 2025-11-22
+--------------------
+
+### Acknowledgement
+
+Thank you to Nick Anderegg for allowing me to use the queueio name for this project.
+
+### Added
+
+- `routine` decorator to declare sync or async functions as background routines.
+- `activate` context manager to activate the queueio system.
+- `pause` to coordinate a pause of a routine to queueio.
+- `gather` to run multiple routines concurrently and gather the results.
+- `Routine.submit()` method to submit a routine invocation to the queue.
+- Configuration in the `tool.queueio` section of `pyproject.toml`.
+  - `pika` configures the pika library to connect to the AMQP broker.
+  - `register` configures the modules that declare routines.
+- `QUEUEIO_PIKA` environment variable
+  to override the `pika` configuration in `pyproject.toml`.
+- `queueio sync` command to synchronize queues to the broker.
+- `queueio run` command to run the queueio worker.
+- The queuespec syntax to `queue run` to consume multiple queues with shared capacity.
+- `queueio monitor` command to monitor activity in the queueio system.
+
+[0.2.0]: https://github.com/ryanhiebert/queueio/releases/tag/0.2

queueio-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Ryan Hiebert
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

queueio-0.2.0/PKG-INFO

@@ -0,0 +1,118 @@
+Metadata-Version: 2.4
+Name: queueio
+Version: 0.2.0
+Summary: Python background queues with an async twist
+Author-email: Ryan Hiebert <ryan@ryanhiebert.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: async,background,queue,rabbitmq,tasks
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.14
+Requires-Dist: dill>=0.3.9
+Requires-Dist: pika>=1.3.2
+Requires-Dist: textual>=3.2.0
+Requires-Dist: typer>=0.15.4
+Description-Content-Type: text/markdown
+
+![queueio](logo.svg)
+
+Python background queues with an async twist
+============================================
+
+Use async functions to manage complex background task workflows,
+and keep using synchronous functions for everything else.
+
+Getting Started
+---------------
+
+Install `queueio`:
+
+```sh
+pip install queueio
+```
+
+Create your routines:
+
+```python
+# basic.py
+from time import sleep
+
+from queueio import gather
+from queueio import pause
+from queueio import routine
+
+
+@routine(name="blocking", queue="queueio")
+def blocking():
+    sleep(0.1)  # Regular blocking call
+
+
+@routine(name="yielding", queue="queueio")
+async def yielding(iterations: int):
+    # Do them two at a time
+    for _ in range(iterations // 2):
+        await gather(blocking(), blocking())
+        await pause(0.2)  # Release processing capacity
+    if iterations % 2 == 1:
+        await blocking()
+```
+
+Add the configuration to your `pyproject.toml`:
+
+```toml
+[tool.queueio]
+# Configure RabbitMQ
+pika = "amqp://guest:guest@localhost:5672/"
+# Register the modules that the worker should load to find your routines
+register = ["basic"]
+```
+
+The pika configuration can be overridden with an environment variable
+to allow a project to be deployed in multiple environments.
+
+```sh
+QUEUEIO_PIKA='amqp://guest:guest@localhost:5672/'
+```
+
+Sync the queues to the broker:
+
+```python
+queueio sync
+```
+
+Submit the routine to run on a worker:
+
+```python
+from queueio import activate
+from basic import yielding
+
+with activate():
+    yielding(7).submit()
+```
+
+Then run the worker to process submitted routines:
+
+```sh
+queueio run queueio=4
+```
+
+Monitor the status of active routine invocations:
+
+```sh
+queueio monitor
+```
+
+Stability
+---------
+
+The design of the public API is under active development
+and is likely to change with any release.
+Release notes will provide upgrade instructions,
+but backward compatibility and deprecation warnings
+will not generally be implemented.

queueio-0.2.0/README.md

@@ -0,0 +1,96 @@
+![queueio](logo.svg)
+
+Python background queues with an async twist
+============================================
+
+Use async functions to manage complex background task workflows,
+and keep using synchronous functions for everything else.
+
+Getting Started
+---------------
+
+Install `queueio`:
+
+```sh
+pip install queueio
+```
+
+Create your routines:
+
+```python
+# basic.py
+from time import sleep
+
+from queueio import gather
+from queueio import pause
+from queueio import routine
+
+
+@routine(name="blocking", queue="queueio")
+def blocking():
+    sleep(0.1)  # Regular blocking call
+
+
+@routine(name="yielding", queue="queueio")
+async def yielding(iterations: int):
+    # Do them two at a time
+    for _ in range(iterations // 2):
+        await gather(blocking(), blocking())
+        await pause(0.2)  # Release processing capacity
+    if iterations % 2 == 1:
+        await blocking()
+```
+
+Add the configuration to your `pyproject.toml`:
+
+```toml
+[tool.queueio]
+# Configure RabbitMQ
+pika = "amqp://guest:guest@localhost:5672/"
+# Register the modules that the worker should load to find your routines
+register = ["basic"]
+```
+
+The pika configuration can be overridden with an environment variable
+to allow a project to be deployed in multiple environments.
+
+```sh
+QUEUEIO_PIKA='amqp://guest:guest@localhost:5672/'
+```
+
+Sync the queues to the broker:
+
+```python
+queueio sync
+```
+
+Submit the routine to run on a worker:
+
+```python
+from queueio import activate
+from basic import yielding
+
+with activate():
+    yielding(7).submit()
+```
+
+Then run the worker to process submitted routines:
+
+```sh
+queueio run queueio=4
+```
+
+Monitor the status of active routine invocations:
+
+```sh
+queueio monitor
+```
+
+Stability
+---------
+
+The design of the public API is under active development
+and is likely to change with any release.
+Release notes will provide upgrade instructions,
+but backward compatibility and deprecation warnings
+will not generally be implemented.

queueio-0.2.0/dill.pyi

@@ -0,0 +1,4 @@
+from typing import Any
+
+def dumps(obj: Any) -> bytes: ...
+def loads(data: bytes) -> Any: ...

queueio-0.2.0/logo.svg

@@ -0,0 +1 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg width="100%" height="100%" viewBox="0 0 3750 1250" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" xmlns:serif="http://www.serif.com/" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:2;"><path d="M338.012,467.371c55.9,-7.825 112.888,7.725 154.194,46.346l1.633,-34.906l110.412,0.348l0.429,621.113l-119.094,0.729l0.14,-84.292c0.581,-38.062 0.669,-76.146 0.262,-114.229c-26.765,20.792 -52.321,33.083 -86.125,38.792c-59.756,9.646 -120.892,-4.896 -169.906,-40.417c-123.606,-89.792 -124.921,-294.833 -5.21,-388.29c33.813,-26.398 71.26,-39.381 113.265,-45.194Zm19.171,112.15c-155.729,31.208 -124.015,270.292 35.14,252.375c31.623,-7.125 63.613,-22.396 78.979,-52c48.587,-93.646 0.254,-206.665 -114.119,-200.375Z" style="fill:#0384d3;"/><path d="M3268.646,460.052c134.458,-6.86 249.125,96.36 256.396,230.802c7.271,134.438 -95.604,249.417 -230.021,257.104c-135,7.708 -250.583,-95.708 -257.896,-230.729c-7.292,-135.021 96.458,-250.287 231.521,-257.177Zm2.479,120.469c-68.125,5.438 -118.958,65.021 -113.625,133.146c5.354,68.125 64.854,119.063 133,113.813c68.271,-5.271 119.312,-64.917 113.958,-133.167c-5.354,-68.271 -65.083,-119.229 -133.333,-113.792Z" style="fill:#f36c17;"/><path d="M2429.667,469.504c12.625,-1.277 25.333,-1.637 38,-1.077c58.917,2.092 114.5,27.933 154.062,71.644c52.979,58.929 65.167,125.804 61.271,202.013c-112.188,0.062 -236.083,-2.688 -347.25,0.688c33.104,99.271 122.188,114.187 212.771,76.75c14.417,-5.958 26.458,-14.292 39.333,-23.104l19.854,17.875c17.688,15.375 35.479,30.604 53.417,45.667c-45.021,45.646 -88.25,70.958 -151.542,80.979c-136.083,21.812 -272.917,-55.979 -289.583,-199.979c-16.167,-139.792 69.271,-254.075 209.667,-271.454Zm6.521,105.933c-53.875,11.125 -79.604,36.646 -97.854,87l131.833,0l93.062,-0.021c-20.5,-65.292 -58.521,-91.863 -127.042,-86.979Z" style="fill:#0384d3;"/><path d="M1393.312,469.496c13.946,-1.45 27.981,-1.844 41.985,-1.177c124.958,6.165 211.754,108.598 215.608,230.931c0.454,14.458 -0.396,28.729 -1.135,43.146c-113.421,0.146 -233.358,-1.979 -346.019,0.917c35.075,119.604 166.117,114.438 252.931,54.021l19.162,16.812l52.948,45.75c-93.298,94.875 -235.873,114.75 -348.002,38.292c-108.408,-73.938 -123.404,-237.167 -46.944,-338.815c39.348,-52.308 95.038,-80.494 159.465,-89.877Zm11.287,106.067c-52.808,10.479 -81.34,36.708 -98.487,87.188l128.262,-0.188l95.665,0.021c-20.037,-66.396 -57.262,-91.356 -125.44,-87.021Z" style="fill:#0384d3;"/><path d="M2024.729,479.54l116.958,-0.354l0.021,138.523l-0.167,311.458l-108.396,-0.312c-0.146,-11.854 -0.792,-24.396 -1.208,-36.292c-9.125,8.25 -18.333,16.771 -28.583,23.604c-67.708,45.271 -173.667,36.542 -231.077,-21.188c-20.035,-20.333 -34.302,-45.604 -41.333,-73.271c-11.271,-42.875 -8.121,-116.021 -8.058,-162.333l-0.331,-180.123l117.487,0.158c0.854,63.542 -0.438,127.694 0.25,191.319c0.729,34.896 -1.833,68.792 3.146,103.437c9.833,72.146 114.625,83.875 158.271,37.771c31.292,-33.042 23.438,-107.792 23.417,-150.979l-0.396,-181.419Z" style="fill:#0384d3;fill-rule:nonzero;"/><path d="M994.065,479.233l117.913,-0.075l0.004,138.404l0.092,311.167c-36.994,-0.792 -73.996,-1.208 -111,-1.229l-0.89,-34.312c-37.987,35.167 -67.31,49.958 -120.646,51.5c-48.3,1.417 -96.542,-13.354 -131.633,-47.625c-21.715,-21.229 -37.135,-48.062 -44.563,-77.542c-9.981,-38.771 -7.425,-111.354 -7.448,-154.083l-0.317,-186.194l116.79,0.156l0.242,182.517c0.017,35.979 -2.15,80.188 3.8,114.854c14.915,86.896 161.917,80.75 174.977,-4.979c5.375,-35.271 3.119,-75.542 3.006,-111.167l-0.327,-181.392Z" style="fill:#0384d3;fill-rule:nonzero;"/><path d="M2915.604,464.727c6.417,0.619 11.042,0.294 16.167,4.017c2.625,12.019 1.146,114.131 1.083,136.027l0.208,324.417l-125.042,0.062l-0.167,-39.146l0.188,-258.479c-20.562,0.042 -39.271,0.104 -59.875,1.292c30.375,-33.896 68,-66.692 100.604,-99.329c20.229,-20.242 46.75,-51.537 66.833,-68.86Z" style="fill:#f36c17;fill-rule:nonzero;"/><path d="M2859.583,240.567c42.292,-5.871 81.313,23.763 87.021,66.09c5.688,42.329 -24.104,81.225 -66.437,86.752c-42.125,5.496 -80.75,-24.088 -86.417,-66.175c-5.667,-42.085 23.771,-80.831 65.833,-86.667Z" style="fill:#f36c17;fill-rule:nonzero;"/></svg>

queueio-0.2.0/principles.md

@@ -0,0 +1,28 @@
+Principles
+==========
+
+There's been a great deal of momentum to embrace async functions for IO,
+but they split the world and are confusing, especially to new developers.
+Threads or green threads are often a better choice for concurrency
+because it doesn't require learning about coroutines
+before you even need concurrency.
+
+Still, async functions are a powerful way to implement state machines.
+Complex workflows can be represented as async functions
+to allow thinking about things in a pull-based fashion,
+while also scaling well for processes that must take a long time.
+Ideally, running coroutines could be serialized and resumed,
+perhaps even on a different machine.
+
+This vision motivates queueio.
+Routines need to coordinate with the worker
+because queue-level concurrency must be carefully managed.
+Instead of complex construction of pipelines to make this possible,
+queueio allows you to write complex workflows
+in a traditional imperative style.
+
+queueio encourages you to use synchronous IO in routines.
+The yield points in async functions are only needed
+when a routine needs to coordinate with the worker,
+such as to indicate that the worker can use
+the capacity reserved for it to do other work.

queueio-0.2.0/pyproject.toml

@@ -0,0 +1,66 @@
+[project]
+name = "queueio"
+version = "0.2.0"
+description = "Python background queues with an async twist"
+readme = "README.md"
+license = "MIT"
+homepage = "https://github.com/ryanhiebert/queueio"
+repository = "https://github.com/ryanhiebert/queueio"
+authors = [
+    { name = "Ryan Hiebert", email = "ryan@ryanhiebert.com" },
+]
+keywords = ["async", "queue", "background", "tasks", "rabbitmq"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: System :: Distributed Computing",
+]
+requires-python = ">=3.14"
+dependencies = [
+    "dill>=0.3.9",
+    "pika>=1.3.2",
+    "textual>=3.2.0",
+    "typer>=0.15.4",
+]
+
+[project.scripts]
+queueio = "queueio.__main__:app"
+
+[dependency-groups]
+dev = [
+    "basedpyright>=1.32.1",
+    "pytest>=8.4.1",
+    "pytest-cov>=6.0.0",
+    "pytest-timeout>=2.4.0",
+    "ruff>=0.9.6",
+    "textual-dev>=1.7.0",
+    "types-pika>=1.2.0b1",
+]
+
+[tool.ruff.lint]
+select = ["E", "F", "UP", "B", "SIM", "I"]
+
+[tool.ruff.lint.isort]
+force-single-line = true
+
+[tool.queueio]
+register = [
+    "queueio.samples.basic",
+    "queueio.samples.expanded",
+]
+pika = "amqp://localhost:5672"
+
+[tool.pytest.ini_options]
+timeout = 30
+
+[tool.coverage.run]
+branch = true
+source = ["queueio"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"

queueio-0.2.0/pyrightconfig.json

@@ -0,0 +1,4 @@
+{
+  "pythonVersion": "3.14",
+  "typeCheckingMode": "standard"
+}

queueio-0.2.0/queueio/__init__.py

@@ -0,0 +1,12 @@
+from contextlib import contextmanager
+
+from .gather import gather as gather
+from .pause import pause as pause
+from .queueio import QueueIO as RealQueueIO
+from .registry import routine as routine
+
+
+@contextmanager
+def activate():
+    with RealQueueIO().activate():
+        yield

queueio-0.2.0/queueio/__main__.py

@@ -0,0 +1,155 @@
+from typing import Annotated
+
+from typer import Argument
+from typer import Typer
+
+from .monitor import Monitor
+from .queueio import QueueIO
+from .queuespec import QueueSpec
+from .worker import Worker
+
+app = Typer()
+
+routine_app = Typer()
+queue_app = Typer()
+
+app.add_typer(
+    routine_app,
+    name="routine",
+    help="A function to coordinate background execution.",
+    rich_help_panel="Entities",
+)
+app.add_typer(
+    queue_app,
+    name="queue",
+    help="An ordered collection of work items to process.",
+    rich_help_panel="Entities",
+)
+
+
+@routine_app.command("list")
+def routine_list():
+    """Show all registered routines."""
+    queueio = QueueIO()
+    try:
+        routines = queueio.routines()
+
+        if not routines:
+            print("No routines registered.")
+            return
+
+        # Calculate column widths
+        name_width = max(len("Name"), max(len(routine.name) for routine in routines))
+        function_paths = []
+        for routine in routines:
+            module = routine.fn.__module__
+            qualname = routine.fn.__qualname__
+            function_paths.append(f"{module}.{qualname}")
+        path_width = max(len("Path"), max(len(path) for path in function_paths))
+
+        print(f"{'Name':<{name_width}} | {'Path':<{path_width}}")
+        print(f"{'-' * name_width}-+-{'-' * path_width}")
+        for routine, path in zip(routines, function_paths, strict=False):
+            print(f"{routine.name:<{name_width}} | {path:<{path_width}}")
+    finally:
+        queueio.shutdown()
+
+
+@app.command(rich_help_panel="Commands")
+def monitor(raw: bool = False):
+    """Monitor queueio events.
+
+    Show a live view of queueio activity. Use --raw for detailed event output.
+    """
+    if raw:
+        queueio = QueueIO()
+        events = queueio.subscribe({object})
+        try:
+            while True:
+                print(events.get())
+        except KeyboardInterrupt:
+            print("Shutting down gracefully.")
+        finally:
+            queueio.shutdown()
+    else:
+        Monitor().run()
+
+
+@app.command(rich_help_panel="Commands")
+def run(
+    queuespec: Annotated[
+        QueueSpec,
+        Argument(
+            parser=QueueSpec.parse,
+            help="Queue configuration in format 'queue=concurrency'. "
+            "Examples: 'production=10', 'api,background=5'",
+            metavar="QUEUE[,QUEUE2,...]=CONCURRENCY",
+        ),
+    ],
+):
+    """Run a worker to process from a queue.
+
+    The worker will process invocations from the specified queue,
+    as many at a time as specified by the concurrency.
+    """
+    queueio = QueueIO()
+    Worker(queueio, queuespec)()
+
+
+@app.command(rich_help_panel="Commands")
+def sync():
+    """Sync known queues to the broker."""
+    queueio = QueueIO()
+    try:
+        routines = queueio.routines()
+
+        if not routines:
+            print("No routines registered.")
+            return
+
+        queues = sorted({routine.queue for routine in routines})
+
+        print(f"Syncing queues for {len(routines)} routine(s):")
+        for queue in queues:
+            print(f"  Ensuring queue exists: {queue}")
+            queueio.create(queue=queue)
+
+        print(f"Successfully synced {len(queues)} queue(s)")
+    finally:
+        queueio.shutdown()
+
+
+@queue_app.command("purge")
+def queue_purge(
+    queues: Annotated[
+        str,
+        Argument(
+            help="Comma-separated list of queues to purge. "
+            "Examples: 'queueio', 'production,background'",
+            metavar="QUEUE[,QUEUE2,...]",
+        ),
+    ],
+):
+    """Purge all messages from some queues.
+
+    This will remove all pending messages from the given queues.
+    Use with caution as this operation cannot be undone.
+    """
+    queueio = QueueIO()
+    try:
+        queue_list = [q.strip() for q in queues.split(",") if q.strip()]
+        if not queue_list:
+            print("Error: No valid queue names provided")
+            return
+
+        for queue in queue_list:
+            print(f"Purging queue: {queue}")
+            queueio.purge(queue=queue)
+
+        print(f"Successfully purged {len(queue_list)} queue(s)")
+    finally:
+        queueio.shutdown()
+
+
+if __name__ == "__main__":
+    app()