pgqueuer 0.0.0__tar.gz

pgqueuer-0.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 janbjorge
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pgqueuer-0.0.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.1
+Name: pgqueuer
+Version: 0.0.0
+Summary: Pgqueuer is a Python library leveraging PostgreSQL for efficient job queuing.
+Author: janbjorge
+License: MIT License
+Project-URL: Documentation, https://github.com/janbjorge/pgqueuer/wiki
+Project-URL: Homepage, https://github.com/janbjorge/pgqueuer/
+Project-URL: Issues, https://github.com/janbjorge/pgqueuer/issues
+Project-URL: Repository, https://github.com/janbjorge/pgqueuer/
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Web Environment
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Utilities
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anyio>=4.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: tabulate>=0.9.0
+Provides-Extra: asyncpg
+Requires-Dist: asyncpg; extra == "asyncpg"
+Provides-Extra: psycopg
+Requires-Dist: psycopg>=3.2.0; extra == "psycopg"
+Provides-Extra: dev
+Requires-Dist: asyncpg; extra == "dev"
+Requires-Dist: asyncpg-stubs; extra == "dev"
+Requires-Dist: fastapi; extra == "dev"
+Requires-Dist: httpx; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: mypy-extensions; extra == "dev"
+Requires-Dist: psycopg>=3.2.0; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: tqdm; extra == "dev"
+Requires-Dist: types-tabulate; extra == "dev"
+Requires-Dist: uvicorn; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: myst-parser; extra == "docs"
+Requires-Dist: sphinx; extra == "docs"
+Requires-Dist: sphinx-rtd-theme; extra == "docs"
+
+### Readme
+## 🚀 pgqueuer - Building Smoother Workflows One Queue at a Time 🚀
+[![CI](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml?query=branch%3Amain)
+[![pypi](https://img.shields.io/pypi/v/pgqueuer.svg)](https://pypi.python.org/pypi/pgqueuer)
+[![downloads](https://static.pepy.tech/badge/pgqueuer/month)](https://pepy.tech/project/pgqueuer)
+[![versions](https://img.shields.io/pypi/pyversions/pgqueuer.svg)](https://github.com/janbjorge/pgqueuer)
+
+---
+
+📚 **Documentation**: [Explore the Docs 📖](https://pgqueuer.readthedocs.io/en/latest/)
+
+🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/pgqueuer/)
+
+💬 **Join the Discussion**: [Discord Community](https://discord.gg/C7YMBzcRMQ)
+
+---
+
+## pgqueuer
+
+pgqueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, pgqueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
+
+### Features
+
+- **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
+- **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
+- **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
+
+### Installation
+
+To install pgqueuer, simply install with pip the following command:
+
+```bash
+pip install pgqueuer
+```
+
+### Example Usage
+
+Here's how you can use pgqueuer in a typical scenario processing incoming data messages:
+
+#### Write and run a consumer
+Start a long-lived consumer that will begin processing jobs as soon as they are enqueued by another process. In this case we want to be a bit more carefull as we want gracefull shutdowns, `pgqueuer run` will setup signals to
+ensure this.
+
+```python
+from __future__ import annotations
+
+import asyncpg
+from pgqueuer.db import AsyncpgDriver, dsn
+from pgqueuer.models import Job
+from pgqueuer.qm import QueueManager
+
+
+async def main() -> QueueManager:
+    connection = await asyncpg.connect(dsn())
+    driver = AsyncpgDriver(connection)
+    qm = QueueManager(driver)
+
+    # Setup the 'fetch' entrypoint
+    @qm.entrypoint("fetch")
+    async def process_message(job: Job) -> None:
+        print(f"Processed message: {job}")
+
+    return qm
+```
+
+```bash
+python3 -m pgqueuer run tools.consumer.main
+```
+
+#### Write and run a producer
+Start a short-lived producer that will enqueue 10,000 jobs.
+```python
+from __future__ import annotations
+
+import asyncio
+import sys
+
+import asyncpg
+from pgqueuer.db import AsyncpgDriver
+from pgqueuer.queries import Queries
+
+
+async def main(N: int) -> None:
+    connection = await asyncpg.connect()
+    driver = AsyncpgDriver(connection)
+    queries = Queries(driver)
+    await queries.enqueue(
+        ["fetch"] * N,
+        [f"this is from me: {n}".encode() for n in range(1, N+1)],
+        [0] * N,
+    )
+
+
+if __name__ == "__main__":
+    print(sys.argv)
+    N = 1_000 if len(sys.argv) == 1 else int(sys.argv[1])
+    asyncio.run(main(N))
+```
+
+```bash
+python3 tools/producer.py 10000
+```

pgqueuer-0.0.0/README.md

@@ -0,0 +1,102 @@
+### Readme
+## 🚀 pgqueuer - Building Smoother Workflows One Queue at a Time 🚀
+[![CI](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml?query=branch%3Amain)
+[![pypi](https://img.shields.io/pypi/v/pgqueuer.svg)](https://pypi.python.org/pypi/pgqueuer)
+[![downloads](https://static.pepy.tech/badge/pgqueuer/month)](https://pepy.tech/project/pgqueuer)
+[![versions](https://img.shields.io/pypi/pyversions/pgqueuer.svg)](https://github.com/janbjorge/pgqueuer)
+
+---
+
+📚 **Documentation**: [Explore the Docs 📖](https://pgqueuer.readthedocs.io/en/latest/)
+
+🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/pgqueuer/)
+
+💬 **Join the Discussion**: [Discord Community](https://discord.gg/C7YMBzcRMQ)
+
+---
+
+## pgqueuer
+
+pgqueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, pgqueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
+
+### Features
+
+- **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
+- **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
+- **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
+
+### Installation
+
+To install pgqueuer, simply install with pip the following command:
+
+```bash
+pip install pgqueuer
+```
+
+### Example Usage
+
+Here's how you can use pgqueuer in a typical scenario processing incoming data messages:
+
+#### Write and run a consumer
+Start a long-lived consumer that will begin processing jobs as soon as they are enqueued by another process. In this case we want to be a bit more carefull as we want gracefull shutdowns, `pgqueuer run` will setup signals to
+ensure this.
+
+```python
+from __future__ import annotations
+
+import asyncpg
+from pgqueuer.db import AsyncpgDriver, dsn
+from pgqueuer.models import Job
+from pgqueuer.qm import QueueManager
+
+
+async def main() -> QueueManager:
+    connection = await asyncpg.connect(dsn())
+    driver = AsyncpgDriver(connection)
+    qm = QueueManager(driver)
+
+    # Setup the 'fetch' entrypoint
+    @qm.entrypoint("fetch")
+    async def process_message(job: Job) -> None:
+        print(f"Processed message: {job}")
+
+    return qm
+```
+
+```bash
+python3 -m pgqueuer run tools.consumer.main
+```
+
+#### Write and run a producer
+Start a short-lived producer that will enqueue 10,000 jobs.
+```python
+from __future__ import annotations
+
+import asyncio
+import sys
+
+import asyncpg
+from pgqueuer.db import AsyncpgDriver
+from pgqueuer.queries import Queries
+
+
+async def main(N: int) -> None:
+    connection = await asyncpg.connect()
+    driver = AsyncpgDriver(connection)
+    queries = Queries(driver)
+    await queries.enqueue(
+        ["fetch"] * N,
+        [f"this is from me: {n}".encode() for n in range(1, N+1)],
+        [0] * N,
+    )
+
+
+if __name__ == "__main__":
+    print(sys.argv)
+    N = 1_000 if len(sys.argv) == 1 else int(sys.argv[1])
+    asyncio.run(main(N))
+```
+
+```bash
+python3 tools/producer.py 10000
+```

pgqueuer-0.0.0/pgqueuer/__main__.py

@@ -0,0 +1,8 @@
+import asyncio
+import contextlib
+
+from . import cli
+
+if __name__ == "__main__":
+    with contextlib.suppress(KeyboardInterrupt):
+        asyncio.run(cli.main())

pgqueuer-0.0.0/pgqueuer/__version__.py

@@ -0,0 +1,16 @@
+# file generated by setuptools_scm
+# don't change, don't track in version control
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple, Union
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+else:
+    VERSION_TUPLE = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+
+__version__ = version = '0.7.1.dev1+g4c5074e.d20240824'
+__version_tuple__ = version_tuple = (0, 7, 1, 'dev1', 'g4c5074e.d20240824')

pgqueuer-0.0.0/pgqueuer/_version.py

@@ -0,0 +1,16 @@
+# file generated by setuptools_scm
+# don't change, don't track in version control
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from typing import Tuple, Union
+    VERSION_TUPLE = Tuple[Union[int, str], ...]
+else:
+    VERSION_TUPLE = object
+
+version: str
+__version__: str
+__version_tuple__: VERSION_TUPLE
+version_tuple: VERSION_TUPLE
+
+__version__ = version = '0.7.1.dev1+g4c5074e.d20240824'
+__version_tuple__ = version_tuple = (0, 7, 1, 'dev1', 'g4c5074e.d20240824')

pgqueuer-0.0.0/pgqueuer/buffers.py

@@ -0,0 +1,90 @@
+from __future__ import annotations
+
+import asyncio
+import dataclasses
+from datetime import datetime, timedelta
+from typing import Awaitable, Callable
+
+from .helpers import perf_counter_dt
+from .logconfig import logger
+from .models import STATUS_LOG, Job
+
+
+@dataclasses.dataclass
+class JobBuffer:
+    """
+    A buffer class that accumulates jobs and their statuses until a specified
+    capacity or timeout is reached, at which point it flushes them using a
+    provided callback function.
+
+    Attributes:
+        max_size (int): Maximum number of jobs the buffer can hold before
+            triggering a flush.
+        timeout (timedelta): Maximum time to wait before flushing the buffer,
+            regardless of the buffer size.
+        flush_callback (Callable[[list[tuple[Job, STATUS_LOG]]], Awaitable[None]]):
+            Asynchronous callback function to process jobs when the buffer is flushed.
+    """
+
+    max_size: int
+    timeout: timedelta
+    flush_callback: Callable[
+        [list[tuple[Job, STATUS_LOG]]],
+        Awaitable[None],
+    ]
+
+    alive: bool = dataclasses.field(
+        init=False,
+        default=True,
+    )
+    events: list[tuple[Job, STATUS_LOG]] = dataclasses.field(
+        init=False,
+        default_factory=list,
+    )
+    last_event_time: datetime = dataclasses.field(
+        init=False,
+        default_factory=perf_counter_dt,
+    )
+    lock: asyncio.Lock = dataclasses.field(
+        init=False,
+        default_factory=asyncio.Lock,
+    )
+
+    async def add_job(self, job: Job, status: STATUS_LOG) -> None:
+        """
+        Adds a job and its status to the buffer and flushes the buffer
+        if it reaches maximum size.
+        """
+        async with self.lock:
+            self.events.append((job, status))
+            self.last_event_time = perf_counter_dt()
+            if len(self.events) >= self.max_size:
+                await self.flush_jobs()
+
+    async def flush_jobs(self) -> None:
+        """
+        Flushes the buffer by calling the flush callback with all accumulated jobs
+        and statuses. Clears the buffer after flushing.
+        """
+        while self.events:
+            try:
+                await self.flush_callback(self.events)
+            except Exception:
+                logger.exception(
+                    "Exception during buffer flush, waiting: %s seconds before retry.",
+                    self.timeout.total_seconds(),
+                )
+                await asyncio.sleep(self.timeout.total_seconds())
+            else:
+                self.events.clear()
+
+    async def monitor(self) -> None:
+        """
+        Periodically checks if the buffer needs to be flushed based on the timeout.
+        Runs until the `alive` event is cleared.
+        """
+        while self.alive:
+            await asyncio.sleep(self.timeout.total_seconds())
+            async with self.lock:
+                if perf_counter_dt() - self.last_event_time >= self.timeout:
+                    await self.flush_jobs()