PgQueuer 0.2.0__tar.gz → 0.2.2__tar.gz

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/.github/workflows/ci.yml

@@ -7,6 +7,37 @@ on:
     branches: [main]
 
 jobs:
+  benchmark:
+
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11"]
+        os: [ubuntu-latest]
+
+    name: PgQueuer Benchmark
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Build custom PostgreSQL Docker image
+        run: |
+          docker-compose up --build -d pgq
+
+      - name: Install PgQueuer
+        run: |
+          pip install pip -U
+          pip install .[dev]
+
+      - name: Benchmark
+        run: PGUSER=testuser PGPASSWORD=testpassword PGDATABASE=testdb python3 tools/benchmark.py
+
   ci:
     strategy:
       fail-fast: false

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: PgQueuer
-Version: 0.2.0
+Version: 0.2.2
 Summary: PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.
 Author: janbjorge
 License: MIT License
@@ -26,6 +26,7 @@ Classifier: Topic :: System :: Distributed Computing
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: anyio>=4.0
 Requires-Dist: asyncpg>=0.27.0
 Requires-Dist: pgcachewatch>=0.4
 Requires-Dist: pydantic>=2.0.0
@@ -37,43 +38,40 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
 
+##  🚀 PgQueuer - Building Smoother Workflows One Queue at a Time 🚀
+[![CI](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml?query=branch%3Amain)
+[![pypi](https://img.shields.io/pypi/v/PgQueuer.svg)](https://pypi.python.org/pypi/PgQueuer)
+[![downloads](https://static.pepy.tech/badge/PgQueuer/month)](https://pepy.tech/project/PgQueuer)
+[![versions](https://img.shields.io/pypi/pyversions/PgQueuer.svg)](https://github.com/janbjorge/PgQueuer)
+
+---
+
+📚 **Documentation**: [Explore the Docs 📖](https://github.com/janbjorge/PgQueuer/wiki/)
+
+🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/PgQueuer/)
+
+---
 ## PgQueuer
 
-PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's native features like transactions, row locking, and LISTEN/NOTIFY to manage job queues effortlessly.
+PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
 
 ### Features
 
 - **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
 - **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
 - **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
-- **Scalable**: Built to efficiently scale with your application's needs.
 
-### Getting Started
+### Installation
 
-#### Installation
-
-To install PgQueuer, simply run the following command:
+To install PgQueuer, simply install with pip the following command:
 
 ```bash
 pip install PgQueuer
 ```
 
-This command installs PgQueuer along with its dependencies, setting up everything you need to start integrating it into your application.
-
-#### Database Configuration
-
-Configure your database connection settings by setting the appropriate environment variables for your PostgreSQL credentials:
-
-```bash
-export PGHOST='localhost'
-export PGDATABASE='your_database'
-export PGUSER='your_username'
-export PGPASSWORD='your_password'
-```
-
-#### CLI Usage
+### Database Configuration
 
-PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your environment variables or use the appropriate flags to specify your database credentials.
+PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your [environment variables](https://magicstack.github.io/asyncpg/current/api/index.html#connection) or use the appropriate flags to specify your database credentials.
 
 - **Installing PgQueuer Database Components**:
   ```bash
@@ -87,7 +85,7 @@ PgQueuer provides a command-line interface for easy management of installation a
 
 The CLI supports several flags to customize the connection settings. Use `--help` to see all available options.
 
-### Use-Case: Processing Incoming Data Messages
+### Show-Case: Processing Incoming Data Messages
 
 In this scenario, the system is designed to manage a queue of incoming data messages that need to be processed. Each message is encapsulated as a job, and these jobs are then processed by a designated function linked to an entrypoint. This is typical in systems where large volumes of data are collected continuously, such as telemetry data from IoT devices, logs from web servers, or transaction data in financial systems.
 
@@ -127,3 +125,12 @@ async def main() -> None:
 if __name__ == "__main__":
     asyncio.run(main())
 ```
+
+### Benchmark Summary
+
+The PgQueuer underwent basic benchmark testing to assess its performance across varying job volumes and concurrency levels.
+
+#### Key Observations:
+- **Scalability**: Performance increases with higher concurrency, demonstrating the library's ability to efficiently manage larger workloads.
+- **Consistency**: PgQueuer maintains consistent throughput across different job counts, ensuring reliable performance.
+- **Optimal Performance**: The highest throughput observed was 5,224 jobs per second at a concurrency level of 5, highlighting the library's robust handling capabilities.

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/README.md

@@ -1,40 +1,37 @@
+##  🚀 PgQueuer - Building Smoother Workflows One Queue at a Time 🚀
+[![CI](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml?query=branch%3Amain)
+[![pypi](https://img.shields.io/pypi/v/PgQueuer.svg)](https://pypi.python.org/pypi/PgQueuer)
+[![downloads](https://static.pepy.tech/badge/PgQueuer/month)](https://pepy.tech/project/PgQueuer)
+[![versions](https://img.shields.io/pypi/pyversions/PgQueuer.svg)](https://github.com/janbjorge/PgQueuer)
+
+---
+
+📚 **Documentation**: [Explore the Docs 📖](https://github.com/janbjorge/PgQueuer/wiki/)
+
+🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/PgQueuer/)
+
+---
 ## PgQueuer
 
-PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's native features like transactions, row locking, and LISTEN/NOTIFY to manage job queues effortlessly.
+PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
 
 ### Features
 
 - **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
 - **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
 - **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
-- **Scalable**: Built to efficiently scale with your application's needs.
 
-### Getting Started
+### Installation
 
-#### Installation
-
-To install PgQueuer, simply run the following command:
+To install PgQueuer, simply install with pip the following command:
 
 ```bash
 pip install PgQueuer
 ```
 
-This command installs PgQueuer along with its dependencies, setting up everything you need to start integrating it into your application.
-
-#### Database Configuration
+### Database Configuration
 
-Configure your database connection settings by setting the appropriate environment variables for your PostgreSQL credentials:
-
-```bash
-export PGHOST='localhost'
-export PGDATABASE='your_database'
-export PGUSER='your_username'
-export PGPASSWORD='your_password'
-```
-
-#### CLI Usage
-
-PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your environment variables or use the appropriate flags to specify your database credentials.
+PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your [environment variables](https://magicstack.github.io/asyncpg/current/api/index.html#connection) or use the appropriate flags to specify your database credentials.
 
 - **Installing PgQueuer Database Components**:
   ```bash
@@ -48,7 +45,7 @@ PgQueuer provides a command-line interface for easy management of installation a
 
 The CLI supports several flags to customize the connection settings. Use `--help` to see all available options.
 
-### Use-Case: Processing Incoming Data Messages
+### Show-Case: Processing Incoming Data Messages
 
 In this scenario, the system is designed to manage a queue of incoming data messages that need to be processed. Each message is encapsulated as a job, and these jobs are then processed by a designated function linked to an entrypoint. This is typical in systems where large volumes of data are collected continuously, such as telemetry data from IoT devices, logs from web servers, or transaction data in financial systems.
 
@@ -87,4 +84,13 @@ async def main() -> None:
 
 if __name__ == "__main__":
     asyncio.run(main())
-```
+```
+
+### Benchmark Summary
+
+The PgQueuer underwent basic benchmark testing to assess its performance across varying job volumes and concurrency levels.
+
+#### Key Observations:
+- **Scalability**: Performance increases with higher concurrency, demonstrating the library's ability to efficiently manage larger workloads.
+- **Consistency**: PgQueuer maintains consistent throughput across different job counts, ensuring reliable performance.
+- **Optimal Performance**: The highest throughput observed was 5,224 jobs per second at a concurrency level of 5, highlighting the library's robust handling capabilities.

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/pyproject.toml

@@ -30,6 +30,7 @@ classifiers = [
 ]
 
 dependencies = [
+    "anyio>=4.0",
     "asyncpg>=0.27.0",
     "pgcachewatch>=0.4",
     "pydantic>=2.0.0",

pgqueuer-0.2.2/src/PgQueuer/qm.py

@@ -0,0 +1,163 @@
+from __future__ import annotations
+
+import asyncio
+import dataclasses
+import functools
+from datetime import timedelta
+from typing import (
+    TYPE_CHECKING,
+    Awaitable,
+    Callable,
+    TypeAlias,
+    TypeGuard,
+    TypeVar,
+    overload,
+)
+
+import anyio
+import anyio.to_thread
+import asyncpg
+from pgcachewatch.listeners import PGEventQueue
+from pgcachewatch.models import PGChannel
+
+from .logconfig import logger
+from .models import Job
+from .queries import DBSettings, Queries
+from .tm import TaskManager
+
+if TYPE_CHECKING:
+    AsyncEntrypoint: TypeAlias = Callable[[Job], Awaitable[None]]
+    SyncEntrypoint: TypeAlias = Callable[[Job], None]
+    Entrypoint: TypeAlias = AsyncEntrypoint | SyncEntrypoint
+    T = TypeVar("T", bound=Entrypoint)
+
+
+@overload
+def is_async_callable(obj: AsyncEntrypoint) -> TypeGuard[AsyncEntrypoint]: ...
+
+
+@overload
+def is_async_callable(obj: SyncEntrypoint) -> TypeGuard[SyncEntrypoint]: ...
+
+
+def is_async_callable(obj: object) -> bool:
+    # Inspired by:
+    # https://github.com/encode/starlette/blob/9f16bf5c25e126200701f6e04330864f4a91a898/starlette/_utils.py#L38
+    while isinstance(obj, functools.partial):
+        obj = obj.func
+
+    return asyncio.iscoroutinefunction(obj) or (
+        callable(obj) and asyncio.iscoroutinefunction(obj.__call__)
+    )
+
+
+@dataclasses.dataclass
+class QueueManager:
+    """
+    Manages job queues and dispatches jobs to registered entry points,
+    handling database connections and events.
+    """
+
+    pool: asyncpg.Pool
+    queries: Queries = dataclasses.field(init=False)
+    channel: PGChannel = dataclasses.field(
+        default=PGChannel(DBSettings().channel),
+    )
+    alive: bool = dataclasses.field(
+        init=False,
+        default=True,
+    )
+    # Should registry be a weakref?
+    registry: dict[str, Entrypoint] = dataclasses.field(
+        init=False,
+        default_factory=dict,
+    )
+
+    def __post_init__(self) -> None:
+        """
+        Initializes database query handlers and validates pool size upon
+        instance creation.
+        """
+        if self.pool.get_min_size() < 1:
+            raise ValueError("... min size must be gt 1.")
+        self.queries = Queries(self.pool)
+
+    def entrypoint(self, name: str) -> Callable[[T], T]:
+        """
+        Decorator to register a function as an entrypoint for
+        handling specific job types.
+        """
+
+        if name in self.registry:
+            raise RuntimeError(f"{name} already in registry, name must be unique.")
+
+        def register(func: T) -> T:
+            self.registry[name] = func
+            return func
+
+        return register
+
+    async def run(
+        self,
+        dequeue_timeout: timedelta = timedelta(seconds=30),
+    ) -> None:
+        """
+        Starts the event listener and continuously dispatches jobs to
+        registered entry points until stopped.
+        """
+        async with (
+            self.pool.acquire() as conn,
+            TaskManager() as tm,
+        ):
+            listener = PGEventQueue()
+            await listener.connect(conn, self.channel)
+
+            while self.alive:
+                while self.alive and (job := await self.queries.dequeue()):
+                    tm.add(asyncio.create_task(self._dispatch(job)))
+
+                try:
+                    await asyncio.wait_for(
+                        listener.get(),
+                        timeout=dequeue_timeout.total_seconds(),
+                    )
+                except asyncio.TimeoutError:
+                    logger.debug(
+                        "Timeout after %r without receiving an event.",
+                        dequeue_timeout,
+                    )
+
+            await conn.reset()
+
+    async def _dispatch(self, job: Job) -> None:
+        """
+        Internal method to asynchronously handle job dispatch,
+        including exception logging and job status updates.
+        """
+
+        logger.debug(
+            "Dispatching entrypoint/id: %s/%s",
+            job.entrypoint,
+            job.id,
+        )
+
+        try:
+            fn = self.registry[job.entrypoint]
+            if is_async_callable(fn):
+                await fn(job)
+            else:
+                await anyio.to_thread.run_sync(fn, job)
+        except Exception:
+            logger.exception(
+                "Exception while processing entrypoint/id: %s/%s",
+                job.entrypoint,
+                job.id,
+            )
+            await self.queries.log_job(job, "exception")
+        else:
+            logger.debug(
+                "Dispatching entrypoint/id: %s/%s - successful",
+                job.entrypoint,
+                job.id,
+            )
+            await self.queries.log_job(job, "successful")

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/PgQueuer/queries.py

@@ -79,7 +79,8 @@ class Queries:
                 entrypoint TEXT NOT NULL,
                 payload BYTEA
             );
-            CREATE UNIQUE INDEX ON {self.settings.queue_table} (priority, id) WHERE status != 'picked';
+            CREATE INDEX ON {self.settings.queue_table} (priority ASC, id DESC)
+                INCLUDE (id) WHERE status = 'queued';
 
             CREATE TYPE {self.settings.log_table_status_type} AS ENUM ('exception', 'successful');
             CREATE TABLE {self.settings.log_table} (
@@ -92,16 +93,42 @@ class Queries:
             );
 
             CREATE FUNCTION {self.settings.function}() RETURNS TRIGGER AS $$
+            DECLARE
+                to_emit BOOLEAN := false;  -- Flag to decide whether to emit a notification
             BEGIN
-                PERFORM pg_notify(
-                '{self.settings.channel}',
-                json_build_object(
-                    'channel', '{self.settings.channel}',
-                    'operation', lower(TG_OP),
-                    'sent_at', NOW(),
-                    'table', TG_TABLE_NAME
-                )::text);
-                RETURN NEW;
+                -- Check operation type and set the emit flag accordingly
+                IF TG_OP = 'UPDATE' AND OLD IS DISTINCT FROM NEW THEN
+                    to_emit := true;
+                ELSIF TG_OP = 'DELETE' THEN
+                    to_emit := true;
+                ELSIF TG_OP = 'INSERT' THEN
+                    to_emit := true;
+                ELSIF TG_OP = 'TRUNCATE' THEN
+                    to_emit := true;
+                END IF;
+
+                -- Perform notification if the emit flag is set
+                IF to_emit THEN
+                    PERFORM pg_notify(
+                        '{self.settings.channel}',
+                        json_build_object(
+                            'channel', '{self.settings.channel}',
+                            'operation', lower(TG_OP),
+                            'sent_at', NOW(),
+                            'table', TG_TABLE_NAME
+                        )::text
+                    );
+                END IF;
+
+                -- Return appropriate value based on the operation
+                IF TG_OP IN ('INSERT', 'UPDATE') THEN
+                    RETURN NEW;
+                ELSIF TG_OP = 'DELETE' THEN
+                    RETURN OLD;
+                ELSE
+                    RETURN NULL; -- For TRUNCATE and other non-row-specific contexts
+                END IF;
+
             END;
             $$ LANGUAGE plpgsql;
 
@@ -135,14 +162,10 @@ class Queries:
         """
         query = f"""
             WITH next_job AS (
-                SELECT p1.id
-                FROM {self.settings.queue_table} p1
-                WHERE p1.status = 'queued' AND NOT EXISTS (
-                    SELECT FROM
-                    {self.settings.queue_table} p2
-                    WHERE p1.entrypoint = p2.entrypoint AND p2.status = 'picked'
-                )
-                ORDER BY priority, id
+                SELECT id
+                FROM {self.settings.queue_table}
+                WHERE status = 'queued'
+                ORDER BY priority DESC, id ASC
                 FOR UPDATE SKIP LOCKED
                 LIMIT 1
             )
@@ -158,20 +181,39 @@ class Queries:
 
     async def enqueue(
         self,
-        entrypoint: str,
-        payload: bytes | None,
-        priority: int = 0,
+        entrypoint: str | list[str],
+        payload: bytes | None | list[bytes | None],
+        priority: int | list[int] = 0,
     ) -> None:
         """
         Inserts a new job into the queue with the specified
         entrypoint, payload, and priority, marking it as 'queued'.
+        This method ensures that if any of entrypoint, payload, or priority is a list,
+        all list arguments must be of the same length.
+
+        If two or more arguments are lists, they must be the same length;
+            otherwise, an ValueError is raised.
         """
+
+        # If they are not lists, create single-item lists for uniform processing
+        normed_entrypoint = entrypoint if isinstance(entrypoint, list) else [entrypoint]
+        normed_payload = payload if isinstance(payload, list) else [payload]
+        normed_priority = priority if isinstance(priority, list) else [priority]
+
         query = f"""
             INSERT INTO {self.settings.queue_table} (priority, status, entrypoint, payload)
             VALUES ($1, 'queued', $2, $3)
         """  # noqa: E501
 
-        await self.pool.execute(query, priority, entrypoint, payload)
+        await self.pool.executemany(
+            query,
+            zip(
+                normed_priority,
+                normed_entrypoint,
+                normed_payload,
+                strict=True,
+            ),
+        )
 
     async def clear_queue(self, entrypoint: str | list[str] | None = None) -> None:
         """

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/PgQueuer/tm.py

@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import asyncio
 import dataclasses
 
@@ -21,3 +23,9 @@ class TaskManager:
         """
         self.tasks.add(task)
         task.add_done_callback(self.tasks.remove)
+
+    async def __aenter__(self) -> TaskManager:
+        return self
+
+    async def __aexit__(self, *_: object) -> None:
+        await asyncio.gather(*self.tasks)

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/PgQueuer.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: PgQueuer
-Version: 0.2.0
+Version: 0.2.2
 Summary: PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.
 Author: janbjorge
 License: MIT License
@@ -26,6 +26,7 @@ Classifier: Topic :: System :: Distributed Computing
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: anyio>=4.0
 Requires-Dist: asyncpg>=0.27.0
 Requires-Dist: pgcachewatch>=0.4
 Requires-Dist: pydantic>=2.0.0
@@ -37,43 +38,40 @@ Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
 
+##  🚀 PgQueuer - Building Smoother Workflows One Queue at a Time 🚀
+[![CI](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/PgQueuer/actions/workflows/ci.yml?query=branch%3Amain)
+[![pypi](https://img.shields.io/pypi/v/PgQueuer.svg)](https://pypi.python.org/pypi/PgQueuer)
+[![downloads](https://static.pepy.tech/badge/PgQueuer/month)](https://pepy.tech/project/PgQueuer)
+[![versions](https://img.shields.io/pypi/pyversions/PgQueuer.svg)](https://github.com/janbjorge/PgQueuer)
+
+---
+
+📚 **Documentation**: [Explore the Docs 📖](https://github.com/janbjorge/PgQueuer/wiki/)
+
+🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/PgQueuer/)
+
+---
 ## PgQueuer
 
-PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's native features like transactions, row locking, and LISTEN/NOTIFY to manage job queues effortlessly.
+PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
 
 ### Features
 
 - **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
 - **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
 - **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
-- **Scalable**: Built to efficiently scale with your application's needs.
 
-### Getting Started
+### Installation
 
-#### Installation
-
-To install PgQueuer, simply run the following command:
+To install PgQueuer, simply install with pip the following command:
 
 ```bash
 pip install PgQueuer
 ```
 
-This command installs PgQueuer along with its dependencies, setting up everything you need to start integrating it into your application.
-
-#### Database Configuration
-
-Configure your database connection settings by setting the appropriate environment variables for your PostgreSQL credentials:
-
-```bash
-export PGHOST='localhost'
-export PGDATABASE='your_database'
-export PGUSER='your_username'
-export PGPASSWORD='your_password'
-```
-
-#### CLI Usage
+### Database Configuration
 
-PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your environment variables or use the appropriate flags to specify your database credentials.
+PgQueuer provides a command-line interface for easy management of installation and uninstallation. Ensure you have configured your [environment variables](https://magicstack.github.io/asyncpg/current/api/index.html#connection) or use the appropriate flags to specify your database credentials.
 
 - **Installing PgQueuer Database Components**:
   ```bash
@@ -87,7 +85,7 @@ PgQueuer provides a command-line interface for easy management of installation a
 
 The CLI supports several flags to customize the connection settings. Use `--help` to see all available options.
 
-### Use-Case: Processing Incoming Data Messages
+### Show-Case: Processing Incoming Data Messages
 
 In this scenario, the system is designed to manage a queue of incoming data messages that need to be processed. Each message is encapsulated as a job, and these jobs are then processed by a designated function linked to an entrypoint. This is typical in systems where large volumes of data are collected continuously, such as telemetry data from IoT devices, logs from web servers, or transaction data in financial systems.
 
@@ -127,3 +125,12 @@ async def main() -> None:
 if __name__ == "__main__":
     asyncio.run(main())
 ```
+
+### Benchmark Summary
+
+The PgQueuer underwent basic benchmark testing to assess its performance across varying job volumes and concurrency levels.
+
+#### Key Observations:
+- **Scalability**: Performance increases with higher concurrency, demonstrating the library's ability to efficiently manage larger workloads.
+- **Consistency**: PgQueuer maintains consistent throughput across different job counts, ensuring reliable performance.
+- **Optimal Performance**: The highest throughput observed was 5,224 jobs per second at a concurrency level of 5, highlighting the library's robust handling capabilities.

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/PgQueuer.egg-info/SOURCES.txt

@@ -27,4 +27,5 @@ test/test_qm.py
 test/test_queries.py
 test/test_tm.py
 test/db/Dockerfile
-test/db/init_db.sh
+test/db/init_db.sh
+tools/benchmark.py

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/PgQueuer.egg-info/requires.txt

@@ -1,3 +1,4 @@
+anyio>=4.0
 asyncpg>=0.27.0
 pgcachewatch>=0.4
 pydantic>=2.0.0

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/src/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.2.0'
-__version_tuple__ = version_tuple = (0, 2, 0)
+__version__ = version = '0.2.2'
+__version_tuple__ = version_tuple = (0, 2, 2)

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/test/conftest.py

@@ -27,7 +27,7 @@ async def pgpool() -> AsyncGenerator[asyncpg.Pool, None]:
     async with (
         asyncpg.create_pool(dsn=dsn(database="postgres")) as p1,
         create_test_database(database, p1),
-        asyncpg.create_pool(dsn(database=database), max_size=40) as p2,
+        asyncpg.create_pool(dsn(database=database), max_size=50) as p2,
     ):
         yield p2
 

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/test/test_qm.py

@@ -1,4 +1,5 @@
 import asyncio
+import time
 
 import asyncpg
 import pytest
@@ -7,7 +8,7 @@ from PgQueuer.qm import QueueManager
 from PgQueuer.queries import Queries
 
 
-@pytest.mark.parametrize("N", (1, 2, 32, 500))
+@pytest.mark.parametrize("N", (1, 2, 32))
 async def test_job_queing(
     pgpool: asyncpg.Pool,
     N: int,
@@ -33,7 +34,7 @@ async def test_job_queing(
     assert seen == list(range(N))
 
 
-@pytest.mark.parametrize("N", (1, 2, 32, 512))
+@pytest.mark.parametrize("N", (1, 2, 32))
 @pytest.mark.parametrize("concurrency", (1, 2, 3, 4))
 async def test_job_fetch(
     pgpool: asyncpg.Pool,
@@ -66,3 +67,39 @@ async def test_job_fetch(
         timeout=10,
     )
     assert sorted(seen) == list(range(N))
+
+
+@pytest.mark.parametrize("N", (1, 2, 32))
+@pytest.mark.parametrize("concurrency", (1, 2, 3, 4))
+async def test_sync_entrypoint(
+    pgpool: asyncpg.Pool,
+    N: int,
+    concurrency: int,
+) -> None:
+    q = Queries(pgpool)
+    qmpool = [QueueManager(pgpool) for _ in range(concurrency)]
+    seen = list[int]()
+
+    for qm in qmpool:
+
+        @qm.entrypoint("fetch")
+        def fetch(context: Job) -> None:
+            time.sleep(2)  # Sim. heavy CPU/IO.
+            if context.payload is None:
+                for qm in qmpool:
+                    qm.alive = False
+                return
+            assert context
+            seen.append(int(context.payload))
+
+    for n in range(N):
+        await q.enqueue("fetch", f"{n}".encode())
+
+    # Stop flag
+    await q.enqueue("fetch", None)
+
+    await asyncio.wait_for(
+        asyncio.gather(*[qm.run() for qm in qmpool]),
+        timeout=10,
+    )
+    assert sorted(seen) == list(range(N))

{pgqueuer-0.2.0 → pgqueuer-0.2.2}/test/test_queries.py

@@ -2,7 +2,7 @@ import asyncio
 
 import asyncpg
 import pytest
-from PgQueuer import queries
+from PgQueuer import models, queries
 
 
 @pytest.mark.parametrize("N", (1, 2, 64))
@@ -37,7 +37,7 @@ async def test_queries_next_jobs(
     assert seen == list(range(N))
 
 
-@pytest.mark.parametrize("N", (1, 2, 64, 256))
+@pytest.mark.parametrize("N", (1, 2, 64))
 @pytest.mark.parametrize("concurrency", (1, 2, 4, 16))
 async def test_queries_next_jobs_concurrent(
     pgpool: asyncpg.Pool,
@@ -79,7 +79,7 @@ async def test_queries_clear(pgpool: asyncpg.Pool) -> None:
     assert sum(x.count for x in await q.queue_size()) == 0
 
 
-@pytest.mark.parametrize("N", (1, 2, 64, 256))
+@pytest.mark.parametrize("N", (1, 2, 64))
 async def test_move_job_log(
     pgpool: asyncpg.Pool,
     N: int,
@@ -128,3 +128,21 @@ async def test_clear_queue(
     assert sum(x.count for x in await q.queue_size()) == N
     await q.clear_queue("placeholer0")
     assert sum(x.count for x in await q.queue_size()) == N - 1
+
+
+@pytest.mark.parametrize("N", (1, 2, 64))
+async def test_queue_priority(
+    pgpool: asyncpg.Pool,
+    N: int,
+) -> None:
+    q = queries.Queries(pgpool)
+    jobs = list[models.Job]()
+
+    for n in range(N):
+        await q.enqueue("placeholer", f"{n}".encode(), priority=n)
+
+    while next_job := await q.dequeue():
+        jobs.append(next_job)
+        await q.log_job(next_job, status="successful")
+
+    assert jobs == sorted(jobs, key=lambda x: x.priority, reverse=True)

pgqueuer-0.2.2/test/test_tm.py

@@ -0,0 +1,44 @@
+import asyncio
+
+import pytest
+from PgQueuer import qm
+
+
+@pytest.mark.parametrize("N", (1, 2, 3, 5, 64))
+async def test_task_manager(N: int) -> None:
+    future = asyncio.Future[None]()
+
+    tm = qm.TaskManager()
+    assert len(tm.tasks) == 0
+
+    async def waiter(future: asyncio.Future) -> None:
+        return await future
+
+    for _ in range(N):
+        tm.add(asyncio.create_task(waiter(future)))
+
+    assert len(tm.tasks) == N
+
+    future.set_result(None)
+    await asyncio.gather(*tm.tasks)
+    assert len(tm.tasks) == 0
+
+
+@pytest.mark.parametrize("N", (1, 2, 3, 5, 64))
+async def test_task_manager_ctx_mngr(N: int) -> None:
+    async def waiter(future: asyncio.Future) -> None:
+        return await future
+
+    future = asyncio.Future[None]()
+
+    async with qm.TaskManager() as tm:
+        assert len(tm.tasks) == 0
+
+        for _ in range(N):
+            tm.add(asyncio.create_task(waiter(future)))
+
+        assert len(tm.tasks) == N
+
+        future.set_result(None)
+
+    assert len(tm.tasks) == 0

pgqueuer-0.2.2/tools/benchmark.py

@@ -0,0 +1,99 @@
+import asyncio
+import contextlib
+import time
+from datetime import timedelta
+from typing import Callable, Generator
+
+import asyncpg
+from PgQueuer.models import Job
+from PgQueuer.qm import QueueManager
+from PgQueuer.queries import Queries
+
+
+@contextlib.contextmanager
+def timer() -> (
+    Generator[
+        Callable[[], float],
+        None,
+        None,
+    ]
+):
+    enter = time.perf_counter()
+    done: float | None = None
+    try:
+        yield lambda: done - enter if done else time.perf_counter() - enter
+    finally:
+        done = time.perf_counter()
+
+
+async def jobs_per_second(pool: asyncpg.Pool) -> float:
+    qm = QueueManager(pool)
+    seen = 0
+
+    @qm.entrypoint("async")
+    async def asyncfetch(job: Job) -> None:
+        nonlocal qm, seen
+        seen += 1
+        if job.payload is None:
+            qm.alive = False
+
+    @qm.entrypoint("sync")
+    def syncfetch(job: Job) -> None:
+        nonlocal qm, seen
+        seen += 1
+        if job.payload is None:
+            qm.alive = False
+
+    with timer() as elapsed:
+        await qm.run(timedelta(seconds=0))
+
+    return seen / elapsed()
+
+
+async def benchmark(
+    entrypoint: str,
+    concurrecy: int,
+    N: int,
+    pool: asyncpg.Pool,
+) -> None:
+    queries = Queries(pool)
+
+    await queries.clear_log()
+    await queries.clear_queue()
+
+    await queries.enqueue(
+        [entrypoint] * N,
+        [f"{n}".encode() for n in range(N)],
+        [0] * N,
+    )
+    assert sum(x.count for x in await queries.queue_size()) == N
+
+    await queries.enqueue(
+        [entrypoint] * concurrecy**2,
+        [None] * concurrecy**2,
+        [0] * concurrecy**2,
+    )
+
+    jobs = [jobs_per_second(pool) for _ in range(concurrecy)]
+    results = await asyncio.gather(*jobs)
+
+    print(
+        f"Entrypoint: {entrypoint:<5} ",
+        f"Concurrecy: {concurrecy:<2} "
+        f"Jobs: {N:<5} "
+        f"JPS: {sum(results)/1_000:.1f}k",
+    )
+
+
+async def main() -> None:
+    async with asyncpg.create_pool(
+        min_size=20,
+        max_size=99,
+    ) as pool:
+        for entrypoint in ("sync", "async"):
+            for concurrecy in range(1, 6):
+                await benchmark(entrypoint, concurrecy, 1_000 * concurrecy, pool)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

pgqueuer-0.2.0/src/PgQueuer/qm.py

@@ -1,118 +0,0 @@
-from __future__ import annotations
-
-import asyncio
-import dataclasses
-from typing import TYPE_CHECKING, Awaitable, Callable, TypeAlias, TypeVar
-
-import asyncpg
-from pgcachewatch.listeners import PGEventQueue
-from pgcachewatch.models import PGChannel
-
-from .logconfig import logger
-from .models import Job
-from .queries import Queries
-from .tm import TaskManager
-
-if TYPE_CHECKING:
-    EntrypointFn: TypeAlias = Callable[[Job], Awaitable[None]]
-    T = TypeVar("T", bound=EntrypointFn)
-
-
-@dataclasses.dataclass
-class QueueManager:
-    """
-    Manages job queues and dispatches jobs to registered entry points,
-    handling database connections and events.
-    """
-
-    pool: asyncpg.Pool
-    queries: Queries = dataclasses.field(init=False)
-
-    channel: PGChannel = dataclasses.field(
-        default=PGChannel("ch_pgqueuer"),
-        init=False,
-    )
-    alive: bool = dataclasses.field(
-        init=False,
-        default=True,
-    )
-    # Should registry be a weakref?
-    registry: dict[str, EntrypointFn] = dataclasses.field(
-        init=False,
-        default_factory=dict,
-    )
-    tm: TaskManager = dataclasses.field(
-        init=False,
-        default_factory=TaskManager,
-    )
-
-    def __post_init__(self) -> None:
-        """
-        Initializes database query handlers and validates pool size upon
-        instance creation.
-        """
-        if self.pool.get_min_size() < 1:
-            raise ValueError("... min size must be gt 1.")
-        self.queries = Queries(self.pool)
-
-    def entrypoint(self, name: str) -> Callable[[T], T]:
-        """
-        Decorator to register a function as an entrypoint for
-        handling specific job types.
-        """
-
-        if name in self.registry:
-            raise RuntimeError(f"{name} already in registry, name must be unique.")
-
-        def register(func: T) -> T:
-            self.registry[name] = func
-            return func
-
-        return register
-
-    async def run(self) -> None:
-        """
-        Starts the event listener and continuously dispatches jobs to
-        registered entry points until stopped.
-        """
-        async with self.pool.acquire() as conn:
-            listener = PGEventQueue()
-            await listener.connect(conn, self.channel)
-
-            while self.alive:
-                while job := await self.queries.dequeue():
-                    self._dispatch(job)
-                await listener.get()
-
-            await asyncio.gather(*self.tm.tasks)
-
-    def _dispatch(self, job: Job) -> None:
-        """
-        Internal method to asynchronously handle job dispatch,
-        including exception logging and job status updates.
-        """
-
-        async def runit() -> None:
-            logger.debug(
-                "Dispatching entrypoint/id: %s/%s",
-                job.entrypoint,
-                job.id,
-            )
-            try:
-                await self.registry[job.entrypoint](job)
-            except Exception:
-                logger.exception(
-                    "Exception while processing entrypoint/id: %s/%s",
-                    job.entrypoint,
-                    job.id,
-                )
-                await self.queries.log_job(job, "exception")
-            else:
-                logger.debug(
-                    "Dispatching entrypoint/id: %s/%s - successful",
-                    job.entrypoint,
-                    job.id,
-                )
-                await self.queries.log_job(job, "successful")
-
-        self.tm.add(asyncio.create_task(runit()))

pgqueuer-0.2.0/test/test_tm.py

@@ -1,24 +0,0 @@
-import asyncio
-
-import pytest
-from PgQueuer import qm
-
-
-@pytest.mark.parametrize("N", (1, 2, 64, 1024))
-async def test_task_manager(N: int) -> None:
-    future = asyncio.Future[None]()
-
-    tm = qm.TaskManager()
-    assert len(tm.tasks) == 0
-
-    async def waiter(future: asyncio.Future) -> None:
-        return await future
-
-    for _ in range(N):
-        tm.add(asyncio.create_task(waiter(future)))
-
-    assert len(tm.tasks) == N
-
-    future.set_result(None)
-    await asyncio.gather(*tm.tasks)
-    assert len(tm.tasks) == 0