beanqueue 0.1.0__tar.gz → 0.1.2__tar.gz

beanqueue-0.1.2/PKG-INFO

@@ -0,0 +1,235 @@
+Metadata-Version: 2.1
+Name: beanqueue
+Version: 0.1.2
+Summary: BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library
+License: MIT
+Author: Fang-Pen Lin
+Author-email: fangpen@launchplatform.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: click (>=8.1.7,<9.0.0)
+Requires-Dist: dependency-injector (>=4.41.0,<5.0.0)
+Requires-Dist: pg-activity (>=3.5.1,<4.0.0)
+Requires-Dist: pydantic-settings (>=2.2.1,<3.0.0)
+Requires-Dist: sqlalchemy (>=2.0.30,<3.0.0)
+Requires-Dist: venusian (>=3.1.0,<4.0.0)
+Description-Content-Type: text/markdown
+
+# BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
+BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), [PostgreSQL SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+
+**Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
+
+## Features
+
+- **Super lightweight**: Under 1K lines
+- **Easy-to-deploy**: Only rely on PostgreSQL
+- **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
+- **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
+- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is dead, the others will reschedule the tasks
+- **Customizable**: Use it as an library and build your own worker queue
+- **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
+
+## Install
+
+```bash
+pip install beanqueue
+```
+
+## Usage
+
+You can define a task processor like this
+
+```python
+from sqlalchemy.orm import Session
+
+from bq.processors.registry import processor
+from bq import models
+from .. import my_models
+from .. import image_utils
+
+@processor(channel="images")
+def resize_image(db: Session, task: models.Task, width: int, height: int):
+    image = db.query(my_models.Image).filter(my_models.Image.task == task).one()
+    image_utils.resize(image, size=(width, height))
+    db.add(image)
+    # by default the `processor` decorator has `auto_complete` flag turns on,
+    # so it will commit the db changes for us automatically
+```
+
+The `db` and `task` keyword arguments are optional.
+If you don't need to access the task object, you can simply define the function without these two parameters.
+
+To submit a task, you can either use `bq.models.Task` model object to construct the task object, insert into the
+database session and commit.
+
+```python
+from bq import models
+from .db import Session
+from .. import my_models
+
+db = Session()
+task = models.Task(
+    channel="files",
+    module="my_pkgs.files.processors",
+    name="upload_to_s3_for_backup",
+)
+file = my_models.File(
+    task=task,
+    blob_name="...",
+)
+db.add(task)
+db.add(file)
+db.commit()
+```
+
+Or, you can use the `run` helper like this:
+
+```python
+from .processors import resize_image
+from .db import Session
+from .. import my_models
+
+db = Session()
+# a Task model generated for invoking resize_image function
+task = resize_image.run(width=200, height=300)
+# associate task with your own models
+image = my_models.Image(task=task, blob_name="...")
+db.add(image)
+# we have Task model SQLALchemy event handler to send NOTIFY "<channel>" statement for you,
+# so that the workers will be woken up immediately
+db.add(task)
+# commit will make the task visible to worker immediately
+db.commit()
+```
+
+To run the worker, you can do this:
+
+```bash
+BQ_PROCESSOR_PACKAGES='["my_pkgs.processors"]' python -m bq.cmds.process images
+```
+
+To submit a task for testing purpose, you can do
+
+```bash
+python -m bq.cmds.submit images my_pkgs.processors resize_image -k '{"width": 200, "height": 300}'
+```
+
+To create tables for BeanQueue, you can run
+
+```bash
+python -m bq.cmds.create_tables
+```
+
+### Configurations
+
+Configurations can be modified by setting environment variables with `BQ_` prefix.
+For example, to set the python packages to scan for processors, you can set `BQ_PROCESSOR_PACKAGES`.
+To change the PostgreSQL database to connect to, you can set `BQ_DATABASE_URL`.
+The complete definition of configurations can be found at the [bq/config.py](bq/config.py) module.
+For now, the configurations only affect command line tools.
+
+If you want to configure BeanQueue programmatically for the command lines, you can override our [dependency-injector](https://python-dependency-injector.ets-labs.org/)'s container defined at [bq/container.py](bq/container.py) and call the command function manually.
+For example:
+
+```python
+import bq.cmds.process
+from bq.container import Container
+from bq.config import Config
+
+container = Container()
+container.wire(modules=[bq.cmds.process])
+with container.config.override(
+    Config(
+        PROCESSOR_PACKAGES=["my_pkgs.processors"],
+        DATABASE_URL="postgresql://...",
+        BATCH_SIZE=10,
+    )
+):
+    bq.cmds.process.process_tasks(channels=("images",))
+```
+
+Many other behaviors of this framework can also be modified by overriding the container defined at [bq/container.py](bq/container.py).
+
+## Why?
+
+There are countless worker queue projects. Why make yet another one?
+The primary issue with most worker queue tools is their reliance on a standalone broker server.
+Our worker queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
+However, integrating an external worker queue into the system presents a risk.
+The worker queue and the database don't share the same data view, potentially compromising data integrity and reliability.
+
+For example, you have a table of `images` to keep the user-uploaded images.
+And you have a background worker queue for resizing the uploaded images into different thumbnail sizes.
+So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the worker queue.
+
+Say you push the task to the worker queue immediately after you insert the `images` table then commit like this:
+
+```
+1. Insert into the "images" table
+2. Push resizing task to the worker queue
+3. Commit db changes
+```
+
+While this might seem like the right way to do it, there's a hidden bug.
+If the worker starts too fast before the transaction commits at step 3, it will not be able to see the new row in `images` as it has not been committed yet.
+One may need to make the task retry a few times to ensure that even if the first attempt failed, it could see the image row in the following attempt.
+But this adds complexity to the system and also increases the latency if the first attempt fails.
+Also, if the commit step fails, you will have a failed worker queue job trying to fetch a row from the database that will never exist.
+
+Another approach is to push the resize task after the database changes are committed. It works like this:
+
+```
+1. Insert into the "images" table
+2. Commit db changes
+3. Push resizing task to the worker queue
+```
+
+With this approach, we don't need to worry about workers picking up the task too early.
+However, there's another drawback.
+If step 3 for pushing a new task to the worker queue fails, the newly inserted `images` row will never be processed.
+There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the worker queue storage.
+Things will be much easier if we have a worker queue that shares the same consistent view with the worker queue.
+
+By using a database as the data storage, all the problems are gone.
+You can simply do the following:
+
+```
+1. Insert into the "images" table
+2. Insert the image resizing task into the `tasks` table
+3. Commit db changes
+```
+
+It's all or nothing!
+By doing so, you don't need to maintain another worker queue backend.
+You are probably using a database anyway, so this worker queue comes for free.
+
+Usually, a database is inefficient as the worker queues data storage because of the potential lock contention and the need for constant querying.
+However, things have changed since the [introduction of the SKIP LOCKED](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) / [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) features in PostgreSQL or other databases.
+
+This project is inspired by many of the SKIP-LOCKED-based worker queue successors.
+Why don't we just use those existing tools?
+Well, because while they work great as worker queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
+Many provide an abstraction function or gRPC method of pushing tasks into the database instead of opening it up for the user to insert the row directly with other rows and commit altogether.
+
+With BeanQueue, we don't abstract away the logic of publishing a new task into the queue.
+Instead, we open it up to let the user insert the row and choose when and what to commit to the task.
+
+## Sponsor
+
+<p align="center">
+  <a href="https://beanhub.io"><img src="https://github.com/LaunchPlatform/bq/raw/master/assets/beanhub.svg?raw=true" alt="BeanHub logo" /></a>
+</p>
+
+A modern accounting book service based on the most popular open source version control system [Git](https://git-scm.com/) and text-based double entry accounting book software [Beancount](https://beancount.github.io/docs/index.html).
+
+## Alternatives
+
+- [solid_queue](https://github.com/rails/solid_queue)
+- [postgres-tq](https://github.com/flix-tech/postgres-tq)
+- [PgQueuer](https://github.com/janbjorge/PgQueuer)
+- [hatchet](https://github.com/hatchet-dev/hatchet)
+

beanqueue-0.1.2/README.md

@@ -0,0 +1,214 @@
+# BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
+BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), [PostgreSQL SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+
+**Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
+
+## Features
+
+- **Super lightweight**: Under 1K lines
+- **Easy-to-deploy**: Only rely on PostgreSQL
+- **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
+- **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
+- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is dead, the others will reschedule the tasks
+- **Customizable**: Use it as an library and build your own worker queue
+- **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
+
+## Install
+
+```bash
+pip install beanqueue
+```
+
+## Usage
+
+You can define a task processor like this
+
+```python
+from sqlalchemy.orm import Session
+
+from bq.processors.registry import processor
+from bq import models
+from .. import my_models
+from .. import image_utils
+
+@processor(channel="images")
+def resize_image(db: Session, task: models.Task, width: int, height: int):
+    image = db.query(my_models.Image).filter(my_models.Image.task == task).one()
+    image_utils.resize(image, size=(width, height))
+    db.add(image)
+    # by default the `processor` decorator has `auto_complete` flag turns on,
+    # so it will commit the db changes for us automatically
+```
+
+The `db` and `task` keyword arguments are optional.
+If you don't need to access the task object, you can simply define the function without these two parameters.
+
+To submit a task, you can either use `bq.models.Task` model object to construct the task object, insert into the
+database session and commit.
+
+```python
+from bq import models
+from .db import Session
+from .. import my_models
+
+db = Session()
+task = models.Task(
+    channel="files",
+    module="my_pkgs.files.processors",
+    name="upload_to_s3_for_backup",
+)
+file = my_models.File(
+    task=task,
+    blob_name="...",
+)
+db.add(task)
+db.add(file)
+db.commit()
+```
+
+Or, you can use the `run` helper like this:
+
+```python
+from .processors import resize_image
+from .db import Session
+from .. import my_models
+
+db = Session()
+# a Task model generated for invoking resize_image function
+task = resize_image.run(width=200, height=300)
+# associate task with your own models
+image = my_models.Image(task=task, blob_name="...")
+db.add(image)
+# we have Task model SQLALchemy event handler to send NOTIFY "<channel>" statement for you,
+# so that the workers will be woken up immediately
+db.add(task)
+# commit will make the task visible to worker immediately
+db.commit()
+```
+
+To run the worker, you can do this:
+
+```bash
+BQ_PROCESSOR_PACKAGES='["my_pkgs.processors"]' python -m bq.cmds.process images
+```
+
+To submit a task for testing purpose, you can do
+
+```bash
+python -m bq.cmds.submit images my_pkgs.processors resize_image -k '{"width": 200, "height": 300}'
+```
+
+To create tables for BeanQueue, you can run
+
+```bash
+python -m bq.cmds.create_tables
+```
+
+### Configurations
+
+Configurations can be modified by setting environment variables with `BQ_` prefix.
+For example, to set the python packages to scan for processors, you can set `BQ_PROCESSOR_PACKAGES`.
+To change the PostgreSQL database to connect to, you can set `BQ_DATABASE_URL`.
+The complete definition of configurations can be found at the [bq/config.py](bq/config.py) module.
+For now, the configurations only affect command line tools.
+
+If you want to configure BeanQueue programmatically for the command lines, you can override our [dependency-injector](https://python-dependency-injector.ets-labs.org/)'s container defined at [bq/container.py](bq/container.py) and call the command function manually.
+For example:
+
+```python
+import bq.cmds.process
+from bq.container import Container
+from bq.config import Config
+
+container = Container()
+container.wire(modules=[bq.cmds.process])
+with container.config.override(
+    Config(
+        PROCESSOR_PACKAGES=["my_pkgs.processors"],
+        DATABASE_URL="postgresql://...",
+        BATCH_SIZE=10,
+    )
+):
+    bq.cmds.process.process_tasks(channels=("images",))
+```
+
+Many other behaviors of this framework can also be modified by overriding the container defined at [bq/container.py](bq/container.py).
+
+## Why?
+
+There are countless worker queue projects. Why make yet another one?
+The primary issue with most worker queue tools is their reliance on a standalone broker server.
+Our worker queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
+However, integrating an external worker queue into the system presents a risk.
+The worker queue and the database don't share the same data view, potentially compromising data integrity and reliability.
+
+For example, you have a table of `images` to keep the user-uploaded images.
+And you have a background worker queue for resizing the uploaded images into different thumbnail sizes.
+So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the worker queue.
+
+Say you push the task to the worker queue immediately after you insert the `images` table then commit like this:
+
+```
+1. Insert into the "images" table
+2. Push resizing task to the worker queue
+3. Commit db changes
+```
+
+While this might seem like the right way to do it, there's a hidden bug.
+If the worker starts too fast before the transaction commits at step 3, it will not be able to see the new row in `images` as it has not been committed yet.
+One may need to make the task retry a few times to ensure that even if the first attempt failed, it could see the image row in the following attempt.
+But this adds complexity to the system and also increases the latency if the first attempt fails.
+Also, if the commit step fails, you will have a failed worker queue job trying to fetch a row from the database that will never exist.
+
+Another approach is to push the resize task after the database changes are committed. It works like this:
+
+```
+1. Insert into the "images" table
+2. Commit db changes
+3. Push resizing task to the worker queue
+```
+
+With this approach, we don't need to worry about workers picking up the task too early.
+However, there's another drawback.
+If step 3 for pushing a new task to the worker queue fails, the newly inserted `images` row will never be processed.
+There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the worker queue storage.
+Things will be much easier if we have a worker queue that shares the same consistent view with the worker queue.
+
+By using a database as the data storage, all the problems are gone.
+You can simply do the following:
+
+```
+1. Insert into the "images" table
+2. Insert the image resizing task into the `tasks` table
+3. Commit db changes
+```
+
+It's all or nothing!
+By doing so, you don't need to maintain another worker queue backend.
+You are probably using a database anyway, so this worker queue comes for free.
+
+Usually, a database is inefficient as the worker queues data storage because of the potential lock contention and the need for constant querying.
+However, things have changed since the [introduction of the SKIP LOCKED](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) / [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) features in PostgreSQL or other databases.
+
+This project is inspired by many of the SKIP-LOCKED-based worker queue successors.
+Why don't we just use those existing tools?
+Well, because while they work great as worker queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
+Many provide an abstraction function or gRPC method of pushing tasks into the database instead of opening it up for the user to insert the row directly with other rows and commit altogether.
+
+With BeanQueue, we don't abstract away the logic of publishing a new task into the queue.
+Instead, we open it up to let the user insert the row and choose when and what to commit to the task.
+
+## Sponsor
+
+<p align="center">
+  <a href="https://beanhub.io"><img src="https://github.com/LaunchPlatform/bq/raw/master/assets/beanhub.svg?raw=true" alt="BeanHub logo" /></a>
+</p>
+
+A modern accounting book service based on the most popular open source version control system [Git](https://git-scm.com/) and text-based double entry accounting book software [Beancount](https://beancount.github.io/docs/index.html).
+
+## Alternatives
+
+- [solid_queue](https://github.com/rails/solid_queue)
+- [postgres-tq](https://github.com/flix-tech/postgres-tq)
+- [PgQueuer](https://github.com/janbjorge/PgQueuer)
+- [hatchet](https://github.com/hatchet-dev/hatchet)

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/db/base.py

@@ -1,5 +1,5 @@
-from sqlalchemy.orm import DeclarativeBase
-
-
-class Base(DeclarativeBase):
-    pass
+from sqlalchemy.orm import DeclarativeBase
+
+
+class Base(DeclarativeBase):
+    pass

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/db/session.py

@@ -1,5 +1,5 @@
-from sqlalchemy.orm import scoped_session
-from sqlalchemy.orm import sessionmaker
-
-SessionMaker = sessionmaker()
-Session = scoped_session(SessionMaker)
+from sqlalchemy.orm import scoped_session
+from sqlalchemy.orm import sessionmaker
+
+SessionMaker = sessionmaker()
+Session = scoped_session(SessionMaker)

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/models/__init__.py

@@ -1,4 +1,4 @@
-from .task import Task
-from .task import TaskState
-from .worker import Worker
-from .worker import WorkerState
+from .task import Task
+from .task import TaskState
+from .worker import Worker
+from .worker import WorkerState

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/models/helpers.py

@@ -1,5 +1,5 @@
-import typing
-
-
-def make_repr_attrs(items: typing.Sequence[typing.Tuple[str, typing.Any]]) -> str:
-    return " ".join(map(lambda item: "=".join([item[0], str(item[1])]), items))
+import typing
+
+
+def make_repr_attrs(items: typing.Sequence[typing.Tuple[str, typing.Any]]) -> str:
+    return " ".join(map(lambda item: "=".join([item[0], str(item[1])]), items))

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/models/worker.py

@@ -1,69 +1,69 @@
-import enum
-
-from sqlalchemy import Column
-from sqlalchemy import DateTime
-from sqlalchemy import Enum
-from sqlalchemy import func
-from sqlalchemy import String
-from sqlalchemy.dialects.postgresql import ARRAY
-from sqlalchemy.dialects.postgresql import UUID
-from sqlalchemy.orm import relationship
-
-from ..db.base import Base
-from .helpers import make_repr_attrs
-
-
-class WorkerState(enum.Enum):
-    # the worker is running
-    RUNNING = "RUNNING"
-    # the worker shuts down normally
-    SHUTDOWN = "SHUTDOWN"
-    # The worker has no heartbeat for a while
-    NO_HEARTBEAT = "NO_HEARTBEAT"
-
-
-class Worker(Base):
-    id = Column(
-        UUID(as_uuid=True), primary_key=True, server_default=func.gen_random_uuid()
-    )
-    # current state of the worker
-    state = Column(
-        Enum(WorkerState),
-        nullable=False,
-        default=WorkerState.RUNNING,
-        server_default=WorkerState.RUNNING.value,
-        index=True,
-    )
-    # name of the worker
-    name = Column(String, nullable=False)
-    # the channels we are processing
-    channels = Column(ARRAY(String), nullable=False)
-    # last heartbeat of this worker
-    last_heartbeat = Column(
-        DateTime(timezone=True),
-        nullable=False,
-        server_default=func.now(),
-        index=True,
-    )
-    # created datetime of the worker
-    created_at = Column(
-        DateTime(timezone=True), nullable=False, server_default=func.now()
-    )
-
-    tasks = relationship(
-        "Task",
-        back_populates="worker",
-        cascade="all,delete",
-        order_by="Task.created_at",
-    )
-
-    __tablename__ = "bq_workers"
-
-    def __repr__(self) -> str:
-        items = [
-            ("id", self.id),
-            ("name", self.name),
-            ("channels", self.channels),
-            ("state", self.state),
-        ]
-        return f"<{self.__class__.__name__} {make_repr_attrs(items)}>"
+import enum
+
+from sqlalchemy import Column
+from sqlalchemy import DateTime
+from sqlalchemy import Enum
+from sqlalchemy import func
+from sqlalchemy import String
+from sqlalchemy.dialects.postgresql import ARRAY
+from sqlalchemy.dialects.postgresql import UUID
+from sqlalchemy.orm import relationship
+
+from ..db.base import Base
+from .helpers import make_repr_attrs
+
+
+class WorkerState(enum.Enum):
+    # the worker is running
+    RUNNING = "RUNNING"
+    # the worker shuts down normally
+    SHUTDOWN = "SHUTDOWN"
+    # The worker has no heartbeat for a while
+    NO_HEARTBEAT = "NO_HEARTBEAT"
+
+
+class Worker(Base):
+    id = Column(
+        UUID(as_uuid=True), primary_key=True, server_default=func.gen_random_uuid()
+    )
+    # current state of the worker
+    state = Column(
+        Enum(WorkerState),
+        nullable=False,
+        default=WorkerState.RUNNING,
+        server_default=WorkerState.RUNNING.value,
+        index=True,
+    )
+    # name of the worker
+    name = Column(String, nullable=False)
+    # the channels we are processing
+    channels = Column(ARRAY(String), nullable=False)
+    # last heartbeat of this worker
+    last_heartbeat = Column(
+        DateTime(timezone=True),
+        nullable=False,
+        server_default=func.now(),
+        index=True,
+    )
+    # created datetime of the worker
+    created_at = Column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+    tasks = relationship(
+        "Task",
+        back_populates="worker",
+        cascade="all,delete",
+        order_by="Task.created_at",
+    )
+
+    __tablename__ = "bq_workers"
+
+    def __repr__(self) -> str:
+        items = [
+            ("id", self.id),
+            ("name", self.name),
+            ("channels", self.channels),
+            ("state", self.state),
+        ]
+        return f"<{self.__class__.__name__} {make_repr_attrs(items)}>"

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/processors/registry.py

@@ -1,136 +1,136 @@
-import collections
-import dataclasses
-import inspect
-import logging
-import typing
-
-import venusian
-from sqlalchemy.orm import object_session
-
-from bq import models
-
-BQ_PROCESSOR_CATEGORY = "bq_processor"
-
-
-@dataclasses.dataclass(frozen=True)
-class Processor:
-    channel: str
-    module: str
-    name: str
-    func: typing.Callable
-    # should we auto complete the task or not
-    auto_complete: bool = True
-    # should we auto rollback the transaction when encounter unhandled exception
-    auto_rollback_on_exc: bool = True
-
-
-class ProcessorHelper:
-    def __init__(self, processor: Processor, task_cls: typing.Type = models.Task):
-        self._processor = processor
-        self._task_cls = task_cls
-
-    def __call__(self, *args, **kwargs):
-        return self._processor.func(*args, **kwargs)
-
-    def run(self, **kwargs) -> models.Task:
-        return self._task_cls(
-            channel=self._processor.channel,
-            module=self._processor.module,
-            func_name=self._processor.name,
-            kwargs=kwargs,
-        )
-
-
-def process_task(task: models.Task, processor: Processor):
-    logger = logging.getLogger(__name__)
-    db = object_session(task)
-    func_signature = inspect.signature(processor.func)
-    base_kwargs = {}
-    if "task" in func_signature.parameters:
-        base_kwargs["task"] = task
-    if "db" in func_signature.parameters:
-        base_kwargs["db"] = db
-    with db.begin_nested() as savepoint:
-        try:
-            result = processor.func(**base_kwargs, **task.kwargs)
-            savepoint.commit()
-        except Exception as exc:
-            logger.error("Unhandled exception for task %s", task.id, exc_info=True)
-            if processor.auto_rollback_on_exc:
-                savepoint.rollback()
-            # TODO: add error event
-            task.state = models.TaskState.FAILED
-            task.error_message = str(exc)
-            db.add(task)
-            return
-    if processor.auto_complete:
-        logger.info("Task %s auto complete", task.id)
-        task.state = models.TaskState.DONE
-        task.result = result
-        db.add(task)
-    return result
-
-
-class Registry:
-    def __init__(self):
-        self.logger = logging.getLogger(__name__)
-        self.processors = collections.defaultdict(lambda: collections.defaultdict(dict))
-
-    def add(self, processor: Processor):
-        self.processors[processor.channel][processor.module][processor.name] = processor
-
-    def process(self, task: models.Task) -> typing.Any:
-        modules = self.processors.get(task.channel, {})
-        functions = modules.get(task.module, {})
-        processor = functions.get(task.func_name)
-        db = object_session(task)
-        if processor is None:
-            self.logger.error(
-                "Cannot find processor for task %s with module=%s, func=%s",
-                task.id,
-                task.module,
-                task.func_name,
-            )
-            # TODO: add error event
-            task.state = models.TaskState.FAILED
-            task.error_message = f"Cannot find processor for task with module={task.module}, func={task.func_name}"
-            db.add(task)
-            return
-        return process_task(task, processor)
-
-
-def processor(
-    channel: str,
-    auto_complete: bool = True,
-    auto_rollback_on_exc: bool = True,
-    task_cls: typing.Type = models.Task,
-) -> typing.Callable:
-    def decorator(wrapped: typing.Callable):
-        processor = Processor(
-            module=wrapped.__module__,
-            name=wrapped.__name__,
-            channel=channel,
-            func=wrapped,
-            auto_complete=auto_complete,
-            auto_rollback_on_exc=auto_rollback_on_exc,
-        )
-        helper_obj = ProcessorHelper(processor, task_cls=task_cls)
-
-        def callback(scanner: venusian.Scanner, name: str, ob: typing.Callable):
-            if processor.name != name:
-                raise ValueError("Name is not the same")
-            scanner.registry.add(processor)
-
-        venusian.attach(helper_obj, callback, category=BQ_PROCESSOR_CATEGORY)
-        return helper_obj
-
-    return decorator
-
-
-def collect(packages: list[typing.Any], registry: Registry | None = None) -> Registry:
-    if registry is None:
-        registry = Registry()
-    scanner = venusian.Scanner(registry=registry)
-    for package in packages:
-        scanner.scan(package, categories=(BQ_PROCESSOR_CATEGORY,))
-    return registry
+import collections
+import dataclasses
+import inspect
+import logging
+import typing
+
+import venusian
+from sqlalchemy.orm import object_session
+
+from bq import models
+
+BQ_PROCESSOR_CATEGORY = "bq_processor"
+
+
+@dataclasses.dataclass(frozen=True)
+class Processor:
+    channel: str
+    module: str
+    name: str
+    func: typing.Callable
+    # should we auto complete the task or not
+    auto_complete: bool = True
+    # should we auto rollback the transaction when encounter unhandled exception
+    auto_rollback_on_exc: bool = True
+
+
+class ProcessorHelper:
+    def __init__(self, processor: Processor, task_cls: typing.Type = models.Task):
+        self._processor = processor
+        self._task_cls = task_cls
+
+    def __call__(self, *args, **kwargs):
+        return self._processor.func(*args, **kwargs)
+
+    def run(self, **kwargs) -> models.Task:
+        return self._task_cls(
+            channel=self._processor.channel,
+            module=self._processor.module,
+            func_name=self._processor.name,
+            kwargs=kwargs,
+        )
+
+
+def process_task(task: models.Task, processor: Processor):
+    logger = logging.getLogger(__name__)
+    db = object_session(task)
+    func_signature = inspect.signature(processor.func)
+    base_kwargs = {}
+    if "task" in func_signature.parameters:
+        base_kwargs["task"] = task
+    if "db" in func_signature.parameters:
+        base_kwargs["db"] = db
+    with db.begin_nested() as savepoint:
+        try:
+            result = processor.func(**base_kwargs, **task.kwargs)
+            savepoint.commit()
+        except Exception as exc:
+            logger.error("Unhandled exception for task %s", task.id, exc_info=True)
+            if processor.auto_rollback_on_exc:
+                savepoint.rollback()
+            # TODO: add error event
+            task.state = models.TaskState.FAILED
+            task.error_message = str(exc)
+            db.add(task)
+            return
+    if processor.auto_complete:
+        logger.info("Task %s auto complete", task.id)
+        task.state = models.TaskState.DONE
+        task.result = result
+        db.add(task)
+    return result
+
+
+class Registry:
+    def __init__(self):
+        self.logger = logging.getLogger(__name__)
+        self.processors = collections.defaultdict(lambda: collections.defaultdict(dict))
+
+    def add(self, processor: Processor):
+        self.processors[processor.channel][processor.module][processor.name] = processor
+
+    def process(self, task: models.Task) -> typing.Any:
+        modules = self.processors.get(task.channel, {})
+        functions = modules.get(task.module, {})
+        processor = functions.get(task.func_name)
+        db = object_session(task)
+        if processor is None:
+            self.logger.error(
+                "Cannot find processor for task %s with module=%s, func=%s",
+                task.id,
+                task.module,
+                task.func_name,
+            )
+            # TODO: add error event
+            task.state = models.TaskState.FAILED
+            task.error_message = f"Cannot find processor for task with module={task.module}, func={task.func_name}"
+            db.add(task)
+            return
+        return process_task(task, processor)
+
+
+def processor(
+    channel: str,
+    auto_complete: bool = True,
+    auto_rollback_on_exc: bool = True,
+    task_cls: typing.Type = models.Task,
+) -> typing.Callable:
+    def decorator(wrapped: typing.Callable):
+        processor = Processor(
+            module=wrapped.__module__,
+            name=wrapped.__name__,
+            channel=channel,
+            func=wrapped,
+            auto_complete=auto_complete,
+            auto_rollback_on_exc=auto_rollback_on_exc,
+        )
+        helper_obj = ProcessorHelper(processor, task_cls=task_cls)
+
+        def callback(scanner: venusian.Scanner, name: str, ob: typing.Callable):
+            if processor.name != name:
+                raise ValueError("Name is not the same")
+            scanner.registry.add(processor)
+
+        venusian.attach(helper_obj, callback, category=BQ_PROCESSOR_CATEGORY)
+        return helper_obj
+
+    return decorator
+
+
+def collect(packages: list[typing.Any], registry: Registry | None = None) -> Registry:
+    if registry is None:
+        registry = Registry()
+    scanner = venusian.Scanner(registry=registry)
+    for package in packages:
+        scanner.scan(package, categories=(BQ_PROCESSOR_CATEGORY,))
+    return registry

{beanqueue-0.1.0 → beanqueue-0.1.2}/bq/services/worker.py

@@ -1,69 +1,69 @@
-import datetime
-import typing
-
-from sqlalchemy import func
-from sqlalchemy.orm import Query
-from sqlalchemy.orm import Session
-
-from .. import models
-
-
-class WorkerService:
-    def __init__(self, session: Session):
-        self.session = session
-
-    def update_heartbeat(self, worker: models.Worker):
-        worker.last_heartbeat = func.now()
-        self.session.add(worker)
-
-    def make_dead_worker_query(self, timeout: int, limit: int = 5) -> Query:
-        return (
-            self.session.query(models.Worker.id)
-            .filter(
-                models.Worker.last_heartbeat
-                < (func.now() - datetime.timedelta(seconds=timeout))
-            )
-            .filter(models.Worker.state == models.WorkerState.RUNNING)
-            .limit(limit)
-            .with_for_update(skip_locked=True)
-        )
-
-    def make_update_dead_worker_query(self, worker_query: typing.Any):
-        return (
-            models.Worker.__table__.update()
-            .where(models.Worker.id.in_(worker_query))
-            .values(
-                state=models.WorkerState.NO_HEARTBEAT,
-            )
-            .returning(models.Worker.id)
-        )
-
-    def fetch_dead_workers(self, timeout: int, limit: int = 5) -> Query:
-        dead_worker_query = self.make_dead_worker_query(timeout=timeout, limit=limit)
-        dead_worker_subquery = dead_worker_query.scalar_subquery()
-        worker_ids = [
-            item[0]
-            for item in self.session.execute(
-                self.make_update_dead_worker_query(dead_worker_subquery)
-            )
-        ]
-        # TODO: ideally returning with (models.Task) should return the whole model, but SQLAlchemy is returning
-        #       it columns in rows. We can save a round trip if we can find out how to solve this
-        return self.session.query(models.Worker).filter(
-            models.Worker.id.in_(worker_ids)
-        )
-
-    def make_update_tasks_query(self, worker_query: typing.Any):
-        return (
-            models.Task.__table__.update()
-            .where(models.Task.worker_id.in_(worker_query))
-            .where(models.Task.state == models.TaskState.PROCESSING)
-            .values(
-                state=models.TaskState.PENDING,
-            )
-        )
-
-    def reschedule_dead_tasks(self, worker_query: typing.Any) -> int:
-        update_dead_task_query = self.make_update_tasks_query(worker_query=worker_query)
-        res = self.session.execute(update_dead_task_query)
-        return res.rowcount
+import datetime
+import typing
+
+from sqlalchemy import func
+from sqlalchemy.orm import Query
+from sqlalchemy.orm import Session
+
+from .. import models
+
+
+class WorkerService:
+    def __init__(self, session: Session):
+        self.session = session
+
+    def update_heartbeat(self, worker: models.Worker):
+        worker.last_heartbeat = func.now()
+        self.session.add(worker)
+
+    def make_dead_worker_query(self, timeout: int, limit: int = 5) -> Query:
+        return (
+            self.session.query(models.Worker.id)
+            .filter(
+                models.Worker.last_heartbeat
+                < (func.now() - datetime.timedelta(seconds=timeout))
+            )
+            .filter(models.Worker.state == models.WorkerState.RUNNING)
+            .limit(limit)
+            .with_for_update(skip_locked=True)
+        )
+
+    def make_update_dead_worker_query(self, worker_query: typing.Any):
+        return (
+            models.Worker.__table__.update()
+            .where(models.Worker.id.in_(worker_query))
+            .values(
+                state=models.WorkerState.NO_HEARTBEAT,
+            )
+            .returning(models.Worker.id)
+        )
+
+    def fetch_dead_workers(self, timeout: int, limit: int = 5) -> Query:
+        dead_worker_query = self.make_dead_worker_query(timeout=timeout, limit=limit)
+        dead_worker_subquery = dead_worker_query.scalar_subquery()
+        worker_ids = [
+            item[0]
+            for item in self.session.execute(
+                self.make_update_dead_worker_query(dead_worker_subquery)
+            )
+        ]
+        # TODO: ideally returning with (models.Task) should return the whole model, but SQLAlchemy is returning
+        #       it columns in rows. We can save a round trip if we can find out how to solve this
+        return self.session.query(models.Worker).filter(
+            models.Worker.id.in_(worker_ids)
+        )
+
+    def make_update_tasks_query(self, worker_query: typing.Any):
+        return (
+            models.Task.__table__.update()
+            .where(models.Task.worker_id.in_(worker_query))
+            .where(models.Task.state == models.TaskState.PROCESSING)
+            .values(
+                state=models.TaskState.PENDING,
+            )
+        )
+
+    def reschedule_dead_tasks(self, worker_query: typing.Any) -> int:
+        update_dead_task_query = self.make_update_tasks_query(worker_query=worker_query)
+        res = self.session.execute(update_dead_task_query)
+        return res.rowcount

{beanqueue-0.1.0 → beanqueue-0.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "beanqueue"
-version = "0.1.0"
+version = "0.1.2"
 description = "BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library"
 authors = ["Fang-Pen Lin <fangpen@launchplatform.com>"]
 license = "MIT"

beanqueue-0.1.0/PKG-INFO

@@ -1,23 +0,0 @@
-Metadata-Version: 2.1
-Name: beanqueue
-Version: 0.1.0
-Summary: BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library
-License: MIT
-Author: Fang-Pen Lin
-Author-email: fangpen@launchplatform.com
-Requires-Python: >=3.11,<4.0
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: click (>=8.1.7,<9.0.0)
-Requires-Dist: dependency-injector (>=4.41.0,<5.0.0)
-Requires-Dist: pg-activity (>=3.5.1,<4.0.0)
-Requires-Dist: pydantic-settings (>=2.2.1,<3.0.0)
-Requires-Dist: sqlalchemy (>=2.0.30,<3.0.0)
-Requires-Dist: venusian (>=3.1.0,<4.0.0)
-Description-Content-Type: text/markdown
-
-# bq
-BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library
-

beanqueue-0.1.0/README.md

@@ -1,2 +0,0 @@
-# bq
-BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library