beanqueue 0.1.2__tar.gz → 0.2.0__tar.gz

{beanqueue-0.1.2 → beanqueue-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: beanqueue
-Version: 0.1.2
+Version: 0.2.0
 Summary: BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library
 License: MIT
 Author: Fang-Pen Lin
@@ -10,8 +10,8 @@ Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: blinker (>=1.8.2,<2.0.0)
 Requires-Dist: click (>=8.1.7,<9.0.0)
-Requires-Dist: dependency-injector (>=4.41.0,<5.0.0)
 Requires-Dist: pg-activity (>=3.5.1,<4.0.0)
 Requires-Dist: pydantic-settings (>=2.2.1,<3.0.0)
 Requires-Dist: sqlalchemy (>=2.0.30,<3.0.0)
@@ -19,7 +19,7 @@ Requires-Dist: venusian (>=3.1.0,<4.0.0)
 Description-Content-Type: text/markdown
 
 # BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
-BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), [PostgreSQL SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
 
 **Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
 
@@ -29,7 +29,7 @@ BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://ww
 - **Easy-to-deploy**: Only rely on PostgreSQL
 - **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
 - **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
-- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is dead, the others will reschedule the tasks
+- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is found dead, the others will reschedule the tasks
 - **Customizable**: Use it as an library and build your own worker queue
 - **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
 
@@ -46,14 +46,15 @@ You can define a task processor like this
 ```python
 from sqlalchemy.orm import Session
 
-from bq.processors.registry import processor
-from bq import models
-from .. import my_models
+import bq
+from .. import models
 from .. import image_utils
 
-@processor(channel="images")
-def resize_image(db: Session, task: models.Task, width: int, height: int):
-    image = db.query(my_models.Image).filter(my_models.Image.task == task).one()
+app = bq.BeanQueue()
+
+@app.processor(channel="images")
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    image = db.query(models.Image).filter(models.Image.task == task).one()
     image_utils.resize(image, size=(width, height))
     db.add(image)
     # by default the `processor` decorator has `auto_complete` flag turns on,
@@ -62,22 +63,23 @@ def resize_image(db: Session, task: models.Task, width: int, height: int):
 
 The `db` and `task` keyword arguments are optional.
 If you don't need to access the task object, you can simply define the function without these two parameters.
+We also provide an optional `savepoint` argument in case if you want to rollback database changes you made.
 
-To submit a task, you can either use `bq.models.Task` model object to construct the task object, insert into the
+To submit a task, you can either use `bq.Task` model object to construct the task object, insert into the
 database session and commit.
 
 ```python
-from bq import models
+import bq
 from .db import Session
-from .. import my_models
+from .. import models
 
 db = Session()
-task = models.Task(
+task = bq.Task(
     channel="files",
     module="my_pkgs.files.processors",
     name="upload_to_s3_for_backup",
 )
-file = my_models.File(
+file = models.File(
     task=task,
     blob_name="...",
 )
@@ -112,6 +114,7 @@ To run the worker, you can do this:
 BQ_PROCESSOR_PACKAGES='["my_pkgs.processors"]' python -m bq.cmds.process images
 ```
 
+The `BQ_PROCESSOR_PACKAGES` is a JSON list contains the Python packages where you define your processors (the functions you decorated with `bq.processors.registry.processor`).
 To submit a task for testing purpose, you can do
 
 ```bash
@@ -130,29 +133,116 @@ Configurations can be modified by setting environment variables with `BQ_` prefi
 For example, to set the python packages to scan for processors, you can set `BQ_PROCESSOR_PACKAGES`.
 To change the PostgreSQL database to connect to, you can set `BQ_DATABASE_URL`.
 The complete definition of configurations can be found at the [bq/config.py](bq/config.py) module.
-For now, the configurations only affect command line tools.
 
-If you want to configure BeanQueue programmatically for the command lines, you can override our [dependency-injector](https://python-dependency-injector.ets-labs.org/)'s container defined at [bq/container.py](bq/container.py) and call the command function manually.
+If you want to configure BeanQueue programmatically, you can pass in `Config` object to the `bq.BeanQueue` object when creating.
 For example:
 
 ```python
-import bq.cmds.process
-from bq.container import Container
-from bq.config import Config
-
-container = Container()
-container.wire(modules=[bq.cmds.process])
-with container.config.override(
-    Config(
-        PROCESSOR_PACKAGES=["my_pkgs.processors"],
-        DATABASE_URL="postgresql://...",
-        BATCH_SIZE=10,
+import bq
+from .my_config import config
+
+container = bq.Container()
+container.wire(packages=[bq])
+config = bq.Config(
+    PROCESSOR_PACKAGES=["my_pkgs.processors"],
+    DATABASE_URL=str(config.DATABASE_URL),
+    BATCH_SIZE=10,
+)
+app = bq.BeanQueue(config=config)
+```
+
+Then you can pass `--app` argument pointing to the app object to the process command like this:
+
+```bash
+python -m bq.cmds.process -a my_pkgs.bq.app images
+```
+
+Or if you prefer to define your own process command, you can also call `process_tasks` of the `BeanQueue` object directly like this:
+
+```python
+app.process_tasks(channels=("images",))
+```
+
+### Define your own tables
+
+BeanQueue is designed to be as customizable as much as possible.
+Of course, you can define your own SQLAlchemy model instead of using the ones we provided. 
+
+To make defining your own `Task` model or `Worker` model much easier, you can use our mixin classes:
+
+- `bq.TaskModelMixin`: provides task model columns
+- `bq.TaskModelRefWorkerMixin`: provides foreign key column and relationship to `bq.Worker`
+- `bq.WorkerModelMixin`: provides worker model columns
+- `bq.WorkerRefMixin`: provides relationship to `bq.Task`
+
+Here's an example for defining your own Task model:
+
+```python
+import uuid
+
+import bq
+from sqlalchemy import ForeignKey
+from sqlalchemy.dialects.postgresql import UUID
+from sqlalchemy.orm import Mapped
+from sqlalchemy.orm import mapped_column
+from sqlalchemy.orm import relationship
+
+from .base_class import Base
+
+
+class Task(bq.TaskModelMixin, Base):
+    __tablename__ = "task"
+    worker_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        ForeignKey("worker.id", onupdate="CASCADE"),
+        nullable=True,
+        index=True,
+    )
+
+    worker: Mapped["Worker"] = relationship(
+        "Worker", back_populates="tasks", uselist=False
     )
-):
-    bq.cmds.process.process_tasks(channels=("images",))
 ```
 
-Many other behaviors of this framework can also be modified by overriding the container defined at [bq/container.py](bq/container.py).
+To make task insert and update with state changing to `PENDING` send out NOTIFY "channel" statement automatically, you can also use `bq.models.task.listen_events` helper to register our SQLAlchemy event handlers automatically like this
+
+```python
+from bq.models.task import listen_events
+listen_events(Task)
+```
+
+You just see how easy it is to define your Task model. Now, here's an example for defining your own Worker model:
+
+```python
+import bq
+from sqlalchemy.orm import Mapped
+from sqlalchemy.orm import relationship
+
+from .base_class import Base
+
+
+class Worker(bq.WorkerModelMixin, Base):
+    __tablename__ = "worker"
+
+    tasks: Mapped[list["Task"]] = relationship(
+        "Task",
+        back_populates="worker",
+        cascade="all,delete",
+        order_by="Task.created_at",
+    )
+```
+
+With the model class ready, you only need to change the `TASK_MODEL` and `WORKER_MODEL` of `Config` to the full Python module name plus the class name like this.
+
+```python
+import bq
+config = bq.Config(
+    TASK_MODEL="my_pkgs.models.Task",
+    WORKER_MODEL="my_pkgs.models.Worker",
+    # ... other configs
+)
+app = bq.BeanQueue(config)
+```
 
 ## Why?
 
@@ -230,6 +320,7 @@ A modern accounting book service based on the most popular open source version c
 
 - [solid_queue](https://github.com/rails/solid_queue)
 - [postgres-tq](https://github.com/flix-tech/postgres-tq)
+- [pq](https://github.com/malthe/pq/)
 - [PgQueuer](https://github.com/janbjorge/PgQueuer)
 - [hatchet](https://github.com/hatchet-dev/hatchet)
 

{beanqueue-0.1.2 → beanqueue-0.2.0}/README.md

@@ -1,5 +1,5 @@
 # BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
-BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), [PostgreSQL SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
 
 **Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
 
@@ -9,7 +9,7 @@ BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://ww
 - **Easy-to-deploy**: Only rely on PostgreSQL
 - **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
 - **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
-- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is dead, the others will reschedule the tasks
+- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is found dead, the others will reschedule the tasks
 - **Customizable**: Use it as an library and build your own worker queue
 - **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
 
@@ -26,14 +26,15 @@ You can define a task processor like this
 ```python
 from sqlalchemy.orm import Session
 
-from bq.processors.registry import processor
-from bq import models
-from .. import my_models
+import bq
+from .. import models
 from .. import image_utils
 
-@processor(channel="images")
-def resize_image(db: Session, task: models.Task, width: int, height: int):
-    image = db.query(my_models.Image).filter(my_models.Image.task == task).one()
+app = bq.BeanQueue()
+
+@app.processor(channel="images")
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    image = db.query(models.Image).filter(models.Image.task == task).one()
     image_utils.resize(image, size=(width, height))
     db.add(image)
     # by default the `processor` decorator has `auto_complete` flag turns on,
@@ -42,22 +43,23 @@ def resize_image(db: Session, task: models.Task, width: int, height: int):
 
 The `db` and `task` keyword arguments are optional.
 If you don't need to access the task object, you can simply define the function without these two parameters.
+We also provide an optional `savepoint` argument in case if you want to rollback database changes you made.
 
-To submit a task, you can either use `bq.models.Task` model object to construct the task object, insert into the
+To submit a task, you can either use `bq.Task` model object to construct the task object, insert into the
 database session and commit.
 
 ```python
-from bq import models
+import bq
 from .db import Session
-from .. import my_models
+from .. import models
 
 db = Session()
-task = models.Task(
+task = bq.Task(
     channel="files",
     module="my_pkgs.files.processors",
     name="upload_to_s3_for_backup",
 )
-file = my_models.File(
+file = models.File(
     task=task,
     blob_name="...",
 )
@@ -92,6 +94,7 @@ To run the worker, you can do this:
 BQ_PROCESSOR_PACKAGES='["my_pkgs.processors"]' python -m bq.cmds.process images
 ```
 
+The `BQ_PROCESSOR_PACKAGES` is a JSON list contains the Python packages where you define your processors (the functions you decorated with `bq.processors.registry.processor`).
 To submit a task for testing purpose, you can do
 
 ```bash
@@ -110,29 +113,116 @@ Configurations can be modified by setting environment variables with `BQ_` prefi
 For example, to set the python packages to scan for processors, you can set `BQ_PROCESSOR_PACKAGES`.
 To change the PostgreSQL database to connect to, you can set `BQ_DATABASE_URL`.
 The complete definition of configurations can be found at the [bq/config.py](bq/config.py) module.
-For now, the configurations only affect command line tools.
 
-If you want to configure BeanQueue programmatically for the command lines, you can override our [dependency-injector](https://python-dependency-injector.ets-labs.org/)'s container defined at [bq/container.py](bq/container.py) and call the command function manually.
+If you want to configure BeanQueue programmatically, you can pass in `Config` object to the `bq.BeanQueue` object when creating.
 For example:
 
 ```python
-import bq.cmds.process
-from bq.container import Container
-from bq.config import Config
-
-container = Container()
-container.wire(modules=[bq.cmds.process])
-with container.config.override(
-    Config(
-        PROCESSOR_PACKAGES=["my_pkgs.processors"],
-        DATABASE_URL="postgresql://...",
-        BATCH_SIZE=10,
+import bq
+from .my_config import config
+
+container = bq.Container()
+container.wire(packages=[bq])
+config = bq.Config(
+    PROCESSOR_PACKAGES=["my_pkgs.processors"],
+    DATABASE_URL=str(config.DATABASE_URL),
+    BATCH_SIZE=10,
+)
+app = bq.BeanQueue(config=config)
+```
+
+Then you can pass `--app` argument pointing to the app object to the process command like this:
+
+```bash
+python -m bq.cmds.process -a my_pkgs.bq.app images
+```
+
+Or if you prefer to define your own process command, you can also call `process_tasks` of the `BeanQueue` object directly like this:
+
+```python
+app.process_tasks(channels=("images",))
+```
+
+### Define your own tables
+
+BeanQueue is designed to be as customizable as much as possible.
+Of course, you can define your own SQLAlchemy model instead of using the ones we provided. 
+
+To make defining your own `Task` model or `Worker` model much easier, you can use our mixin classes:
+
+- `bq.TaskModelMixin`: provides task model columns
+- `bq.TaskModelRefWorkerMixin`: provides foreign key column and relationship to `bq.Worker`
+- `bq.WorkerModelMixin`: provides worker model columns
+- `bq.WorkerRefMixin`: provides relationship to `bq.Task`
+
+Here's an example for defining your own Task model:
+
+```python
+import uuid
+
+import bq
+from sqlalchemy import ForeignKey
+from sqlalchemy.dialects.postgresql import UUID
+from sqlalchemy.orm import Mapped
+from sqlalchemy.orm import mapped_column
+from sqlalchemy.orm import relationship
+
+from .base_class import Base
+
+
+class Task(bq.TaskModelMixin, Base):
+    __tablename__ = "task"
+    worker_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        ForeignKey("worker.id", onupdate="CASCADE"),
+        nullable=True,
+        index=True,
+    )
+
+    worker: Mapped["Worker"] = relationship(
+        "Worker", back_populates="tasks", uselist=False
     )
-):
-    bq.cmds.process.process_tasks(channels=("images",))
 ```
 
-Many other behaviors of this framework can also be modified by overriding the container defined at [bq/container.py](bq/container.py).
+To make task insert and update with state changing to `PENDING` send out NOTIFY "channel" statement automatically, you can also use `bq.models.task.listen_events` helper to register our SQLAlchemy event handlers automatically like this
+
+```python
+from bq.models.task import listen_events
+listen_events(Task)
+```
+
+You just see how easy it is to define your Task model. Now, here's an example for defining your own Worker model:
+
+```python
+import bq
+from sqlalchemy.orm import Mapped
+from sqlalchemy.orm import relationship
+
+from .base_class import Base
+
+
+class Worker(bq.WorkerModelMixin, Base):
+    __tablename__ = "worker"
+
+    tasks: Mapped[list["Task"]] = relationship(
+        "Task",
+        back_populates="worker",
+        cascade="all,delete",
+        order_by="Task.created_at",
+    )
+```
+
+With the model class ready, you only need to change the `TASK_MODEL` and `WORKER_MODEL` of `Config` to the full Python module name plus the class name like this.
+
+```python
+import bq
+config = bq.Config(
+    TASK_MODEL="my_pkgs.models.Task",
+    WORKER_MODEL="my_pkgs.models.Worker",
+    # ... other configs
+)
+app = bq.BeanQueue(config)
+```
 
 ## Why?
 
@@ -210,5 +300,6 @@ A modern accounting book service based on the most popular open source version c
 
 - [solid_queue](https://github.com/rails/solid_queue)
 - [postgres-tq](https://github.com/flix-tech/postgres-tq)
+- [pq](https://github.com/malthe/pq/)
 - [PgQueuer](https://github.com/janbjorge/PgQueuer)
 - [hatchet](https://github.com/hatchet-dev/hatchet)

beanqueue-0.2.0/bq/__init__.py

@@ -0,0 +1,10 @@
+from .app import BeanQueue
+from .config import Config  # noqa
+from .models import Task  # noqa
+from .models import TaskModelMixin
+from .models import TaskModelRefWorkerMixin
+from .models import TaskState  # noqa
+from .models import Worker  # noqa
+from .models import WorkerModelMixin  # noqa
+from .models import WorkerRefMixin  # noqa
+from .models import WorkerState  # noqa