beanqueue 0.2.2__tar.gz → 1.0.0__tar.gz

{beanqueue-0.2.2 → beanqueue-1.0.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: beanqueue
-Version: 0.2.2
-Summary: BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library
+Version: 1.0.0
+Summary: BeanQueue or BQ for short, PostgreSQL SKIP LOCK and SQLAlchemy based worker queue library
 License: MIT
 Author: Fang-Pen Lin
 Author-email: fangpen@launchplatform.com
@@ -19,7 +19,7 @@ Requires-Dist: venusian (>=3.1.0,<4.0.0)
 Description-Content-Type: text/markdown
 
 # BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
-BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+BeanQueue, a lightweight Python task queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
 
 **Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
 
@@ -29,8 +29,10 @@ BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://ww
 - **Easy-to-deploy**: Only rely on PostgreSQL
 - **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
 - **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
+- **Retry**: Built-in and customizable retry-policies
+- **Schedule**: Schedule task to run later
 - **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is found dead, the others will reschedule the tasks
-- **Customizable**: Use it as an library and build your own worker queue
+- **Customizable**: Use it as an library and build your own work queue
 - **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
 
 ## Install
@@ -127,6 +129,78 @@ To create tables for BeanQueue, you can run
 python -m bq.cmds.create_tables
 ```
 
+### Schedule
+
+In most cases, a task will be executed as soon as possible after it is created.
+To run a task later, you can set a datetime value to the `scheduled_at` attribute of the task model.
+For example:
+
+```python
+import datetime
+
+db = Session()
+task = resize_image.run(width=200, height=300)
+task.scheduled_at = func.now() + datetime.timedelta(minutes=3)
+db.add(task)
+```
+
+Please note that currently, workers won't wake up at the next exact moment when the scheduled tasks are ready to run.
+It has to wait until the polling times out, and eventually, it will see the task's scheduled_at time exceeds the current datetime.
+Therefore, depending on your `POLL_TIMEOUT` setting and the number of your workers when they started processing, the actual execution may be inaccurate.
+If you set the `POLL_TIMEOUT` to 60 seconds, please expect less than 60 seconds of delay.
+
+### Retry
+
+To automatically retry a task after failure, you can specify a retry policy to the processor.
+
+```python
+import datetime
+import bq
+from sqlalchemy.orm import Session
+
+app = bq.BeanQueue()
+delay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))
+
+@app.processor(channel="images", retry_policy=delay_retry)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
+Currently, we provide some simple common retry policies such as `DelayRetry` and `ExponentialBackoffRetry`.
+Surely, you can define your retry policy easily by making a function that returns an optional object at the next scheduled time for retry.
+
+```python
+def my_retry_policy(task: bq.Task) -> typing.Any:
+    # calculate delay based on task model ...
+    return func.now() + datetime.timedelta(seconds=delay)
+```
+
+To cap how many attempts is allowed, you can also use `LimitAttempt` like this:
+
+```python
+delay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))
+capped_delay_retry = bq.LimitAttempt(3, delay_retry)
+
+@app.processor(channel="images", retry_policy=capped_delay_retry)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
+You can also retry only for specific exception classes with the `retry_exceptions` argument.
+
+```python
+@app.processor(
+    channel="images",
+    retry_policy=delay_retry,
+    retry_exceptions=ValueError,
+)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
 ### Configurations
 
 Configurations can be modified by setting environment variables with `BQ_` prefix.
@@ -145,7 +219,7 @@ container = bq.Container()
 container.wire(packages=[bq])
 config = bq.Config(
     PROCESSOR_PACKAGES=["my_pkgs.processors"],
-    DATABASE_URL=str(config.DATABASE_URL),
+    DATABASE_URL=config.DATABASE_URL,
     BATCH_SIZE=10,
 )
 app = bq.BeanQueue(config=config)
@@ -168,12 +242,16 @@ app.process_tasks(channels=("images",))
 BeanQueue is designed to be as customizable as much as possible.
 Of course, you can define your own SQLAlchemy model instead of using the ones we provided. 
 
-To make defining your own `Task` model or `Worker` model much easier, you can use our mixin classes:
+To make defining your own `Task`, `Worker` or `Event` model much easier, you can use our mixin classes:
 
 - `bq.TaskModelMixin`: provides task model columns
 - `bq.TaskModelRefWorkerMixin`: provides foreign key column and relationship to `bq.Worker`
+- `bq.TaskModelRefParentMixin`: provides foreign key column and relationship to children `bq.Task` created during processing
+- `bq.TaskModelRefEventMixin`: provides foreign key column and relationship to `bq.Event`
 - `bq.WorkerModelMixin`: provides worker model columns
 - `bq.WorkerRefMixin`: provides relationship to `bq.Task`
+- `bq.EventModelMixin`: provides event model columns
+- `bq.EventModelRefTaskMixin`: provides foreign key column and relationship to `bq.Task`
 
 Here's an example for defining your own Task model:
 
@@ -232,13 +310,14 @@ class Worker(bq.WorkerModelMixin, Base):
     )
 ```
 
-With the model class ready, you only need to change the `TASK_MODEL` and `WORKER_MODEL` of `Config` to the full Python module name plus the class name like this.
+With the model class ready, you only need to change the `TASK_MODEL`, `WORKER_MODEL` and `EVENT_MODEL` of `Config` to the full Python module name plus the class name like this.
 
 ```python
 import bq
 config = bq.Config(
     TASK_MODEL="my_pkgs.models.Task",
     WORKER_MODEL="my_pkgs.models.Worker",
+    EVENT_MODEL="my_pkgs.models.Event",
     # ... other configs
 )
 app = bq.BeanQueue(config)
@@ -246,21 +325,21 @@ app = bq.BeanQueue(config)
 
 ## Why?
 
-There are countless worker queue projects. Why make yet another one?
-The primary issue with most worker queue tools is their reliance on a standalone broker server.
-Our worker queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
-However, integrating an external worker queue into the system presents a risk.
-The worker queue and the database don't share the same data view, potentially compromising data integrity and reliability.
+There are countless work queue projects. Why make yet another one?
+The primary issue with most work queue tools is their reliance on a standalone broker server.
+Our work queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
+However, integrating an external work queue into the system presents a risk.
+The work queue and the database don't share the same data view, potentially compromising data integrity and reliability.
 
 For example, you have a table of `images` to keep the user-uploaded images.
-And you have a background worker queue for resizing the uploaded images into different thumbnail sizes.
-So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the worker queue.
+And you have a background work queue for resizing the uploaded images into different thumbnail sizes.
+So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the work queue.
 
-Say you push the task to the worker queue immediately after you insert the `images` table then commit like this:
+Say you push the task to the work queue immediately after you insert the `images` table then commit like this:
 
 ```
 1. Insert into the "images" table
-2. Push resizing task to the worker queue
+2. Push resizing task to the work queue
 3. Commit db changes
 ```
 
@@ -268,21 +347,21 @@ While this might seem like the right way to do it, there's a hidden bug.
 If the worker starts too fast before the transaction commits at step 3, it will not be able to see the new row in `images` as it has not been committed yet.
 One may need to make the task retry a few times to ensure that even if the first attempt failed, it could see the image row in the following attempt.
 But this adds complexity to the system and also increases the latency if the first attempt fails.
-Also, if the commit step fails, you will have a failed worker queue job trying to fetch a row from the database that will never exist.
+Also, if the commit step fails, you will have a failed work queue job trying to fetch a row from the database that will never exist.
 
 Another approach is to push the resize task after the database changes are committed. It works like this:
 
 ```
 1. Insert into the "images" table
 2. Commit db changes
-3. Push resizing task to the worker queue
+3. Push resizing task to the work queue
 ```
 
 With this approach, we don't need to worry about workers picking up the task too early.
 However, there's another drawback.
-If step 3 for pushing a new task to the worker queue fails, the newly inserted `images` row will never be processed.
-There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the worker queue storage.
-Things will be much easier if we have a worker queue that shares the same consistent view with the worker queue.
+If step 3 for pushing a new task to the work queue fails, the newly inserted `images` row will never be processed.
+There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the work queue storage.
+Things will be much easier if we have a work queue that shares the same consistent view with the database.
 
 By using a database as the data storage, all the problems are gone.
 You can simply do the following:
@@ -294,15 +373,15 @@ You can simply do the following:
 ```
 
 It's all or nothing!
-By doing so, you don't need to maintain another worker queue backend.
-You are probably using a database anyway, so this worker queue comes for free.
+By doing so, you don't need to maintain another work queue backend.
+You are probably using a database anyway, so this work queue comes for free.
 
-Usually, a database is inefficient as the worker queues data storage because of the potential lock contention and the need for constant querying.
+Usually, a database is inefficient as the work queues data storage because of the potential lock contention and the need for constant querying.
 However, things have changed since the [introduction of the SKIP LOCKED](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) / [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) features in PostgreSQL or other databases.
 
-This project is inspired by many of the SKIP-LOCKED-based worker queue successors.
+This project is inspired by many of the SKIP-LOCKED-based work queue successors.
 Why don't we just use those existing tools?
-Well, because while they work great as worker queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
+Well, because while they work great as work queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
 Many provide an abstraction function or gRPC method of pushing tasks into the database instead of opening it up for the user to insert the row directly with other rows and commit altogether.
 
 With BeanQueue, we don't abstract away the logic of publishing a new task into the queue.
@@ -319,6 +398,8 @@ A modern accounting book service based on the most popular open source version c
 ## Alternatives
 
 - [solid_queue](https://github.com/rails/solid_queue)
+- [good_job](https://github.com/bensheldon/good_job)
+- [graphile-worker](https://github.com/graphile/worker)
 - [postgres-tq](https://github.com/flix-tech/postgres-tq)
 - [pq](https://github.com/malthe/pq/)
 - [PgQueuer](https://github.com/janbjorge/PgQueuer)

{beanqueue-0.2.2 → beanqueue-1.0.0}/README.md

@@ -1,5 +1,5 @@
 # BeanQueue  [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/beanhub-extract/tree/master)
-BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
+BeanQueue, a lightweight Python task queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.
 
 **Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.
 
@@ -9,8 +9,10 @@ BeanQueue, a lightweight worker queue framework based on [SQLAlchemy](https://ww
 - **Easy-to-deploy**: Only rely on PostgreSQL
 - **Easy-to-use**: Provide command line tools for processing tasks, also helpers for generating tasks models
 - **Auto-notify**: Notify will automatically be generated and send for inserted or update tasks
+- **Retry**: Built-in and customizable retry-policies
+- **Schedule**: Schedule task to run later
 - **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is found dead, the others will reschedule the tasks
-- **Customizable**: Use it as an library and build your own worker queue
+- **Customizable**: Use it as an library and build your own work queue
 - **Native DB operations**: Commit your tasks with other db entries altogether without worrying about data inconsistent issue
 
 ## Install
@@ -107,6 +109,78 @@ To create tables for BeanQueue, you can run
 python -m bq.cmds.create_tables
 ```
 
+### Schedule
+
+In most cases, a task will be executed as soon as possible after it is created.
+To run a task later, you can set a datetime value to the `scheduled_at` attribute of the task model.
+For example:
+
+```python
+import datetime
+
+db = Session()
+task = resize_image.run(width=200, height=300)
+task.scheduled_at = func.now() + datetime.timedelta(minutes=3)
+db.add(task)
+```
+
+Please note that currently, workers won't wake up at the next exact moment when the scheduled tasks are ready to run.
+It has to wait until the polling times out, and eventually, it will see the task's scheduled_at time exceeds the current datetime.
+Therefore, depending on your `POLL_TIMEOUT` setting and the number of your workers when they started processing, the actual execution may be inaccurate.
+If you set the `POLL_TIMEOUT` to 60 seconds, please expect less than 60 seconds of delay.
+
+### Retry
+
+To automatically retry a task after failure, you can specify a retry policy to the processor.
+
+```python
+import datetime
+import bq
+from sqlalchemy.orm import Session
+
+app = bq.BeanQueue()
+delay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))
+
+@app.processor(channel="images", retry_policy=delay_retry)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
+Currently, we provide some simple common retry policies such as `DelayRetry` and `ExponentialBackoffRetry`.
+Surely, you can define your retry policy easily by making a function that returns an optional object at the next scheduled time for retry.
+
+```python
+def my_retry_policy(task: bq.Task) -> typing.Any:
+    # calculate delay based on task model ...
+    return func.now() + datetime.timedelta(seconds=delay)
+```
+
+To cap how many attempts is allowed, you can also use `LimitAttempt` like this:
+
+```python
+delay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))
+capped_delay_retry = bq.LimitAttempt(3, delay_retry)
+
+@app.processor(channel="images", retry_policy=capped_delay_retry)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
+You can also retry only for specific exception classes with the `retry_exceptions` argument.
+
+```python
+@app.processor(
+    channel="images",
+    retry_policy=delay_retry,
+    retry_exceptions=ValueError,
+)
+def resize_image(db: Session, task: bq.Task, width: int, height: int):
+    # resize iamge here ...
+    pass
+```
+
 ### Configurations
 
 Configurations can be modified by setting environment variables with `BQ_` prefix.
@@ -125,7 +199,7 @@ container = bq.Container()
 container.wire(packages=[bq])
 config = bq.Config(
     PROCESSOR_PACKAGES=["my_pkgs.processors"],
-    DATABASE_URL=str(config.DATABASE_URL),
+    DATABASE_URL=config.DATABASE_URL,
     BATCH_SIZE=10,
 )
 app = bq.BeanQueue(config=config)
@@ -148,12 +222,16 @@ app.process_tasks(channels=("images",))
 BeanQueue is designed to be as customizable as much as possible.
 Of course, you can define your own SQLAlchemy model instead of using the ones we provided. 
 
-To make defining your own `Task` model or `Worker` model much easier, you can use our mixin classes:
+To make defining your own `Task`, `Worker` or `Event` model much easier, you can use our mixin classes:
 
 - `bq.TaskModelMixin`: provides task model columns
 - `bq.TaskModelRefWorkerMixin`: provides foreign key column and relationship to `bq.Worker`
+- `bq.TaskModelRefParentMixin`: provides foreign key column and relationship to children `bq.Task` created during processing
+- `bq.TaskModelRefEventMixin`: provides foreign key column and relationship to `bq.Event`
 - `bq.WorkerModelMixin`: provides worker model columns
 - `bq.WorkerRefMixin`: provides relationship to `bq.Task`
+- `bq.EventModelMixin`: provides event model columns
+- `bq.EventModelRefTaskMixin`: provides foreign key column and relationship to `bq.Task`
 
 Here's an example for defining your own Task model:
 
@@ -212,13 +290,14 @@ class Worker(bq.WorkerModelMixin, Base):
     )
 ```
 
-With the model class ready, you only need to change the `TASK_MODEL` and `WORKER_MODEL` of `Config` to the full Python module name plus the class name like this.
+With the model class ready, you only need to change the `TASK_MODEL`, `WORKER_MODEL` and `EVENT_MODEL` of `Config` to the full Python module name plus the class name like this.
 
 ```python
 import bq
 config = bq.Config(
     TASK_MODEL="my_pkgs.models.Task",
     WORKER_MODEL="my_pkgs.models.Worker",
+    EVENT_MODEL="my_pkgs.models.Event",
     # ... other configs
 )
 app = bq.BeanQueue(config)
@@ -226,21 +305,21 @@ app = bq.BeanQueue(config)
 
 ## Why?
 
-There are countless worker queue projects. Why make yet another one?
-The primary issue with most worker queue tools is their reliance on a standalone broker server.
-Our worker queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
-However, integrating an external worker queue into the system presents a risk.
-The worker queue and the database don't share the same data view, potentially compromising data integrity and reliability.
+There are countless work queue projects. Why make yet another one?
+The primary issue with most work queue tools is their reliance on a standalone broker server.
+Our work queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.
+However, integrating an external work queue into the system presents a risk.
+The work queue and the database don't share the same data view, potentially compromising data integrity and reliability.
 
 For example, you have a table of `images` to keep the user-uploaded images.
-And you have a background worker queue for resizing the uploaded images into different thumbnail sizes.
-So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the worker queue.
+And you have a background work queue for resizing the uploaded images into different thumbnail sizes.
+So, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the work queue.
 
-Say you push the task to the worker queue immediately after you insert the `images` table then commit like this:
+Say you push the task to the work queue immediately after you insert the `images` table then commit like this:
 
 ```
 1. Insert into the "images" table
-2. Push resizing task to the worker queue
+2. Push resizing task to the work queue
 3. Commit db changes
 ```
 
@@ -248,21 +327,21 @@ While this might seem like the right way to do it, there's a hidden bug.
 If the worker starts too fast before the transaction commits at step 3, it will not be able to see the new row in `images` as it has not been committed yet.
 One may need to make the task retry a few times to ensure that even if the first attempt failed, it could see the image row in the following attempt.
 But this adds complexity to the system and also increases the latency if the first attempt fails.
-Also, if the commit step fails, you will have a failed worker queue job trying to fetch a row from the database that will never exist.
+Also, if the commit step fails, you will have a failed work queue job trying to fetch a row from the database that will never exist.
 
 Another approach is to push the resize task after the database changes are committed. It works like this:
 
 ```
 1. Insert into the "images" table
 2. Commit db changes
-3. Push resizing task to the worker queue
+3. Push resizing task to the work queue
 ```
 
 With this approach, we don't need to worry about workers picking up the task too early.
 However, there's another drawback.
-If step 3 for pushing a new task to the worker queue fails, the newly inserted `images` row will never be processed.
-There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the worker queue storage.
-Things will be much easier if we have a worker queue that shares the same consistent view with the worker queue.
+If step 3 for pushing a new task to the work queue fails, the newly inserted `images` row will never be processed.
+There are many solutions to this problem, but these are all caused by inconsistent data views between the database and the work queue storage.
+Things will be much easier if we have a work queue that shares the same consistent view with the database.
 
 By using a database as the data storage, all the problems are gone.
 You can simply do the following:
@@ -274,15 +353,15 @@ You can simply do the following:
 ```
 
 It's all or nothing!
-By doing so, you don't need to maintain another worker queue backend.
-You are probably using a database anyway, so this worker queue comes for free.
+By doing so, you don't need to maintain another work queue backend.
+You are probably using a database anyway, so this work queue comes for free.
 
-Usually, a database is inefficient as the worker queues data storage because of the potential lock contention and the need for constant querying.
+Usually, a database is inefficient as the work queues data storage because of the potential lock contention and the need for constant querying.
 However, things have changed since the [introduction of the SKIP LOCKED](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) / [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) features in PostgreSQL or other databases.
 
-This project is inspired by many of the SKIP-LOCKED-based worker queue successors.
+This project is inspired by many of the SKIP-LOCKED-based work queue successors.
 Why don't we just use those existing tools?
-Well, because while they work great as worker queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
+Well, because while they work great as work queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.
 Many provide an abstraction function or gRPC method of pushing tasks into the database instead of opening it up for the user to insert the row directly with other rows and commit altogether.
 
 With BeanQueue, we don't abstract away the logic of publishing a new task into the queue.
@@ -299,6 +378,8 @@ A modern accounting book service based on the most popular open source version c
 ## Alternatives
 
 - [solid_queue](https://github.com/rails/solid_queue)
+- [good_job](https://github.com/bensheldon/good_job)
+- [graphile-worker](https://github.com/graphile/worker)
 - [postgres-tq](https://github.com/flix-tech/postgres-tq)
 - [pq](https://github.com/malthe/pq/)
 - [PgQueuer](https://github.com/janbjorge/PgQueuer)

beanqueue-1.0.0/bq/__init__.py

@@ -0,0 +1,19 @@
+from .app import BeanQueue
+from .config import Config  # noqa
+from .models import Event
+from .models import EventModelMixin
+from .models import EventModelRefTaskMixin
+from .models import EventType
+from .models import Task  # noqa
+from .models import TaskModelMixin
+from .models import TaskModelRefEventMixin
+from .models import TaskModelRefParentMixin
+from .models import TaskModelRefWorkerMixin
+from .models import TaskState  # noqa
+from .models import Worker  # noqa
+from .models import WorkerModelMixin  # noqa
+from .models import WorkerRefMixin  # noqa
+from .models import WorkerState  # noqa
+from .processors.retry_policies import DelayRetry
+from .processors.retry_policies import ExponentialBackoffRetry
+from .processors.retry_policies import LimitAttempt

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/app.py

@@ -7,6 +7,8 @@ import sys
 import threading
 import time
 import typing
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version
 from wsgiref.simple_server import make_server
 from wsgiref.simple_server import WSGIRequestHandler
 
@@ -84,6 +86,12 @@ class BeanQueue:
     def worker_model(self) -> typing.Type[models.Worker]:
         return load_module_var(self.config.WORKER_MODEL)
 
+    @property
+    def event_model(self) -> typing.Type[models.Event] | None:
+        if self.config.EVENT_MODEL is None:
+            return
+        return load_module_var(self.config.EVENT_MODEL)
+
     def _make_worker_service(self, session: DBSession):
         return self.worker_service_cls(
             session=session, task_model=self.task_model, worker_model=self.worker_model
@@ -96,7 +104,8 @@ class BeanQueue:
         self,
         channel: str = constants.DEFAULT_CHANNEL,
         auto_complete: bool = True,
-        auto_rollback_on_exc: bool = True,
+        retry_policy: typing.Callable | None = None,
+        retry_exceptions: typing.Type | typing.Tuple[typing.Type, ...] | None = None,
         task_model: typing.Type | None = None,
     ) -> typing.Callable:
         def decorator(wrapped: typing.Callable):
@@ -106,7 +115,8 @@ class BeanQueue:
                 channel=channel,
                 func=wrapped,
                 auto_complete=auto_complete,
-                auto_rollback_on_exc=auto_rollback_on_exc,
+                retry_policy=retry_policy,
+                retry_exceptions=retry_exceptions,
             )
             helper_obj = ProcessorHelper(
                 processor,
@@ -241,6 +251,15 @@ class BeanQueue:
         self,
         channels: tuple[str, ...],
     ):
+        try:
+            bq_version = version(__name__.split(".")[0])
+        except PackageNotFoundError:
+            bq_version = "unknown"
+
+        logger.info(
+            "Starting processing tasks, bq_version=%s",
+            bq_version,
+        )
         db = self.make_session()
         if not channels:
             channels = [constants.DEFAULT_CHANNEL]
@@ -318,7 +337,7 @@ class BeanQueue:
                             task.func_name,
                         )
                         # TODO: support processor pool and other approaches to dispatch the workload
-                        registry.process(task)
+                        registry.process(task, event_cls=self.event_model)
                     if not tasks:
                         # we should try to keep dispatching until we cannot find tasks
                         break

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/config.py

@@ -30,6 +30,9 @@ class Config(BaseSettings):
     # which worker model to use
     WORKER_MODEL: str = "bq.Worker"
 
+    # which event model to use
+    EVENT_MODEL: str | None = "bq.Event"
+
     # Enable metrics HTTP server
     METRICS_HTTP_SERVER_ENABLED: bool = True
 

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/events.py

@@ -1,3 +1,5 @@
 import blinker
 
 worker_init = blinker.signal("worker-init")
+
+task_failure = blinker.signal("task-failure")

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/models/__init__.py

@@ -1,5 +1,11 @@
+from .event import Event
+from .event import EventModelMixin
+from .event import EventModelRefTaskMixin
+from .event import EventType
 from .task import Task
 from .task import TaskModelMixin
+from .task import TaskModelRefEventMixin
+from .task import TaskModelRefParentMixin
 from .task import TaskModelRefWorkerMixin
 from .task import TaskState
 from .worker import Worker

beanqueue-1.0.0/bq/models/event.py

@@ -0,0 +1,76 @@
+import datetime
+import enum
+import typing
+import uuid
+
+from sqlalchemy import DateTime
+from sqlalchemy import Enum
+from sqlalchemy import ForeignKey
+from sqlalchemy import func
+from sqlalchemy import String
+from sqlalchemy.dialects.postgresql import UUID
+from sqlalchemy.orm import declared_attr
+from sqlalchemy.orm import Mapped
+from sqlalchemy.orm import mapped_column
+from sqlalchemy.orm import relationship
+
+from ..db.base import Base
+from .helpers import make_repr_attrs
+
+
+class EventType(enum.Enum):
+    # task failed
+    FAILED = "FAILED"
+    # task failed and retry scheduled
+    FAILED_RETRY_SCHEDULED = "FAILED_RETRY_SCHEDULED"
+    # task complete
+    COMPLETE = "COMPLETE"
+
+
+class EventModelMixin:
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, server_default=func.gen_random_uuid()
+    )
+    # type of the event
+    type: Mapped[EventType] = mapped_column(
+        Enum(EventType),
+        nullable=False,
+        index=True,
+    )
+    # Error message
+    error_message: Mapped[typing.Optional[str]] = mapped_column(String, nullable=True)
+    # the scheduled at time for retry
+    scheduled_at: Mapped[datetime.datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=True,
+    )
+    # created datetime of the event
+    created_at: Mapped[datetime.datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class EventModelRefTaskMixin:
+    # foreign key id of the task
+    task_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        ForeignKey("bq_tasks.id", name="fk_event_task_id"),
+        nullable=True,
+    )
+
+    @declared_attr
+    def task(cls) -> Mapped["Task"]:
+        return relationship("Task", back_populates="events", uselist=False)
+
+
+class Event(EventModelMixin, EventModelRefTaskMixin, Base):
+    __tablename__ = "bq_events"
+
+    def __repr__(self) -> str:
+        items = [
+            ("id", self.id),
+            ("type", self.type),
+            ("created_at", self.created_at),
+            ("scheduled_at", self.scheduled_at),
+        ]
+        return f"<{self.__class__.__name__} {make_repr_attrs(items)}>"

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/models/task.py

@@ -24,7 +24,7 @@ from .helpers import make_repr_attrs
 
 
 class TaskState(enum.Enum):
-    # task just created, not scheduled yet
+    # task just created, not dispatched yet. or, the task failed and is waiting for a retry.
     PENDING = "PENDING"
     # a worker is processing the task right now
     PROCESSING = "PROCESSING"
@@ -62,6 +62,11 @@ class TaskModelMixin:
     created_at: Mapped[datetime.datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()
     )
+    # scheduled to run at a specific time
+    scheduled_at: Mapped[datetime.datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=True,
+    )
 
 
 class TaskModelRefWorkerMixin:
@@ -77,7 +82,44 @@ class TaskModelRefWorkerMixin:
         return relationship("Worker", back_populates="tasks", uselist=False)
 
 
-class Task(TaskModelMixin, TaskModelRefWorkerMixin, Base):
+class TaskModelRefParentMixin:
+    # foreign key id of the source task which created the current task while we are processing it
+    parent_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        ForeignKey("bq_tasks.id", name="fk_task_parent_task_id"),
+        nullable=True,
+    )
+
+    @declared_attr
+    def parent(cls) -> Mapped[typing.Optional["Task"]]:
+        return relationship(
+            "Task",
+            back_populates="children",
+            remote_side=[cls.id],
+            foreign_keys=[cls.parent_id],
+            uselist=False,
+        )
+
+    @declared_attr
+    def children(cls) -> Mapped[list["Task"]]:
+        return relationship(
+            "Task", foreign_keys=[cls.parent_id], back_populates="parent"
+        )
+
+
+class TaskModelRefEventMixin:
+    @declared_attr
+    def events(cls) -> Mapped[list["Event"]]:
+        return relationship("Event", back_populates="task")
+
+
+class Task(
+    TaskModelMixin,
+    TaskModelRefWorkerMixin,
+    TaskModelRefEventMixin,
+    TaskModelRefParentMixin,
+    Base,
+):
     __tablename__ = "bq_tasks"
 
     def __repr__(self) -> str:
@@ -85,6 +127,8 @@ class Task(TaskModelMixin, TaskModelRefWorkerMixin, Base):
             ("id", self.id),
             ("state", self.state),
             ("channel", self.channel),
+            ("module", self.module),
+            ("func_name", self.func_name),
         ]
         return f"<{self.__class__.__name__} {make_repr_attrs(items)}>"
 

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/models/worker.py

@@ -2,7 +2,6 @@ import datetime
 import enum
 import uuid
 
-from sqlalchemy import Column
 from sqlalchemy import DateTime
 from sqlalchemy import Enum
 from sqlalchemy import func
@@ -12,7 +11,6 @@ from sqlalchemy.dialects.postgresql import UUID
 from sqlalchemy.orm import declared_attr
 from sqlalchemy.orm import Mapped
 from sqlalchemy.orm import mapped_column
-from sqlalchemy.orm import Mapper
 from sqlalchemy.orm import relationship
 
 from ..db.base import Base

beanqueue-1.0.0/bq/processors/processor.py

@@ -0,0 +1,120 @@
+import contextvars
+import dataclasses
+import datetime
+import inspect
+import logging
+import typing
+
+from sqlalchemy import select
+from sqlalchemy.orm import object_session
+
+from .. import events
+from .. import models
+
+logger = logging.getLogger(__name__)
+current_task = contextvars.ContextVar("current_task")
+
+
+@dataclasses.dataclass(frozen=True)
+class Processor:
+    channel: str
+    module: str
+    name: str
+    func: typing.Callable
+    # should we auto complete the task or not
+    auto_complete: bool = True
+    # The retry policy function for returning a new scheduled time for next attempt
+    retry_policy: typing.Callable | None = None
+    # The exceptions we suppose to retry when encountered
+    retry_exceptions: typing.Type | typing.Tuple[typing.Type, ...] | None = None
+
+    def process(self, task: models.Task, event_cls: typing.Type | None = None):
+        ctx_token = current_task.set(task)
+        try:
+            db = object_session(task)
+            func_signature = inspect.signature(self.func)
+            base_kwargs = {}
+            if "task" in func_signature.parameters:
+                base_kwargs["task"] = task
+            if "db" in func_signature.parameters:
+                base_kwargs["db"] = db
+            try:
+                with db.begin_nested() as savepoint:
+                    if "savepoint" in func_signature.parameters:
+                        base_kwargs["savepoint"] = savepoint
+                    result = self.func(**base_kwargs, **task.kwargs)
+            except Exception as exc:
+                logger.error("Unhandled exception for task %s", task.id, exc_info=True)
+                events.task_failure.send(self, task=task, exception=exc)
+                task.state = models.TaskState.FAILED
+                task.error_message = str(exc)
+                retry_scheduled_at = None
+                if (
+                    self.retry_exceptions is None
+                    or isinstance(exc, self.retry_exceptions)
+                ) and self.retry_policy is not None:
+                    retry_scheduled_at = self.retry_policy(task)
+                    if retry_scheduled_at is not None:
+                        task.state = models.TaskState.PENDING
+                        task.scheduled_at = retry_scheduled_at
+                        if isinstance(retry_scheduled_at, datetime.datetime):
+                            retry_scheduled_at_value = retry_scheduled_at
+                        else:
+                            retry_scheduled_at_value = db.scalar(
+                                select(retry_scheduled_at)
+                            )
+                        logger.info(
+                            "Schedule task %s for retry at %s",
+                            task.id,
+                            retry_scheduled_at_value,
+                        )
+                if event_cls is not None:
+                    event = event_cls(
+                        task=task,
+                        type=models.EventType.FAILED
+                        if retry_scheduled_at is None
+                        else models.EventType.FAILED_RETRY_SCHEDULED,
+                        error_message=task.error_message,
+                        scheduled_at=retry_scheduled_at,
+                    )
+                    db.add(event)
+                db.add(task)
+                return
+            if self.auto_complete:
+                logger.info("Task %s auto complete", task.id)
+                task.state = models.TaskState.DONE
+                task.result = result
+                if event_cls is not None:
+                    event = event_cls(
+                        task=task,
+                        type=models.EventType.COMPLETE,
+                    )
+                    db.add(event)
+                db.add(task)
+            return result
+        finally:
+            current_task.reset(ctx_token)
+
+
+class ProcessorHelper:
+    """Helper function to replace the decorated processor function and make creating Task model much easier"""
+
+    def __init__(self, processor: Processor, task_cls: typing.Type = models.Task):
+        self._processor = processor
+        self._task_cls = task_cls
+
+    def __call__(self, *args, **kwargs):
+        return self._processor.func(*args, **kwargs)
+
+    def run(self, **kwargs) -> models.Task:
+        try:
+            parent = current_task.get()
+        except LookupError:
+            parent = None
+        return self._task_cls(
+            channel=self._processor.channel,
+            module=self._processor.module,
+            func_name=self._processor.name,
+            kwargs=kwargs,
+            parent=parent,
+        )

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/processors/registry.py

@@ -18,10 +18,14 @@ class Registry:
     def add(self, processor: Processor):
         self.processors[processor.channel][processor.module][processor.name] = processor
 
-    def process(self, task: models.Task) -> typing.Any:
+    def process(
+        self,
+        task: models.Task,
+        event_cls: typing.Type | None = None,
+    ) -> typing.Any:
         modules = self.processors.get(task.channel, {})
         functions = modules.get(task.module, {})
-        processor = functions.get(task.func_name)
+        processor: Processor = functions.get(task.func_name)
         db = object_session(task)
         if processor is None:
             self.logger.error(
@@ -30,12 +34,18 @@ class Registry:
                 task.module,
                 task.func_name,
             )
-            # TODO: add error event
             task.state = models.TaskState.FAILED
             task.error_message = f"Cannot find processor for task with module={task.module}, func={task.func_name}"
+            if event_cls is not None:
+                event = event_cls(
+                    task=task,
+                    type=models.EventType.FAILED,
+                    error_message=task.error_message,
+                )
+                db.add(event)
             db.add(task)
             return
-        return processor.process(task)
+        return processor.process(task, event_cls=event_cls)
 
 
 def collect(packages: list[typing.Any], registry: Registry | None = None) -> Registry:

beanqueue-1.0.0/bq/processors/retry_policies.py

@@ -0,0 +1,55 @@
+import datetime
+import typing
+
+from sqlalchemy import func
+from sqlalchemy import inspect
+from sqlalchemy.orm import object_session
+
+from .. import models
+
+
+def get_failure_times(task: models.Task) -> int:
+    db = object_session(task)
+    task_info = inspect(task.__class__)
+    event_cls = task_info.attrs["events"].entity.class_
+    return (
+        db.query(event_cls)
+        .filter(event_cls.task == task)
+        .filter(event_cls.type == models.EventType.FAILED_RETRY_SCHEDULED)
+    ).count()
+
+
+class DelayRetry:
+    def __init__(self, delay: datetime.timedelta):
+        self.delay = delay
+
+    def __call__(self, task: models.Task) -> typing.Any:
+        return func.now() + self.delay
+
+
+class ExponentialBackoffRetry:
+    def __init__(
+        self, base: float = 2, exponent_offset: float = 0, exponent_scalar: float = 1.0
+    ):
+        self.base = base
+        self.exponent_offset = exponent_offset
+        self.exponent_scalar = exponent_scalar
+
+    def __call__(self, task: models.Task) -> typing.Any:
+        failure_times = get_failure_times(task)
+        delay_seconds = self.base ** (
+            self.exponent_offset + (self.exponent_scalar * (failure_times + 1))
+        )
+        return func.now() + datetime.timedelta(seconds=delay_seconds)
+
+
+class LimitAttempt:
+    def __init__(self, maximum_attempt: int, retry_policy: typing.Callable):
+        self.maximum_attempt = maximum_attempt
+        self.retry_policy = retry_policy
+
+    def __call__(self, task: models.Task) -> typing.Any:
+        failure_times = get_failure_times(task)
+        if (failure_times + 1) >= self.maximum_attempt:
+            return None
+        return self.retry_policy(task)

{beanqueue-0.2.2 → beanqueue-1.0.0}/bq/services/dispatch.py

@@ -3,6 +3,9 @@ import select
 import typing
 import uuid
 
+from sqlalchemy import func
+from sqlalchemy import null
+from sqlalchemy import or_
 from sqlalchemy.orm import Query
 
 from .. import models
@@ -21,11 +24,22 @@ class DispatchService:
         self.session = session
         self.task_model: typing.Type[models.Task] = task_model
 
-    def make_task_query(self, channels: typing.Sequence[str], limit: int = 1) -> Query:
+    def make_task_query(
+        self,
+        channels: typing.Sequence[str],
+        limit: int = 1,
+        now: typing.Any = func.now(),
+    ) -> Query:
         return (
             self.session.query(self.task_model.id)
             .filter(self.task_model.channel.in_(channels))
             .filter(self.task_model.state == models.TaskState.PENDING)
+            .filter(
+                or_(
+                    self.task_model.scheduled_at.is_(null()),
+                    now >= self.task_model.scheduled_at,
+                )
+            )
             .order_by(self.task_model.created_at)
             .limit(limit)
             .with_for_update(skip_locked=True)
@@ -43,9 +57,13 @@ class DispatchService:
         )
 
     def dispatch(
-        self, channels: typing.Sequence[str], worker_id: uuid.UUID, limit: int = 1
+        self,
+        channels: typing.Sequence[str],
+        worker_id: uuid.UUID,
+        limit: int = 1,
+        now: typing.Any = func.now(),
     ) -> Query:
-        task_query = self.make_task_query(channels, limit=limit)
+        task_query = self.make_task_query(channels, limit=limit, now=now)
         task_subquery = task_query.scalar_subquery()
         task_ids = [
             item[0]

{beanqueue-0.2.2 → beanqueue-1.0.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [tool.poetry]
 name = "beanqueue"
-version = "0.2.2"
-description = "BeanQueue or BQ for short, PostgreSQL SKIP LOCK based worker queue library"
+version = "1.0.0"
+description = "BeanQueue or BQ for short, PostgreSQL SKIP LOCK and SQLAlchemy based worker queue library"
 authors = ["Fang-Pen Lin <fangpen@launchplatform.com>"]
 license = "MIT"
 readme = "README.md"

beanqueue-0.2.2/bq/__init__.py

@@ -1,10 +0,0 @@
-from .app import BeanQueue
-from .config import Config  # noqa
-from .models import Task  # noqa
-from .models import TaskModelMixin
-from .models import TaskModelRefWorkerMixin
-from .models import TaskState  # noqa
-from .models import Worker  # noqa
-from .models import WorkerModelMixin  # noqa
-from .models import WorkerRefMixin  # noqa
-from .models import WorkerState  # noqa

beanqueue-0.2.2/bq/processors/processor.py

@@ -1,70 +0,0 @@
-import dataclasses
-import inspect
-import logging
-import typing
-
-from sqlalchemy.orm import object_session
-
-from .. import models
-
-logger = logging.getLogger(__name__)
-
-
-@dataclasses.dataclass(frozen=True)
-class Processor:
-    channel: str
-    module: str
-    name: str
-    func: typing.Callable
-    # should we auto complete the task or not
-    auto_complete: bool = True
-    # should we auto rollback the transaction when encounter unhandled exception
-    auto_rollback_on_exc: bool = True
-
-    def process(self, task: models.Task):
-        db = object_session(task)
-        func_signature = inspect.signature(self.func)
-        base_kwargs = {}
-        if "task" in func_signature.parameters:
-            base_kwargs["task"] = task
-        if "db" in func_signature.parameters:
-            base_kwargs["db"] = db
-        with db.begin_nested() as savepoint:
-            if "savepoint" in func_signature.parameters:
-                base_kwargs["savepoint"] = savepoint
-            try:
-                result = self.func(**base_kwargs, **task.kwargs)
-            except Exception as exc:
-                logger.error("Unhandled exception for task %s", task.id, exc_info=True)
-                if self.auto_rollback_on_exc:
-                    savepoint.rollback()
-                # TODO: add error event
-                task.state = models.TaskState.FAILED
-                task.error_message = str(exc)
-                db.add(task)
-                return
-        if self.auto_complete:
-            logger.info("Task %s auto complete", task.id)
-            task.state = models.TaskState.DONE
-            task.result = result
-            db.add(task)
-        return result
-
-
-class ProcessorHelper:
-    """Helper function to replace the decorated processor function and make creating Task model much easier"""
-
-    def __init__(self, processor: Processor, task_cls: typing.Type = models.Task):
-        self._processor = processor
-        self._task_cls = task_cls
-
-    def __call__(self, *args, **kwargs):
-        return self._processor.func(*args, **kwargs)
-
-    def run(self, **kwargs) -> models.Task:
-        return self._task_cls(
-            channel=self._processor.channel,
-            module=self._processor.module,
-            func_name=self._processor.name,
-            kwargs=kwargs,
-        )