django-taskq 1.0.0__tar.gz

django-taskq-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Aivars Kalvāns
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

django-taskq-1.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include LICENSE
+include README.rst

django-taskq-1.0.0/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.1
+Name: django-taskq
+Version: 1.0.0
+Summary: A simple Celery-like database-backed task queue for Django
+Home-page: https://github.com/aivarsk/django-taskq
+Author: Aivars Kalvans
+Author-email: aivars.kalvans@gmail.com
+License: MIT
+Classifier: Environment :: Web Environment
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 4.0
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+License-File: LICENSE
+
+Django database task queue
+
+Django Transaction Task Queue
+=============================
+
+A short and simple Celery replacement for my Django projects.
+
+* *Database is the only backend.* The task is a simple Django model, it uses the same transaction and connection as other models. No more ``transaction.on_commit`` hooks to schedule tasks.
+* *Tasks do not produce any results*, there is no ``get`` or ``join`` and there is no "result backend".  This is not a distrubuted await. If you need to store results, pass a unique key into the task and store the result in some DIY model.
+* *No workflows and chaining of jobs.*
+* *ETA (estimated time of arrival) is a first-class citizen.* It does not depend on whether backends support the feature or not.
+* *The worker is a single-threaded process*, you start several of those to scale. I have seen too many issues with autoscaling workers, worker processes killed by OS, workers stuck: simple is better.
+* *No prefetching or visibility timeouts*. The worker picks the first available task and processes it.
+* *Dead letter queue* built in. You get access to failed tasks and can retry them from Django admin.
+* *Django admin for monitoring.* You can view pending tasks, future, failed, and "dirty" (crashed in the middle of work).
+* *Easy to get the metrics* from Django shell and export to your favorite monitoring tool
+* *Task records are removed after successful execution*. Unlike Celery SQLAlchemy's backend, records are removed so you don't have to care about archiving. It also keeps the table small, properly indexed and efficient.
+
+Celery API
+----------
+
+The main API is the Celery API (``shared_task``) with ``delay``, ``apply_async`` and ``s``. Just to make switching between implementations easier.
+
+.. code-block:: python
+  
+  from django_taskq.celery import shared_task
+
+  @shared_task(autoretry_for(MyException,), retry_kwargs={"max_retries": 10, "countdown": 5})
+  def my_task(foo, bar=None):
+      ...
+
+.. code-block:: python
+  
+  my_task.delay(1,bar=2)
+  my_task.appy_async((1,2))
+  my_task.s(1,2).apply_async()
+
+
+Internals
+---------
+
+Adding a new task to the queue is just creating a new instance of the Task model.
+
+Executing a task is a bit more expensive:
+# A task is picked up from a queue and the state is updated to "started" within a single transaction.
+# Python code is executed, a background thread updates "alive at" field every second ("a liveness probe").
+# Successful tasks are deleted from the table. Failed tasks are marked as such and retried (based on configuration).
+
+This is a bit more expensive than necessary but:
+* we can recognize running tasks - the task is "started" and the record is updated in the last couple seconds.
+* we can recognize "dirty" tasks that got killed or lost database connection in the middle - the task is "started" and the record has not been updated for a while.
+
+In an ideal world tasks should be idempotent but things happen and I prefer to know which tasks crashed and double-check if some cleanup is necessary.
+
+Performance
+-----------
+
+A single process can execute around 150 dummy tasks per second which is more than enough. After years of struggling with Celery, correctness and observability are more important.
+On the other hand, to handle more "tasks" you probably want to store many events not tasks and have a single task that processes them in batches.

django-taskq-1.0.0/README.rst

@@ -0,0 +1,59 @@
+Django database task queue
+
+Django Transaction Task Queue
+=============================
+
+A short and simple Celery replacement for my Django projects.
+
+* *Database is the only backend.* The task is a simple Django model, it uses the same transaction and connection as other models. No more ``transaction.on_commit`` hooks to schedule tasks.
+* *Tasks do not produce any results*, there is no ``get`` or ``join`` and there is no "result backend".  This is not a distrubuted await. If you need to store results, pass a unique key into the task and store the result in some DIY model.
+* *No workflows and chaining of jobs.*
+* *ETA (estimated time of arrival) is a first-class citizen.* It does not depend on whether backends support the feature or not.
+* *The worker is a single-threaded process*, you start several of those to scale. I have seen too many issues with autoscaling workers, worker processes killed by OS, workers stuck: simple is better.
+* *No prefetching or visibility timeouts*. The worker picks the first available task and processes it.
+* *Dead letter queue* built in. You get access to failed tasks and can retry them from Django admin.
+* *Django admin for monitoring.* You can view pending tasks, future, failed, and "dirty" (crashed in the middle of work).
+* *Easy to get the metrics* from Django shell and export to your favorite monitoring tool
+* *Task records are removed after successful execution*. Unlike Celery SQLAlchemy's backend, records are removed so you don't have to care about archiving. It also keeps the table small, properly indexed and efficient.
+
+Celery API
+----------
+
+The main API is the Celery API (``shared_task``) with ``delay``, ``apply_async`` and ``s``. Just to make switching between implementations easier.
+
+.. code-block:: python
+  
+  from django_taskq.celery import shared_task
+
+  @shared_task(autoretry_for(MyException,), retry_kwargs={"max_retries": 10, "countdown": 5})
+  def my_task(foo, bar=None):
+      ...
+
+.. code-block:: python
+  
+  my_task.delay(1,bar=2)
+  my_task.appy_async((1,2))
+  my_task.s(1,2).apply_async()
+
+
+Internals
+---------
+
+Adding a new task to the queue is just creating a new instance of the Task model.
+
+Executing a task is a bit more expensive:
+# A task is picked up from a queue and the state is updated to "started" within a single transaction.
+# Python code is executed, a background thread updates "alive at" field every second ("a liveness probe").
+# Successful tasks are deleted from the table. Failed tasks are marked as such and retried (based on configuration).
+
+This is a bit more expensive than necessary but:
+* we can recognize running tasks - the task is "started" and the record is updated in the last couple seconds.
+* we can recognize "dirty" tasks that got killed or lost database connection in the middle - the task is "started" and the record has not been updated for a while.
+
+In an ideal world tasks should be idempotent but things happen and I prefer to know which tasks crashed and double-check if some cleanup is necessary.
+
+Performance
+-----------
+
+A single process can execute around 150 dummy tasks per second which is more than enough. After years of struggling with Celery, correctness and observability are more important.
+On the other hand, to handle more "tasks" you probably want to store many events not tasks and have a single task that processes them in batches.

django-taskq-1.0.0/django_taskq/admin.py

@@ -0,0 +1,84 @@
+from django.contrib import admin, messages
+
+from django_taskq.models import (ActiveTask, DirtyTask, FailedTask, FutureTask,
+                                 PendingTask)
+
+
+@admin.register(PendingTask, ActiveTask, FutureTask)
+class PendingTaskAdmin(admin.ModelAdmin):
+    fields = ("id", "execute_at", "queue", "repr", "created_at")
+    list_display = ("id", "execute_at", "queue", "func", "args", "kwargs")
+    readonly_fields = (
+        "id",
+        "execute_at",
+        "queue",
+        "func",
+        "args",
+        "kwargs",
+        "created_at",
+    )
+    list_filter = (
+        "queue",
+        "func",
+        ("execute_at", admin.DateFieldListFilter),
+        ("created_at", admin.DateFieldListFilter),
+        ("started_at", admin.DateFieldListFilter),
+    )
+
+    def has_add_permission(self, request, obj=None):
+        return False
+
+    def has_delete_permission(self, request, obj=None):
+        return False
+
+
+@admin.register(DirtyTask, FailedTask)
+class RestartableTaskAdmin(admin.ModelAdmin):
+    fields = (
+        "id",
+        "execute_at",
+        "queue",
+        "func",
+        "args",
+        "kwargs",
+        "retries",
+        "created_at",
+        "alive_at",
+        "traceback",
+    )
+    list_display = ("id", "execute_at", "queue", "repr", "retries")
+    readonly_fields = (
+        "id",
+        "execute_at",
+        "func",
+        "retries",
+        "created_at",
+        "alive_at",
+        "traceback",
+    )
+    list_filter = (
+        "queue",
+        "func",
+        ("execute_at", admin.DateFieldListFilter),
+        ("created_at", admin.DateFieldListFilter),
+        ("alive_at", admin.DateFieldListFilter),
+    )
+    actions = ("force_retry",)
+
+    def has_add_permission(self, request, obj=None):
+        return False
+
+    def has_delete_permission(self, request, obj=None):
+        return True
+
+    @admin.action(description="Retry selected tasks")
+    def force_retry(self, request, queryset):
+        count = 0
+        for task in queryset.iterator():
+            count += 1
+            task.retry()
+        self.message_user(
+            request,
+            f"{count} task(s) will be retried",
+            messages.SUCCESS,
+        )

django-taskq-1.0.0/django_taskq/apps.py

@@ -0,0 +1,8 @@
+from django.apps import AppConfig
+
+
+class TaskQConfig(AppConfig):
+    name = "django_taskq"
+    label = "taskq"
+    verbose_name = "Task Queue"
+    default_auto_field = "django.db.models.BigAutoField"

django-taskq-1.0.0/django_taskq/celery.py

@@ -0,0 +1,123 @@
+import datetime
+import inspect
+from functools import wraps
+from typing import Union
+
+from django.conf import settings
+from django.utils import timezone
+
+from django_taskq.models import Retry, Task
+
+__all__ = ["shared_task", "Ignore", "Retry"]
+
+
+def _apply_async(
+    func,
+    args: tuple = None,
+    kwargs: dict = None,
+    countdown: float = None,
+    eta: datetime.datetime = None,
+    expires: Union[float, datetime.datetime] = None,
+    queue: str = None,
+):
+    if countdown:
+        eta = timezone.now() + timezone.timedelta(seconds=int(countdown))
+    if expires and isinstance(expires, (int, float)):
+        expires = timezone.now() + timezone.timedelta(seconds=int(expires))
+
+    task = Task(
+        queue=queue or Task.DEFAULTQ,
+        func=".".join((inspect.getmodule(func).__name__, func.__name__)),
+        args=args or (),
+        kwargs=kwargs or {},
+        execute_at=eta,
+        expires_at=expires,
+    )
+    if getattr(settings, "CELERY_TASK_ALWAYS_EAGER", False):
+        task.execute()
+    else:
+        task.save()
+
+
+def _retry(
+    args=None,
+    kwargs=None,
+    exc=None,
+    throw=True,
+    eta=None,
+    countdown=None,
+    max_retries=None,
+    **options
+):
+    pass
+
+
+class Signature:
+    def __init__(self, func, args, kwargs):
+        self.func = func
+        self.args = args
+        self.kwargs = kwargs
+        self.options = {}
+
+    def set(self, **kwargs):
+        self.options.update(kwargs)
+        return self
+
+    def delay(self):
+        _apply_async(self.func, self.args, self.kwargs, **self.options)
+
+    def apply_async(self, **kwargs):
+        self.options.update(kwargs)
+        _apply_async(self.func, self.args, self.kwargs, **self.options)
+
+
+def _retry(exc=None, eta=None, countdown=None, max_retries=None):
+    if not eta:
+        if countdown is None:
+            countdown = 3 * 60
+        eta = timezone.now() + timezone.timedelta(seconds=int(countdown))
+    raise Retry(exc=exc, execute_at=eta, max_retries=max_retries)
+
+
+def _wrap_autoretry(func, **options):
+    autoretry_for = tuple(options.get("autoretry_for", ()))
+    dont_autoretry_for = tuple(options.get("dont_autoretry_for", ()))
+    retry_kwargs = options.get("retry_kwargs", {})
+    max_retries = retry_kwargs.get("max_retries", 3)
+    countdown = retry_kwargs.get("countdown", 3 * 60)
+    eta = timezone.now() + timezone.timedelta(seconds=int(countdown))
+
+    if autoretry_for:
+
+        @wraps(func)
+        def run(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except Retry:
+                raise
+            except dont_autoretry_for:
+                raise
+            except autoretry_for as exc:
+                raise Retry(exc=exc, execute_at=eta, max_retries=max_retries)
+
+        return run
+
+    return func
+
+
+def shared_task(*args, **kwargs):
+    def create_shared_task(**options):
+        def run(func):
+            func.delay = lambda *args, **kwargs: _apply_async(func, args, kwargs)
+            func.apply_async = lambda *args, **kwargs: _apply_async(
+                func, *args, **kwargs
+            )
+            func.s = lambda *args, **kwargs: Signature(func, args, kwargs)
+            func.retry = lambda *args, **kwargs: _retry(*args, **kwargs)
+            return _wrap_autoretry(func, **options)
+
+        return run
+
+    if len(args) and callable(args[0]):
+        return create_shared_task(**kwargs)(args[0])
+    return create_shared_task(*args, **kwargs)

django-taskq-1.0.0/django_taskq/management/commands/taskq.py

@@ -0,0 +1,71 @@
+import threading
+import time
+
+from django.core.management.base import BaseCommand
+
+from django_taskq.models import Retry, Task
+
+
+class Heartbeat(threading.Thread):
+    def __init__(self, command):
+        super().__init__()
+        self.finished = threading.Event()
+        self.command = command
+
+    def run(self):
+        previous = None
+        while not self.finished.is_set():
+            self.finished.wait(1)
+            current = self.command.task_id
+
+            # Skip the first second update
+            # TODO: make this smarter
+            if current and current == previous:
+                Task.alive(current)
+                self.command.stdout.write(
+                    self.command.style.SUCCESS(f"Task({current}) alive")
+                )
+            previous = current
+
+    def cancel(self):
+        self.finished.set()
+
+
+class Command(BaseCommand):
+    help = "Process tasks from a queue specified by -Q or 'default'"
+    task_id = None
+
+    def add_arguments(self, parser):
+        parser.add_argument("-Q", action="store", dest="queue_name", help="Queue name")
+
+    def handle(self, *_, **options):
+        heartbeat = Heartbeat(self)
+        heartbeat.start()
+        try:
+            while True:
+                task = Task.next_task(queue=options.get("queue_name"))
+                if not task:
+                    self.stdout.write(self.style.SUCCESS("No new tasks"))
+                    time.sleep(1)
+                else:
+                    self._execute_one(task)
+        finally:
+            heartbeat.cancel()
+
+    def _execute_one(self, task):
+        self.stdout.write(self.style.SUCCESS(f"Processing Task({task.pk}) {task.repr}"))
+
+        try:
+            self.task_id = task.id
+            task.execute()
+
+            self.stdout.write(self.style.SUCCESS(f"Completed Task({task.pk})"))
+            task.delete()
+        except Retry as retry:
+            self.stdout.write(self.style.ERROR(f"Failed Task({task.pk}), will retry"))
+            task.retry(retry)
+        except Exception as exc:
+            self.stdout.write(self.style.ERROR(f"Failed Task({task.pk}): {exc!r}"))
+            task.fail(exc)
+        finally:
+            self.task_id = None

django-taskq-1.0.0/django_taskq/migrations/0001_initial.py

@@ -0,0 +1,103 @@
+# Generated by Django 5.1 on 2024-08-13 19:30
+
+import picklefield.fields
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = []
+
+    operations = [
+        migrations.CreateModel(
+            name="Task",
+            fields=[
+                (
+                    "id",
+                    models.BigAutoField(
+                        auto_created=True,
+                        primary_key=True,
+                        serialize=False,
+                        verbose_name="ID",
+                    ),
+                ),
+                ("queue", models.CharField(max_length=64)),
+                ("func", models.CharField(max_length=256)),
+                ("args", picklefield.fields.PickledObjectField(editable=False)),
+                ("kwargs", picklefield.fields.PickledObjectField(editable=False)),
+                ("repr", models.CharField(max_length=512)),
+                ("created_at", models.DateTimeField(auto_now_add=True)),
+                ("execute_at", models.DateTimeField(auto_now_add=True)),
+                ("expires_at", models.DateTimeField(blank=True, null=True)),
+                (
+                    "alive_at",
+                    models.DateTimeField(blank=True, editable=False, null=True),
+                ),
+                ("retries", models.PositiveIntegerField(default=0, editable=False)),
+                ("traceback", models.TextField(blank=True, editable=False, null=True)),
+                ("started", models.BooleanField(default=False, editable=False)),
+                ("failed", models.BooleanField(default=False, editable=False)),
+            ],
+            options={
+                "indexes": [
+                    models.Index(
+                        condition=models.Q(("failed", False), ("started", False)),
+                        fields=["execute_at", "queue"],
+                        name="taskq_execute_at_queue_idx",
+                    )
+                ],
+            },
+        ),
+        migrations.CreateModel(
+            name="ActiveTask",
+            fields=[],
+            options={
+                "proxy": True,
+                "indexes": [],
+                "constraints": [],
+            },
+            bases=("taskq.task",),
+        ),
+        migrations.CreateModel(
+            name="DirtyTask",
+            fields=[],
+            options={
+                "proxy": True,
+                "indexes": [],
+                "constraints": [],
+            },
+            bases=("taskq.task",),
+        ),
+        migrations.CreateModel(
+            name="FailedTask",
+            fields=[],
+            options={
+                "proxy": True,
+                "indexes": [],
+                "constraints": [],
+            },
+            bases=("taskq.task",),
+        ),
+        migrations.CreateModel(
+            name="FutureTask",
+            fields=[],
+            options={
+                "proxy": True,
+                "indexes": [],
+                "constraints": [],
+            },
+            bases=("taskq.task",),
+        ),
+        migrations.CreateModel(
+            name="PendingTask",
+            fields=[],
+            options={
+                "proxy": True,
+                "indexes": [],
+                "constraints": [],
+            },
+            bases=("taskq.task",),
+        ),
+    ]

django-taskq-1.0.0/django_taskq/models.py

@@ -0,0 +1,208 @@
+import datetime
+import traceback
+
+from django.db import models, transaction
+from django.utils import timezone
+from picklefield.fields import PickledObjectField
+
+
+class MaxRetriesExceededError(Exception): ...
+
+
+class Retry(Exception):
+    exc = None
+    execute_at = None
+    max_retries = None
+
+    def __init__(self, exc=None, execute_at=None, max_retries=None):
+        super().__init__()
+        self.exc = exc
+        self.execute_at = execute_at
+        self.max_retries = max_retries
+
+
+class Task(models.Model):
+    DEFAULTQ = "default"
+    queue = models.CharField(max_length=64)
+
+    func = models.CharField(max_length=256)
+    args = PickledObjectField()
+    kwargs = PickledObjectField()
+    repr = models.CharField(max_length=512)
+
+    created_at = models.DateTimeField(auto_now_add=True, editable=False)
+    execute_at = models.DateTimeField(auto_now_add=True)
+    expires_at = models.DateTimeField(null=True, blank=True)
+    alive_at = models.DateTimeField(null=True, blank=True, editable=False)
+
+    retries = models.PositiveIntegerField(default=0, editable=False)
+    traceback = models.TextField(null=True, blank=True, editable=False)
+
+    started = models.BooleanField(default=False, editable=False)
+    failed = models.BooleanField(default=False, editable=False)
+
+    def save(self, *args, **kwargs):
+        strargs = [str(arg) for arg in self.args]
+        strargs += [str(key) + "=" + str(value) for key, value in self.kwargs.items()]
+        self.repr = f"{self.func}({', '.join(strargs)})"
+        super().save(*args, **kwargs)
+
+    @classmethod
+    def next_task(cls, queue=None):
+        queue = queue or "default"
+        while True:
+            with transaction.atomic():
+                task = (
+                    cls.objects.select_for_update(skip_locked=True)
+                    .filter(
+                        failed=False,
+                        started=False,
+                        execute_at__lte=timezone.now(),
+                        queue=queue,
+                    )
+                    .order_by("execute_at")
+                    .first()
+                )
+                if not task:
+                    return task
+
+                if task.expires_at and task.expires_at <= timezone.now():
+                    task.delete()
+                else:
+                    task.started = True
+                    task.alive_at = timezone.now()
+                    task.save(update_fields=["started", "alive_at"])
+                    return task
+
+    @classmethod
+    def alive(cls, task_id):
+        cls.obects.filter(pk=task_id).update(alive_at=timezone.now())
+
+    def fail(self, exc):
+        self.started = False
+        self.failed = True
+        self.traceback = traceback.format_exception(exc)
+        self.save(update_fields=["started", "failed", "traceback"])
+
+    def retry(self, retry_info: Retry):
+
+        self.retries += 1
+        if retry_info.max_retries is not None and self.retries > retry_info.max_retries:
+            if retry_info.exc:
+                self.fail(retry_info.exc)
+            else:
+                self.fail(MaxRetriesExceededError())
+            return False
+
+        self.started = False
+        self.failed = False
+        if retry_info.exc:
+            self.traceback = traceback.format_exception(retry_info.exc)
+        else:
+            self.traceback = None
+        self.execute_at = retry_info.execute_at
+        self.save(
+            update_fields=["started", "failed", "traceback", "retries", "execute_at"]
+        )
+        return True
+
+    def execute(self):
+        last_dot = self.func.rindex(".")
+        mod = self.func[:last_dot]
+        func = self.func[last_dot + 1 :]
+
+        f = getattr(__import__(mod, fromlist=(func,)), func)
+        f(*self.args, **self.kwargs)
+
+    class Meta:
+        indexes = [
+            models.Index(
+                name="taskq_execute_at_queue_idx",
+                fields=("execute_at", "queue"),
+                condition=models.Q(failed=False, started=False),
+            )
+        ]
+
+
+class PendingTaskManager(models.Manager):
+    def get_queryset(self):
+        return (
+            super()
+            .get_queryset()
+            .filter(failed=False, started=False, execute_at__lte=timezone.now())
+            .order_by("execute_at")
+        )
+
+
+class PendingTask(Task):
+    objects = PendingTaskManager()
+
+    class Meta:
+        proxy = True
+
+
+class FutureTaskManager(models.Manager):
+    def get_queryset(self):
+        return (
+            super()
+            .get_queryset()
+            .filter(failed=False, started=False, execute_at__gt=timezone.now())
+            .order_by("execute_at")
+        )
+
+
+class FutureTask(Task):
+    objects = FutureTaskManager()
+
+    class Meta:
+        proxy = True
+
+
+class ActiveTaskManager(models.Manager):
+    def get_queryset(self):
+        return (
+            super()
+            .get_queryset()
+            .filter(
+                started=True,
+                alive_at__gt=timezone.now() - datetime.timedelta(seconds=3),
+            )
+        )
+
+
+class ActiveTask(Task):
+    objects = ActiveTaskManager()
+
+    class Meta:
+        proxy = True
+
+
+class DirtyTaskManager(models.Manager):
+    def get_queryset(self):
+        return (
+            super()
+            .get_queryset()
+            .filter(
+                started=True,
+                alive_at__lte=timezone.now() - datetime.timedelta(seconds=3),
+            )
+        )
+
+
+class DirtyTask(Task):
+    objects = DirtyTaskManager()
+
+    class Meta:
+        proxy = True
+
+
+class FailedTaskManager(models.Manager):
+    def get_queryset(self):
+        return super().get_queryset().filter(failed=True)
+
+
+class FailedTask(Task):
+    objects = FailedTaskManager()
+
+    class Meta:
+        proxy = True

django-taskq-1.0.0/django_taskq/tests.py

@@ -0,0 +1,157 @@
+import datetime
+from unittest import TestCase
+
+from django.utils import timezone
+
+from django_taskq.celery import shared_task
+from django_taskq.models import Retry, Task
+
+
+class TaskTestCase(TestCase):
+    def test_repr_is_created_on_save(self):
+        task = Task.objects.create(func="foo", args=(1, 2), kwargs={"a": 3, "b": 4})
+        self.assertEqual(task.repr, "foo(1, 2, a=3, b=4)")
+
+    def test_datetime_parameter_can_be_serialized(self):
+        timestamp = datetime.datetime.now()
+        task = Task.objects.create(func="foo", args=(timestamp,), kwargs={})
+        task.refresh_from_db()
+        self.assertEqual(task.args[0], timestamp)
+
+
+@shared_task
+def dummy(a, *, b=None):
+    pass
+
+
+@shared_task
+def failing_task():
+    raise KeyError()
+
+
+@shared_task(autoretry_for=(ValueError,))
+def retry_task():
+    raise ValueError()
+
+
+@shared_task
+def self_retry_task():
+    self_retry_task.retry()
+
+
+@shared_task(autoretry_for=(ValueError,))
+def retry_count():
+    raise ValueError()
+
+
+@shared_task(
+    autoretry_for=(ValueError,), retry_kwargs={"max_retries": 2, "countdown": 5}
+)
+def retry_params():
+    raise ValueError()
+
+
+class CeleryInterfaceTestCase(TestCase):
+    def test_shared_task_delay(self):
+        dummy.delay(1, 2)
+
+        task = Task.objects.last()
+        self.assertEqual(task.repr, "django_taskq.tests.dummy(1, 2)")
+
+    def test_shared_task_apply_async(self):
+        dummy.apply_async(args=(1, 2))
+
+        task = Task.objects.last()
+        self.assertEqual(task.repr, "django_taskq.tests.dummy(1, 2)")
+
+    def test_shared_task_signature_delay(self):
+        dummy.s(1, 2).delay()
+
+        task = Task.objects.last()
+        self.assertEqual(task.repr, "django_taskq.tests.dummy(1, 2)")
+
+    def test_shared_task_signature_apply_async(self):
+        dummy.s(1, 2).apply_async()
+
+        task = Task.objects.last()
+        self.assertEqual(task.repr, "django_taskq.tests.dummy(1, 2)")
+
+    def test_shared_task_fail(self):
+        failing_task.delay()
+
+        task = Task.objects.last()
+        with self.assertRaises(KeyError):
+            task.execute()
+
+    def test_shared_task_retry(self):
+        retry_task.delay()
+
+        task = Task.objects.last()
+        with self.assertRaises(Retry):
+            task.execute()
+
+    def test_shared_task_self_retry(self):
+        self_retry_task.delay()
+
+        task = Task.objects.last()
+        with self.assertRaises(Retry):
+            task.execute()
+
+    def test_shared_task_self_retry_counter(self):
+        retry_count.delay()
+
+        task = Task.objects.last()
+        task.execute_at = timezone.now()
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertFalse(task.failed)
+        self.assertAlmostEqual(
+            (task.execute_at - timezone.now()).seconds, 3 * 60, delta=1
+        )
+
+        task.execute_at = timezone.now()
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertFalse(task.failed)
+        self.assertAlmostEqual(
+            (task.execute_at - timezone.now()).seconds, 3 * 60, delta=1
+        )
+
+        task.execute_at = timezone.now()
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertFalse(task.failed)
+        self.assertAlmostEqual(
+            (task.execute_at - timezone.now()).seconds, 3 * 60, delta=1
+        )
+
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertTrue(task.failed)
+
+    def test_shared_task_self_retry_params(self):
+        retry_params.delay()
+
+        task = Task.objects.last()
+        task.execute_at = timezone.now()
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertFalse(task.failed)
+        self.assertAlmostEqual((task.execute_at - timezone.now()).seconds, 5, delta=1)
+
+        task.execute_at = timezone.now()
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertFalse(task.failed)
+        self.assertAlmostEqual((task.execute_at - timezone.now()).seconds, 5, delta=1)
+
+        with self.assertRaises(Retry) as exc_info:
+            task.execute()
+        task.retry(exc_info.exception)
+        self.assertTrue(task.failed)

django-taskq-1.0.0/django_taskq.egg-info/PKG-INFO

@@ -0,0 +1,82 @@
+Metadata-Version: 2.1
+Name: django-taskq
+Version: 1.0.0
+Summary: A simple Celery-like database-backed task queue for Django
+Home-page: https://github.com/aivarsk/django-taskq
+Author: Aivars Kalvans
+Author-email: aivars.kalvans@gmail.com
+License: MIT
+Classifier: Environment :: Web Environment
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 4.0
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+License-File: LICENSE
+
+Django database task queue
+
+Django Transaction Task Queue
+=============================
+
+A short and simple Celery replacement for my Django projects.
+
+* *Database is the only backend.* The task is a simple Django model, it uses the same transaction and connection as other models. No more ``transaction.on_commit`` hooks to schedule tasks.
+* *Tasks do not produce any results*, there is no ``get`` or ``join`` and there is no "result backend".  This is not a distrubuted await. If you need to store results, pass a unique key into the task and store the result in some DIY model.
+* *No workflows and chaining of jobs.*
+* *ETA (estimated time of arrival) is a first-class citizen.* It does not depend on whether backends support the feature or not.
+* *The worker is a single-threaded process*, you start several of those to scale. I have seen too many issues with autoscaling workers, worker processes killed by OS, workers stuck: simple is better.
+* *No prefetching or visibility timeouts*. The worker picks the first available task and processes it.
+* *Dead letter queue* built in. You get access to failed tasks and can retry them from Django admin.
+* *Django admin for monitoring.* You can view pending tasks, future, failed, and "dirty" (crashed in the middle of work).
+* *Easy to get the metrics* from Django shell and export to your favorite monitoring tool
+* *Task records are removed after successful execution*. Unlike Celery SQLAlchemy's backend, records are removed so you don't have to care about archiving. It also keeps the table small, properly indexed and efficient.
+
+Celery API
+----------
+
+The main API is the Celery API (``shared_task``) with ``delay``, ``apply_async`` and ``s``. Just to make switching between implementations easier.
+
+.. code-block:: python
+  
+  from django_taskq.celery import shared_task
+
+  @shared_task(autoretry_for(MyException,), retry_kwargs={"max_retries": 10, "countdown": 5})
+  def my_task(foo, bar=None):
+      ...
+
+.. code-block:: python
+  
+  my_task.delay(1,bar=2)
+  my_task.appy_async((1,2))
+  my_task.s(1,2).apply_async()
+
+
+Internals
+---------
+
+Adding a new task to the queue is just creating a new instance of the Task model.
+
+Executing a task is a bit more expensive:
+# A task is picked up from a queue and the state is updated to "started" within a single transaction.
+# Python code is executed, a background thread updates "alive at" field every second ("a liveness probe").
+# Successful tasks are deleted from the table. Failed tasks are marked as such and retried (based on configuration).
+
+This is a bit more expensive than necessary but:
+* we can recognize running tasks - the task is "started" and the record is updated in the last couple seconds.
+* we can recognize "dirty" tasks that got killed or lost database connection in the middle - the task is "started" and the record has not been updated for a while.
+
+In an ideal world tasks should be idempotent but things happen and I prefer to know which tasks crashed and double-check if some cleanup is necessary.
+
+Performance
+-----------
+
+A single process can execute around 150 dummy tasks per second which is more than enough. After years of struggling with Celery, correctness and observability are more important.
+On the other hand, to handle more "tasks" you probably want to store many events not tasks and have a single task that processes them in batches.

django-taskq-1.0.0/django_taskq.egg-info/SOURCES.txt

@@ -0,0 +1,22 @@
+LICENSE
+MANIFEST.in
+README.rst
+pyproject.toml
+setup.cfg
+setup.py
+django_taskq/__init__.py
+django_taskq/admin.py
+django_taskq/apps.py
+django_taskq/celery.py
+django_taskq/models.py
+django_taskq/tests.py
+django_taskq.egg-info/PKG-INFO
+django_taskq.egg-info/SOURCES.txt
+django_taskq.egg-info/dependency_links.txt
+django_taskq.egg-info/requires.txt
+django_taskq.egg-info/top_level.txt
+django_taskq/management/__init__.py
+django_taskq/management/commands/__init__.py
+django_taskq/management/commands/taskq.py
+django_taskq/migrations/0001_initial.py
+django_taskq/migrations/__init__.py

django-taskq-1.0.0/django_taskq.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

django-taskq-1.0.0/django_taskq.egg-info/requires.txt

@@ -0,0 +1,2 @@
+Django>=4.0
+django-picklefield>=3.2

django-taskq-1.0.0/django_taskq.egg-info/top_level.txt

@@ -0,0 +1 @@
+django_taskq

django-taskq-1.0.0/pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ['setuptools>=40.8.0']
+build-backend = 'setuptools.build_meta'

django-taskq-1.0.0/setup.cfg

@@ -0,0 +1,35 @@
+[metadata]
+name = django-taskq
+version = 1.0.0
+description = A simple Celery-like database-backed task queue for Django
+long_description = file: README.rst
+url = https://github.com/aivarsk/django-taskq
+author = Aivars Kalvans
+author_email = aivars.kalvans@gmail.com
+license = MIT
+classifiers = 
+	Environment :: Web Environment
+	Framework :: Django
+	Framework :: Django :: 4.0
+	Intended Audience :: Developers
+	License :: OSI Approved :: MIT License
+	Operating System :: OS Independent
+	Programming Language :: Python
+	Programming Language :: Python :: 3
+	Programming Language :: Python :: 3 :: Only
+	Programming Language :: Python :: 3.10
+	Programming Language :: Python :: 3.11
+	Programming Language :: Python :: 3.12
+
+[options]
+include_package_data = true
+packages = find:
+python_requires = >=3.10
+install_requires = 
+	Django >= 4.0
+	django-picklefield >= 3.2
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

django-taskq-1.0.0/setup.py

@@ -0,0 +1,3 @@
+from setuptools import setup
+
+setup()