threadmill 0.3.1__tar.gz → 0.4.0__tar.gz

{threadmill-0.3.1 → threadmill-0.4.0}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: threadmill
-Version: 0.3.1
+Version: 0.4.0
 Summary: The most reliable backend for Django's task framework.
 Keywords: Django,tasks,worker
 Author-email: Johannes Maron <johannes@maron.family>
-Requires-Python: >=3.12
+Requires-Python: >=3.14
 Description-Content-Type: text/markdown
 Classifier: Development Status :: 4 - Beta
 Classifier: Programming Language :: Python
@@ -20,11 +20,8 @@ Classifier: Topic :: Software Development
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Framework :: Django
-Classifier: Framework :: Django :: 6.0
 Classifier: Framework :: Django :: 6.1
 License-File: LICENSE
 Requires-Dist: django>=6.0
@@ -59,13 +56,9 @@ Provides-Extra: redis
 
 ## Design Principles
 
-- **Durability** – We recover from any failures, even poorly written tasks.
-- **Consistency** – We never lose data, even if someone unplugs the power or network.
-- **Utilization** – We keep the CPU saturated with tasks, not with idle time or waiting for locks.
-
-## Sponsors
-
-[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+- **Durability** – Recover from any failures, even poorly written tasks.
+- **Consistency** – Never lose data, even if someone unplugs the power or network.
+- **Utilization** – Keep the CPU saturated with tasks, not with idle time or waiting for locks.
 
 ## Setup
 
@@ -176,5 +169,78 @@ every multi-key operation — including the cross-queue acquire — runs on a si
 shard. Scale horizontally by running additional backend aliases, not by relying
 on cross-slot operations.
 
+### Retrying failed tasks
+
+Pass a `retry` callback to `@task()` to retry failed tasks with a delay.
+The callback receives a `TaskContext` — use `context.attempt` for the current
+attempt count and `context.task_result.errors[-1]` for the latest error.
+Return a `timedelta` to schedule the next attempt, or `None` to stop retrying.
+
+The worker re-queues the failed task, preserving its ID and error history;
+the broker promotes it back to the ready queue once the delay elapses.
+
+#### Built-in `ExponentialBackoff`
+
+`threadmill.retry.ExponentialBackoff` provides a serializable exponential
+backoff strategy out of the box. It caps the delay at `max_delay`, stops
+after `max_retries` attempts, and only retries exceptions listed in
+`expected_exceptions`.
+
+```python
+import datetime
+
+from django.tasks import task
+from requests import HTTPError
+
+from threadmill.retry import ExponentialBackoff
+
+
+@task(
+    retry=ExponentialBackoff(
+        base_delay=datetime.timedelta(seconds=1),
+        max_delay=datetime.timedelta(minutes=5),
+        factor=2.0,
+        max_retries=5,
+        expected_exceptions=(HTTPError,),
+    )
+)
+def fetch_github_api(url: str): ...
+```
+
+#### Custom retry callbacks
+
+For cases that need logic beyond what `ExponentialBackoff` supports,
+write a callable that accepts a `TaskContext` and returns a `timedelta`
+or `None`. Use `TaskError.exception_class` to filter by exception type:
+
+```python
+import datetime
+
+from django.tasks import task
+from django.tasks.base import TaskContext
+from requests import HTTPError
+
+
+def retry_on_rate_limit(context: TaskContext) -> datetime.timedelta | None:
+    """Retry HTTP 429 responses with exponential backoff, up to 5 attempts."""
+    if context.attempt >= 5:
+        return None
+    error = context.task_result.errors[-1]
+    if not issubclass(error.exception_class, HTTPError):
+        return None
+    return min(
+        datetime.timedelta(seconds=2**context.attempt),
+        datetime.timedelta(seconds=60),
+    )
+
+
+@task(retry=retry_on_rate_limit)
+def fetch_github_api(url: str): ...
+```
+
+## Sponsors
+
+[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+
 [django-tasks]: https://docs.djangoproject.com/en/stable/topics/tasks/
 

{threadmill-0.3.1 → threadmill-0.4.0}/README.md

@@ -17,13 +17,9 @@
 
 ## Design Principles
 
-- **Durability** – We recover from any failures, even poorly written tasks.
-- **Consistency** – We never lose data, even if someone unplugs the power or network.
-- **Utilization** – We keep the CPU saturated with tasks, not with idle time or waiting for locks.
-
-## Sponsors
-
-[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+- **Durability** – Recover from any failures, even poorly written tasks.
+- **Consistency** – Never lose data, even if someone unplugs the power or network.
+- **Utilization** – Keep the CPU saturated with tasks, not with idle time or waiting for locks.
 
 ## Setup
 
@@ -134,4 +130,77 @@ every multi-key operation — including the cross-queue acquire — runs on a si
 shard. Scale horizontally by running additional backend aliases, not by relying
 on cross-slot operations.
 
+### Retrying failed tasks
+
+Pass a `retry` callback to `@task()` to retry failed tasks with a delay.
+The callback receives a `TaskContext` — use `context.attempt` for the current
+attempt count and `context.task_result.errors[-1]` for the latest error.
+Return a `timedelta` to schedule the next attempt, or `None` to stop retrying.
+
+The worker re-queues the failed task, preserving its ID and error history;
+the broker promotes it back to the ready queue once the delay elapses.
+
+#### Built-in `ExponentialBackoff`
+
+`threadmill.retry.ExponentialBackoff` provides a serializable exponential
+backoff strategy out of the box. It caps the delay at `max_delay`, stops
+after `max_retries` attempts, and only retries exceptions listed in
+`expected_exceptions`.
+
+```python
+import datetime
+
+from django.tasks import task
+from requests import HTTPError
+
+from threadmill.retry import ExponentialBackoff
+
+
+@task(
+    retry=ExponentialBackoff(
+        base_delay=datetime.timedelta(seconds=1),
+        max_delay=datetime.timedelta(minutes=5),
+        factor=2.0,
+        max_retries=5,
+        expected_exceptions=(HTTPError,),
+    )
+)
+def fetch_github_api(url: str): ...
+```
+
+#### Custom retry callbacks
+
+For cases that need logic beyond what `ExponentialBackoff` supports,
+write a callable that accepts a `TaskContext` and returns a `timedelta`
+or `None`. Use `TaskError.exception_class` to filter by exception type:
+
+```python
+import datetime
+
+from django.tasks import task
+from django.tasks.base import TaskContext
+from requests import HTTPError
+
+
+def retry_on_rate_limit(context: TaskContext) -> datetime.timedelta | None:
+    """Retry HTTP 429 responses with exponential backoff, up to 5 attempts."""
+    if context.attempt >= 5:
+        return None
+    error = context.task_result.errors[-1]
+    if not issubclass(error.exception_class, HTTPError):
+        return None
+    return min(
+        datetime.timedelta(seconds=2**context.attempt),
+        datetime.timedelta(seconds=60),
+    )
+
+
+@task(retry=retry_on_rate_limit)
+def fetch_github_api(url: str): ...
+```
+
+## Sponsors
+
+[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+
 [django-tasks]: https://docs.djangoproject.com/en/stable/topics/tasks/

{threadmill-0.3.1 → threadmill-0.4.0}/pyproject.toml

@@ -26,14 +26,11 @@ classifiers = [
   "Programming Language :: Python",
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3 :: Only",
-  "Programming Language :: Python :: 3.12",
-  "Programming Language :: Python :: 3.13",
   "Programming Language :: Python :: 3.14",
   "Framework :: Django",
-  "Framework :: Django :: 6.0",
   "Framework :: Django :: 6.1",
 ]
-requires-python = ">=3.12"
+requires-python = ">=3.14"
 dependencies = ["django>=6.0"]
 
 [project.optional-dependencies]
@@ -59,7 +56,7 @@ name = "threadmill"
 write_to = "threadmill/_version.py"
 
 [tool.pytest.ini_options]
-minversion = "6.0"
+minversion = "6.1a1"
 addopts = "--cov --cov-report=xml --cov-report=term --tb=short -rxs --benchmark-autosave --benchmark-group-by=fullname --benchmark-min-rounds=10"
 testpaths = ["tests"]
 DJANGO_SETTINGS_MODULE = "tests.testapp.settings"

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.3.1'
-__version_tuple__ = version_tuple = (0, 3, 1)
+__version__ = version = '0.4.0'
+__version_tuple__ = version_tuple = (0, 4, 0)
 
-__commit_id__ = commit_id = 'g32b680b8b'
+__commit_id__ = commit_id = 'gf9749f870'

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/backends/base.py

@@ -1,39 +1,63 @@
-from __future__ import annotations
-
 import collections.abc
 import dataclasses
 import datetime
 import json
 import threading
+import typing
 from abc import ABC
 
-import django
 from django.core.serializers.json import DjangoJSONEncoder
 from django.tasks import DEFAULT_TASK_QUEUE_NAME, Task, TaskResult, TaskResultStatus
 from django.tasks.backends.base import BaseTaskBackend
-from django.tasks.base import TaskError
+from django.tasks.base import TaskContext, TaskError
+from django.tasks.exceptions import InvalidTask
+from django.utils.inspect import is_module_level_function
 from django.utils.module_loading import import_string
 
-if django.VERSION == (6, 0):
-    # https://github.com/django/django/commit/8c8b833d32c02d3ae6f43b04bb1e45968796b402
-    @dataclasses.dataclass(frozen=True, slots=True, kw_only=True)
-    class Task(Task):
-        @classmethod
-        def _reconstruct(cls, kwargs):
-            func_path = kwargs["func"]
-            try:
-                func = import_string(func_path)
-                kwargs["func"] = func.func
-            except (ImportError, AttributeError) as e:
-                msg = f"Expected {func_path!r} to point to a Task instance."
-                raise ValueError(msg) from e
-            return cls(**kwargs)
 
-        def __reduce__(self):
-            kwargs = {f.name: getattr(self, f.name) for f in dataclasses.fields(self)}
-            kwargs["func"] = self.module_path
+@dataclasses.dataclass(frozen=True, slots=True, kw_only=True)
+class RetryTask(Task):
+    """Task with an optional retry callback for backoff scheduling.
+
+    Usage:
+
+        def retry_callback(context: TaskContext) -> datetime.timedelta | None:
+            if context.attempt < 3:
+                return datetime.timedelta(seconds=60 * (2 ** context.attempt))
 
-            return (self.__class__._reconstruct, (kwargs,))
+        @task(retry=retry_callback)
+        def my_task():
+            ...
+
+    """
+
+    retry: collections.abc.Callable[[TaskContext], datetime.timedelta | None] | None = (
+        None
+    )
+
+    @property
+    def retry_path(self) -> tuple[str, tuple, dict[str, typing.Any]] | str | None:
+        """Importable dotted path of the retry callback, or None."""
+        if self.retry:
+            if hasattr(self.retry, "deconstruct"):
+                return self.retry.deconstruct()
+            return f"{self.retry.__module__}.{self.retry.__qualname__}"
+
+    @classmethod
+    def _reconstruct(cls, kwargs):
+        if retry_path := kwargs.pop("retry", None):
+            try:
+                path, pos_args, kw_args = retry_path
+            except ValueError:
+                kwargs["retry"] = import_string(retry_path)
+            else:
+                kwargs["retry"] = import_string(path)(*pos_args, **kw_args)
+        return super()._reconstruct(kwargs)
+
+    def __reduce__(self):
+        reconstructor, (kwargs,) = super().__reduce__()
+        kwargs["retry"] = self.retry_path
+        return (reconstructor, (kwargs,))
 
 
 @dataclasses.dataclass(kw_only=True, slots=True)
@@ -116,19 +140,20 @@ class TaskResultEncoder(DjangoJSONEncoder):
                 field.name: getattr(o, field.name)
                 for field in dataclasses.fields(type(o))
             }
-        if isinstance(o, Task):
-            return {
+        if isinstance(o, RetryTask):
+            data = {
                 field.name: getattr(o, field.name)
-                for field in dataclasses.fields(Task)
-                if field.name != "func"  # Exclude the function object itself
-            } | {"func": o.module_path}
+                for field in dataclasses.fields(RetryTask)
+                if field.name not in {"func", "retry"}
+            } | {"func": o.module_path, "retry": o.retry_path}
+            return data
         return super().default(o)
 
 
 class ThreadmillTaskBackend(BaseTaskBackend, ABC):
     """Interface for task queues to be processed by the executor."""
 
-    task_class = Task  # can be removed in the future when Django 6.0 support is dropped
+    task_class = RetryTask
     supports_async_task = True
     supports_get_result = True
     broker_class: type[Broker] | None = None
@@ -144,18 +169,8 @@ class ThreadmillTaskBackend(BaseTaskBackend, ABC):
         def _object_hook(d: dict) -> dict | TaskResult:
             if "task" in d and isinstance(d["task"], dict) and "func" in d["task"]:
                 task_data = d["task"]
-                func = import_string(task_data["func"])
-                if isinstance(func, cls.task_class):
-                    func = func.func
-                d["task"] = cls.task_class(
-                    func=func,
-                    **{
-                        field.name: _parse_datetime(task_data[field.name])
-                        for field in dataclasses.fields(cls.task_class)
-                        if field.name not in {"func", "takes_context"}
-                        and field.name in task_data
-                    },
-                )
+                task_data["run_after"] = _parse_datetime(task_data["run_after"])
+                d["task"] = cls.task_class._reconstruct(task_data)
                 d["status"] = TaskResultStatus(d["status"])
                 d["errors"] = [TaskError(**error) for error in d["errors"]]
                 return_value = d.pop("_return_value", None)
@@ -168,6 +183,15 @@ class ThreadmillTaskBackend(BaseTaskBackend, ABC):
 
         return json.loads(data, object_hook=_object_hook)
 
+    def validate_task(self, task: RetryTask) -> None:
+        super().validate_task(task)
+        if task.retry is not None and not (
+            is_module_level_function(task.retry) or hasattr(task.retry, "deconstruct")
+        ):
+            raise InvalidTask(
+                "Task's retry function must be defined at a module level or be a deconstructible callable."
+            )
+
     def acquire(
         self,
         *queue_names: str,
@@ -192,15 +216,19 @@ class ThreadmillTaskBackend(BaseTaskBackend, ABC):
         """Remove the task from the queue and publish the result."""
         raise NotImplementedError
 
+    def requeue(self, task_result: TaskResult, run_after: datetime.datetime) -> None:
+        """Re-queue a failed task result for a retry attempt after `run_after`."""
+        raise NotImplementedError
+
     def peek(
         self,
         queue_name: str = DEFAULT_TASK_QUEUE_NAME,
         *,
         status: TaskResultStatus,
         count: int = 1,
-    ) -> collections.abc.Generator[TaskResult, None, None]:
+    ) -> collections.abc.Generator[TaskResult]:
         """
-        Yield up to ``count`` tasks from a queue in the given status segment.
+        Yield up to `count` tasks from a queue in the given status segment.
 
         Args:
             queue_name: The name of the queue to peek into.

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/backends/redis.py

@@ -1,7 +1,6 @@
 """Redis-backed durable priority queue backend for Django's task framework."""
 
-from __future__ import annotations
-
+import dataclasses
 import datetime
 import logging
 import queue
@@ -315,6 +314,34 @@ class RedisTaskBackend(ThreadmillTaskBackend):
             ],
         )
 
+    def requeue(self, task_result: TaskResult, run_after: datetime.datetime) -> None:
+        task_result = dataclasses.replace(
+            task_result,
+            status=TaskResultStatus.READY,
+            started_at=None,
+            finished_at=None,
+        )
+        serialized = self.serialize_task_result(task_result)
+        running_key = self.RUNNING_KEY.format(
+            prefix=self.key_prefix, queue_name=task_result.task.queue_name
+        )
+        deferred_key = self.DEFERRED_KEY.format(
+            prefix=self.key_prefix, queue_name=task_result.task.queue_name
+        )
+        task_key = self.TASK_KEY.format(prefix=self.key_prefix, task_id=task_result.id)
+        score = self._compute_score(task_result.task.priority, task_result.enqueued_at)
+        run_after_ms = run_after.timestamp() * 1000
+        task_data_ttl = int(
+            self.lease_ttl.total_seconds() * 3 + self.result_ttl.total_seconds()
+        )
+
+        pipe = self.client.pipeline()
+        pipe.zrem(running_key, task_result.id)
+        pipe.hset(task_key, mapping={"data": serialized, "score": str(score)})
+        pipe.expire(task_key, task_data_ttl)
+        pipe.zadd(deferred_key, {task_result.id: run_after_ms})
+        pipe.execute()
+
     def peek(
         self,
         queue_name: str = DEFAULT_TASK_QUEUE_NAME,

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/exceptions.py

@@ -1,7 +1,5 @@
 """Custom exceptions for the threadmill task framework."""
 
-from __future__ import annotations
-
 
 class AcknowledgementTimeout(Exception):
     """Raised when a task's lease has expired before it could be acknowledged."""

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/executor.py

@@ -1,7 +1,5 @@
 """Task worker executor implementation."""
 
-from __future__ import annotations
-
 import asyncio
 import dataclasses
 import datetime
@@ -205,17 +203,32 @@ class WorkerThread(threading.Thread):
                     timeout=datetime.timedelta(seconds=1),
                     worker=self.name,
                 )
-            except (Empty, TimeoutError):
+            except Empty, TimeoutError:
                 if self.worker.shutdown_requested.is_set() or self.worker.exit_empty:
                     return
                 continue
 
             try:
                 result = self.execute_task_result(task_result)
-                self.backend.acknowledge(result)
+                if (
+                    result.status is TaskResultStatus.FAILED
+                    and (delay := self.retry_delay(result)) is not None
+                ):
+                    self.backend.requeue(result, timezone.now() + delay)
+                else:
+                    self.backend.acknowledge(result)
             finally:
                 self.worker.record_task()
 
+    @staticmethod
+    def retry_delay(task_result: TaskResult) -> datetime.timedelta | None:
+        """Return the retry delay for a failed task, or None to stop retrying."""
+        if task_result.task.retry:
+            try:
+                return task_result.task.retry(TaskContext(task_result=task_result))
+            except Exception:
+                logger.exception("Retry callback failed for task %r", task_result.id)
+
     def execute_task_result(self, task_result: TaskResult) -> TaskResult:
         """Execute task from task result and update result lifecycle state."""
         logger.info("Executing task %r", task_result.id)

{threadmill-0.3.1 → threadmill-0.4.0}/threadmill/inspector/app.py

@@ -1,7 +1,5 @@
 """Textual app for the inspector TUI."""
 
-from __future__ import annotations
-
 import datetime
 import logging
 import math

threadmill-0.4.0/threadmill/retry.py

@@ -0,0 +1,71 @@
+"""Reusable callables for retrying tasks."""
+
+import dataclasses
+import datetime
+import typing
+
+if typing.TYPE_CHECKING:
+    from django.tasks import TaskContext
+
+
+@dataclasses.dataclass(frozen=True, slots=True, kw_only=True)
+class ExponentialBackoff:
+    """Delay retries exponentially, capped at ``max_delay`` and limited to ``max_retries`` attempts.
+
+    Only retry exceptions listed in ``expected_exceptions``.
+
+    Usage:
+
+        @task(retry=ExponentialBackoff(base_delay=datetime.timedelta(seconds=1), max_delay=datetime.timedelta(minutes=5), factor=2.0, max_retries=5))
+        def my_task():
+            ...
+    """
+
+    base_delay: datetime.timedelta = dataclasses.field(
+        default=datetime.timedelta(seconds=1), doc="Base delay in seconds"
+    )
+    max_delay: datetime.timedelta = dataclasses.field(
+        default=datetime.timedelta(minutes=60), doc="Maximum delay in seconds"
+    )
+    factor: float = dataclasses.field(default=2.0, doc="Exponential factor for backoff")
+    max_retries: int = dataclasses.field(default=5, doc="Maximum number of retries")
+    expected_exceptions: tuple[type[Exception], ...] = dataclasses.field(
+        default=(Exception,), doc="Tuple of exception classes that trigger a retry"
+    )
+
+    def __post_init__(self) -> None:
+        """Resolve deserialized fields back to their original types."""
+        for field_name in ("base_delay", "max_delay"):
+            value = getattr(self, field_name)
+            if isinstance(value, (int, float)):
+                object.__setattr__(self, field_name, datetime.timedelta(seconds=value))
+        resolved = []
+        for exc in self.expected_exceptions:
+            if isinstance(exc, str):
+                from django.utils.module_loading import import_string
+
+                exc = import_string(exc)
+            resolved.append(exc)
+        object.__setattr__(self, "expected_exceptions", tuple(resolved))
+
+    def __call__(self, context: TaskContext) -> datetime.timedelta | None:
+        if context.attempt < self.max_retries and issubclass(
+            context.task_result.errors[-1].exception_class, self.expected_exceptions
+        ):
+            return min(self.base_delay * (self.factor**context.attempt), self.max_delay)
+
+    def deconstruct(self):
+        return (
+            f"{self.__class__.__module__}.{self.__class__.__qualname__}",
+            (),
+            {
+                "base_delay": self.base_delay.total_seconds(),
+                "max_delay": self.max_delay.total_seconds(),
+                "factor": self.factor,
+                "max_retries": self.max_retries,
+                "expected_exceptions": tuple(
+                    f"{exc.__module__}.{exc.__qualname__}"
+                    for exc in self.expected_exceptions
+                ),
+            },
+        )