threadmill 0.2.0__tar.gz → 0.4.0__tar.gz

{threadmill-0.2.0 → threadmill-0.4.0}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: threadmill
-Version: 0.2.0
+Version: 0.4.0
 Summary: The most reliable backend for Django's task framework.
 Keywords: Django,tasks,worker
 Author-email: Johannes Maron <johannes@maron.family>
-Requires-Python: >=3.12
+Requires-Python: >=3.14
 Description-Content-Type: text/markdown
 Classifier: Development Status :: 4 - Beta
 Classifier: Programming Language :: Python
@@ -20,13 +20,12 @@ Classifier: Topic :: Software Development
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Framework :: Django
 Classifier: Framework :: Django :: 6.1
 License-File: LICENSE
-Requires-Dist: django>=6.1a1
+Requires-Dist: django>=6.0
+Requires-Dist: textual>=8.2.7 ; extra == "inspector"
 Requires-Dist: redis>=5.0 ; extra == "redis"
 Project-URL: Changelog, https://github.com/codingjoe/threadmill/releases
 Project-URL: Documentation, https://github.com/codingjoe/threadmill/
@@ -35,15 +34,14 @@ Project-URL: Homepage, https://github.com/codingjoe/threadmill
 Project-URL: Issues, https://github.com/codingjoe/threadmill/issues
 Project-URL: Releasenotes, https://github.com/codingjoe/threadmill/releases/latest
 Project-URL: Source, https://github.com/codingjoe/threadmill
+Provides-Extra: inspector
 Provides-Extra: redis
 
-# Threadmill
-
 <p align="center">
   <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/codingjoe/threadmill/raw/main/images/logo-dark.svg">
-    <source media="(prefers-color-scheme: light)" srcset="https://github.com/codingjoe/threadmill/raw/main/images/logo-light.svg">
-    <img alt="Threadmill: A queue agnostic worker for Django's task framework." src="https://github.com/codingjoe/threadmill/raw/main/images/logo-light.svg">
+    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-dark.svg">
+    <source media="(prefers-color-scheme: light)" srcset="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-light.svg">
+    <img alt="Threadmill: Durable high-performance backend for Django's task framework." src="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-light.svg">
   </picture>
 <br>
   <a href="https://github.com/codingjoe/threadmill/">Documentation</a> |
@@ -52,24 +50,15 @@ Provides-Extra: redis
   <a href="https://github.com/sponsors/codingjoe">Funding</a> 💚
 </p>
 
-**A queue agnostic worker for Django's task framework.**
-
-## Design Principles
-
-- **Durability** – We recover from any failures, even poorly written tasks.
-- **Consistency** – We never lose data, even if someone unplugs the power or network.
-- **Utilization** – We keep the CPU saturated with tasks, not with idle time or waiting for locks.
-
-> [!WARNING]
-> Threadmill requires a development version of Django and is in a preview stage.
+# Threadmill [![PyPi Version](https://img.shields.io/pypi/v/threadmill.svg)](https://pypi.python.org/pypi/threadmill/) [![Test Coverage](https://codecov.io/gh/codingjoe/threadmill/branch/main/graph/badge.svg)](https://codecov.io/gh/codingjoe/threadmill) [![GitHub License](https://img.shields.io/github/license/codingjoe/threadmill)](https://raw.githubusercontent.com/codingjoe/threadmill/master/LICENSE)
 
-[![PyPi Version](https://img.shields.io/pypi/v/threadmill.svg)](https://pypi.python.org/pypi/threadmill/)
-[![Test Coverage](https://codecov.io/gh/codingjoe/threadmill/branch/main/graph/badge.svg)](https://codecov.io/gh/codingjoe/threadmill)
-[![GitHub License](https://img.shields.io/github/license/codingjoe/threadmill)](https://raw.githubusercontent.com/codingjoe/threadmill/master/LICENSE)
+**Durable high-performance backend for Django's task framework.**
 
-## Sponsors
+## Design Principles
 
-[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+- **Durability** – Recover from any failures, even poorly written tasks.
+- **Consistency** – Never lose data, even if someone unplugs the power or network.
+- **Utilization** – Keep the CPU saturated with tasks, not with idle time or waiting for locks.
 
 ## Setup
 
@@ -100,10 +89,16 @@ TASKS = {
 }
 ```
 
-Finally, you launch the worker pool:
+Optionally, install the inspector dependency if you want the TUI:
+
+```console
+uv add threadmill[inspector]
+```
+
+Then launch the worker pool:
 
 ```console
-uv run manage.py threadmill
+uv run manage.py threadmill worker
 ```
 
 ## Usage
@@ -118,7 +113,7 @@ Depending on your workload, you can tweak the number of processes and threads.
 Processes allow for parallel compute (no GIL) while threads are great for low-memory concurrent IO.
 
 ```console
-uv run manage.py threadmill --processes 4 --threads 2
+uv run manage.py threadmill worker --processes 4 --threads 2
 ```
 
 #### Health
@@ -126,7 +121,7 @@ uv run manage.py threadmill --processes 4 --threads 2
 If your tasks leak memory, you can recycle (restart) the workers after a certain number of tasks have been processed:
 
 ```console
-uv run manage.py threadmill --max-tasks 1000 --max-tasks-jitter 100
+uv run manage.py threadmill worker --max-tasks 1000 --max-tasks-jitter 100
 ```
 
 This will restart the workers after 1000 tasks have been processed, with a random jitter of up to 100 tasks to avoid all workers restarting at the same time.
@@ -141,6 +136,18 @@ All workers will finish the tasks they acquired and acknowledge them.
 You can use `--exit-empty` to exit immediately after all tasks have been processed,
 which might be useful for draining a one-off queue.
 
+### Inspector
+
+![Inspector TUI screenshot](https://github.com/codingjoe/threadmill/raw/main/docs/images/TUI-screenshot.svg)
+
+The optional TUI inspector lets you watch queues, tasks, and task details in real-time.
+Install it with the `inspector` extra and launch it from a separate terminal:
+
+```console
+uv add threadmill[inspector]
+uv run manage.py threadmill inspector
+```
+
 ### Redis Backend Options
 
 The `RedisTaskBackend` accepts the following options under `OPTIONS` in your
@@ -162,5 +169,78 @@ every multi-key operation — including the cross-queue acquire — runs on a si
 shard. Scale horizontally by running additional backend aliases, not by relying
 on cross-slot operations.
 
+### Retrying failed tasks
+
+Pass a `retry` callback to `@task()` to retry failed tasks with a delay.
+The callback receives a `TaskContext` — use `context.attempt` for the current
+attempt count and `context.task_result.errors[-1]` for the latest error.
+Return a `timedelta` to schedule the next attempt, or `None` to stop retrying.
+
+The worker re-queues the failed task, preserving its ID and error history;
+the broker promotes it back to the ready queue once the delay elapses.
+
+#### Built-in `ExponentialBackoff`
+
+`threadmill.retry.ExponentialBackoff` provides a serializable exponential
+backoff strategy out of the box. It caps the delay at `max_delay`, stops
+after `max_retries` attempts, and only retries exceptions listed in
+`expected_exceptions`.
+
+```python
+import datetime
+
+from django.tasks import task
+from requests import HTTPError
+
+from threadmill.retry import ExponentialBackoff
+
+
+@task(
+    retry=ExponentialBackoff(
+        base_delay=datetime.timedelta(seconds=1),
+        max_delay=datetime.timedelta(minutes=5),
+        factor=2.0,
+        max_retries=5,
+        expected_exceptions=(HTTPError,),
+    )
+)
+def fetch_github_api(url: str): ...
+```
+
+#### Custom retry callbacks
+
+For cases that need logic beyond what `ExponentialBackoff` supports,
+write a callable that accepts a `TaskContext` and returns a `timedelta`
+or `None`. Use `TaskError.exception_class` to filter by exception type:
+
+```python
+import datetime
+
+from django.tasks import task
+from django.tasks.base import TaskContext
+from requests import HTTPError
+
+
+def retry_on_rate_limit(context: TaskContext) -> datetime.timedelta | None:
+    """Retry HTTP 429 responses with exponential backoff, up to 5 attempts."""
+    if context.attempt >= 5:
+        return None
+    error = context.task_result.errors[-1]
+    if not issubclass(error.exception_class, HTTPError):
+        return None
+    return min(
+        datetime.timedelta(seconds=2**context.attempt),
+        datetime.timedelta(seconds=60),
+    )
+
+
+@task(retry=retry_on_rate_limit)
+def fetch_github_api(url: str): ...
+```
+
+## Sponsors
+
+[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+
 [django-tasks]: https://docs.djangoproject.com/en/stable/topics/tasks/
 

threadmill-0.4.0/README.md

@@ -0,0 +1,206 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-dark.svg">
+    <source media="(prefers-color-scheme: light)" srcset="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-light.svg">
+    <img alt="Threadmill: Durable high-performance backend for Django's task framework." src="https://github.com/codingjoe/threadmill/raw/main/docs/images/logo-light.svg">
+  </picture>
+<br>
+  <a href="https://github.com/codingjoe/threadmill/">Documentation</a> |
+  <a href="https://github.com/codingjoe/threadmill/issues/new/choose">Issues</a> |
+  <a href="https://github.com/codingjoe/threadmill/releases">Changelog</a> |
+  <a href="https://github.com/sponsors/codingjoe">Funding</a> 💚
+</p>
+
+# Threadmill [![PyPi Version](https://img.shields.io/pypi/v/threadmill.svg)](https://pypi.python.org/pypi/threadmill/) [![Test Coverage](https://codecov.io/gh/codingjoe/threadmill/branch/main/graph/badge.svg)](https://codecov.io/gh/codingjoe/threadmill) [![GitHub License](https://img.shields.io/github/license/codingjoe/threadmill)](https://raw.githubusercontent.com/codingjoe/threadmill/master/LICENSE)
+
+**Durable high-performance backend for Django's task framework.**
+
+## Design Principles
+
+- **Durability** – Recover from any failures, even poorly written tasks.
+- **Consistency** – Never lose data, even if someone unplugs the power or network.
+- **Utilization** – Keep the CPU saturated with tasks, not with idle time or waiting for locks.
+
+## Setup
+
+You need to have [Django's Task framework][django-tasks] set up properly.
+
+```console
+uv add threadmill[redis]
+```
+
+Add `threadmill` to your `INSTALLED_APPS` in `settings.py`
+and configure the task backend:
+
+```python
+# settings.py
+import os
+
+INSTALLED_APPS = [
+    "threadmill",
+    # ...
+]
+
+TASKS = {
+    "default": {
+        "BACKEND": "threadmill.backends.redis.RedisTaskBackend",
+        "REDIS_URL": os.getenv("REDIS_URL", "redis://localhost:6379/0"),
+    },
+    # ...
+}
+```
+
+Optionally, install the inspector dependency if you want the TUI:
+
+```console
+uv add threadmill[inspector]
+```
+
+Then launch the worker pool:
+
+```console
+uv run manage.py threadmill worker
+```
+
+## Usage
+
+### Workers
+
+The workers are inspired by Gunicorn, and the CLI is very similar.
+
+#### Utilization
+
+Depending on your workload, you can tweak the number of processes and threads.
+Processes allow for parallel compute (no GIL) while threads are great for low-memory concurrent IO.
+
+```console
+uv run manage.py threadmill worker --processes 4 --threads 2
+```
+
+#### Health
+
+If your tasks leak memory, you can recycle (restart) the workers after a certain number of tasks have been processed:
+
+```console
+uv run manage.py threadmill worker --max-tasks 1000 --max-tasks-jitter 100
+```
+
+This will restart the workers after 1000 tasks have been processed, with a random jitter of up to 100 tasks to avoid all workers restarting at the same time.
+
+Should a worker crash or be killed, the pool will automatically restart it.
+
+#### Shutdown
+
+A graceful shutdown is possible with the `SIGTERM` or a keyboard interrupt.
+All workers will finish the tasks they acquired and acknowledge them.
+
+You can use `--exit-empty` to exit immediately after all tasks have been processed,
+which might be useful for draining a one-off queue.
+
+### Inspector
+
+![Inspector TUI screenshot](https://github.com/codingjoe/threadmill/raw/main/docs/images/TUI-screenshot.svg)
+
+The optional TUI inspector lets you watch queues, tasks, and task details in real-time.
+Install it with the `inspector` extra and launch it from a separate terminal:
+
+```console
+uv add threadmill[inspector]
+uv run manage.py threadmill inspector
+```
+
+### Redis Backend Options
+
+The `RedisTaskBackend` accepts the following options under `OPTIONS` in your
+`TASKS` configuration:
+
+| Option            | Default                | Description                                                  |
+| ----------------- | ---------------------- | ------------------------------------------------------------ |
+| `lease_ttl`       | `timedelta(hours=1)`   | Max processing time before a started task is marked FAILED.  |
+| `result_ttl`      | `timedelta(days=1)`    | How long task results are retained before automatic removal. |
+| `broker_interval` | `timedelta(seconds=1)` | Interval between background broker maintenance passes.       |
+| `batch_size`      | `100`                  | Max tasks to move or requeue per broker pass.                |
+
+A task that is started but never acknowledged (lease expired) is marked FAILED
+with an `AcknowledgementTimeout` error. Set `lease_ttl` comfortably above your
+worst-case task runtime.
+
+All keys for one backend alias share a Redis Cluster hash tag (`{alias}`), so
+every multi-key operation — including the cross-queue acquire — runs on a single
+shard. Scale horizontally by running additional backend aliases, not by relying
+on cross-slot operations.
+
+### Retrying failed tasks
+
+Pass a `retry` callback to `@task()` to retry failed tasks with a delay.
+The callback receives a `TaskContext` — use `context.attempt` for the current
+attempt count and `context.task_result.errors[-1]` for the latest error.
+Return a `timedelta` to schedule the next attempt, or `None` to stop retrying.
+
+The worker re-queues the failed task, preserving its ID and error history;
+the broker promotes it back to the ready queue once the delay elapses.
+
+#### Built-in `ExponentialBackoff`
+
+`threadmill.retry.ExponentialBackoff` provides a serializable exponential
+backoff strategy out of the box. It caps the delay at `max_delay`, stops
+after `max_retries` attempts, and only retries exceptions listed in
+`expected_exceptions`.
+
+```python
+import datetime
+
+from django.tasks import task
+from requests import HTTPError
+
+from threadmill.retry import ExponentialBackoff
+
+
+@task(
+    retry=ExponentialBackoff(
+        base_delay=datetime.timedelta(seconds=1),
+        max_delay=datetime.timedelta(minutes=5),
+        factor=2.0,
+        max_retries=5,
+        expected_exceptions=(HTTPError,),
+    )
+)
+def fetch_github_api(url: str): ...
+```
+
+#### Custom retry callbacks
+
+For cases that need logic beyond what `ExponentialBackoff` supports,
+write a callable that accepts a `TaskContext` and returns a `timedelta`
+or `None`. Use `TaskError.exception_class` to filter by exception type:
+
+```python
+import datetime
+
+from django.tasks import task
+from django.tasks.base import TaskContext
+from requests import HTTPError
+
+
+def retry_on_rate_limit(context: TaskContext) -> datetime.timedelta | None:
+    """Retry HTTP 429 responses with exponential backoff, up to 5 attempts."""
+    if context.attempt >= 5:
+        return None
+    error = context.task_result.errors[-1]
+    if not issubclass(error.exception_class, HTTPError):
+        return None
+    return min(
+        datetime.timedelta(seconds=2**context.attempt),
+        datetime.timedelta(seconds=60),
+    )
+
+
+@task(retry=retry_on_rate_limit)
+def fetch_github_api(url: str): ...
+```
+
+## Sponsors
+
+[![Sponsors](https://django.the-box.sh/sponsors/codingjoe/threadmill.svg)](https://github.com/sponsors/codingjoe)
+
+[django-tasks]: https://docs.djangoproject.com/en/stable/topics/tasks/

{threadmill-0.2.0 → threadmill-0.4.0}/pyproject.toml

@@ -26,17 +26,18 @@ classifiers = [
   "Programming Language :: Python",
   "Programming Language :: Python :: 3",
   "Programming Language :: Python :: 3 :: Only",
-  "Programming Language :: Python :: 3.12",
-  "Programming Language :: Python :: 3.13",
   "Programming Language :: Python :: 3.14",
   "Framework :: Django",
   "Framework :: Django :: 6.1",
 ]
-requires-python = ">=3.12"
-dependencies = ["django>=6.1a1"]
+requires-python = ">=3.14"
+dependencies = ["django>=6.0"]
 
 [project.optional-dependencies]
 redis = ["redis>=5.0"]
+inspector = [
+    "textual>=8.2.7",
+]
 
 [project.urls]
 # https://packaging.python.org/en/latest/specifications/well-known-project-urls/#well-known-labels
@@ -55,7 +56,7 @@ name = "threadmill"
 write_to = "threadmill/_version.py"
 
 [tool.pytest.ini_options]
-minversion = "6.0"
+minversion = "6.1a1"
 addopts = "--cov --cov-report=xml --cov-report=term --tb=short -rxs --benchmark-autosave --benchmark-group-by=fullname --benchmark-min-rounds=10"
 testpaths = ["tests"]
 DJANGO_SETTINGS_MODULE = "tests.testapp.settings"

{threadmill-0.2.0 → threadmill-0.4.0}/threadmill/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0'
-__version_tuple__ = version_tuple = (0, 2, 0)
+__version__ = version = '0.4.0'
+__version_tuple__ = version_tuple = (0, 4, 0)
 
-__commit_id__ = commit_id = 'gb135558ff'
+__commit_id__ = commit_id = 'gf9749f870'