django-ox 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- django_ox-0.1.0/.gitignore +12 -0
- django_ox-0.1.0/CHANGELOG.md +85 -0
- django_ox-0.1.0/LICENSE +29 -0
- django_ox-0.1.0/PKG-INFO +253 -0
- django_ox-0.1.0/README.md +225 -0
- django_ox-0.1.0/pyproject.toml +119 -0
- django_ox-0.1.0/src/django_ox/__init__.py +1 -0
- django_ox-0.1.0/src/django_ox/apps.py +7 -0
- django_ox-0.1.0/src/django_ox/backend.py +104 -0
- django_ox-0.1.0/src/django_ox/cron.py +272 -0
- django_ox-0.1.0/src/django_ox/exceptions.py +6 -0
- django_ox-0.1.0/src/django_ox/management/__init__.py +0 -0
- django_ox-0.1.0/src/django_ox/management/commands/__init__.py +0 -0
- django_ox-0.1.0/src/django_ox/management/commands/ox_health.py +107 -0
- django_ox-0.1.0/src/django_ox/management/commands/ox_prune.py +120 -0
- django_ox-0.1.0/src/django_ox/management/commands/ox_worker.py +93 -0
- django_ox-0.1.0/src/django_ox/migrations/0001_initial.py +69 -0
- django_ox-0.1.0/src/django_ox/migrations/0002_oxscheduletick.py +47 -0
- django_ox-0.1.0/src/django_ox/migrations/__init__.py +0 -0
- django_ox-0.1.0/src/django_ox/models.py +96 -0
- django_ox-0.1.0/src/django_ox/py.typed +0 -0
- django_ox-0.1.0/src/django_ox/results.py +81 -0
- django_ox-0.1.0/src/django_ox/schedules.py +152 -0
- django_ox-0.1.0/src/django_ox/stats.py +156 -0
- django_ox-0.1.0/src/django_ox/worker.py +674 -0
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [0.1.0] - 2026-08-13
|
|
9
|
+
|
|
10
|
+
Initial release.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `OxBackend`, a database-backed backend for Django's Tasks framework
|
|
15
|
+
(`django.tasks`, Django 6.0+). Tasks are stored in the application database;
|
|
16
|
+
no broker required.
|
|
17
|
+
- Transactional enqueue: `enqueue()` is a single INSERT on the caller's
|
|
18
|
+
connection, so a task enqueued inside `transaction.atomic()` commits or
|
|
19
|
+
rolls back with the business data.
|
|
20
|
+
- `ox_worker` management command: claims tasks with
|
|
21
|
+
`SELECT ... FOR UPDATE SKIP LOCKED` where supported (PostgreSQL, MySQL 8+)
|
|
22
|
+
and an atomic compare-and-set UPDATE elsewhere (including SQLite).
|
|
23
|
+
Configurable via `--backend`, `--queues`, `--concurrency` (thread pool),
|
|
24
|
+
`--interval`, and `--lock-timeout`.
|
|
25
|
+
- Retries with exponential backoff (`MAX_ATTEMPTS`, `BACKOFF_INITIAL`,
|
|
26
|
+
`BACKOFF_MAX`), keeping the full traceback of every attempt.
|
|
27
|
+
- Reaper: tasks whose worker died are returned to the queue after
|
|
28
|
+
`LOCK_TIMEOUT` and count as a failed attempt.
|
|
29
|
+
- Graceful drain: on SIGTERM/SIGINT the worker stops claiming, finishes
|
|
30
|
+
in-flight tasks, then exits; a second signal forces an immediate exit.
|
|
31
|
+
- Priorities (-100 to 100, higher first) and deferred tasks (`run_after`),
|
|
32
|
+
with the corresponding `supports_*` flags declared on the backend.
|
|
33
|
+
- Result store: `get_result()`, `refresh()`, and the async variants, with
|
|
34
|
+
status, return value, and per-attempt errors readable from the database.
|
|
35
|
+
- `ox_prune` management command: batched deletion of finished task rows
|
|
36
|
+
(`--older-than`, `--include-failed`, `--batch-size`, `--dry-run`).
|
|
37
|
+
- `django_ox.stats`: read-only queue metrics as plain ORM queries, on
|
|
38
|
+
both supported databases: per-queue status counts, backlog depth and
|
|
39
|
+
age, throughput and failure rate over a trailing window, and time
|
|
40
|
+
since the last task claim.
|
|
41
|
+
- `ox_health` management command: exits non-zero with a one-line reason
|
|
42
|
+
when the database is unreachable or a `--max-backlog`, `--max-age` or
|
|
43
|
+
`--worker-timeout` threshold is breached; built for cron alerting and
|
|
44
|
+
container probes.
|
|
45
|
+
- Structured logging: worker lifecycle events (claim, start, success,
|
|
46
|
+
retry, failure, reclaim, dispatch, shutdown) log to the `django_ox`
|
|
47
|
+
logger with stable extra keys (`event`, `task_id`, `queue`, `attempt`,
|
|
48
|
+
`duration_ms`, ...) for JSON log handlers.
|
|
49
|
+
- Recurring tasks: cron schedules declared in the `TASKS` setting
|
|
50
|
+
(`SCHEDULES` option), dispatched by the workers themselves; a unique
|
|
51
|
+
constraint on (schedule, tick) makes each tick fire exactly once across
|
|
52
|
+
any number of workers. Five-field cron syntax plus `@hourly`-style
|
|
53
|
+
shortcuts; misconfigured schedules fail at startup and in
|
|
54
|
+
`manage.py check`. On recovery after downtime, only the latest missed
|
|
55
|
+
tick fires.
|
|
56
|
+
- System check `django_ox.E003`: a schedule name defined on more than
|
|
57
|
+
one backend is rejected, at worker startup and in `manage.py check`,
|
|
58
|
+
because the tick log is keyed by schedule name alone and shared names
|
|
59
|
+
would let the backends suppress each other's ticks.
|
|
60
|
+
|
|
61
|
+
### Fixed
|
|
62
|
+
|
|
63
|
+
Findings from a pre-release adversarial review:
|
|
64
|
+
|
|
65
|
+
- A tick row dated in the future (for example written by a worker with a
|
|
66
|
+
fast clock) no longer suppresses schedule dispatch fleet-wide; ticks
|
|
67
|
+
that are due now fire regardless, and the unique constraint still
|
|
68
|
+
protects the future instant itself.
|
|
69
|
+
- Cron step values larger than the field's range (such as `*/61` in the
|
|
70
|
+
minute field, which silently collapsed to minute 0) are rejected at
|
|
71
|
+
parse time.
|
|
72
|
+
|
|
73
|
+
### Security
|
|
74
|
+
|
|
75
|
+
- A stored `task_path` is now required to resolve to a `django.tasks` Task
|
|
76
|
+
(a function registered with `@task`). A row naming any other importable
|
|
77
|
+
callable is rejected as an un-runnable task instead of being executed, so
|
|
78
|
+
the worker never invokes an arbitrary dotted path pulled from the table.
|
|
79
|
+
`SECURITY.md` documents the full trust model, the JSON-only
|
|
80
|
+
serialization, and the guidance to keep secrets out of task arguments.
|
|
81
|
+
- Added an API stability and deprecation policy (`docs/stability.md`):
|
|
82
|
+
the public API surface, the pre-1.0 SemVer rule, the deprecation window,
|
|
83
|
+
and the supported Python and Django matrix.
|
|
84
|
+
|
|
85
|
+
[0.1.0]: https://github.com/oxpull/django-ox/releases/tag/v0.1.0
|
django_ox-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, [COMPANY]
|
|
4
|
+
All rights reserved.
|
|
5
|
+
|
|
6
|
+
Redistribution and use in source and binary forms, with or without
|
|
7
|
+
modification, are permitted provided that the following conditions are met:
|
|
8
|
+
|
|
9
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
10
|
+
list of conditions and the following disclaimer.
|
|
11
|
+
|
|
12
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
13
|
+
this list of conditions and the following disclaimer in the documentation
|
|
14
|
+
and/or other materials provided with the distribution.
|
|
15
|
+
|
|
16
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
17
|
+
contributors may be used to endorse or promote products derived from
|
|
18
|
+
this software without specific prior written permission.
|
|
19
|
+
|
|
20
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
21
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
22
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
23
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
24
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
25
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
26
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
27
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
28
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
29
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
django_ox-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,253 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: django-ox
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Database-backed worker backend for Django's Tasks framework.
|
|
5
|
+
Project-URL: Homepage, https://github.com/oxpull/django-ox
|
|
6
|
+
Project-URL: Repository, https://github.com/oxpull/django-ox
|
|
7
|
+
Project-URL: Changelog, https://github.com/oxpull/django-ox/blob/main/CHANGELOG.md
|
|
8
|
+
Project-URL: Issues, https://github.com/oxpull/django-ox/issues
|
|
9
|
+
Author: Oxpull
|
|
10
|
+
License-Expression: BSD-3-Clause
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: background-tasks,database,django,queue,task-queue,tasks,worker
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Environment :: Web Environment
|
|
15
|
+
Classifier: Framework :: Django
|
|
16
|
+
Classifier: Framework :: Django :: 6.0
|
|
17
|
+
Classifier: Framework :: Django :: 6.1
|
|
18
|
+
Classifier: Intended Audience :: Developers
|
|
19
|
+
Classifier: Operating System :: OS Independent
|
|
20
|
+
Classifier: Programming Language :: Python :: 3
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
24
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
25
|
+
Requires-Python: >=3.12
|
|
26
|
+
Requires-Dist: django>=6.0
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
|
|
29
|
+
# django-ox
|
|
30
|
+
|
|
31
|
+
A database-backed worker backend for Django's Tasks framework (`django.tasks`, Django 6.0+).
|
|
32
|
+
|
|
33
|
+
Django 6.0 ships the Tasks API but no production backend: the built-in
|
|
34
|
+
`ImmediateBackend` and `DummyBackend` are for development and testing only.
|
|
35
|
+
django-ox stores tasks in your existing database and runs them with a
|
|
36
|
+
separate worker process, so you get a durable queue without adding a broker.
|
|
37
|
+
|
|
38
|
+
## Durability model
|
|
39
|
+
|
|
40
|
+
`enqueue()` is a single INSERT on your default database connection, so it
|
|
41
|
+
participates in the caller's open transaction. A task enqueued inside
|
|
42
|
+
`transaction.atomic()` becomes visible to workers only when the transaction
|
|
43
|
+
commits, and disappears on rollback. There is no window where business data
|
|
44
|
+
exists without its task, or a task without its data, and no
|
|
45
|
+
`transaction.on_commit()` boilerplate. Execution is at-least-once: workers
|
|
46
|
+
claim tasks with `SELECT ... FOR UPDATE SKIP LOCKED` on databases that support
|
|
47
|
+
it (PostgreSQL, MySQL 8+) and an atomic compare-and-set UPDATE elsewhere
|
|
48
|
+
(including SQLite), and a reaper returns tasks whose worker died to the queue.
|
|
49
|
+
Failed tasks retry with exponential backoff up to a configurable attempt
|
|
50
|
+
limit, keeping the full traceback of every attempt.
|
|
51
|
+
|
|
52
|
+
## Install
|
|
53
|
+
|
|
54
|
+
Requires Python 3.12+ and Django 6.0+.
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
pip install django-ox
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
```python
|
|
61
|
+
INSTALLED_APPS = [
|
|
62
|
+
# ...
|
|
63
|
+
"django_ox",
|
|
64
|
+
]
|
|
65
|
+
|
|
66
|
+
TASKS = {
|
|
67
|
+
"default": {
|
|
68
|
+
"BACKEND": "django_ox.backend.OxBackend",
|
|
69
|
+
"QUEUES": ["default", "emails"], # [] allows any queue name
|
|
70
|
+
"OPTIONS": {
|
|
71
|
+
"MAX_ATTEMPTS": 3, # executions per task before FAILED
|
|
72
|
+
"LOCK_TIMEOUT": 300, # seconds before a dead worker's task is reclaimed
|
|
73
|
+
"BACKOFF_INITIAL": 5, # first retry delay, seconds; doubles per attempt
|
|
74
|
+
"BACKOFF_MAX": 600, # retry delay ceiling, seconds
|
|
75
|
+
},
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Then run migrations:
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
python manage.py migrate django_ox
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Quickstart
|
|
87
|
+
|
|
88
|
+
Tasks are plain `django.tasks` tasks; django-ox adds nothing to learn on the
|
|
89
|
+
producer side.
|
|
90
|
+
|
|
91
|
+
```python
|
|
92
|
+
from django.tasks import task
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
@task
|
|
96
|
+
def send_welcome_email(user_id): ...
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
result = send_welcome_email.enqueue(user_id=42)
|
|
100
|
+
result.refresh() # later: status, return_value, errors
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Run a worker:
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
python manage.py ox_worker
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
## Worker CLI
|
|
110
|
+
|
|
111
|
+
| Flag | Default | Meaning |
|
|
112
|
+
| --- | --- | --- |
|
|
113
|
+
| `--backend` | `default` | Backend alias from the `TASKS` setting. |
|
|
114
|
+
| `--queues` | all configured queues | Comma-separated queue names to process. |
|
|
115
|
+
| `--concurrency` | `1` | Tasks executed concurrently (thread pool). |
|
|
116
|
+
| `--interval` | `1.0` | Polling interval in seconds when idle. |
|
|
117
|
+
| `--lock-timeout` | backend `LOCK_TIMEOUT` | Seconds before a stuck task is reclaimed. |
|
|
118
|
+
|
|
119
|
+
On SIGTERM or SIGINT the worker stops claiming, finishes in-flight tasks, then
|
|
120
|
+
exits. A second signal forces an immediate exit.
|
|
121
|
+
|
|
122
|
+
## Pruning
|
|
123
|
+
|
|
124
|
+
Finished task rows stay in the table until pruned. Run `ox_prune` on your
|
|
125
|
+
own schedule (cron, systemd timer):
|
|
126
|
+
|
|
127
|
+
```
|
|
128
|
+
python manage.py ox_prune --older-than 7d
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
| Flag | Default | Meaning |
|
|
132
|
+
| --- | --- | --- |
|
|
133
|
+
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
|
|
134
|
+
| `--include-failed` | off | Also delete FAILED rows. By default they are kept: they hold the per-attempt tracebacks. |
|
|
135
|
+
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. |
|
|
136
|
+
| `--dry-run` | off | Report how many rows would be deleted without deleting any. |
|
|
137
|
+
|
|
138
|
+
Only SUCCESSFUL rows (and, with `--include-failed`, FAILED rows) past the
|
|
139
|
+
cutoff are deleted. READY and RUNNING rows are never touched, whatever their
|
|
140
|
+
age. Old rows from the recurring-schedule tick log are cleared with the same
|
|
141
|
+
cutoff, always keeping each schedule's most recent tick.
|
|
142
|
+
|
|
143
|
+
## Health and monitoring
|
|
144
|
+
|
|
145
|
+
`django_ox.stats` exposes queue metrics as plain functions, each a single
|
|
146
|
+
ORM query: per-queue status counts, backlog depth and age, throughput,
|
|
147
|
+
and failure rate. The `ox_health` command turns thresholds on those
|
|
148
|
+
numbers into an exit code for cron alerting and container probes:
|
|
149
|
+
|
|
150
|
+
```
|
|
151
|
+
python manage.py ox_health --max-backlog 1000 --max-age 600
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
| Flag | Default | Meaning |
|
|
155
|
+
| --- | --- | --- |
|
|
156
|
+
| `--queue` | all queues | Restrict the checks to one queue. |
|
|
157
|
+
| `--max-backlog` | off | Fail when more than this many READY tasks are eligible to run. |
|
|
158
|
+
| `--max-age` | off | Fail when the oldest waiting task has waited longer than this many seconds. |
|
|
159
|
+
| `--worker-timeout` | off | Fail when no worker has claimed a task within this many seconds. |
|
|
160
|
+
|
|
161
|
+
Worker lifecycle events (claim, start, success, retry, failure, reclaim,
|
|
162
|
+
shutdown) log to the `django_ox` logger with stable extra keys (task id,
|
|
163
|
+
queue, attempt, duration), ready for JSON log handlers.
|
|
164
|
+
|
|
165
|
+
## Recurring tasks
|
|
166
|
+
|
|
167
|
+
Schedules are declared in settings, next to the backend they enqueue
|
|
168
|
+
through, and deploy with your code. There are no rows to edit by hand and
|
|
169
|
+
no separate scheduler process to keep alive:
|
|
170
|
+
|
|
171
|
+
```python
|
|
172
|
+
TASKS = {
|
|
173
|
+
"default": {
|
|
174
|
+
"BACKEND": "django_ox.backend.OxBackend",
|
|
175
|
+
"QUEUES": ["default", "emails"],
|
|
176
|
+
"OPTIONS": {
|
|
177
|
+
"SCHEDULES": {
|
|
178
|
+
"nightly-report": {
|
|
179
|
+
"task": "reports.tasks.build_report",
|
|
180
|
+
"cron": "0 3 * * *",
|
|
181
|
+
"kwargs": {"full": True},
|
|
182
|
+
},
|
|
183
|
+
"warm-cache": {
|
|
184
|
+
"task": "core.tasks.warm_cache",
|
|
185
|
+
"cron": "*/15 * * * *",
|
|
186
|
+
},
|
|
187
|
+
},
|
|
188
|
+
},
|
|
189
|
+
}
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Each tick enqueues a normal task instance, which workers claim and execute
|
|
194
|
+
through the ordinary queue: retries, backoff, priorities and the result
|
|
195
|
+
store all apply unchanged. Every running worker doubles as the scheduler,
|
|
196
|
+
and a unique constraint on (schedule name, tick time) makes each tick fire
|
|
197
|
+
exactly once however many workers are polling.
|
|
198
|
+
|
|
199
|
+
| Key | Required | Meaning |
|
|
200
|
+
| --- | --- | --- |
|
|
201
|
+
| `task` | yes | Dotted path to a `@task` callable, e.g. `"reports.tasks.build_report"`. |
|
|
202
|
+
| `cron` | yes | Five-field cron expression. |
|
|
203
|
+
| `args`, `kwargs` | no | JSON-serializable arguments passed to each enqueue. |
|
|
204
|
+
| `queue_name` | no | Queue override; defaults to the task's own queue. |
|
|
205
|
+
| `priority` | no | Priority override (-100 to 100). |
|
|
206
|
+
|
|
207
|
+
Cron expressions use the classic five-field syntax: `*`, lists (`1,15`),
|
|
208
|
+
ranges (`mon-fri`), steps (`*/15`), month and weekday names, 0 or 7 for
|
|
209
|
+
Sunday, and the `@hourly`, `@daily`, `@weekly`, `@monthly` and `@yearly`
|
|
210
|
+
shortcuts. When both day-of-month and day-of-week are restricted, a day
|
|
211
|
+
matches if either field does, as in vixie cron. Times are wall-clock in
|
|
212
|
+
your `TIME_ZONE`.
|
|
213
|
+
|
|
214
|
+
Misconfigured schedules (a task path that does not import, an expression
|
|
215
|
+
that can never fire) fail at worker startup and in `manage.py check`, not
|
|
216
|
+
silently at dispatch time.
|
|
217
|
+
|
|
218
|
+
Missed ticks: if every worker was down when a tick passed, the latest
|
|
219
|
+
missed tick fires once on recovery and older ones are skipped, so a
|
|
220
|
+
nightly job still runs after an unlucky deploy window but a backlog never
|
|
221
|
+
stampedes. A newly deployed schedule waits for its next tick rather than
|
|
222
|
+
firing for a time before it existed.
|
|
223
|
+
|
|
224
|
+
## Behavior details
|
|
225
|
+
|
|
226
|
+
- `run_after` (deferred tasks), `priority` (-100 to 100, higher runs first),
|
|
227
|
+
`get_result()` and the async variants are all supported; the backend
|
|
228
|
+
declares `supports_defer`, `supports_priority`, `supports_get_result` and
|
|
229
|
+
`supports_async_task` accordingly.
|
|
230
|
+
- Retry state is visible in the database: attempts, per-attempt tracebacks,
|
|
231
|
+
and the next scheduled run (`run_after`).
|
|
232
|
+
- Because execution is at-least-once, tasks should be idempotent. A task is
|
|
233
|
+
retried both when it raises and when its worker dies mid-run.
|
|
234
|
+
- Concurrency uses a thread pool. That fits I/O-bound tasks (email, HTTP,
|
|
235
|
+
ORM); for CPU-bound work, run multiple worker processes with
|
|
236
|
+
`--concurrency=1` instead.
|
|
237
|
+
|
|
238
|
+
## Not yet supported
|
|
239
|
+
|
|
240
|
+
- Task revocation or cancellation after enqueue.
|
|
241
|
+
- Multi-database routing (tasks are stored on the default database for the
|
|
242
|
+
model).
|
|
243
|
+
- Rate limiting, batching, and a dashboard.
|
|
244
|
+
|
|
245
|
+
## Stability
|
|
246
|
+
|
|
247
|
+
What counts as public API, the pre-1.0 versioning and deprecation policy,
|
|
248
|
+
and the supported Python and Django versions are documented in
|
|
249
|
+
[docs/stability.md](docs/stability.md).
|
|
250
|
+
|
|
251
|
+
## License
|
|
252
|
+
|
|
253
|
+
BSD 3-Clause.
|
|
@@ -0,0 +1,225 @@
|
|
|
1
|
+
# django-ox
|
|
2
|
+
|
|
3
|
+
A database-backed worker backend for Django's Tasks framework (`django.tasks`, Django 6.0+).
|
|
4
|
+
|
|
5
|
+
Django 6.0 ships the Tasks API but no production backend: the built-in
|
|
6
|
+
`ImmediateBackend` and `DummyBackend` are for development and testing only.
|
|
7
|
+
django-ox stores tasks in your existing database and runs them with a
|
|
8
|
+
separate worker process, so you get a durable queue without adding a broker.
|
|
9
|
+
|
|
10
|
+
## Durability model
|
|
11
|
+
|
|
12
|
+
`enqueue()` is a single INSERT on your default database connection, so it
|
|
13
|
+
participates in the caller's open transaction. A task enqueued inside
|
|
14
|
+
`transaction.atomic()` becomes visible to workers only when the transaction
|
|
15
|
+
commits, and disappears on rollback. There is no window where business data
|
|
16
|
+
exists without its task, or a task without its data, and no
|
|
17
|
+
`transaction.on_commit()` boilerplate. Execution is at-least-once: workers
|
|
18
|
+
claim tasks with `SELECT ... FOR UPDATE SKIP LOCKED` on databases that support
|
|
19
|
+
it (PostgreSQL, MySQL 8+) and an atomic compare-and-set UPDATE elsewhere
|
|
20
|
+
(including SQLite), and a reaper returns tasks whose worker died to the queue.
|
|
21
|
+
Failed tasks retry with exponential backoff up to a configurable attempt
|
|
22
|
+
limit, keeping the full traceback of every attempt.
|
|
23
|
+
|
|
24
|
+
## Install
|
|
25
|
+
|
|
26
|
+
Requires Python 3.12+ and Django 6.0+.
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
pip install django-ox
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
INSTALLED_APPS = [
|
|
34
|
+
# ...
|
|
35
|
+
"django_ox",
|
|
36
|
+
]
|
|
37
|
+
|
|
38
|
+
TASKS = {
|
|
39
|
+
"default": {
|
|
40
|
+
"BACKEND": "django_ox.backend.OxBackend",
|
|
41
|
+
"QUEUES": ["default", "emails"], # [] allows any queue name
|
|
42
|
+
"OPTIONS": {
|
|
43
|
+
"MAX_ATTEMPTS": 3, # executions per task before FAILED
|
|
44
|
+
"LOCK_TIMEOUT": 300, # seconds before a dead worker's task is reclaimed
|
|
45
|
+
"BACKOFF_INITIAL": 5, # first retry delay, seconds; doubles per attempt
|
|
46
|
+
"BACKOFF_MAX": 600, # retry delay ceiling, seconds
|
|
47
|
+
},
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Then run migrations:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
python manage.py migrate django_ox
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Quickstart
|
|
59
|
+
|
|
60
|
+
Tasks are plain `django.tasks` tasks; django-ox adds nothing to learn on the
|
|
61
|
+
producer side.
|
|
62
|
+
|
|
63
|
+
```python
|
|
64
|
+
from django.tasks import task
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
@task
|
|
68
|
+
def send_welcome_email(user_id): ...
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
result = send_welcome_email.enqueue(user_id=42)
|
|
72
|
+
result.refresh() # later: status, return_value, errors
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Run a worker:
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
python manage.py ox_worker
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Worker CLI
|
|
82
|
+
|
|
83
|
+
| Flag | Default | Meaning |
|
|
84
|
+
| --- | --- | --- |
|
|
85
|
+
| `--backend` | `default` | Backend alias from the `TASKS` setting. |
|
|
86
|
+
| `--queues` | all configured queues | Comma-separated queue names to process. |
|
|
87
|
+
| `--concurrency` | `1` | Tasks executed concurrently (thread pool). |
|
|
88
|
+
| `--interval` | `1.0` | Polling interval in seconds when idle. |
|
|
89
|
+
| `--lock-timeout` | backend `LOCK_TIMEOUT` | Seconds before a stuck task is reclaimed. |
|
|
90
|
+
|
|
91
|
+
On SIGTERM or SIGINT the worker stops claiming, finishes in-flight tasks, then
|
|
92
|
+
exits. A second signal forces an immediate exit.
|
|
93
|
+
|
|
94
|
+
## Pruning
|
|
95
|
+
|
|
96
|
+
Finished task rows stay in the table until pruned. Run `ox_prune` on your
|
|
97
|
+
own schedule (cron, systemd timer):
|
|
98
|
+
|
|
99
|
+
```
|
|
100
|
+
python manage.py ox_prune --older-than 7d
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
| Flag | Default | Meaning |
|
|
104
|
+
| --- | --- | --- |
|
|
105
|
+
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
|
|
106
|
+
| `--include-failed` | off | Also delete FAILED rows. By default they are kept: they hold the per-attempt tracebacks. |
|
|
107
|
+
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. |
|
|
108
|
+
| `--dry-run` | off | Report how many rows would be deleted without deleting any. |
|
|
109
|
+
|
|
110
|
+
Only SUCCESSFUL rows (and, with `--include-failed`, FAILED rows) past the
|
|
111
|
+
cutoff are deleted. READY and RUNNING rows are never touched, whatever their
|
|
112
|
+
age. Old rows from the recurring-schedule tick log are cleared with the same
|
|
113
|
+
cutoff, always keeping each schedule's most recent tick.
|
|
114
|
+
|
|
115
|
+
## Health and monitoring
|
|
116
|
+
|
|
117
|
+
`django_ox.stats` exposes queue metrics as plain functions, each a single
|
|
118
|
+
ORM query: per-queue status counts, backlog depth and age, throughput,
|
|
119
|
+
and failure rate. The `ox_health` command turns thresholds on those
|
|
120
|
+
numbers into an exit code for cron alerting and container probes:
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
python manage.py ox_health --max-backlog 1000 --max-age 600
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
| Flag | Default | Meaning |
|
|
127
|
+
| --- | --- | --- |
|
|
128
|
+
| `--queue` | all queues | Restrict the checks to one queue. |
|
|
129
|
+
| `--max-backlog` | off | Fail when more than this many READY tasks are eligible to run. |
|
|
130
|
+
| `--max-age` | off | Fail when the oldest waiting task has waited longer than this many seconds. |
|
|
131
|
+
| `--worker-timeout` | off | Fail when no worker has claimed a task within this many seconds. |
|
|
132
|
+
|
|
133
|
+
Worker lifecycle events (claim, start, success, retry, failure, reclaim,
|
|
134
|
+
shutdown) log to the `django_ox` logger with stable extra keys (task id,
|
|
135
|
+
queue, attempt, duration), ready for JSON log handlers.
|
|
136
|
+
|
|
137
|
+
## Recurring tasks
|
|
138
|
+
|
|
139
|
+
Schedules are declared in settings, next to the backend they enqueue
|
|
140
|
+
through, and deploy with your code. There are no rows to edit by hand and
|
|
141
|
+
no separate scheduler process to keep alive:
|
|
142
|
+
|
|
143
|
+
```python
|
|
144
|
+
TASKS = {
|
|
145
|
+
"default": {
|
|
146
|
+
"BACKEND": "django_ox.backend.OxBackend",
|
|
147
|
+
"QUEUES": ["default", "emails"],
|
|
148
|
+
"OPTIONS": {
|
|
149
|
+
"SCHEDULES": {
|
|
150
|
+
"nightly-report": {
|
|
151
|
+
"task": "reports.tasks.build_report",
|
|
152
|
+
"cron": "0 3 * * *",
|
|
153
|
+
"kwargs": {"full": True},
|
|
154
|
+
},
|
|
155
|
+
"warm-cache": {
|
|
156
|
+
"task": "core.tasks.warm_cache",
|
|
157
|
+
"cron": "*/15 * * * *",
|
|
158
|
+
},
|
|
159
|
+
},
|
|
160
|
+
},
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Each tick enqueues a normal task instance, which workers claim and execute
|
|
166
|
+
through the ordinary queue: retries, backoff, priorities and the result
|
|
167
|
+
store all apply unchanged. Every running worker doubles as the scheduler,
|
|
168
|
+
and a unique constraint on (schedule name, tick time) makes each tick fire
|
|
169
|
+
exactly once however many workers are polling.
|
|
170
|
+
|
|
171
|
+
| Key | Required | Meaning |
|
|
172
|
+
| --- | --- | --- |
|
|
173
|
+
| `task` | yes | Dotted path to a `@task` callable, e.g. `"reports.tasks.build_report"`. |
|
|
174
|
+
| `cron` | yes | Five-field cron expression. |
|
|
175
|
+
| `args`, `kwargs` | no | JSON-serializable arguments passed to each enqueue. |
|
|
176
|
+
| `queue_name` | no | Queue override; defaults to the task's own queue. |
|
|
177
|
+
| `priority` | no | Priority override (-100 to 100). |
|
|
178
|
+
|
|
179
|
+
Cron expressions use the classic five-field syntax: `*`, lists (`1,15`),
|
|
180
|
+
ranges (`mon-fri`), steps (`*/15`), month and weekday names, 0 or 7 for
|
|
181
|
+
Sunday, and the `@hourly`, `@daily`, `@weekly`, `@monthly` and `@yearly`
|
|
182
|
+
shortcuts. When both day-of-month and day-of-week are restricted, a day
|
|
183
|
+
matches if either field does, as in vixie cron. Times are wall-clock in
|
|
184
|
+
your `TIME_ZONE`.
|
|
185
|
+
|
|
186
|
+
Misconfigured schedules (a task path that does not import, an expression
|
|
187
|
+
that can never fire) fail at worker startup and in `manage.py check`, not
|
|
188
|
+
silently at dispatch time.
|
|
189
|
+
|
|
190
|
+
Missed ticks: if every worker was down when a tick passed, the latest
|
|
191
|
+
missed tick fires once on recovery and older ones are skipped, so a
|
|
192
|
+
nightly job still runs after an unlucky deploy window but a backlog never
|
|
193
|
+
stampedes. A newly deployed schedule waits for its next tick rather than
|
|
194
|
+
firing for a time before it existed.
|
|
195
|
+
|
|
196
|
+
## Behavior details
|
|
197
|
+
|
|
198
|
+
- `run_after` (deferred tasks), `priority` (-100 to 100, higher runs first),
|
|
199
|
+
`get_result()` and the async variants are all supported; the backend
|
|
200
|
+
declares `supports_defer`, `supports_priority`, `supports_get_result` and
|
|
201
|
+
`supports_async_task` accordingly.
|
|
202
|
+
- Retry state is visible in the database: attempts, per-attempt tracebacks,
|
|
203
|
+
and the next scheduled run (`run_after`).
|
|
204
|
+
- Because execution is at-least-once, tasks should be idempotent. A task is
|
|
205
|
+
retried both when it raises and when its worker dies mid-run.
|
|
206
|
+
- Concurrency uses a thread pool. That fits I/O-bound tasks (email, HTTP,
|
|
207
|
+
ORM); for CPU-bound work, run multiple worker processes with
|
|
208
|
+
`--concurrency=1` instead.
|
|
209
|
+
|
|
210
|
+
## Not yet supported
|
|
211
|
+
|
|
212
|
+
- Task revocation or cancellation after enqueue.
|
|
213
|
+
- Multi-database routing (tasks are stored on the default database for the
|
|
214
|
+
model).
|
|
215
|
+
- Rate limiting, batching, and a dashboard.
|
|
216
|
+
|
|
217
|
+
## Stability
|
|
218
|
+
|
|
219
|
+
What counts as public API, the pre-1.0 versioning and deprecation policy,
|
|
220
|
+
and the supported Python and Django versions are documented in
|
|
221
|
+
[docs/stability.md](docs/stability.md).
|
|
222
|
+
|
|
223
|
+
## License
|
|
224
|
+
|
|
225
|
+
BSD 3-Clause.
|