dj-queue 0.3.0__tar.gz → 0.5.0__tar.gz

{dj_queue-0.3.0 → dj_queue-0.5.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: dj-queue
-Version: 0.3.0
-Summary: Database-backed task queue backend for Django's django.tasks framework
+Version: 0.5.0
+Summary: Database-backed task queue backend for Django’s Tasks framework.
 License-Expression: MIT
 License-File: LICENSE
 Classifier: Development Status :: 4 - Beta
@@ -17,11 +17,13 @@ Requires-Dist: croniter>=6.2.2
 Requires-Dist: django>=6.0.0
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: psycopg>=3.3.3 ; extra == 'postgres'
+Requires-Dist: prometheus-client>=0.4.0 ; extra == 'prometheus'
 Requires-Python: >=3.12
 Project-URL: Homepage, https://github.com/coriocactus/dj_queue
 Project-URL: Repository, https://github.com/coriocactus/dj_queue
 Project-URL: Issues, https://github.com/coriocactus/dj_queue/issues
 Provides-Extra: postgres
+Provides-Extra: prometheus
 Description-Content-Type: text/markdown
 
 # dj_queue
@@ -43,7 +45,7 @@ It keeps the queue, live execution state, runtime metadata, and task results in
 - immediate, scheduled, recurring, and concurrency-limited work
 
 `dj_queue` is inspired by Rails' [Solid Queue](https://github.com/rails/solid_queue),
-but shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
+shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
 
 ## Why dj_queue
 
@@ -71,20 +73,13 @@ Install the package:
 pip install dj-queue
 ```
 
-Backend-specific extras are available when you want `dj_queue` to install a
-database adapter for you:
+Optional extras:
 
 ```bash
-pip install "dj-queue[postgres]"
+pip install "dj-queue[postgres]"    # psycopg for PostgreSQL + LISTEN/NOTIFY
+pip install "dj-queue[prometheus]"  # prometheus_client for /dj_queue/metrics
 ```
 
-Notes:
-
-- `postgres` installs `psycopg`, which Django's PostgreSQL backend and
-  `dj_queue`'s optional `LISTEN/NOTIFY` wakeups use
-- for MySQL or MariaDB, install and configure a Django-compatible driver in
-  your application following Django's database docs
-
 Add `dj_queue` to `INSTALLED_APPS`, register the router, and point Django's task
 backend at `DjQueueBackend`:
 
@@ -164,9 +159,9 @@ If Django admin is installed, `dj_queue` adds an operator dashboard at
 - queue, process, recurring-task, and semaphore overview
 - backend-aware dashboard and raw changelists
 - queue controls: pause, resume, clear ready
-- job detail action: enqueue a fresh copy of any stored job
-- pause detail action: resume the paused queue from the raw pause row
-- failed-job actions: retry and discard from list and detail views
+- job actions: enqueue a fresh copy of any stored job
+- failed jobs: retry and discard from list and detail views
+- unschedule dynamic recurring tasks
 - queue drill-down pages for state-specific inspection
 
 **Dashboard overview**
@@ -217,8 +212,9 @@ results = process_item.get_backend().enqueue_all(
 
 ### Enqueue after commit
 
-`enqueue()` writes immediately. If a task depends on rows that are still inside
-the current transaction, use `enqueue_on_commit()`:
+`enqueue()` writes immediately and returns a real persisted task result ID. If a
+task depends on rows that are still inside the current transaction, use
+`enqueue_on_commit()`:
 
 ```python
 from django.db import transaction
@@ -230,6 +226,10 @@ with transaction.atomic():
   enqueue_on_commit(send_receipt, order.id)
 ```
 
+`dj_queue` does not defer inserts implicitly or return placeholder result IDs for
+uncommitted work. Use the helper above or `transaction.on_commit()` directly
+when the job must not exist before commit.
+
 ### Examples
 
 The repository ships real runnable examples in `examples/`.
@@ -274,14 +274,18 @@ python manage.py dj_queue --skip-recurring
 Mode and topology notes:
 
 - `fork` is the default standalone mode
-- `async` runs supervised actors in threads inside one process
+- `async` is also supported as a standalone mode and runs supervised actors in threads inside one process
 - `--only-work` starts workers without dispatchers or scheduler
 - `--only-dispatch` starts dispatchers without workers or scheduler
 - `--skip-recurring` starts without the scheduler
 
 `fork` runs each worker, dispatcher, and scheduler as a separate OS process.
 `async` runs them as threads in one process, i.e., lower memory, less isolation.
-Default is `fork`. Use `async` for embedded mode or memory-constrained environments.
+Default is `fork`. Use standalone `async` when you want one-process supervision
+with lower memory use and less isolation, or embedded `async` when `dj_queue`
+should live inside an ASGI or Gunicorn server process.
+
+In `async` mode, worker `processes > 1` is ignored and normalized to `1`.
 
 ### Claiming order
 
@@ -294,6 +298,39 @@ For example, a worker configured with `queues: ["email", "default"]` will
 prefer ready work from `email` before `default`, even if `default` contains
 higher-priority rows.
 
+If you combine queue order with priorities, queue selector order still wins
+across queues. Prefer one primary scheduling mechanism per worker when you can.
+
+### Signals and recovery
+
+In standalone mode, both `fork` and `async` `python manage.py dj_queue`
+supervisors own runtime signal handling:
+
+- `SIGTERM` and `SIGINT` request graceful shutdown
+- `SIGQUIT` takes the immediate hard-exit path
+- `shutdown_timeout` controls how long the runtime waits for in-flight work to drain
+- `supervisor_pidfile` can prevent duplicate standalone supervisors on one host
+
+Runners heartbeat into the queue database. If claimed work is left behind,
+`dj_queue` preserves it as failed work that operators can inspect and retry:
+
+- `ProcessExitError`: a supervised runner exited unexpectedly
+- `ProcessPrunedError`: a runner heartbeat expired and the process was pruned
+- `ProcessMissingError`: claimed work was found without its registered process
+
+Use `python manage.py dj_queue_health` to check whether any fresh runtime
+process rows exist for a backend.
+
+### Data Contract
+
+Job payloads and persisted return values are stored in JSON columns, so they must be JSON round-trippable.
+
+- enqueueing args or kwargs that cannot round-trip through JSON fails immediately
+- returning a non-JSON-serializable value marks the job failed instead of
+  leaving it claimed forever
+
+If you need to pass model instances, files, or custom objects, store them elsewhere and pass identifiers or serialized data instead.
+
 ## Database Support
 
 | Backend | Support level | Notes |
@@ -303,20 +340,11 @@ higher-priority rows.
 | MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
 | SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
 
-Polling is the portability path everywhere. Backend-specific features improve
-latency and throughput but are not correctness requirements.
-
-## Data Contract
-
-Job payloads and persisted return values are stored in JSON columns, so they
-must be JSON round-trippable.
+For MySQL or MariaDB, install and configure a Django-compatible driver following Django's database docs.
 
-- enqueueing args or kwargs that cannot round-trip through JSON fails immediately
-- returning a non-JSON-serializable value marks the job failed instead of
-  leaving it claimed forever
+Polling is the portability path everywhere. Backend-specific features improve latency and throughput but are not correctness requirements.
 
-If you need to pass model instances, files, or custom objects, store them
-elsewhere and pass identifiers or serialized data instead.
+For production PostgreSQL operational guidance, see [Postgres Queue Health](#postgres-queue-health).
 
 ## Recurring Tasks
 
@@ -373,6 +401,17 @@ config.
 The scheduler is part of the normal `dj_queue` runtime. You do not run a
 separate recurring service.
 
+Notes:
+
+- schedules are cron expressions
+- recurring task keys are scoped per backend alias
+- only dynamic tasks can be unscheduled at runtime; unscheduling a static task returns `0`
+- Django admin exposes the same unschedule operation on recurring-task list and detail views
+- multiple schedulers sharing the same recurring config dedupe firing in the database
+- finished-job cleanup runs as internal scheduler maintenance when `preserve_finished_jobs=True` and `clear_finished_jobs_after` is set
+- failed-job cleanup can run as internal scheduler maintenance when `clear_failed_jobs_after` is set
+- recurring execution reservation cleanup can run as internal scheduler maintenance when `clear_recurring_executions_after` is set
+
 ## Concurrency Controls
 
 Tasks can opt into database-backed concurrency limits.
@@ -399,6 +438,10 @@ With this configuration:
 - later jobs for the same key can block until capacity is released
 - `on_conflict = "discard"` turns the same pattern into singleton-style work
 
+Semaphore rows remain shared on the queue database. If you want per-backend
+isolation for a limit, express that in the `concurrency_key` itself rather than
+expecting one semaphore namespace per backend alias.
+
 ## Queue Operations
 
 `QueueInfo` exposes operational queue controls without bypassing the queue
@@ -418,15 +461,32 @@ orders.resume()
 orders.clear()
 ```
 
+Queue control notes:
+
+- pausing a queue stops future claims, not enqueueing or already-claimed work
+- pause rows are scoped per backend alias
+- `clear()` discards ready jobs only
+- pass `backend_alias=` when you want to target a non-default `TASKS` alias
+
 Operational commands:
 
 ```bash
 python manage.py dj_queue_health
 python manage.py dj_queue_health --max-age 120
 python manage.py dj_queue_prune --older-than 86400
+python manage.py dj_queue_prune --failed-older-than 604800
+python manage.py dj_queue_prune --recurring-older-than 2592000
 python manage.py dj_queue_prune --task-path myapp.tasks.cleanup
+python manage.py dj_queue_prune --task-key nightly_cleanup
 ```
 
+The runtime, health, and prune commands all accept `--backend` to target a
+non-default backend alias.
+
+For `dj_queue_prune`, `--task-path` filters finished and failed job cleanup by
+task import path, while `--task-key` filters recurring execution cleanup by
+recurring task key.
+
 ## Failed Jobs
 
 When a task raises, `dj_queue` keeps the job and its failed execution row in the
@@ -443,8 +503,52 @@ retry_failed_job(job_id)
 discard_failed_job(job_id)
 ```
 
+Model helpers are available too:
+
+```python
+from dj_queue.exceptions import UndiscardableError
+from dj_queue.models import ClaimedExecution, FailedExecution
+
+failed = FailedExecution.objects.get(job_id=job_id)
+failed.retry()
+failed.discard()
+
+FailedExecution.retry_all(FailedExecution.objects.order_by("job_id"))
+FailedExecution.discard_all_in_batches()
+
+try:
+  ClaimedExecution.discard_all_in_batches()
+except UndiscardableError:
+  pass
+```
+
 Failures stay inspectable until you act on them.
 
+## Errors When Enqueuing
+
+`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
+backend-side validation failures instead of silently dropping work.
+
+Common reasons include:
+
+- args or kwargs are not JSON round-trippable
+- `concurrency_key` is set without `concurrency_limit`
+- `concurrency_key` cannot be resolved from the enqueue arguments
+- `concurrency_key` does not resolve to a non-empty string up to 255 chars
+- `on_conflict` is not `"block"` or `"discard"`
+
+```python
+from dj_queue.exceptions import EnqueueError
+
+try:
+  sync_account.enqueue(account_id, "refresh")
+except EnqueueError as exc:
+  handle_enqueue_error(exc)
+```
+
+Task execution errors are different: they become failed jobs and stay
+inspectable in the queue database.
+
 ## Multi-Database Setup
 
 `dj_queue` can keep queue tables on a dedicated database alias.
@@ -487,6 +591,28 @@ python manage.py migrate dj_queue --database queue
 With this setup, `dj_queue`'s ORM queries and raw SQL helpers stay on the queue
 database.
 
+## Postgres Queue Health
+
+Operational and configuration guidance for scaling with `dj_queue` in
+production PostgreSQL deployments, covering dedicated database setup, retention
+policy, and autovacuum tuning.
+
+- Use a dedicated queue database via `database_alias`. Keep reporting and
+  long-running transactions off the queue database.
+- Keep retention short. Set `preserve_finished_jobs = False` if you do not need
+  successful results. Otherwise use bounded `clear_finished_jobs_after`,
+  `clear_failed_jobs_after`, and `clear_recurring_executions_after` values.
+- Run `python manage.py dj_queue_prune` regularly for stricter cleanup.
+- Keep `use_skip_locked = True` and `listen_notify = True` unless you have a
+  specific reason not to.
+- Tune autovacuum for `dj_queue_jobs` and the high-churn
+  `dj_queue_*_executions` tables, often default OLTP settings are too
+  conservative for queue workloads.
+- Keep transactions short across workers and the rest of your app. Long-lived
+  transactions pin dead tuples and delay vacuum.
+- Monitor dead tuples, autovacuum frequency, and long-running queries before
+  reaching for partitioning or bulk-ingest paths.
+
 ## Embedded Server Mode
 
 `dj_queue` can run inside an existing server process via embedded async
@@ -557,12 +683,19 @@ Start with these options:
 - `dispatchers`: scheduled promotion and concurrency maintenance settings
 - `scheduler`: dynamic recurring polling settings
 - `database_alias`: database alias for queue tables and runtime activity
-- `preserve_finished_jobs` and `clear_finished_jobs_after`: result retention and cleanup
+- `preserve_finished_jobs` and `clear_finished_jobs_after`: successful result retention and cleanup
+- `clear_failed_jobs_after`: optional failed-job retention window
+- `clear_recurring_executions_after`: optional recurring reservation retention window
 
-Additional operational tuning is available when needed, including
-`use_skip_locked`, `listen_notify`, `silence_polling`,
-`process_heartbeat_interval`, `process_alive_threshold`, `shutdown_timeout`, and
-`on_thread_error`.
+Additional operational tuning is available when needed:
+
+- `use_skip_locked`: use `SKIP LOCKED` when the active backend supports it
+- `listen_notify`: PostgreSQL-only worker wakeup optimization layered on top of polling
+- `silence_polling`: suppress `dj_queue`'s own poll-cycle noise without mutating Django's global SQL logger
+- `process_heartbeat_interval` and `process_alive_threshold`: process liveness reporting and stale-runner detection
+- `shutdown_timeout`: graceful drain window before standalone shutdown gives up on waiting
+- `supervisor_pidfile`: optional pidfile guard for standalone supervisors
+- `on_thread_error`: dotted callback path for runtime infrastructure exceptions
 
 On PostgreSQL, `listen_notify` uses the same Django PostgreSQL driver
 configuration as the main database connection. Install a compatible driver in
@@ -587,15 +720,21 @@ python manage.py dj_queue --config /etc/dj_queue.yml
 DJ_QUEUE_CONFIG=/etc/dj_queue.yml python manage.py dj_queue
 ```
 
-The YAML file should contain a single mapping of backend option values. It uses
-the same shape as `TASKS[backend_alias]["OPTIONS"]`, not the full Django
-`TASKS` structure:
+The YAML file is an overlay on `TASKS[backend_alias]["OPTIONS"]`. It supports
+two shapes:
+
+- a flat mapping of option values for the selected backend alias
+- a `backends` mapping keyed by backend alias, where only the selected alias is applied
+
+Flat mapping example:
 
 ```yaml
 mode: async
 database_alias: queue
 preserve_finished_jobs: true
 clear_finished_jobs_after: 86400
+clear_failed_jobs_after: null
+clear_recurring_executions_after: null
 listen_notify: true
 silence_polling: true
 
@@ -624,8 +763,28 @@ recurring:
     description: nightly cleanup
 ```
 
-This file is merged on top of `TASKS[backend_alias]["OPTIONS"]`, then any
-environment-variable and CLI overrides win after that.
+Multi-backend overlay example:
+
+```yaml
+backends:
+  default:
+    mode: async
+    database_alias: default
+    workers:
+      - queues: ["default", "email*"]
+        threads: 8
+        processes: 1
+        polling_interval: 0.1
+
+  critical:
+    mode: fork
+    database_alias: queue
+    workers:
+      - queues: ["alerts", "critical-review"]
+        threads: 2
+        processes: 1
+        polling_interval: 0.05
+```
 
 Environment overrides currently supported by `dj_queue` itself:
 
@@ -633,6 +792,97 @@ Environment overrides currently supported by `dj_queue` itself:
 - `DJ_QUEUE_MODE`
 - `DJ_QUEUE_SKIP_RECURRING`
 
+## Lifecycle Hooks
+
+Register hooks before starting the runtime, typically during Django startup.
+Each callback receives the live supervisor or runner instance.
+
+```python
+from dj_queue.hooks import on_start, on_worker_start, register_hook
+
+@on_start
+def supervisor_started(process):
+  print(process.name)
+
+@on_worker_start
+def worker_started(process):
+  print(process.metadata)
+
+@register_hook("scheduler.exit")
+def scheduler_exited(process):
+  print(process.name)
+```
+
+Available hook helpers:
+
+- supervisor: `on_start`, `on_stop`, `on_exit`
+- worker: `on_worker_start`, `on_worker_stop`, `on_worker_exit`
+- dispatcher: `on_dispatcher_start`, `on_dispatcher_stop`, `on_dispatcher_exit`
+- scheduler: `on_scheduler_start`, `on_scheduler_stop`, `on_scheduler_exit`
+- generic events: `register_hook("worker.start")`, `register_hook("dispatcher.stop")`, and so on
+
+Hook notes:
+
+- hooks fire in registration order
+- hook failures do not block later hooks
+- hook failures are isolated and routed through `on_thread_error`
+
+### Runtime infrastructure errors
+
+Set `on_thread_error` to a dotted callable path when you want custom handling
+for queue-runtime exceptions:
+
+```python
+TASKS = {
+  "default": {
+    "BACKEND": "dj_queue.backend.DjQueueBackend",
+    "QUEUES": [],
+    "OPTIONS": {
+      "on_thread_error": "myapp.queue.report_runtime_error",
+    },
+  },
+}
+```
+
+The callback receives the raised exception object for background runtime issues
+such as hook failures, heartbeat failures, notify-watcher failures, and managed
+runner crashes. It is not used for exceptions raised by your task code; those
+become failed jobs instead.
+
+## Monitoring
+
+Queue statistics are available in JSON via `/dj_queue/stats.json` and in
+Prometheus text format via `/dj_queue/metrics`. 
+
+Include `dj_queue.urls` to expose them:
+
+```python
+urlpatterns += [path("dj_queue/", include("dj_queue.urls"))]
+```
+
+The `/dj_queue/metrics` endpoint requires the `prometheus` extra:
+
+```bash
+pip install "dj-queue[prometheus]"
+```
+
+Exported metric families:
+
+- `dj_queue_queue_jobs{backend,queue,state}`
+- `dj_queue_queue_paused{backend,queue}`
+- `dj_queue_queue_latency_seconds{backend,queue}`
+- `dj_queue_queue_live_workers{backend,queue}`
+- `dj_queue_runner_processes{backend,status}`
+- `dj_queue_runner_processes_by_kind{backend,kind,status}`
+- `dj_queue_recurring_tasks{backend}`
+- `dj_queue_semaphores{queue_database}`
+- `dj_queue_process_rows{backend}`
+
+Both endpoints support bearer token authentication. Set
+`DJ_QUEUE_OBSERVABILITY_TOKEN` in `settings.py` and include it as
+`Authorization: Bearer <token>`. Leave it unset if you protect these URLs at
+the network or proxy layer.
+
 ## License
 
 MIT