dj-queue 0.4.0__tar.gz → 0.5.1__tar.gz

{dj_queue-0.4.0 → dj_queue-0.5.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: dj-queue
-Version: 0.4.0
-Summary: Database-backed task queue backend for Django's django.tasks framework
+Version: 0.5.1
+Summary: Database-backed task queue backend for Django’s Tasks framework.
 License-Expression: MIT
 License-File: LICENSE
 Classifier: Development Status :: 4 - Beta
@@ -17,11 +17,13 @@ Requires-Dist: croniter>=6.2.2
 Requires-Dist: django>=6.0.0
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: psycopg>=3.3.3 ; extra == 'postgres'
+Requires-Dist: prometheus-client>=0.4.0 ; extra == 'prometheus'
 Requires-Python: >=3.12
 Project-URL: Homepage, https://github.com/coriocactus/dj_queue
 Project-URL: Repository, https://github.com/coriocactus/dj_queue
 Project-URL: Issues, https://github.com/coriocactus/dj_queue/issues
 Provides-Extra: postgres
+Provides-Extra: prometheus
 Description-Content-Type: text/markdown
 
 # dj_queue
@@ -43,7 +45,7 @@ It keeps the queue, live execution state, runtime metadata, and task results in
 - immediate, scheduled, recurring, and concurrency-limited work
 
 `dj_queue` is inspired by Rails' [Solid Queue](https://github.com/rails/solid_queue),
-but shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
+shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
 
 ## Why dj_queue
 
@@ -71,20 +73,13 @@ Install the package:
 pip install dj-queue
 ```
 
-Backend-specific extras are available when you want `dj_queue` to install a
-database adapter for you:
+Optional extras:
 
 ```bash
-pip install "dj-queue[postgres]"
+pip install "dj-queue[postgres]"    # psycopg for PostgreSQL + LISTEN/NOTIFY
+pip install "dj-queue[prometheus]"  # prometheus_client for /dj_queue/metrics
 ```
 
-Notes:
-
-- `postgres` installs `psycopg`, which Django's PostgreSQL backend and
-  `dj_queue`'s optional `LISTEN/NOTIFY` wakeups use
-- for MySQL or MariaDB, install and configure a Django-compatible driver in
-  your application following Django's database docs
-
 Add `dj_queue` to `INSTALLED_APPS`, register the router, and point Django's task
 backend at `DjQueueBackend`:
 
@@ -164,12 +159,10 @@ If Django admin is installed, `dj_queue` adds an operator dashboard at
 - queue, process, recurring-task, and semaphore overview
 - backend-aware dashboard and raw changelists
 - queue controls: pause, resume, clear ready
-- job detail action: enqueue a fresh copy of any stored job
-- recurring-task actions: unschedule from list and detail views
-- pause detail action: resume the paused queue from the raw pause row
-- failed-job actions: retry and discard from list and detail views
+- job actions: enqueue a fresh copy of any stored job
+- failed jobs: retry and discard from list and detail views
+- unschedule dynamic recurring tasks
 - queue drill-down pages for state-specific inspection
-- queue drill-down actions: discard ready, scheduled, and blocked jobs; retry or discard failed jobs; enqueue finished jobs again
 
 **Dashboard overview**
 
@@ -328,54 +321,30 @@ Runners heartbeat into the queue database. If claimed work is left behind,
 Use `python manage.py dj_queue_health` to check whether any fresh runtime
 process rows exist for a backend.
 
-## Database Support
-
-| Backend | Support level | Notes |
-|---|---|---|
-| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
-| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
-| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
-| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
-
-Polling is the portability path everywhere. Backend-specific features improve
-latency and throughput but are not correctness requirements.
-
-## Data Contract
+### Data Contract
 
-Job payloads and persisted return values are stored in JSON columns, so they
-must be JSON round-trippable.
+Job payloads and persisted return values are stored in JSON columns, so they must be JSON round-trippable.
 
 - enqueueing args or kwargs that cannot round-trip through JSON fails immediately
 - returning a non-JSON-serializable value marks the job failed instead of
   leaving it claimed forever
 
-If you need to pass model instances, files, or custom objects, store them
-elsewhere and pass identifiers or serialized data instead.
-
-## Errors When Enqueuing
-
-`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
-backend-side validation failures instead of silently dropping work.
+If you need to pass model instances, files, or custom objects, store them elsewhere and pass identifiers or serialized data instead.
 
-Common reasons include:
+## Database Support
 
-- args or kwargs are not JSON round-trippable
-- `concurrency_key` is set without `concurrency_limit`
-- `concurrency_key` cannot be resolved from the enqueue arguments
-- `concurrency_key` does not resolve to a non-empty string up to 255 chars
-- `on_conflict` is not `"block"` or `"discard"`
+| Backend | Support level | Notes |
+|---|---|---|
+| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
+| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
+| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
+| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
 
-```python
-from dj_queue.exceptions import EnqueueError
+For MySQL or MariaDB, install and configure a Django-compatible driver following Django's database docs.
 
-try:
-  sync_account.enqueue(account_id, "refresh")
-except EnqueueError as exc:
-  handle_enqueue_error(exc)
-```
+Polling is the portability path everywhere. Backend-specific features improve latency and throughput but are not correctness requirements.
 
-Task execution errors are different: they become failed jobs and stay
-inspectable in the queue database.
+For production PostgreSQL operational guidance, see [Postgres Queue Health](#postgres-queue-health).
 
 ## Recurring Tasks
 
@@ -432,9 +401,10 @@ config.
 The scheduler is part of the normal `dj_queue` runtime. You do not run a
 separate recurring service.
 
-Recurring notes:
+Notes:
 
 - schedules are cron expressions
+- recurring task keys are scoped per backend alias
 - only dynamic tasks can be unscheduled at runtime; unscheduling a static task returns `0`
 - Django admin exposes the same unschedule operation on recurring-task list and detail views
 - multiple schedulers sharing the same recurring config dedupe firing in the database
@@ -468,6 +438,10 @@ With this configuration:
 - later jobs for the same key can block until capacity is released
 - `on_conflict = "discard"` turns the same pattern into singleton-style work
 
+Semaphore rows remain shared on the queue database. If you want per-backend
+isolation for a limit, express that in the `concurrency_key` itself rather than
+expecting one semaphore namespace per backend alias.
+
 ## Queue Operations
 
 `QueueInfo` exposes operational queue controls without bypassing the queue
@@ -490,6 +464,7 @@ orders.clear()
 Queue control notes:
 
 - pausing a queue stops future claims, not enqueueing or already-claimed work
+- pause rows are scoped per backend alias
 - `clear()` discards ready jobs only
 - pass `backend_alias=` when you want to target a non-default `TASKS` alias
 
@@ -549,6 +524,66 @@ except UndiscardableError:
 
 Failures stay inspectable until you act on them.
 
+## Errors When Enqueuing
+
+`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
+backend-side validation failures instead of silently dropping work.
+
+Common reasons include:
+
+- args or kwargs are not JSON round-trippable
+- `concurrency_key` is set without `concurrency_limit`
+- `concurrency_key` cannot be resolved from the enqueue arguments
+- `concurrency_key` does not resolve to a non-empty string up to 255 chars
+- `on_conflict` is not `"block"` or `"discard"`
+
+```python
+from dj_queue.exceptions import EnqueueError
+
+try:
+  sync_account.enqueue(account_id, "refresh")
+except EnqueueError as exc:
+  handle_enqueue_error(exc)
+```
+
+Task execution errors are different: they become failed jobs and stay
+inspectable in the queue database.
+
+## Lifecycle Hooks
+
+Register hooks before starting the runtime, typically during Django startup.
+Each callback receives the live supervisor or runner instance.
+
+```python
+from dj_queue.hooks import on_start, on_worker_start, register_hook
+
+@on_start
+def supervisor_started(process):
+  print(process.name)
+
+@on_worker_start
+def worker_started(process):
+  print(process.metadata)
+
+@register_hook("scheduler.exit")
+def scheduler_exited(process):
+  print(process.name)
+```
+
+Available hook helpers:
+
+- supervisor: `on_start`, `on_stop`, `on_exit`
+- worker: `on_worker_start`, `on_worker_stop`, `on_worker_exit`
+- dispatcher: `on_dispatcher_start`, `on_dispatcher_stop`, `on_dispatcher_exit`
+- scheduler: `on_scheduler_start`, `on_scheduler_stop`, `on_scheduler_exit`
+- generic events: `register_hook("worker.start")`, `register_hook("dispatcher.stop")`, and so on
+
+Hook notes:
+
+- hooks fire in registration order
+- hook failures do not block later hooks
+- hook failures are isolated and routed through `on_thread_error`
+
 ## Multi-Database Setup
 
 `dj_queue` can keep queue tables on a dedicated database alias.
@@ -591,6 +626,28 @@ python manage.py migrate dj_queue --database queue
 With this setup, `dj_queue`'s ORM queries and raw SQL helpers stay on the queue
 database.
 
+## Postgres Queue Health
+
+Operational and configuration guidance for scaling with `dj_queue` in
+production PostgreSQL deployments, covering dedicated database setup, retention
+policy, and autovacuum tuning.
+
+- Use a dedicated queue database via `database_alias`. Keep reporting and
+  long-running transactions off the queue database.
+- Keep retention short. Set `preserve_finished_jobs = False` if you do not need
+  successful results. Otherwise use bounded `clear_finished_jobs_after`,
+  `clear_failed_jobs_after`, and `clear_recurring_executions_after` values.
+- Run `python manage.py dj_queue_prune` regularly for stricter cleanup.
+- Keep `use_skip_locked = True` and `listen_notify = True` unless you have a
+  specific reason not to.
+- Tune autovacuum for `dj_queue_jobs` and the high-churn
+  `dj_queue_*_executions` tables, often default OLTP settings are too
+  conservative for queue workloads.
+- Keep transactions short across workers and the rest of your app. Long-lived
+  transactions pin dead tuples and delay vacuum.
+- Monitor dead tuples, autovacuum frequency, and long-running queries before
+  reaching for partitioning or bulk-ingest paths.
+
 ## Embedded Server Mode
 
 `dj_queue` can run inside an existing server process via embedded async
@@ -622,6 +679,21 @@ signal handling to the host server.
 
 ## Configuration
 
+### Queues, backends, and databases
+
+`dj_queue` has three separate routing concepts. Keep them distinct:
+
+- `queue_name`: what kind of work this job is. Use it to route lanes inside one backend, such as `email`, `webhooks`, or `search-index`.
+- `backend_alias`: which logical queue system owns the work. Use it when you want separate runtime config, recurring tasks, pause and process visibility, retention, or admin scoping.
+- `database_alias`: where that backend's queue tables and runtime activity live. Use it when you want a dedicated database connection path or stronger storage isolation.
+
+Common setup choices:
+
+- one backend, one database: simplest and usually enough
+- one backend, separate queue database: good when you want dedicated queue connections
+- multiple backends, same database: good for logical and operational separation without another database
+- multiple backends, multiple databases: use when you need stronger isolation and accept more migration and deployment complexity
+
 ### Deployment topology
 
 Once migrations are in place, start processing jobs with `python manage.py dj_queue`
@@ -698,9 +770,13 @@ python manage.py dj_queue --config /etc/dj_queue.yml
 DJ_QUEUE_CONFIG=/etc/dj_queue.yml python manage.py dj_queue
 ```
 
-The YAML file should contain a single mapping of backend option values. It uses
-the same shape as `TASKS[backend_alias]["OPTIONS"]`, not the full Django
-`TASKS` structure:
+The YAML file is an overlay on `TASKS[backend_alias]["OPTIONS"]`. It supports
+two shapes:
+
+- a flat mapping of option values for the selected backend alias
+- a `backends` mapping keyed by backend alias, where only the selected alias is applied
+
+Flat mapping example:
 
 ```yaml
 mode: async
@@ -737,8 +813,28 @@ recurring:
     description: nightly cleanup
 ```
 
-This file is merged on top of `TASKS[backend_alias]["OPTIONS"]`, then any
-environment-variable and CLI overrides win after that.
+Multi-backend overlay example:
+
+```yaml
+backends:
+  default:
+    mode: async
+    database_alias: default
+    workers:
+      - queues: ["default", "email*"]
+        threads: 8
+        processes: 1
+        polling_interval: 0.1
+
+  critical:
+    mode: fork
+    database_alias: queue
+    workers:
+      - queues: ["alerts", "critical-review"]
+        threads: 2
+        processes: 1
+        polling_interval: 0.05
+```
 
 Environment overrides currently supported by `dj_queue` itself:
 
@@ -746,41 +842,6 @@ Environment overrides currently supported by `dj_queue` itself:
 - `DJ_QUEUE_MODE`
 - `DJ_QUEUE_SKIP_RECURRING`
 
-## Lifecycle Hooks
-
-Register hooks before starting the runtime, typically during Django startup.
-Each callback receives the live supervisor or runner instance.
-
-```python
-from dj_queue.hooks import on_start, on_worker_start, register_hook
-
-@on_start
-def supervisor_started(process):
-  print(process.name)
-
-@on_worker_start
-def worker_started(process):
-  print(process.metadata)
-
-@register_hook("scheduler.exit")
-def scheduler_exited(process):
-  print(process.name)
-```
-
-Available hook helpers:
-
-- supervisor: `on_start`, `on_stop`, `on_exit`
-- worker: `on_worker_start`, `on_worker_stop`, `on_worker_exit`
-- dispatcher: `on_dispatcher_start`, `on_dispatcher_stop`, `on_dispatcher_exit`
-- scheduler: `on_scheduler_start`, `on_scheduler_stop`, `on_scheduler_exit`
-- generic events: `register_hook("worker.start")`, `register_hook("dispatcher.stop")`, and so on
-
-Hook notes:
-
-- hooks fire in registration order
-- hook failures do not block later hooks
-- hook failures are isolated and routed through `on_thread_error`
-
 ### Runtime infrastructure errors
 
 Set `on_thread_error` to a dotted callable path when you want custom handling
@@ -803,6 +864,40 @@ such as hook failures, heartbeat failures, notify-watcher failures, and managed
 runner crashes. It is not used for exceptions raised by your task code; those
 become failed jobs instead.
 
+## Monitoring
+
+Queue statistics are available in JSON via `/dj_queue/stats.json` and in
+Prometheus text format via `/dj_queue/metrics`.
+
+Include `dj_queue.urls` to expose them:
+
+```python
+urlpatterns += [path("dj_queue/", include("dj_queue.urls"))]
+```
+
+The `/dj_queue/metrics` endpoint requires the `prometheus` extra:
+
+```bash
+pip install "dj-queue[prometheus]"
+```
+
+Exported metric families:
+
+- `dj_queue_queue_jobs{backend,queue,state}`
+- `dj_queue_queue_paused{backend,queue}`
+- `dj_queue_queue_latency_seconds{backend,queue}`
+- `dj_queue_queue_live_workers{backend,queue}`
+- `dj_queue_runner_processes{backend,status}`
+- `dj_queue_runner_processes_by_kind{backend,kind,status}`
+- `dj_queue_recurring_tasks{backend}`
+- `dj_queue_semaphores{queue_database}`
+- `dj_queue_process_rows{backend}`
+
+Both endpoints support bearer token authentication. Set
+`DJ_QUEUE_OBSERVABILITY_TOKEN` in `settings.py` and include it as
+`Authorization: Bearer <token>`. Leave it unset if you protect these URLs at
+the network or proxy layer.
+
 ## License
 
 MIT