dj-queue 0.4.0__tar.gz → 0.5.0__tar.gz

dj_queue-0.4.0/README.md → dj_queue-0.5.0/PKG-INFO

@@ -1,3 +1,31 @@
+Metadata-Version: 2.4
+Name: dj-queue
+Version: 0.5.0
+Summary: Database-backed task queue backend for Django’s Tasks framework.
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 6.0
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: croniter>=6.2.2
+Requires-Dist: django>=6.0.0
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: psycopg>=3.3.3 ; extra == 'postgres'
+Requires-Dist: prometheus-client>=0.4.0 ; extra == 'prometheus'
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/coriocactus/dj_queue
+Project-URL: Repository, https://github.com/coriocactus/dj_queue
+Project-URL: Issues, https://github.com/coriocactus/dj_queue/issues
+Provides-Extra: postgres
+Provides-Extra: prometheus
+Description-Content-Type: text/markdown
+
 # dj_queue
 
 [![CI](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml/badge.svg)](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml)
@@ -17,7 +45,7 @@ It keeps the queue, live execution state, runtime metadata, and task results in
 - immediate, scheduled, recurring, and concurrency-limited work
 
 `dj_queue` is inspired by Rails' [Solid Queue](https://github.com/rails/solid_queue),
-but shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
+shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
 
 ## Why dj_queue
 
@@ -45,20 +73,13 @@ Install the package:
 pip install dj-queue
 ```
 
-Backend-specific extras are available when you want `dj_queue` to install a
-database adapter for you:
+Optional extras:
 
 ```bash
-pip install "dj-queue[postgres]"
+pip install "dj-queue[postgres]"    # psycopg for PostgreSQL + LISTEN/NOTIFY
+pip install "dj-queue[prometheus]"  # prometheus_client for /dj_queue/metrics
 ```
 
-Notes:
-
-- `postgres` installs `psycopg`, which Django's PostgreSQL backend and
-  `dj_queue`'s optional `LISTEN/NOTIFY` wakeups use
-- for MySQL or MariaDB, install and configure a Django-compatible driver in
-  your application following Django's database docs
-
 Add `dj_queue` to `INSTALLED_APPS`, register the router, and point Django's task
 backend at `DjQueueBackend`:
 
@@ -138,12 +159,10 @@ If Django admin is installed, `dj_queue` adds an operator dashboard at
 - queue, process, recurring-task, and semaphore overview
 - backend-aware dashboard and raw changelists
 - queue controls: pause, resume, clear ready
-- job detail action: enqueue a fresh copy of any stored job
-- recurring-task actions: unschedule from list and detail views
-- pause detail action: resume the paused queue from the raw pause row
-- failed-job actions: retry and discard from list and detail views
+- job actions: enqueue a fresh copy of any stored job
+- failed jobs: retry and discard from list and detail views
+- unschedule dynamic recurring tasks
 - queue drill-down pages for state-specific inspection
-- queue drill-down actions: discard ready, scheduled, and blocked jobs; retry or discard failed jobs; enqueue finished jobs again
 
 **Dashboard overview**
 
@@ -302,54 +321,30 @@ Runners heartbeat into the queue database. If claimed work is left behind,
 Use `python manage.py dj_queue_health` to check whether any fresh runtime
 process rows exist for a backend.
 
-## Database Support
-
-| Backend | Support level | Notes |
-|---|---|---|
-| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
-| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
-| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
-| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
-
-Polling is the portability path everywhere. Backend-specific features improve
-latency and throughput but are not correctness requirements.
-
-## Data Contract
+### Data Contract
 
-Job payloads and persisted return values are stored in JSON columns, so they
-must be JSON round-trippable.
+Job payloads and persisted return values are stored in JSON columns, so they must be JSON round-trippable.
 
 - enqueueing args or kwargs that cannot round-trip through JSON fails immediately
 - returning a non-JSON-serializable value marks the job failed instead of
   leaving it claimed forever
 
-If you need to pass model instances, files, or custom objects, store them
-elsewhere and pass identifiers or serialized data instead.
+If you need to pass model instances, files, or custom objects, store them elsewhere and pass identifiers or serialized data instead.
 
-## Errors When Enqueuing
-
-`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
-backend-side validation failures instead of silently dropping work.
-
-Common reasons include:
+## Database Support
 
-- args or kwargs are not JSON round-trippable
-- `concurrency_key` is set without `concurrency_limit`
-- `concurrency_key` cannot be resolved from the enqueue arguments
-- `concurrency_key` does not resolve to a non-empty string up to 255 chars
-- `on_conflict` is not `"block"` or `"discard"`
+| Backend | Support level | Notes |
+|---|---|---|
+| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
+| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
+| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
+| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
 
-```python
-from dj_queue.exceptions import EnqueueError
+For MySQL or MariaDB, install and configure a Django-compatible driver following Django's database docs.
 
-try:
-  sync_account.enqueue(account_id, "refresh")
-except EnqueueError as exc:
-  handle_enqueue_error(exc)
-```
+Polling is the portability path everywhere. Backend-specific features improve latency and throughput but are not correctness requirements.
 
-Task execution errors are different: they become failed jobs and stay
-inspectable in the queue database.
+For production PostgreSQL operational guidance, see [Postgres Queue Health](#postgres-queue-health).
 
 ## Recurring Tasks
 
@@ -406,9 +401,10 @@ config.
 The scheduler is part of the normal `dj_queue` runtime. You do not run a
 separate recurring service.
 
-Recurring notes:
+Notes:
 
 - schedules are cron expressions
+- recurring task keys are scoped per backend alias
 - only dynamic tasks can be unscheduled at runtime; unscheduling a static task returns `0`
 - Django admin exposes the same unschedule operation on recurring-task list and detail views
 - multiple schedulers sharing the same recurring config dedupe firing in the database
@@ -442,6 +438,10 @@ With this configuration:
 - later jobs for the same key can block until capacity is released
 - `on_conflict = "discard"` turns the same pattern into singleton-style work
 
+Semaphore rows remain shared on the queue database. If you want per-backend
+isolation for a limit, express that in the `concurrency_key` itself rather than
+expecting one semaphore namespace per backend alias.
+
 ## Queue Operations
 
 `QueueInfo` exposes operational queue controls without bypassing the queue
@@ -464,6 +464,7 @@ orders.clear()
 Queue control notes:
 
 - pausing a queue stops future claims, not enqueueing or already-claimed work
+- pause rows are scoped per backend alias
 - `clear()` discards ready jobs only
 - pass `backend_alias=` when you want to target a non-default `TASKS` alias
 
@@ -523,6 +524,31 @@ except UndiscardableError:
 
 Failures stay inspectable until you act on them.
 
+## Errors When Enqueuing
+
+`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
+backend-side validation failures instead of silently dropping work.
+
+Common reasons include:
+
+- args or kwargs are not JSON round-trippable
+- `concurrency_key` is set without `concurrency_limit`
+- `concurrency_key` cannot be resolved from the enqueue arguments
+- `concurrency_key` does not resolve to a non-empty string up to 255 chars
+- `on_conflict` is not `"block"` or `"discard"`
+
+```python
+from dj_queue.exceptions import EnqueueError
+
+try:
+  sync_account.enqueue(account_id, "refresh")
+except EnqueueError as exc:
+  handle_enqueue_error(exc)
+```
+
+Task execution errors are different: they become failed jobs and stay
+inspectable in the queue database.
+
 ## Multi-Database Setup
 
 `dj_queue` can keep queue tables on a dedicated database alias.
@@ -565,6 +591,28 @@ python manage.py migrate dj_queue --database queue
 With this setup, `dj_queue`'s ORM queries and raw SQL helpers stay on the queue
 database.
 
+## Postgres Queue Health
+
+Operational and configuration guidance for scaling with `dj_queue` in
+production PostgreSQL deployments, covering dedicated database setup, retention
+policy, and autovacuum tuning.
+
+- Use a dedicated queue database via `database_alias`. Keep reporting and
+  long-running transactions off the queue database.
+- Keep retention short. Set `preserve_finished_jobs = False` if you do not need
+  successful results. Otherwise use bounded `clear_finished_jobs_after`,
+  `clear_failed_jobs_after`, and `clear_recurring_executions_after` values.
+- Run `python manage.py dj_queue_prune` regularly for stricter cleanup.
+- Keep `use_skip_locked = True` and `listen_notify = True` unless you have a
+  specific reason not to.
+- Tune autovacuum for `dj_queue_jobs` and the high-churn
+  `dj_queue_*_executions` tables, often default OLTP settings are too
+  conservative for queue workloads.
+- Keep transactions short across workers and the rest of your app. Long-lived
+  transactions pin dead tuples and delay vacuum.
+- Monitor dead tuples, autovacuum frequency, and long-running queries before
+  reaching for partitioning or bulk-ingest paths.
+
 ## Embedded Server Mode
 
 `dj_queue` can run inside an existing server process via embedded async
@@ -672,9 +720,13 @@ python manage.py dj_queue --config /etc/dj_queue.yml
 DJ_QUEUE_CONFIG=/etc/dj_queue.yml python manage.py dj_queue
 ```
 
-The YAML file should contain a single mapping of backend option values. It uses
-the same shape as `TASKS[backend_alias]["OPTIONS"]`, not the full Django
-`TASKS` structure:
+The YAML file is an overlay on `TASKS[backend_alias]["OPTIONS"]`. It supports
+two shapes:
+
+- a flat mapping of option values for the selected backend alias
+- a `backends` mapping keyed by backend alias, where only the selected alias is applied
+
+Flat mapping example:
 
 ```yaml
 mode: async
@@ -711,8 +763,28 @@ recurring:
     description: nightly cleanup
 ```
 
-This file is merged on top of `TASKS[backend_alias]["OPTIONS"]`, then any
-environment-variable and CLI overrides win after that.
+Multi-backend overlay example:
+
+```yaml
+backends:
+  default:
+    mode: async
+    database_alias: default
+    workers:
+      - queues: ["default", "email*"]
+        threads: 8
+        processes: 1
+        polling_interval: 0.1
+
+  critical:
+    mode: fork
+    database_alias: queue
+    workers:
+      - queues: ["alerts", "critical-review"]
+        threads: 2
+        processes: 1
+        polling_interval: 0.05
+```
 
 Environment overrides currently supported by `dj_queue` itself:
 
@@ -777,6 +849,40 @@ such as hook failures, heartbeat failures, notify-watcher failures, and managed
 runner crashes. It is not used for exceptions raised by your task code; those
 become failed jobs instead.
 
+## Monitoring
+
+Queue statistics are available in JSON via `/dj_queue/stats.json` and in
+Prometheus text format via `/dj_queue/metrics`. 
+
+Include `dj_queue.urls` to expose them:
+
+```python
+urlpatterns += [path("dj_queue/", include("dj_queue.urls"))]
+```
+
+The `/dj_queue/metrics` endpoint requires the `prometheus` extra:
+
+```bash
+pip install "dj-queue[prometheus]"
+```
+
+Exported metric families:
+
+- `dj_queue_queue_jobs{backend,queue,state}`
+- `dj_queue_queue_paused{backend,queue}`
+- `dj_queue_queue_latency_seconds{backend,queue}`
+- `dj_queue_queue_live_workers{backend,queue}`
+- `dj_queue_runner_processes{backend,status}`
+- `dj_queue_runner_processes_by_kind{backend,kind,status}`
+- `dj_queue_recurring_tasks{backend}`
+- `dj_queue_semaphores{queue_database}`
+- `dj_queue_process_rows{backend}`
+
+Both endpoints support bearer token authentication. Set
+`DJ_QUEUE_OBSERVABILITY_TOKEN` in `settings.py` and include it as
+`Authorization: Bearer <token>`. Leave it unset if you protect these URLs at
+the network or proxy layer.
+
 ## License
 
 MIT

dj_queue-0.4.0/PKG-INFO → dj_queue-0.5.0/README.md

@@ -1,29 +1,3 @@
-Metadata-Version: 2.4
-Name: dj-queue
-Version: 0.4.0
-Summary: Database-backed task queue backend for Django's django.tasks framework
-License-Expression: MIT
-License-File: LICENSE
-Classifier: Development Status :: 4 - Beta
-Classifier: Framework :: Django
-Classifier: Framework :: Django :: 6.0
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Programming Language :: Python :: 3.14
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Dist: croniter>=6.2.2
-Requires-Dist: django>=6.0.0
-Requires-Dist: pyyaml>=6.0.3
-Requires-Dist: psycopg>=3.3.3 ; extra == 'postgres'
-Requires-Python: >=3.12
-Project-URL: Homepage, https://github.com/coriocactus/dj_queue
-Project-URL: Repository, https://github.com/coriocactus/dj_queue
-Project-URL: Issues, https://github.com/coriocactus/dj_queue/issues
-Provides-Extra: postgres
-Description-Content-Type: text/markdown
-
 # dj_queue
 
 [![CI](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml/badge.svg)](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml)
@@ -43,7 +17,7 @@ It keeps the queue, live execution state, runtime metadata, and task results in
 - immediate, scheduled, recurring, and concurrency-limited work
 
 `dj_queue` is inspired by Rails' [Solid Queue](https://github.com/rails/solid_queue),
-but shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
+shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
 
 ## Why dj_queue
 
@@ -71,20 +45,13 @@ Install the package:
 pip install dj-queue
 ```
 
-Backend-specific extras are available when you want `dj_queue` to install a
-database adapter for you:
+Optional extras:
 
 ```bash
-pip install "dj-queue[postgres]"
+pip install "dj-queue[postgres]"    # psycopg for PostgreSQL + LISTEN/NOTIFY
+pip install "dj-queue[prometheus]"  # prometheus_client for /dj_queue/metrics
 ```
 
-Notes:
-
-- `postgres` installs `psycopg`, which Django's PostgreSQL backend and
-  `dj_queue`'s optional `LISTEN/NOTIFY` wakeups use
-- for MySQL or MariaDB, install and configure a Django-compatible driver in
-  your application following Django's database docs
-
 Add `dj_queue` to `INSTALLED_APPS`, register the router, and point Django's task
 backend at `DjQueueBackend`:
 
@@ -164,12 +131,10 @@ If Django admin is installed, `dj_queue` adds an operator dashboard at
 - queue, process, recurring-task, and semaphore overview
 - backend-aware dashboard and raw changelists
 - queue controls: pause, resume, clear ready
-- job detail action: enqueue a fresh copy of any stored job
-- recurring-task actions: unschedule from list and detail views
-- pause detail action: resume the paused queue from the raw pause row
-- failed-job actions: retry and discard from list and detail views
+- job actions: enqueue a fresh copy of any stored job
+- failed jobs: retry and discard from list and detail views
+- unschedule dynamic recurring tasks
 - queue drill-down pages for state-specific inspection
-- queue drill-down actions: discard ready, scheduled, and blocked jobs; retry or discard failed jobs; enqueue finished jobs again
 
 **Dashboard overview**
 
@@ -328,54 +293,30 @@ Runners heartbeat into the queue database. If claimed work is left behind,
 Use `python manage.py dj_queue_health` to check whether any fresh runtime
 process rows exist for a backend.
 
-## Database Support
-
-| Backend | Support level | Notes |
-|---|---|---|
-| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
-| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
-| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
-| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
-
-Polling is the portability path everywhere. Backend-specific features improve
-latency and throughput but are not correctness requirements.
-
-## Data Contract
+### Data Contract
 
-Job payloads and persisted return values are stored in JSON columns, so they
-must be JSON round-trippable.
+Job payloads and persisted return values are stored in JSON columns, so they must be JSON round-trippable.
 
 - enqueueing args or kwargs that cannot round-trip through JSON fails immediately
 - returning a non-JSON-serializable value marks the job failed instead of
   leaving it claimed forever
 
-If you need to pass model instances, files, or custom objects, store them
-elsewhere and pass identifiers or serialized data instead.
+If you need to pass model instances, files, or custom objects, store them elsewhere and pass identifiers or serialized data instead.
 
-## Errors When Enqueuing
-
-`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
-backend-side validation failures instead of silently dropping work.
-
-Common reasons include:
+## Database Support
 
-- args or kwargs are not JSON round-trippable
-- `concurrency_key` is set without `concurrency_limit`
-- `concurrency_key` cannot be resolved from the enqueue arguments
-- `concurrency_key` does not resolve to a non-empty string up to 255 chars
-- `on_conflict` is not `"block"` or `"discard"`
+| Backend | Support level | Notes |
+|---|---|---|
+| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
+| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
+| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
+| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
 
-```python
-from dj_queue.exceptions import EnqueueError
+For MySQL or MariaDB, install and configure a Django-compatible driver following Django's database docs.
 
-try:
-  sync_account.enqueue(account_id, "refresh")
-except EnqueueError as exc:
-  handle_enqueue_error(exc)
-```
+Polling is the portability path everywhere. Backend-specific features improve latency and throughput but are not correctness requirements.
 
-Task execution errors are different: they become failed jobs and stay
-inspectable in the queue database.
+For production PostgreSQL operational guidance, see [Postgres Queue Health](#postgres-queue-health).
 
 ## Recurring Tasks
 
@@ -432,9 +373,10 @@ config.
 The scheduler is part of the normal `dj_queue` runtime. You do not run a
 separate recurring service.
 
-Recurring notes:
+Notes:
 
 - schedules are cron expressions
+- recurring task keys are scoped per backend alias
 - only dynamic tasks can be unscheduled at runtime; unscheduling a static task returns `0`
 - Django admin exposes the same unschedule operation on recurring-task list and detail views
 - multiple schedulers sharing the same recurring config dedupe firing in the database
@@ -468,6 +410,10 @@ With this configuration:
 - later jobs for the same key can block until capacity is released
 - `on_conflict = "discard"` turns the same pattern into singleton-style work
 
+Semaphore rows remain shared on the queue database. If you want per-backend
+isolation for a limit, express that in the `concurrency_key` itself rather than
+expecting one semaphore namespace per backend alias.
+
 ## Queue Operations
 
 `QueueInfo` exposes operational queue controls without bypassing the queue
@@ -490,6 +436,7 @@ orders.clear()
 Queue control notes:
 
 - pausing a queue stops future claims, not enqueueing or already-claimed work
+- pause rows are scoped per backend alias
 - `clear()` discards ready jobs only
 - pass `backend_alias=` when you want to target a non-default `TASKS` alias
 
@@ -549,6 +496,31 @@ except UndiscardableError:
 
 Failures stay inspectable until you act on them.
 
+## Errors When Enqueuing
+
+`DjQueueBackend.enqueue()` raises `dj_queue.exceptions.EnqueueError` for
+backend-side validation failures instead of silently dropping work.
+
+Common reasons include:
+
+- args or kwargs are not JSON round-trippable
+- `concurrency_key` is set without `concurrency_limit`
+- `concurrency_key` cannot be resolved from the enqueue arguments
+- `concurrency_key` does not resolve to a non-empty string up to 255 chars
+- `on_conflict` is not `"block"` or `"discard"`
+
+```python
+from dj_queue.exceptions import EnqueueError
+
+try:
+  sync_account.enqueue(account_id, "refresh")
+except EnqueueError as exc:
+  handle_enqueue_error(exc)
+```
+
+Task execution errors are different: they become failed jobs and stay
+inspectable in the queue database.
+
 ## Multi-Database Setup
 
 `dj_queue` can keep queue tables on a dedicated database alias.
@@ -591,6 +563,28 @@ python manage.py migrate dj_queue --database queue
 With this setup, `dj_queue`'s ORM queries and raw SQL helpers stay on the queue
 database.
 
+## Postgres Queue Health
+
+Operational and configuration guidance for scaling with `dj_queue` in
+production PostgreSQL deployments, covering dedicated database setup, retention
+policy, and autovacuum tuning.
+
+- Use a dedicated queue database via `database_alias`. Keep reporting and
+  long-running transactions off the queue database.
+- Keep retention short. Set `preserve_finished_jobs = False` if you do not need
+  successful results. Otherwise use bounded `clear_finished_jobs_after`,
+  `clear_failed_jobs_after`, and `clear_recurring_executions_after` values.
+- Run `python manage.py dj_queue_prune` regularly for stricter cleanup.
+- Keep `use_skip_locked = True` and `listen_notify = True` unless you have a
+  specific reason not to.
+- Tune autovacuum for `dj_queue_jobs` and the high-churn
+  `dj_queue_*_executions` tables, often default OLTP settings are too
+  conservative for queue workloads.
+- Keep transactions short across workers and the rest of your app. Long-lived
+  transactions pin dead tuples and delay vacuum.
+- Monitor dead tuples, autovacuum frequency, and long-running queries before
+  reaching for partitioning or bulk-ingest paths.
+
 ## Embedded Server Mode
 
 `dj_queue` can run inside an existing server process via embedded async
@@ -698,9 +692,13 @@ python manage.py dj_queue --config /etc/dj_queue.yml
 DJ_QUEUE_CONFIG=/etc/dj_queue.yml python manage.py dj_queue
 ```
 
-The YAML file should contain a single mapping of backend option values. It uses
-the same shape as `TASKS[backend_alias]["OPTIONS"]`, not the full Django
-`TASKS` structure:
+The YAML file is an overlay on `TASKS[backend_alias]["OPTIONS"]`. It supports
+two shapes:
+
+- a flat mapping of option values for the selected backend alias
+- a `backends` mapping keyed by backend alias, where only the selected alias is applied
+
+Flat mapping example:
 
 ```yaml
 mode: async
@@ -737,8 +735,28 @@ recurring:
     description: nightly cleanup
 ```
 
-This file is merged on top of `TASKS[backend_alias]["OPTIONS"]`, then any
-environment-variable and CLI overrides win after that.
+Multi-backend overlay example:
+
+```yaml
+backends:
+  default:
+    mode: async
+    database_alias: default
+    workers:
+      - queues: ["default", "email*"]
+        threads: 8
+        processes: 1
+        polling_interval: 0.1
+
+  critical:
+    mode: fork
+    database_alias: queue
+    workers:
+      - queues: ["alerts", "critical-review"]
+        threads: 2
+        processes: 1
+        polling_interval: 0.05
+```
 
 Environment overrides currently supported by `dj_queue` itself:
 
@@ -803,6 +821,40 @@ such as hook failures, heartbeat failures, notify-watcher failures, and managed
 runner crashes. It is not used for exceptions raised by your task code; those
 become failed jobs instead.
 
+## Monitoring
+
+Queue statistics are available in JSON via `/dj_queue/stats.json` and in
+Prometheus text format via `/dj_queue/metrics`. 
+
+Include `dj_queue.urls` to expose them:
+
+```python
+urlpatterns += [path("dj_queue/", include("dj_queue.urls"))]
+```
+
+The `/dj_queue/metrics` endpoint requires the `prometheus` extra:
+
+```bash
+pip install "dj-queue[prometheus]"
+```
+
+Exported metric families:
+
+- `dj_queue_queue_jobs{backend,queue,state}`
+- `dj_queue_queue_paused{backend,queue}`
+- `dj_queue_queue_latency_seconds{backend,queue}`
+- `dj_queue_queue_live_workers{backend,queue}`
+- `dj_queue_runner_processes{backend,status}`
+- `dj_queue_runner_processes_by_kind{backend,kind,status}`
+- `dj_queue_recurring_tasks{backend}`
+- `dj_queue_semaphores{queue_database}`
+- `dj_queue_process_rows{backend}`
+
+Both endpoints support bearer token authentication. Set
+`DJ_QUEUE_OBSERVABILITY_TOKEN` in `settings.py` and include it as
+`Authorization: Bearer <token>`. Leave it unset if you protect these URLs at
+the network or proxy layer.
+
 ## License
 
 MIT