dj-queue 0.1.1__tar.gz → 0.2.0__tar.gz

{dj_queue-0.1.1 → dj_queue-0.2.0}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: dj-queue
-Version: 0.1.1
+Version: 0.2.0
 Summary: Database-backed task queue backend for Django's django.tasks framework
 License-Expression: MIT
 License-File: LICENSE
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Framework :: Django
 Classifier: Framework :: Django :: 6.0
 Classifier: Intended Audience :: Developers
@@ -26,27 +26,30 @@ Description-Content-Type: text/markdown
 
 # dj_queue
 
-`dj_queue` is a database-backed task queue backend for Django's `django.tasks`
-framework.
+[![CI](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml/badge.svg)](https://github.com/coriocactus/dj_queue/actions/workflows/ci.yml)
+![PyPI](https://img.shields.io/pypi/v/dj-queue.svg)
+![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dj-queue.svg)
+![PyPI - Status](https://img.shields.io/pypi/status/dj-queue.svg)
+![PyPI - License](https://img.shields.io/pypi/l/dj-queue.svg)
 
-It keeps the queue, live execution state, runtime metadata, and task results in
-your database.
+`dj_queue` is a database-backed task queue backend for the `django.tasks` framework.
+
+It keeps the queue, live execution state, runtime metadata, and task results in your database.
 
 - no Redis, RabbitMQ, or separate result store
 - PostgreSQL is the first-class production backend
 - MySQL 8+, MariaDB 10.6+, and SQLite are supported
 - immediate, scheduled, recurring, and concurrency-limited work
-- fork and async runtime modes
-- multi-database aware from day one
 
-`dj_queue` is inspired by Rails solid_queue, but shaped to fit Django's task
-backend API and long-running process model.
+`dj_queue` is inspired by Rails' [Solid Queue](https://github.com/rails/solid_queue),
+but shaped to fit Django's [task backend API](https://docs.djangoproject.com/en/6.0/topics/tasks/).
 
 ## Why dj_queue
 
-The database is the queue.
+Django applications already depend on the database as the durable system of
+record. `dj_queue` lets background work follow the same model.
 
-That gives `dj_queue` a narrow, explicit shape:
+It has a narrow, explicit shape:
 
 - application code uses Django's `@task` API
 - `DjQueueBackend` stores jobs and results in Django-managed tables
@@ -54,8 +57,8 @@ That gives `dj_queue` a narrow, explicit shape:
 - PostgreSQL can use `LISTEN/NOTIFY` and `SKIP LOCKED` as optimizations
 - polling remains the correctness path on every supported database
 
-If your application already depends on the database being the durable system of
-record, `dj_queue` lets background work follow the same model.
+For detailed comparisons with Celery, RQ, Procrastinate, and other alternatives,
+see [COMPARISONS.md](COMPARISONS.md).
 
 ## Installation
 
@@ -103,6 +106,9 @@ TASKS = {
 }
 ```
 
+The router is optional when using the default database, but harmless to include
+and required for [multi-database setups](#multi-database-setup).
+
 Run migrations:
 
 ```bash
@@ -115,10 +121,8 @@ Define a task with Django's `@task` decorator:
 
 ```python
 # myapp/tasks.py
-
 from django.tasks import task
 
-
 @task
 def add(a, b):
   return a + b
@@ -151,83 +155,6 @@ print(fresh_result.return_value)
 
 When the worker has executed the job, `fresh_result.return_value` will be `10`.
 
-## Data Contract
-
-Job payloads and persisted return values are stored in JSON columns, so they
-must be JSON round-trippable.
-
-- enqueueing args or kwargs that cannot round-trip through JSON fails immediately
-- returning a non-JSON-serializable value marks the job failed instead of
-  leaving it claimed forever
-
-If you need to pass model instances, files, or custom objects, store them
-elsewhere and pass identifiers or serialized data instead.
-
-## How dj_queue runs
-
-`python manage.py dj_queue` starts a supervisor for one backend alias.
-
-Job lifecycle:
-
-`enqueue -> ready | scheduled | blocked -> claimed -> successful | failed`
-
-The runtime has four moving parts:
-
-- `supervisor`: boots and stops the runtime
-- `workers`: claim ready jobs and execute them
-- `dispatchers`: promote due scheduled jobs and run concurrency maintenance
-- `scheduler`: enqueue recurring tasks and finished-job cleanup when configured
-
-Useful command variants:
-
-```bash
-python manage.py dj_queue
-python manage.py dj_queue --mode async
-python manage.py dj_queue --only-work
-python manage.py dj_queue --only-dispatch
-python manage.py dj_queue --skip-recurring
-```
-
-Mode and topology notes:
-
-- `fork` is the default standalone mode
-- `async` runs supervised actors in threads inside one process
-- `--only-work` starts workers without dispatchers or scheduler
-- `--only-dispatch` starts dispatchers without workers or scheduler
-- `--skip-recurring` starts without the scheduler
-
-If you're familiar with Solid Queue, the same high-level tradeoff is described
-in its [fork vs async mode](https://github.com/rails/solid_queue?tab=readme-ov-file#fork-vs-async-mode)
-section.
-
-## Choose a setup
-
-Once migrations are in place, start processing jobs with `python manage.py dj_queue`
-on the machine that should do the work. With the default configuration, this
-starts the supervisor, workers, and dispatcher for the default backend alias and
-processes all queues.
-
-For most deployments, start with a standalone `dj_queue` process. Reach for a
-dedicated queue database before you reach for embedded mode.
-
-- single database, standalone process: easiest way to start. Use the app
-  database and run `python manage.py dj_queue`
-- dedicated queue database: recommended production default. Keep queue tables
-  and runtime traffic on `database_alias`. See [Multi-Database Setup](#multi-database-setup)
-- embedded server mode: run `dj_queue` inside ASGI or Gunicorn when you want
-  queue execution colocated with the server process. See [Embedded Server Mode](#embedded-server-mode)
-
-For small deployments, running `dj_queue` on the same machine as the web server
-is often enough. When you need more capacity, multiple machines can point at
-the same queue database. Full `python manage.py dj_queue` instances coordinate
-through database locking, so workers and dispatchers share load safely and
-recurring firing stays deduplicated across schedulers.
-
-In practice, keep recurring settings identical on every full node and prefer one
-full instance plus additional `python manage.py dj_queue --only-work` nodes.
-Add `--only-dispatch` nodes only when you need more scheduled-job promotion or
-concurrency-maintenance throughput.
-
 ## Common Patterns
 
 ### Scheduled jobs
@@ -236,9 +163,7 @@ Use `run_after` to keep work out of the ready queue until a future time:
 
 ```python
 from datetime import timedelta
-
 from django.utils import timezone
-
 from myapp.tasks import send_digest
 
 future = timezone.now() + timedelta(hours=1)
@@ -268,9 +193,75 @@ results = process_item.get_backend().enqueue_all(
 )
 ```
 
-## Ordering and transactions
+### Enqueue after commit
+
+`enqueue()` writes immediately. If a task depends on rows that are still inside
+the current transaction, use `enqueue_on_commit()`:
+
+```python
+from django.db import transaction
+from dj_queue.api import enqueue_on_commit
+from myapp.tasks import send_receipt
+
+with transaction.atomic():
+  order = create_order()
+  enqueue_on_commit(send_receipt, order.id)
+```
+
+### Examples
+
+The repository ships real runnable examples in `examples/`.
+
+Recommended entry points:
+
+- [examples/ex01_basic_enqueue.py](examples/ex01_basic_enqueue.py)
+- [examples/ex07_basic_enqueue_on_commit.py](examples/ex07_basic_enqueue_on_commit.py)
+- [examples/ex08_basic_recurring.py](examples/ex08_basic_recurring.py)
+- [examples/ex20_advanced_concurrency.py](examples/ex20_advanced_concurrency.py)
+- [examples/ex21_advanced_queue_control.py](examples/ex21_advanced_queue_control.py)
+- [examples/ex24_advanced_multi_db.py](examples/ex24_advanced_multi_db.py)
+- [examples/ex25_advanced_asgi.py](examples/ex25_advanced_asgi.py)
+
+The [examples index](examples/README.md) lists the full progression.
+
+## How it Works
+
+`python manage.py dj_queue` starts a supervisor for one backend alias.
+
+Job lifecycle:
+
+`enqueue -> ready | scheduled | blocked -> claimed -> successful | failed`
+
+The runtime has four moving parts:
+
+- `supervisor`: boots and stops the runtime
+- `workers`: claim ready jobs and execute them
+- `dispatchers`: promote due scheduled jobs and run concurrency maintenance
+- `scheduler`: enqueue recurring tasks and finished-job cleanup when configured
+
+Useful command variants:
+
+```bash
+python manage.py dj_queue
+python manage.py dj_queue --mode async
+python manage.py dj_queue --only-work
+python manage.py dj_queue --only-dispatch
+python manage.py dj_queue --skip-recurring
+```
+
+Mode and topology notes:
+
+- `fork` is the default standalone mode
+- `async` runs supervised actors in threads inside one process
+- `--only-work` starts workers without dispatchers or scheduler
+- `--only-dispatch` starts dispatchers without workers or scheduler
+- `--skip-recurring` starts without the scheduler
+
+`fork` runs each worker, dispatcher, and scheduler as a separate OS process.
+`async` runs them as threads in one process, i.e., lower memory, less isolation.
+Default is `fork`. Use `async` for embedded mode or memory-constrained environments.
 
-Queue ordering rules:
+### Claiming order
 
 - within one selected queue, higher numeric `priority` is claimed first
 - across multiple queue selectors, selector order wins
@@ -281,19 +272,29 @@ For example, a worker configured with `queues: ["email", "default"]` will
 prefer ready work from `email` before `default`, even if `default` contains
 higher-priority rows.
 
-`enqueue()` writes immediately. If a task depends on rows that are still inside
-the current transaction, use `enqueue_on_commit()`:
+## Database Support
 
-```python
-from django.db import transaction
+| Backend | Support level | Notes |
+|---|---|---|
+| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
+| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
+| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
+| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
 
-from dj_queue.api import enqueue_on_commit
-from myapp.tasks import send_receipt
+Polling is the portability path everywhere. Backend-specific features improve
+latency and throughput but are not correctness requirements.
 
-with transaction.atomic():
-  order = create_order()
-  enqueue_on_commit(send_receipt, order.id)
-```
+## Data Contract
+
+Job payloads and persisted return values are stored in JSON columns, so they
+must be JSON round-trippable.
+
+- enqueueing args or kwargs that cannot round-trip through JSON fails immediately
+- returning a non-JSON-serializable value marks the job failed instead of
+  leaving it claimed forever
+
+If you need to pass model instances, files, or custom objects, store them
+elsewhere and pass identifiers or serialized data instead.
 
 ## Recurring Tasks
 
@@ -352,18 +353,18 @@ separate recurring service.
 
 ## Concurrency Controls
 
-Tasks can opt into database-backed concurrency limits by defining concurrency
-metadata on the wrapped function:
+Tasks can opt into database-backed concurrency limits.
+
+`django.tasks` has no standard way to pass backend-specific options through the
+`@task` decorator, so `dj_queue` reads them as attributes on the wrapped function:
 
 ```python
 from django.tasks import task
 
-
 @task
 def sync_account(account_id, action):
   return f"{account_id}:{action}"
 
-
 sync_account.func.concurrency_key = "account:{account_id}"
 sync_account.func.concurrency_limit = 1
 sync_account.func.concurrency_duration = 60
@@ -408,13 +409,13 @@ If Django admin is installed, `dj_queue` also registers the main operational
 models there, including jobs, failed executions, processes, recurring tasks,
 pauses, and semaphores.
 
-## Failed jobs
+## Failed Jobs
 
 When a task raises, `dj_queue` keeps the job and its failed execution row in the
 queue database, including the exception class, message, and traceback.
 
-You can retry or discard failed jobs through Django admin or the operations
-layer:
+You can retry failed jobs through Django admin, or retry and discard them
+through the operations layer:
 
 ```python
 from dj_queue.operations.jobs import discard_failed_job, retry_failed_job
@@ -478,7 +479,6 @@ Wrap your ASGI application with `DjQueueLifespan`:
 
 ```python
 from django.core.asgi import get_asgi_application
-
 from dj_queue.contrib.asgi import DjQueueLifespan
 
 django_application = get_asgi_application()
@@ -491,7 +491,6 @@ Import the provided hooks in your Gunicorn config:
 
 ```python
 # gunicorn.conf.py
-
 from dj_queue.contrib.gunicorn import post_fork, worker_exit
 ```
 
@@ -500,6 +499,36 @@ signal handling to the host server.
 
 ## Configuration
 
+### Deployment topology
+
+Once migrations are in place, start processing jobs with `python manage.py dj_queue`
+on the machine that should do the work. With the default configuration, this
+starts the supervisor, workers, dispatcher, and scheduler for the default
+backend alias and processes all queues.
+
+For most deployments, start with a standalone `dj_queue` process. Reach for a
+dedicated queue database before you reach for embedded mode.
+
+- single database, standalone process: easiest way to start. Use the app
+  database and run `python manage.py dj_queue`
+- dedicated queue database: recommended production default. Keep queue tables
+  and runtime traffic on `database_alias`. See [Multi-Database Setup](#multi-database-setup)
+- embedded server mode: run `dj_queue` inside ASGI or Gunicorn when you want
+  queue execution colocated with the server process. See [Embedded Server Mode](#embedded-server-mode)
+
+For small deployments, running `dj_queue` on the same machine as the web server
+is often enough. When you need more capacity, multiple machines can point at
+the same queue database. Full `python manage.py dj_queue` instances coordinate
+through database locking, so workers and dispatchers share load safely and
+recurring firing stays deduplicated across schedulers.
+
+In practice, keep recurring settings identical on every full node and prefer one
+full instance plus additional `python manage.py dj_queue --only-work` nodes.
+Add `--only-dispatch` nodes only when you need more scheduled-job promotion or
+concurrency-maintenance throughput.
+
+### Options
+
 The main configuration lives in `TASKS[backend_alias]["OPTIONS"]`.
 
 Start with these options:
@@ -509,8 +538,7 @@ Start with these options:
 - `dispatchers`: scheduled promotion and concurrency maintenance settings
 - `scheduler`: dynamic recurring polling settings
 - `database_alias`: database alias for queue tables and runtime activity
-- `preserve_finished_jobs` and `clear_finished_jobs_after`: result retention and
-  cleanup
+- `preserve_finished_jobs` and `clear_finished_jobs_after`: result retention and cleanup
 
 Additional operational tuning is available when needed, including
 `use_skip_locked`, `listen_notify`, `silence_polling`,
@@ -521,6 +549,8 @@ On PostgreSQL, `listen_notify` uses the same Django PostgreSQL driver
 configuration as the main database connection. Install a compatible driver in
 your project, or use `dj-queue[postgres]` to pull in `psycopg`.
 
+### Precedence
+
 Configuration precedence is explicit:
 
 - CLI overrides
@@ -530,11 +560,11 @@ Configuration precedence is explicit:
 
 ### YAML file config
 
-You can point `dj_queue` at a YAML file with either `--config` or
-`DJ_QUEUE_CONFIG`:
-
 ```bash
+# via cli
 python manage.py dj_queue --config /etc/dj_queue.yml
+
+# or via environment variable
 DJ_QUEUE_CONFIG=/etc/dj_queue.yml python manage.py dj_queue
 ```
 
@@ -584,30 +614,6 @@ Environment overrides currently supported by `dj_queue` itself:
 - `DJ_QUEUE_MODE`
 - `DJ_QUEUE_SKIP_RECURRING`
 
-## Database Support
-
-| Backend | Support level | Notes |
-|---|---|---|
-| PostgreSQL | first-class | polling, `SKIP LOCKED`, and optional `LISTEN/NOTIFY` |
-| MySQL 8+ | supported | polling plus `SKIP LOCKED` |
-| MariaDB 10.6+ | supported | polling plus `SKIP LOCKED` |
-| SQLite | supported with limits | polling only, serialized writes, no `SKIP LOCKED`, no `LISTEN/NOTIFY`; practical for development, CI, and smaller deployments |
-
-Polling is the portability path everywhere. Backend-specific features improve
-latency and throughput but are not correctness requirements.
-
-## Examples
-
-The repository ships real runnable examples in `examples/`.
-
-Recommended entry points:
-
-- [examples/ex01_basic_enqueue.py](examples/ex01_basic_enqueue.py)
-- [examples/ex07_basic_enqueue_on_commit.py](examples/ex07_basic_enqueue_on_commit.py)
-- [examples/ex08_basic_recurring.py](examples/ex08_basic_recurring.py)
-- [examples/ex20_advanced_concurrency.py](examples/ex20_advanced_concurrency.py)
-- [examples/ex21_advanced_queue_control.py](examples/ex21_advanced_queue_control.py)
-- [examples/ex24_advanced_multi_db.py](examples/ex24_advanced_multi_db.py)
-- [examples/ex25_advanced_asgi.py](examples/ex25_advanced_asgi.py)
+## License
 
-The [examples index](examples/README.md) lists the full progression.
+MIT