django-database-task 0.3.0__tar.gz → 0.4.0__tar.gz

django_database_task-0.4.0/CHANGELOG.md

@@ -0,0 +1,65 @@
+# Changelog
+
+## 0.4.0
+
+Brokers — the services that trigger execution of a saved task — are now
+separate from the task backend, and Amazon SQS joins Cloud Tasks as one of
+them.
+
+**Existing projects need no changes.** The settings, the URLs, the management
+commands and their defaults all behave as they did in 0.3.
+
+### Added
+
+- **Amazon SQS broker** (`django_database_task.sqs.SQSDatabaseBackend`,
+  `pip install django-database-task[sqs]`). Sends a message naming the task,
+  and `run_database_tasks` receives it. A task deferred beyond the 15 minute
+  SQS delay limit stays in the database for the worker's database sweep.
+- **`--source` for `run_database_tasks`**: `auto` (default), `db`, `broker` or
+  `both`. `auto` means `both` when the backend has a broker a worker can
+  receive from, and `db` otherwise, so the command is run the same way as
+  before either way. `--wait-time` and `--max-messages` go with it.
+- **The broker abstraction** (`django_database_task.brokers`): `TaskBroker`,
+  `HTTPPushBroker` and `PullBroker`. A project attaches its own with the
+  `BROKER` option.
+- **Several authentication handlers per backend**, through
+  `get_auth_handlers()`. A request is accepted as soon as one handler accepts
+  it, so the service that calls the endpoints and an external cron job can use
+  different credentials. Configure them with the `AUTH_HANDLERS` option, and
+  limit one to some endpoints with `ENDPOINTS`.
+- **Bundled authentication handlers** in `django_database_task.auth`:
+  `SharedSecretAuth`, `HMACAuth` and `StaffOnlyAuth`, with `build_signature()`
+  for callers that have to sign a request for `HMACAuth`.
+- **AWS environment detection** in `django_database_task.sqs`:
+  `detect_aws_region()`, `is_lambda()` and `is_ecs()`, alongside the existing
+  Cloud Tasks ones.
+- The brokers themselves are importable, for a project that wants one on a
+  backend of its own: `django_database_task.sqs.SQSBroker` and
+  `django_database_task.cloudtasks.CloudTasksBroker`.
+
+### Fixed
+
+- Enabling Cloud Tasks OIDC no longer locks every other caller out of the task
+  endpoints. Since 0.3.1 the OIDC handler was applied to all of them, so an
+  external cron job calling `/tasks/run/` or `/tasks/purge/` was rejected.
+- The Cloud Tasks tests were skipped on every run, in CI included, because the
+  `cloudtasks` extra was never installed. Four of them had gone stale
+  unnoticed.
+
+### Deprecated
+
+- `get_auth_handler()` (singular). It still works, with a
+  `DeprecationWarning`, and is removed in 0.5. Override `get_auth_handlers()`
+  instead.
+
+### Documentation
+
+- A *Task Brokers* section in the README, and an Amazon SQS one with a
+  sequence diagram beside the ones the database backend and Cloud Tasks
+  already had.
+- An SQS walkthrough in `examples/`, run against a local mock: set
+  `DEMO_BROKER=sqs` to point the demo project at it.
+- `CONTRIBUTING.md`, covering the development setup, the tests, and how to add
+  a broker.
+- This file. Releases before 0.4.0 are in the git history and the GitHub
+  releases.

{django_database_task-0.3.0 → django_database_task-0.4.0}/MANIFEST.in

@@ -1,3 +1,4 @@
 include LICENSE
 include README.md
+include CHANGELOG.md
 recursive-include django_database_task/locale *.po *.mo

{django_database_task-0.3.0/django_database_task.egg-info → django_database_task-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-database-task
-Version: 0.3.0
+Version: 0.4.0
 Summary: A database-backed task queue backend for Django's built-in task framework
 Author-email: Shinya Okano <tokibito@gmail.com>
 Maintainer-email: Shinya Okano <tokibito@gmail.com>
@@ -29,6 +29,8 @@ Requires-Dist: Django>=6.0
 Provides-Extra: cloudtasks
 Requires-Dist: google-cloud-tasks>=2.0.0; extra == "cloudtasks"
 Requires-Dist: google-auth>=2.0.0; extra == "cloudtasks"
+Provides-Extra: sqs
+Requires-Dist: boto3>=1.28; extra == "sqs"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0; extra == "dev"
 Requires-Dist: pytest-django>=4.5; extra == "dev"
@@ -113,6 +115,10 @@ The minimum database versions are the ones Django itself requires, and Django
 
 ```bash
 pip install django-database-task
+
+# With a broker (see Task Brokers)
+pip install django-database-task[cloudtasks]
+pip install django-database-task[sqs]
 ```
 
 ## Quick Start
@@ -282,12 +288,46 @@ python manage.py run_database_tasks [options]
 | `--continuous` | Keep polling even when no tasks |
 | `--interval` | Polling interval in seconds (default: 5) |
 | `--max-tasks` | Maximum number of tasks to process (0=unlimited) |
+| `--source` | Where to look for tasks: `auto` (default), `db`, `broker` or `both`. See [Task sources](#task-sources) |
+| `--wait-time` | Seconds to wait for a broker message before looking again (default: 20) |
+| `--max-messages` | Maximum number of broker messages to receive at a time (default: 1) |
 | `--shutdown-timeout` | Maximum seconds to wait for the running task after `SIGTERM`/`SIGINT` before forcing exit (0=wait indefinitely, default: 0) |
 | `--no-graceful-shutdown` | Do not install signal handlers (terminate immediately, even while a task is running) |
 | `--verbosity` | Output level: `0` silent (errors only), `1` normal (default), `2` also print an idle heartbeat dot per poll |
 
 See [Graceful Shutdown](#graceful-shutdown) for details.
 
+#### Task sources
+
+By default the worker polls the database, which is what it has always done.
+When the backend has a [broker](#task-brokers) a worker can receive from — a
+`PullBroker` — the same command also receives from it, without any change to
+how the command is run.
+
+| `--source` | Behaviour |
+|------------|-----------|
+| `auto` | `both` when the backend has a `PullBroker`, `db` otherwise. The default |
+| `db` | Poll the database only. What the command did before 0.4 |
+| `broker` | Receive from the broker only |
+| `both` | Receive from the broker, and fall back to the database when it is empty |
+
+`both` is the useful combination for a broker that cannot hold a task
+indefinitely. A broker with a delivery delay limit — SQS caps it at 15 minutes
+— cannot carry a task deferred further out than that, so those stay in the
+database until they are due, and the database sweep is what picks them up. It
+is also what recovers tasks the broker never accepted, since a broker failure
+during `enqueue()` is logged and swallowed.
+
+While receiving from a broker, `--wait-time` replaces `--interval` as the idle
+wait: the broker's own wait for a message is the pause, so a message wakes the
+worker as soon as it arrives. `SIGTERM` is still honoured — see
+[Graceful Shutdown](#graceful-shutdown).
+
+A message is acknowledged whenever redelivering it would not help: the task
+ran (whether it succeeded or failed), it no longer exists, or another worker
+already holds it. If the worker itself cannot run the task, the message is
+returned to the broker instead, to be delivered again.
+
 #### Output verbosity
 
 At the default verbosity the worker only prints the startup banner and one
@@ -815,6 +855,186 @@ urlpatterns = [
 ]
 ```
 
+### Backend authentication handlers
+
+Instead of wrapping each view, the backend can supply authentication handlers
+that every endpoint applies automatically. Configure them with the
+`AUTH_HANDLERS` option:
+
+```python
+# settings.py
+TASKS = {
+    "default": {
+        "BACKEND": "django_database_task.backends.DatabaseTaskBackend",
+        "OPTIONS": {
+            "AUTH_HANDLERS": [
+                "django_database_task.auth.SharedSecretAuth",
+            ],
+            "AUTH_HANDLER_OPTIONS": {
+                # Read the token from settings.TASK_API_TOKEN
+                "TOKEN_SETTING": "TASK_API_TOKEN",
+            },
+        },
+    },
+}
+```
+
+```bash
+curl -X POST https://example.com/tasks/run/ \
+  -H "Authorization: Bearer $TASK_API_TOKEN"
+```
+
+**A request is accepted as soon as one handler accepts it.** This lets the
+service that calls the endpoints (Cloud Tasks, for example) and an external
+cron job authenticate differently on the same endpoint:
+
+```python
+TASKS = {
+    "default": {
+        "BACKEND": "django_database_task.cloudtasks.CloudTasksDatabaseBackend",
+        "OPTIONS": {
+            # Cloud Tasks calls /tasks/execute/<id>/ with an OIDC token
+            "OIDC_SERVICE_ACCOUNT_EMAIL": "sa@my-project.iam.gserviceaccount.com",
+            # An external cron job calls /tasks/run/ with a shared secret
+            "AUTH_HANDLERS": [
+                {
+                    "HANDLER": "django_database_task.auth.SharedSecretAuth",
+                    "OPTIONS": {"TOKEN_SETTING": "TASK_CRON_TOKEN"},
+                    "ENDPOINTS": ["run", "run_one", "status", "purge"],
+                },
+            ],
+        },
+    },
+}
+```
+
+`ENDPOINTS` limits a handler to some of the endpoints; omit it to apply the
+handler everywhere. The valid names are `run`, `run_one`, `status`, `execute`
+and `purge`.
+
+#### Bundled handlers
+
+| Handler | Description |
+|---------|-------------|
+| `SharedSecretAuth` | Compares a token in a header. Options: `TOKEN` / `TOKEN_SETTING` / `TOKEN_ENV`, `HEADER` (default `Authorization`), `SCHEME` (default `Bearer`) |
+| `HMACAuth` | Verifies a signature with a timestamp, rejecting replays. Options: `SECRET` / `SECRET_SETTING` / `SECRET_ENV`, `HEADER` (default `X-Task-Signature`), `TIMESTAMP_HEADER` (default `X-Task-Timestamp`), `MAX_AGE` (default `300`), `ALGORITHM` (default `sha256`) |
+| `StaffOnlyAuth` | Accepts a logged in staff user. Requires `AuthenticationMiddleware` |
+
+Prefer `TOKEN_SETTING` / `TOKEN_ENV` over writing the secret into `OPTIONS`.
+
+Callers sign a request for `HMACAuth` with `build_signature()`:
+
+```python
+import time
+import requests
+from django_database_task.auth import build_signature
+
+timestamp = str(int(time.time()))
+body = b'{"max_tasks": 10}'
+signature = build_signature(SECRET, timestamp, "POST", "/tasks/run/", body)
+
+requests.post(
+    "https://example.com/tasks/run/",
+    data=body,
+    headers={
+        "Content-Type": "application/json",
+        "X-Task-Signature": signature,
+        "X-Task-Timestamp": timestamp,
+    },
+)
+```
+
+#### Custom handlers
+
+A handler is any callable that takes a request and returns `None` to accept it
+or a response to reject it. Put one in `AUTH_HANDLERS`, or override
+`get_auth_handlers()` on a backend subclass:
+
+```python
+from django.http import JsonResponse
+from django_database_task.backends import DatabaseTaskBackend
+
+
+def allow_internal_network(request):
+    if request.META.get("REMOTE_ADDR", "").startswith("10."):
+        return None
+    return JsonResponse({"error": "Forbidden"}, status=403)
+
+
+class MyBackend(DatabaseTaskBackend):
+    def get_auth_handlers(self, endpoint=None):
+        return [allow_internal_network, *super().get_auth_handlers(endpoint)]
+```
+
+> **Deprecated:** the single-handler `get_auth_handler()` still works in 0.4
+> but is removed in 0.5. Override `get_auth_handlers()` instead.
+
+## Task Brokers
+
+A broker notifies an external service whenever a task is saved, so that
+service can trigger its execution. The database stays the source of truth: a
+broker only ever carries a task id, never the arguments or the state.
+
+Without a broker — the default — tasks are picked up by `run_database_tasks`
+or the [HTTP endpoints](#http-endpoints-optional). With one, the two are still
+available and become the **fallback when the broker is down**: a broker
+failure is logged and the task is left `READY` in the database, so the next
+worker run or endpoint call picks it up.
+
+Two brokers are bundled. Each has a backend that attaches it, so naming the
+backend is all a project has to do:
+
+| Broker | Backend | Shape |
+|--------|---------|-------|
+| [Cloud Tasks](#google-cloud-tasks-integration) | `django_database_task.cloudtasks.CloudTasksDatabaseBackend` | Push: calls an [HTTP endpoint](#http-endpoints-optional) of your app |
+| [Amazon SQS](#amazon-sqs-integration) | `django_database_task.sqs.SQSDatabaseBackend` | Pull: a worker receives from the queue |
+
+```python
+TASKS = {
+    "default": {
+        "BACKEND": "django_database_task.cloudtasks.CloudTasksDatabaseBackend",
+    },
+}
+```
+
+### Custom brokers
+
+A project can attach its own broker to the plain backend with the `BROKER`
+option:
+
+```python
+TASKS = {
+    "default": {
+        "BACKEND": "django_database_task.backends.DatabaseTaskBackend",
+        "OPTIONS": {"BROKER": "myproject.brokers.MyBroker"},
+    },
+}
+```
+
+The broker receives the backend and its whole `OPTIONS` dict, so it decides
+which options it reads:
+
+```python
+from django_database_task.brokers import HTTPPushBroker
+
+
+class MyBroker(HTTPPushBroker):
+    def enqueue(self, task_result):
+        url = self.get_handler_url(task_result.id)
+        queue = self.resolve_queue(task_result.task.queue_name)
+        my_service.publish(queue, url)
+
+    def get_auth_handlers(self, endpoint=None):
+        # Verify the credentials my_service sends back to the endpoints.
+        return [verify_my_service]
+```
+
+| Base class | Use for |
+|------------|---------|
+| `TaskBroker` | Anything else |
+| `HTTPPushBroker` | Services that call an HTTP endpoint of your app (Cloud Tasks). Provides `get_handler_url()`, `TASK_HANDLER_URL` and `TASK_HANDLER_PATH` |
+| `PullBroker` | Services a worker polls. Defines `receive()`, `ack()` and `nack()` |
+
 ## Google Cloud Tasks Integration
 
 For serverless environments like Google App Engine or Cloud Run, you can use the Cloud Tasks backend to automatically create Cloud Tasks when tasks are enqueued.
@@ -935,7 +1155,9 @@ TASKS = {
 
 ### OIDC Authentication
 
-When `OIDC_SERVICE_ACCOUNT_EMAIL` is configured, Cloud Tasks will send OIDC tokens with each request. The backend automatically verifies these tokens on the `/tasks/execute/` and `/tasks/purge/` endpoints.
+When `OIDC_SERVICE_ACCOUNT_EMAIL` is configured, Cloud Tasks will send OIDC tokens with each request. The backend automatically verifies these tokens on every task endpoint.
+
+To let another caller — an external cron job, for example — reach the endpoints with its own credentials, add handlers with the `AUTH_HANDLERS` option. A request is accepted as soon as one handler accepts it. See [Backend authentication handlers](#backend-authentication-handlers).
 
 #### Required IAM Roles
 
@@ -1024,6 +1246,161 @@ elif is_app_engine():
     print(f"Running on App Engine in project {detect_gcp_project()}")
 ```
 
+## Amazon SQS Integration
+
+Send a message to SQS whenever a task is saved, and let a worker receive those
+messages. Unlike Cloud Tasks, SQS is a **pull** broker: nothing calls your
+application, so there is no HTTP endpoint to expose and nothing to
+authenticate.
+
+### Installation
+
+```bash
+pip install django-database-task[sqs]
+```
+
+### Quick Setup
+
+```python
+# settings.py
+TASKS = {
+    "default": {
+        "BACKEND": "django_database_task.sqs.SQSDatabaseBackend",
+        "QUEUES": [],  # Allow all queue names
+    },
+}
+```
+
+```bash
+python manage.py run_database_tasks --continuous
+```
+
+That is the same worker command as always. With an SQS broker configured it
+receives from the queue and sweeps the database, because `--source` defaults
+to `auto`. See [Task sources](#task-sources).
+
+Credentials come from the usual boto3 chain: the instance or task role, the
+environment, or `~/.aws/credentials`. The task needs `sqs:SendMessage`,
+`sqs:ReceiveMessage`, `sqs:DeleteMessage`, `sqs:ChangeMessageVisibility` and,
+unless you set `SQS_QUEUE_URL_TEMPLATE`, `sqs:GetQueueUrl`.
+
+### How It Works
+
+```mermaid
+sequenceDiagram
+    participant App as Application
+    participant Backend as SQSDatabaseBackend
+    participant DB as Database
+    participant SQS as Amazon SQS
+    participant Worker as Worker Process
+
+    Note over App,Worker: Task Enqueue
+    App->>Backend: task.enqueue(args, kwargs)
+    Backend->>DB: INSERT task (status=READY)
+    DB-->>Backend: Task ID
+    alt No run_after, or within 15 minutes
+        Backend->>SQS: SendMessage (task_id only,<br/>DelaySeconds)
+        SQS-->>Backend: MessageId
+    else Deferred beyond the SQS delay limit
+        Note over Backend,SQS: Not sent. The task waits in the<br/>database for the sweep below
+    end
+    Backend-->>App: TaskResult (id, status=READY)
+
+    Note over App,Worker: Task Execution (the worker receives)
+    loop run_database_tasks --continuous
+        Worker->>SQS: ReceiveMessage (long poll)
+        alt A message is waiting
+            SQS-->>Worker: task_id + ReceiptHandle
+            Worker->>DB: SELECT FOR UPDATE SKIP LOCKED<br/>(id=task_id, status=READY)
+            Worker->>DB: UPDATE status=RUNNING
+            Worker->>Worker: Execute task function
+            Worker->>DB: UPDATE status=SUCCESSFUL / FAILED
+            Worker->>SQS: DeleteMessage (only now)
+        else The queue is empty
+            Worker->>DB: SELECT FOR UPDATE SKIP LOCKED<br/>(status=READY, run_after <= now)
+            Worker->>Worker: Execute task function
+            Worker->>DB: UPDATE status=SUCCESSFUL / FAILED
+        end
+    end
+```
+
+The message carries only the task id, the same as with Cloud Tasks. What that
+buys here:
+
+- **The message is deleted after the task finishes**, not when it is received.
+  A worker that dies mid-task leaves the message to reappear once the
+  visibility timeout expires, so nothing is lost. Should it be delivered twice
+  anyway, the `READY` check and the row lock mean only one worker runs it
+- **The database sweep is the other half of the worker.** It runs tasks SQS
+  could not carry — anything deferred past 15 minutes — along with anything the
+  broker never accepted, since a `SendMessage` failure is logged and swallowed
+  rather than losing the task
+- **Nothing calls the application**, so there is no endpoint to expose and no
+  credentials for SQS to present, unlike the push model Cloud Tasks uses
+
+### Options
+
+| Option | Description |
+|--------|-------------|
+| `AWS_REGION` | Region. Detected from `AWS_REGION` or `AWS_DEFAULT_REGION` when unset |
+| `SQS_QUEUE_URL_TEMPLATE` | Queue URL with a `{queue_name}` placeholder. Set it to skip the `GetQueueUrl` call |
+| `SQS_ENDPOINT_URL` | Endpoint override, for LocalStack |
+| `VISIBILITY_TIMEOUT` | Seconds a received message stays hidden. Leave unset to use the queue's own setting |
+| `MAX_DELAY_SECONDS` | Largest delay to put on a message (default: 900, the SQS limit) |
+
+### Queues
+
+The SQS queue name is the task's `queue_name` attribute, the same as with
+Cloud Tasks:
+
+```python
+@task(queue_name="ranking")
+def rebuild_ranking(tenant_id):
+    ...
+# → sent to the "ranking" SQS queue
+```
+
+Run one worker per queue with `--queue`:
+
+```bash
+python manage.py run_database_tasks --queue ranking --continuous
+```
+
+Use standard queues, not FIFO ones. Duplicate delivery is already handled by
+the task status and `SELECT FOR UPDATE SKIP LOCKED`, and ordering does not
+apply to independent tasks.
+
+Set the queue's visibility timeout to more than your longest task, or a second
+worker will start the same task before the first one finishes. Attach a
+dead letter queue with a `maxReceiveCount` to catch messages that keep coming
+back.
+
+### Deferred tasks
+
+**SQS cannot hold a message for longer than 15 minutes.** A task deferred
+further out than that is not sent to the queue at all:
+
+```python
+send_report.using(run_after=timezone.now() + timedelta(hours=3)).enqueue()
+```
+
+It stays `READY` in the database, and the database sweep the worker already
+performs runs it once it is due. This is why `--source` resolves to `both`
+rather than `broker`, and why the worker should be left running with
+`--continuous`. The same sweep recovers tasks SQS never accepted, since a
+broker failure during `enqueue()` is logged and swallowed.
+
+### Serverless
+
+On Lambda or App Runner, where no worker process can be kept running, put an
+HTTP push in front of the existing
+[`/tasks/execute/<task_id>/` endpoint](#http-endpoints-optional) instead —
+EventBridge Pipes with an SQS source and an API destination target needs no
+code of its own. Authenticate it with the
+[bundled handlers](#backend-authentication-handlers): an EventBridge
+connection sends an API key or basic credentials, which `SharedSecretAuth`
+verifies.
+
 ## Django Admin
 
 The package includes a Django Admin integration to view and manage tasks:
@@ -1047,6 +1424,11 @@ These actions are useful for:
 - Retrying failed tasks after fixing issues
 - Testing task execution during development
 
+## Contributing
+
+Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md)
+for how to set up a development environment, run the tests and add a broker.
+
 ## License
 
 MIT License - see [LICENSE](LICENSE) for details.