django-lambda-tasks 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- django_lambda_tasks-0.1.0/.github/workflows/ci.yml +41 -0
- django_lambda_tasks-0.1.0/.github/workflows/release.yml +51 -0
- django_lambda_tasks-0.1.0/.gitignore +17 -0
- django_lambda_tasks-0.1.0/.kiro/specs/deferred-task-enqueue/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/deferred-task-enqueue/design.md +298 -0
- django_lambda_tasks-0.1.0/.kiro/specs/deferred-task-enqueue/requirements.md +101 -0
- django_lambda_tasks-0.1.0/.kiro/specs/deferred-task-enqueue/tasks.md +185 -0
- django_lambda_tasks-0.1.0/.kiro/specs/eager-mode-example-app/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/eager-mode-example-app/design.md +335 -0
- django_lambda_tasks-0.1.0/.kiro/specs/eager-mode-example-app/requirements.md +65 -0
- django_lambda_tasks-0.1.0/.kiro/specs/eager-mode-example-app/tasks.md +94 -0
- django_lambda_tasks-0.1.0/.kiro/specs/ignore-errors-decorator-option/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/ignore-errors-decorator-option/design.md +290 -0
- django_lambda_tasks-0.1.0/.kiro/specs/ignore-errors-decorator-option/requirements.md +81 -0
- django_lambda_tasks-0.1.0/.kiro/specs/ignore-errors-decorator-option/tasks.md +83 -0
- django_lambda_tasks-0.1.0/.kiro/specs/import-string-task-resolution/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/import-string-task-resolution/design.md +196 -0
- django_lambda_tasks-0.1.0/.kiro/specs/import-string-task-resolution/requirements.md +95 -0
- django_lambda_tasks-0.1.0/.kiro/specs/import-string-task-resolution/tasks.md +39 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks/design.md +521 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks/requirements.md +158 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks/tasks.md +230 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks-bugfix/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks-bugfix/bugfix.md +227 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks-bugfix/design.md +242 -0
- django_lambda_tasks-0.1.0/.kiro/specs/rse-background-tasks-bugfix/tasks.md +48 -0
- django_lambda_tasks-0.1.0/.kiro/specs/task-retry/.config.kiro +1 -0
- django_lambda_tasks-0.1.0/.kiro/specs/task-retry/design.md +297 -0
- django_lambda_tasks-0.1.0/.kiro/specs/task-retry/requirements.md +81 -0
- django_lambda_tasks-0.1.0/.kiro/specs/task-retry/tasks.md +139 -0
- django_lambda_tasks-0.1.0/.kiro/steering/product.md +190 -0
- django_lambda_tasks-0.1.0/.kiro/steering/structure.md +54 -0
- django_lambda_tasks-0.1.0/.kiro/steering/tech.md +54 -0
- django_lambda_tasks-0.1.0/.pre-commit-config.yaml +29 -0
- django_lambda_tasks-0.1.0/.vscode/settings.json +3 -0
- django_lambda_tasks-0.1.0/PKG-INFO +462 -0
- django_lambda_tasks-0.1.0/README.md +452 -0
- django_lambda_tasks-0.1.0/example/README.md +42 -0
- django_lambda_tasks-0.1.0/example/example_app/__init__.py +0 -0
- django_lambda_tasks-0.1.0/example/example_app/apps.py +6 -0
- django_lambda_tasks-0.1.0/example/example_app/tasks.py +6 -0
- django_lambda_tasks-0.1.0/example/example_app/urls.py +6 -0
- django_lambda_tasks-0.1.0/example/example_app/views.py +8 -0
- django_lambda_tasks-0.1.0/example/example_project/__init__.py +0 -0
- django_lambda_tasks-0.1.0/example/example_project/settings.py +68 -0
- django_lambda_tasks-0.1.0/example/example_project/urls.py +7 -0
- django_lambda_tasks-0.1.0/example/example_project/wsgi.py +9 -0
- django_lambda_tasks-0.1.0/example/manage.py +22 -0
- django_lambda_tasks-0.1.0/lambda_tasks/__init__.py +0 -0
- django_lambda_tasks-0.1.0/lambda_tasks/admin.py +42 -0
- django_lambda_tasks-0.1.0/lambda_tasks/apps.py +8 -0
- django_lambda_tasks-0.1.0/lambda_tasks/decorators.py +359 -0
- django_lambda_tasks-0.1.0/lambda_tasks/handler.py +50 -0
- django_lambda_tasks-0.1.0/lambda_tasks/logging.py +34 -0
- django_lambda_tasks-0.1.0/lambda_tasks/migrations/0001_initial.py +73 -0
- django_lambda_tasks-0.1.0/lambda_tasks/migrations/__init__.py +0 -0
- django_lambda_tasks-0.1.0/lambda_tasks/models.py +240 -0
- django_lambda_tasks-0.1.0/lambda_tasks/secret_loader.py +183 -0
- django_lambda_tasks-0.1.0/lambda_tasks/settings.py +76 -0
- django_lambda_tasks-0.1.0/lambda_tasks/timeouts.py +78 -0
- django_lambda_tasks-0.1.0/pyproject.toml +45 -0
- django_lambda_tasks-0.1.0/tests/conftest.py +2 -0
- django_lambda_tasks-0.1.0/tests/settings.py +15 -0
- django_lambda_tasks-0.1.0/tests/test_admin.py +20 -0
- django_lambda_tasks-0.1.0/tests/test_decorator.py +934 -0
- django_lambda_tasks-0.1.0/tests/test_decorators.py +141 -0
- django_lambda_tasks-0.1.0/tests/test_deferred_enqueue.py +142 -0
- django_lambda_tasks-0.1.0/tests/test_handler.py +302 -0
- django_lambda_tasks-0.1.0/tests/test_kwargs_only.py +59 -0
- django_lambda_tasks-0.1.0/tests/test_logging.py +124 -0
- django_lambda_tasks-0.1.0/tests/test_models.py +1503 -0
- django_lambda_tasks-0.1.0/tests/test_secret_loader.py +317 -0
- django_lambda_tasks-0.1.0/tests/test_serializer.py +206 -0
- django_lambda_tasks-0.1.0/tests/test_settings.py +141 -0
- django_lambda_tasks-0.1.0/tests/test_timeouts.py +224 -0
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
pull_request:
|
|
6
|
+
workflow_call:
|
|
7
|
+
|
|
8
|
+
env:
|
|
9
|
+
MINIMUM_PYTHON_VERSION: '3.10'
|
|
10
|
+
|
|
11
|
+
concurrency:
|
|
12
|
+
group: ${{ github.head_ref || github.run_id }}
|
|
13
|
+
cancel-in-progress: true
|
|
14
|
+
|
|
15
|
+
jobs:
|
|
16
|
+
|
|
17
|
+
precommit:
|
|
18
|
+
runs-on: ubuntu-latest
|
|
19
|
+
steps:
|
|
20
|
+
- uses: actions/checkout@v4
|
|
21
|
+
- name: Install uv and set the python version
|
|
22
|
+
uses: astral-sh/setup-uv@v6
|
|
23
|
+
with:
|
|
24
|
+
python-version: ${{ env.MINIMUM_PYTHON_VERSION }}
|
|
25
|
+
- name: Run static code inspections
|
|
26
|
+
run: uv run pre-commit run --all-files
|
|
27
|
+
|
|
28
|
+
build:
|
|
29
|
+
runs-on: ubuntu-latest
|
|
30
|
+
strategy:
|
|
31
|
+
matrix:
|
|
32
|
+
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
|
|
33
|
+
steps:
|
|
34
|
+
- uses: actions/checkout@v4
|
|
35
|
+
- name: Install uv and set the python version
|
|
36
|
+
uses: astral-sh/setup-uv@v6
|
|
37
|
+
with:
|
|
38
|
+
python-version: ${{ matrix.python-version }}
|
|
39
|
+
- name: Run uv
|
|
40
|
+
run: |
|
|
41
|
+
uv run pytest --cov-branch --cov-report term-missing --cov=tests/ --cov=lambda_tasks/
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
name: Publish Package
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
release:
|
|
5
|
+
types:
|
|
6
|
+
- published
|
|
7
|
+
|
|
8
|
+
env:
|
|
9
|
+
MINIMUM_PYTHON_VERSION: '3.10'
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
ci:
|
|
13
|
+
uses: ./.github/workflows/ci.yml
|
|
14
|
+
|
|
15
|
+
build-for-pypi:
|
|
16
|
+
needs: ci
|
|
17
|
+
runs-on: ubuntu-latest
|
|
18
|
+
permissions:
|
|
19
|
+
contents: read
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v4
|
|
22
|
+
- name: Install Python ${{ env.MINIMUM_PYTHON_VERSION }}
|
|
23
|
+
uses: astral-sh/setup-uv@v6
|
|
24
|
+
with:
|
|
25
|
+
python-version: ${{ env.MINIMUM_PYTHON_VERSION }}
|
|
26
|
+
- name: Build the package
|
|
27
|
+
run: uv build
|
|
28
|
+
- name: 'Upload Artifact'
|
|
29
|
+
uses: actions/upload-artifact@v4
|
|
30
|
+
with:
|
|
31
|
+
name: python-dist
|
|
32
|
+
path: dist/
|
|
33
|
+
if-no-files-found: error
|
|
34
|
+
retention-days: 1
|
|
35
|
+
|
|
36
|
+
deploy-to-pypi:
|
|
37
|
+
# IMPORTANT: use a separate job for this as each step has access to the tokens
|
|
38
|
+
needs: build-for-pypi
|
|
39
|
+
runs-on: ubuntu-latest
|
|
40
|
+
environment: pypi
|
|
41
|
+
permissions:
|
|
42
|
+
# IMPORTANT: this permission is mandatory for Trusted Publishing
|
|
43
|
+
id-token: write
|
|
44
|
+
steps:
|
|
45
|
+
- name: Download build artifacts
|
|
46
|
+
uses: actions/download-artifact@v4
|
|
47
|
+
with:
|
|
48
|
+
name: python-dist
|
|
49
|
+
path: dist
|
|
50
|
+
- name: Publish package distributions
|
|
51
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"specId": "607aa163-ef89-4b0c-b886-6b6d0a53b02b", "workflowType": "requirements-first", "specType": "feature"}
|
|
@@ -0,0 +1,298 @@
|
|
|
1
|
+
# Design Document: deferred-task-enqueue
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
This feature adds deferred task enqueuing to `django-lambda-tasks`. A task invocation can be
|
|
6
|
+
serialized into a plain JSON-compatible dict (suitable for a Django `JSONField`), stored, and
|
|
7
|
+
later enqueued from any context — a management command, a scheduled job, another task, etc.
|
|
8
|
+
|
|
9
|
+
`SQSLambdaSQSLambdaTaskMessage` wraps a `SQSLambdaTaskMessage` as an attribute alongside `delay` and `queue`. This
|
|
10
|
+
avoids field duplication and means the deferred path can serialize the embedded `SQSLambdaTaskMessage`
|
|
11
|
+
directly — no changes to `serialize()` or `enqueue()` are needed.
|
|
12
|
+
|
|
13
|
+
`on_commit` and `enqueue_from_json` share a private `_do_enqueue` helper on
|
|
14
|
+
`LambdaTaskWrapper`. `enqueue_deferred` in `enqueuer.py` calls the same lower-level
|
|
15
|
+
`_send_message` function that `enqueue()` already delegates to after serialization.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Architecture
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
Developer code
|
|
23
|
+
│
|
|
24
|
+
├─ wrapper.to_json(**kwargs)
|
|
25
|
+
│ └─ SQSLambdaSQSLambdaTaskMessage(message=SQSLambdaTaskMessage(...), delay=..., queue=...)
|
|
26
|
+
│ stored as dict in JSONField
|
|
27
|
+
│
|
|
28
|
+
├─ wrapper.on_commit(**kwargs) ─┐
|
|
29
|
+
│ ├─ wrapper._do_enqueue(message, delay, queue)
|
|
30
|
+
└─ wrapper.enqueue_from_json(dict) ─┘ │
|
|
31
|
+
└─ enqueuer._send_message(body, delay, queue)
|
|
32
|
+
│
|
|
33
|
+
enqueue_deferred(dict) ─────────────────────────────────────►│
|
|
34
|
+
│
|
|
35
|
+
┌───────────┴───────────┐
|
|
36
|
+
EAGER=True EAGER=False
|
|
37
|
+
execute_message() boto3 SQS send
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Components and Interfaces
|
|
43
|
+
|
|
44
|
+
### `serializer.py` — `SQSLambdaSQSLambdaTaskMessage`
|
|
45
|
+
|
|
46
|
+
New Pydantic model added alongside `SQSLambdaTaskMessage`. No field duplication — `SQSLambdaTaskMessage` is stored
|
|
47
|
+
as a nested attribute:
|
|
48
|
+
|
|
49
|
+
```python
|
|
50
|
+
class SQSLambdaSQSLambdaTaskMessage(BaseModel):
|
|
51
|
+
model_config = ConfigDict(extra="forbid")
|
|
52
|
+
message: SQSLambdaTaskMessage
|
|
53
|
+
delay: int
|
|
54
|
+
queue: str
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`message` is a full `SQSLambdaTaskMessage` (with `task_name`, `invocation_id`, `kwargs`) generated at
|
|
58
|
+
serialization time by `to_json`. The `invocation_id` is stable — the same stored dict always
|
|
59
|
+
carries the same ID, enabling deduplication via `TaskRecord.get_or_create`.
|
|
60
|
+
|
|
61
|
+
`serialize()` and `deserialize()` are unchanged.
|
|
62
|
+
|
|
63
|
+
### `enqueuer.py` — refactor to extract `_send_message`
|
|
64
|
+
|
|
65
|
+
`enqueue()` is refactored to extract a private `_send_message(body, delay, queue)` helper that
|
|
66
|
+
owns the SQS send (and eager execution). `enqueue()` continues to call `serialize()` then
|
|
67
|
+
`_send_message()`. `enqueue_deferred` calls `_send_message()` directly with the pre-serialized
|
|
68
|
+
body from the embedded `SQSLambdaTaskMessage`:
|
|
69
|
+
|
|
70
|
+
```python
|
|
71
|
+
def _send_message(*, body: str, delay: int, queue: str) -> None:
|
|
72
|
+
conf = LambdaTasksSettings()
|
|
73
|
+
if conf.EAGER:
|
|
74
|
+
execute_message(message_body=body)
|
|
75
|
+
else:
|
|
76
|
+
queue_url = conf.QUEUES[queue] # raises ImproperlyConfigured if missing
|
|
77
|
+
boto3.client("sqs").send_message(
|
|
78
|
+
QueueUrl=queue_url, MessageBody=body, DelaySeconds=delay
|
|
79
|
+
)
|
|
80
|
+
|
|
81
|
+
def enqueue(*, task_name, kwargs, delay, queue) -> None:
|
|
82
|
+
body = serialize(task_name=task_name, kwargs=kwargs)
|
|
83
|
+
_send_message(body=body, delay=delay, queue=queue)
|
|
84
|
+
|
|
85
|
+
def enqueue_deferred(*, deferred: dict) -> None:
|
|
86
|
+
msg = SQSLambdaSQSLambdaTaskMessage.model_validate(deferred)
|
|
87
|
+
_send_message(body=msg.message.model_dump_json(), delay=msg.delay, queue=msg.queue)
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
### `decorators.py` — `LambdaTaskWrapper`
|
|
91
|
+
|
|
92
|
+
Three additions:
|
|
93
|
+
|
|
94
|
+
**`to_json(**kwargs) -> dict`**
|
|
95
|
+
- Pops `_delay` / `_queue` overrides (same resolution logic as `on_commit`).
|
|
96
|
+
- Validates remaining kwargs against `_kwargs_model`.
|
|
97
|
+
- Builds a `SQSLambdaTaskMessage` with a fresh UUID4 `invocation_id`.
|
|
98
|
+
- Returns `SQSLambdaSQSLambdaTaskMessage(message=task_message, delay=..., queue=...).model_dump()`.
|
|
99
|
+
|
|
100
|
+
**`_do_enqueue(message: SQSLambdaTaskMessage, delay: int, queue: str) -> None`** (private)
|
|
101
|
+
- Single call site for `enqueuer._send_message(body=message.model_dump_json(), delay=delay, queue=queue)`.
|
|
102
|
+
- Both `on_commit` and `enqueue_from_json` delegate here.
|
|
103
|
+
|
|
104
|
+
**`enqueue_from_json(*, data: dict) -> None`**
|
|
105
|
+
- Validates `data` against `SQSLambdaSQSLambdaTaskMessage` (raises `pydantic.ValidationError` on failure).
|
|
106
|
+
- Calls `self._do_enqueue(msg.message, msg.delay, msg.queue)`.
|
|
107
|
+
|
|
108
|
+
`on_commit` is refactored to build a `SQSLambdaTaskMessage` and call `self._do_enqueue(...)` instead of
|
|
109
|
+
calling `enqueuer.enqueue()` directly.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Data Models
|
|
114
|
+
|
|
115
|
+
### `SQSLambdaSQSLambdaTaskMessage` schema (as stored in JSONField)
|
|
116
|
+
|
|
117
|
+
```json
|
|
118
|
+
{
|
|
119
|
+
"message": {
|
|
120
|
+
"task_name": "myapp.tasks.my_task",
|
|
121
|
+
"invocation_id": "550e8400-e29b-41d4-a716-446655440000",
|
|
122
|
+
"kwargs": {"user_id": 42}
|
|
123
|
+
},
|
|
124
|
+
"delay": 0,
|
|
125
|
+
"queue": "default"
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Extra fields are forbidden at the top level. `message` is validated as a full `SQSLambdaTaskMessage`.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## Correctness Properties
|
|
134
|
+
|
|
135
|
+
### Property 1: `to_json` structural invariant
|
|
136
|
+
|
|
137
|
+
*For any* `LambdaTaskWrapper` and any valid task kwargs (with optional `_delay` / `_queue`
|
|
138
|
+
overrides), `to_json` returns a dict that round-trips through `SQSLambdaSQSLambdaTaskMessage.model_validate`
|
|
139
|
+
and contains a nested `message` dict with `task_name`, `invocation_id`, and `kwargs`, plus
|
|
140
|
+
top-level `delay` and `queue` reflecting the resolved overrides or decorator defaults.
|
|
141
|
+
|
|
142
|
+
**Validates: Requirements 1.1, 1.4, 1.5, 1.6, 1.7, 1.8**
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
### Property 2: `to_json` rejects invalid kwargs
|
|
147
|
+
|
|
148
|
+
*For any* kwargs that fail the task's declared type annotations (wrong type, missing required
|
|
149
|
+
field, or extra field), `to_json` raises `pydantic.ValidationError` and returns no dict.
|
|
150
|
+
|
|
151
|
+
**Validates: Requirements 1.2, 1.3**
|
|
152
|
+
|
|
153
|
+
---
|
|
154
|
+
|
|
155
|
+
### Property 3: `SQSLambdaSQSLambdaTaskMessage` round-trip
|
|
156
|
+
|
|
157
|
+
*For any* valid `SQSLambdaSQSLambdaTaskMessage` instance `m`, constructing a new instance from
|
|
158
|
+
`m.model_dump()` produces an object equal to `m`.
|
|
159
|
+
|
|
160
|
+
**Validates: Requirements 2.4**
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
### Property 4: `SQSLambdaSQSLambdaTaskMessage` rejects invalid and extra fields
|
|
165
|
+
|
|
166
|
+
*For any* dict that is missing a required field, has a field with the wrong type, or contains
|
|
167
|
+
extra top-level keys, `SQSLambdaSQSLambdaTaskMessage.model_validate` raises `pydantic.ValidationError`.
|
|
168
|
+
|
|
169
|
+
**Validates: Requirements 2.2, 2.3**
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
### Property 5: `enqueue_from_json` passes stable `invocation_id` to `_send_message`
|
|
174
|
+
|
|
175
|
+
*For any* valid deferred dict `d`, calling `enqueue_from_json(d)` passes the `invocation_id`
|
|
176
|
+
from `d["message"]` to `_send_message` unchanged, so two calls with the same dict produce the
|
|
177
|
+
same `invocation_id` in the SQS body.
|
|
178
|
+
|
|
179
|
+
**Validates: Requirements 3.4, 3.5, 6.1**
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
### Property 6: `enqueue_from_json` rejects invalid dicts
|
|
184
|
+
|
|
185
|
+
*For any* dict that fails `SQSLambdaSQSLambdaTaskMessage` validation, `enqueue_from_json` raises
|
|
186
|
+
`pydantic.ValidationError` without calling `_send_message`.
|
|
187
|
+
|
|
188
|
+
**Validates: Requirements 3.2, 3.3**
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
### Property 7: `enqueue_from_json` eager mode executes synchronously
|
|
193
|
+
|
|
194
|
+
*For any* valid deferred dict, when `LAMBDA_TASKS_EAGER=True`, `enqueue_from_json` executes
|
|
195
|
+
the task in-process without sending to SQS, identical to the eager behaviour of `on_commit`.
|
|
196
|
+
|
|
197
|
+
**Validates: Requirements 3.6**
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
### Property 8: `on_commit` and `enqueue_from_json` share the same send path
|
|
202
|
+
|
|
203
|
+
*For any* valid task kwargs, both `on_commit(**kwargs)` and `enqueue_from_json(to_json(**kwargs))`
|
|
204
|
+
call `_send_message` with the same `task_name`, `kwargs`, `delay`, and `queue`.
|
|
205
|
+
|
|
206
|
+
**Validates: Requirements 4.1, 4.2**
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
### Property 9: `enqueue_deferred` passes all fields including stable `invocation_id`
|
|
211
|
+
|
|
212
|
+
*For any* valid `SQSLambdaSQSLambdaTaskMessage` dict `d`, `enqueue_deferred(d)` calls `_send_message` with
|
|
213
|
+
a body whose `task_name`, `invocation_id`, and `kwargs` match `d["message"]`, and `delay` and
|
|
214
|
+
`queue` match the top-level fields of `d`.
|
|
215
|
+
|
|
216
|
+
**Validates: Requirements 5.1, 5.3, 5.4, 6.2**
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
### Property 10: `enqueue_deferred` rejects invalid dicts
|
|
221
|
+
|
|
222
|
+
*For any* dict that fails `SQSLambdaSQSLambdaTaskMessage` validation, `enqueue_deferred` raises
|
|
223
|
+
`pydantic.ValidationError` without calling `_send_message`.
|
|
224
|
+
|
|
225
|
+
**Validates: Requirements 5.2**
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
### Property 11: `enqueue_deferred` eager mode executes synchronously
|
|
230
|
+
|
|
231
|
+
*For any* valid deferred dict, when `LAMBDA_TASKS_EAGER=True`, `enqueue_deferred` executes
|
|
232
|
+
the task in-process without sending to SQS.
|
|
233
|
+
|
|
234
|
+
**Validates: Requirements 5.5**
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Error Handling
|
|
239
|
+
|
|
240
|
+
| Situation | Behaviour |
|
|
241
|
+
|---|---|
|
|
242
|
+
| `to_json` called with wrong-type or missing kwargs | `pydantic.ValidationError` raised; no dict returned |
|
|
243
|
+
| `enqueue_from_json` called with invalid dict | `pydantic.ValidationError` raised; `_send_message` not called |
|
|
244
|
+
| `enqueue_deferred` called with invalid dict | `pydantic.ValidationError` raised; `_send_message` not called |
|
|
245
|
+
| Queue name not in `LAMBDA_TASKS_QUEUES` | `ImproperlyConfigured` raised by `_send_message` (same as current `enqueue()` behaviour) |
|
|
246
|
+
| boto3 / SQS error | Exception propagates from `_send_message` (unchanged) |
|
|
247
|
+
|
|
248
|
+
No new exception types are introduced.
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
|
|
252
|
+
## Testing Strategy
|
|
253
|
+
|
|
254
|
+
Tests follow red/green TDD: failing tests are written first, then the implementation makes them
|
|
255
|
+
pass. One new test file: `tests/test_deferred_enqueue.py`. Existing files `test_decorator.py`,
|
|
256
|
+
`test_enqueuer.py`, and `test_serializer.py` receive targeted additions for the touched code.
|
|
257
|
+
|
|
258
|
+
### Unit tests
|
|
259
|
+
|
|
260
|
+
- `to_json` returns a dict that validates as `SQSLambdaSQSLambdaTaskMessage` with correct fields
|
|
261
|
+
- `to_json` uses decorator defaults when `_delay` / `_queue` are omitted
|
|
262
|
+
- `to_json` uses call-site overrides when `_delay` / `_queue` are provided
|
|
263
|
+
- `to_json` raises `ValidationError` for wrong-type kwargs
|
|
264
|
+
- `to_json` raises `ValidationError` for missing required kwargs
|
|
265
|
+
- `SQSLambdaSQSLambdaTaskMessage` accepts a valid dict with nested `message`
|
|
266
|
+
- `SQSLambdaSQSLambdaTaskMessage` rejects a dict missing `message`
|
|
267
|
+
- `SQSLambdaSQSLambdaTaskMessage` rejects a dict with extra top-level fields
|
|
268
|
+
- `enqueue_from_json` passes the stored `invocation_id` through to `_send_message`
|
|
269
|
+
- `enqueue_from_json` raises `ValidationError` for an invalid dict without calling `_send_message`
|
|
270
|
+
- `on_commit` and `enqueue_from_json` both call `_do_enqueue`
|
|
271
|
+
- `enqueue_deferred` calls `_send_message` with all correct fields including `invocation_id`
|
|
272
|
+
- `enqueue_deferred` raises `ValidationError` for an invalid dict without calling `_send_message`
|
|
273
|
+
- `enqueue()` still works correctly after `_send_message` extraction
|
|
274
|
+
- Eager mode: `enqueue_from_json` executes task in-process
|
|
275
|
+
- Eager mode: `enqueue_deferred` executes task in-process
|
|
276
|
+
|
|
277
|
+
### Property-based tests
|
|
278
|
+
|
|
279
|
+
Using `hypothesis` with a minimum of 100 iterations per property.
|
|
280
|
+
|
|
281
|
+
Tag format: `# Feature: deferred-task-enqueue, Property {N}: {property_text}`
|
|
282
|
+
|
|
283
|
+
| Property | Test description | Hypothesis strategy |
|
|
284
|
+
|---|---|---|
|
|
285
|
+
| P1 | `to_json` always returns a valid `SQSLambdaSQSLambdaTaskMessage` dict with correct fields | `st.fixed_dictionaries` for valid kwargs |
|
|
286
|
+
| P2 | `to_json` always raises `ValidationError` for wrong-type kwargs | `st.one_of` wrong types per field |
|
|
287
|
+
| P3 | `SQSLambdaSQSLambdaTaskMessage` round-trip | `st.builds(SQSLambdaSQSLambdaTaskMessage, ...)` |
|
|
288
|
+
| P4 | `SQSLambdaSQSLambdaTaskMessage` rejects missing/wrong-type/extra fields | `st.fixed_dictionaries` with dropped/mutated fields |
|
|
289
|
+
| P5 | `enqueue_from_json` always passes stable `invocation_id` to `_send_message` | `st.fixed_dictionaries` for valid dicts |
|
|
290
|
+
| P6 | `enqueue_from_json` always raises for invalid dicts | `st.fixed_dictionaries` with missing fields |
|
|
291
|
+
| P7 | `enqueue_from_json` with `EAGER=True` always executes in-process | `st.fixed_dictionaries` for valid dicts |
|
|
292
|
+
| P8 | `on_commit` and `enqueue_from_json` always call `_send_message` with same task/kwargs/delay/queue | `st.fixed_dictionaries` for valid kwargs |
|
|
293
|
+
| P9 | `enqueue_deferred` always passes all fields including `invocation_id` to `_send_message` | `st.builds(SQSLambdaSQSLambdaTaskMessage, ...)` |
|
|
294
|
+
| P10 | `enqueue_deferred` always raises for invalid dicts | `st.fixed_dictionaries` with missing fields |
|
|
295
|
+
| P11 | `enqueue_deferred` with `EAGER=True` always executes in-process | `st.builds(SQSLambdaSQSLambdaTaskMessage, ...)` |
|
|
296
|
+
|
|
297
|
+
Each property-based test must run a minimum of 100 iterations (`@settings(max_examples=100)`).
|
|
298
|
+
Each correctness property is implemented by exactly one property-based test.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# Requirements Document
|
|
2
|
+
|
|
3
|
+
## Introduction
|
|
4
|
+
|
|
5
|
+
This feature adds support for deferred task enqueuing in the `django-lambda-tasks` library. Currently, tasks can only be enqueued immediately after a transaction commits via `LambdaTaskWrapper.on_commit()`. This feature allows a task invocation to be serialized into a plain JSON-compatible dict (suitable for storage in a Django `JSONField`), and later loaded and enqueued — from any context (management command, scheduled job, another task, etc.).
|
|
6
|
+
|
|
7
|
+
`on_commit` and `enqueue_from_json` share the same underlying enqueue path. This means eager mode, queue resolution, and SQS sending all behave identically regardless of how the task was triggered. The refactor extracts a shared `_do_enqueue(task_name, kwargs, delay, queue)` helper that both methods call.
|
|
8
|
+
|
|
9
|
+
The serialized form captures the task name, kwargs, enqueue-time options (delay, queue), and a stable `invocation_id` generated at serialization time. Using a stable ID means the same stored invocation can be enqueued multiple times and the second delivery will be deduplicated by `TaskRecord.get_or_create` — the same guarantee the existing SQS path provides.
|
|
10
|
+
|
|
11
|
+
## Glossary
|
|
12
|
+
|
|
13
|
+
- **LambdaTaskWrapper**: The wrapper object produced by `@lambda_task`; exposes `__call__`, `on_commit`, `to_json`, and `enqueue_from_json` methods.
|
|
14
|
+
- **SQSLambdaTask**: A JSON-compatible dict representing a serialized task invocation, including `task_name`, `invocation_id`, `kwargs`, `delay`, and `queue`.
|
|
15
|
+
- **Enqueuer**: The `enqueuer.enqueue()` function in `enqueuer.py` that serializes a `SQSLambdaTaskMessage` and sends it to SQS (or runs eagerly).
|
|
16
|
+
- **Serializer**: The `serializer.py` module containing `SQSLambdaTaskMessage`, `serialize()`, and `deserialize()`.
|
|
17
|
+
- **JSONField**: A Django model field that stores Python dicts as JSON in the database.
|
|
18
|
+
- **SQSLambdaSQSLambdaTaskMessage**: A new Pydantic model in `serializer.py` representing the schema of a serialized deferred task invocation.
|
|
19
|
+
- **`_do_enqueue`**: A private helper on `LambdaTaskWrapper` that performs the actual enqueue call; shared by `on_commit` and `enqueue_from_json`.
|
|
20
|
+
|
|
21
|
+
## Requirements
|
|
22
|
+
|
|
23
|
+
### Requirement 1: Serialize a Task Invocation to a JSON-Compatible Dict
|
|
24
|
+
|
|
25
|
+
**User Story:** As a developer, I want to serialize a task invocation (with its kwargs and enqueue options) into a plain dict, so that I can store it in a Django `JSONField` for later enqueuing.
|
|
26
|
+
|
|
27
|
+
#### Acceptance Criteria
|
|
28
|
+
|
|
29
|
+
1. THE `LambdaTaskWrapper` SHALL expose a `to_json` method that accepts task kwargs plus optional `_delay` and `_queue` override kwargs and returns a JSON-compatible dict.
|
|
30
|
+
2. WHEN `to_json` is called, THE `LambdaTaskWrapper` SHALL validate the provided kwargs against the task's declared parameter types before producing the dict.
|
|
31
|
+
3. WHEN `to_json` is called with invalid kwargs, THE `LambdaTaskWrapper` SHALL raise a `pydantic.ValidationError`.
|
|
32
|
+
4. THE dict returned by `to_json` SHALL contain the fields `task_name`, `invocation_id`, `kwargs`, `delay`, and `queue`.
|
|
33
|
+
5. THE `invocation_id` field in the returned dict SHALL be a freshly generated UUID4 string.
|
|
34
|
+
6. THE `task_name` field in the returned dict SHALL be the fully-qualified dotted name of the wrapped function (e.g. `"myapp.tasks.my_task"`).
|
|
35
|
+
7. WHEN `_delay` is not provided to `to_json`, THE `LambdaTaskWrapper` SHALL use the decorator-level `delay` default for the `delay` field in the returned dict.
|
|
36
|
+
8. WHEN `_queue` is not provided to `to_json`, THE `LambdaTaskWrapper` SHALL use the decorator-level `queue` default for the `queue` field in the returned dict.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
### Requirement 2: Validate and Parse a Deferred Task Dict
|
|
41
|
+
|
|
42
|
+
**User Story:** As a developer, I want the library to validate a dict loaded from a `JSONField` before enqueuing it, so that corrupt or tampered data is rejected with a clear error.
|
|
43
|
+
|
|
44
|
+
#### Acceptance Criteria
|
|
45
|
+
|
|
46
|
+
1. THE `Serializer` SHALL expose a `SQSLambdaSQSLambdaTaskMessage` Pydantic model with required fields: `task_name: str`, `invocation_id: str`, `kwargs: dict`, `delay: int`, `queue: str`.
|
|
47
|
+
2. WHEN a dict is validated against `SQSLambdaSQSLambdaTaskMessage`, THE `Serializer` SHALL raise `pydantic.ValidationError` if any required field is missing or has the wrong type.
|
|
48
|
+
3. THE `SQSLambdaSQSLambdaTaskMessage` model SHALL reject extra fields not in its schema.
|
|
49
|
+
4. FOR ALL valid `SQSLambdaSQSLambdaTaskMessage` instances `m`, constructing `m` from `m.model_dump()` SHALL produce an equivalent object (round-trip property).
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
### Requirement 3: Enqueue a Deferred Task from a Stored Dict
|
|
54
|
+
|
|
55
|
+
**User Story:** As a developer, I want to load a dict from a `JSONField` and enqueue the represented task, so that I can defer task enqueuing to a later point in time (e.g. from a management command or scheduled job).
|
|
56
|
+
|
|
57
|
+
#### Acceptance Criteria
|
|
58
|
+
|
|
59
|
+
1. THE `LambdaTaskWrapper` SHALL expose an `enqueue_from_json` method that accepts a dict (as produced by `to_json`) and enqueues the task by calling the same underlying enqueue path as `on_commit`.
|
|
60
|
+
2. WHEN `enqueue_from_json` is called with a valid dict, THE `LambdaTaskWrapper` SHALL validate the dict against `SQSLambdaSQSLambdaTaskMessage` before enqueuing.
|
|
61
|
+
3. WHEN `enqueue_from_json` is called with an invalid dict, THE `LambdaTaskWrapper` SHALL raise `pydantic.ValidationError` without enqueuing.
|
|
62
|
+
4. WHEN `enqueue_from_json` is called, THE `Enqueuer` SHALL use the `invocation_id`, `delay`, and `queue` values from the dict.
|
|
63
|
+
5. WHEN the same dict is passed to `enqueue_from_json` twice, THE second call SHALL be deduplicated by `TaskRecord.get_or_create` because both calls use the same `invocation_id`.
|
|
64
|
+
6. WHEN `LAMBDA_TASKS_EAGER` is `True`, `enqueue_from_json` SHALL execute the task synchronously in-process, identical to the eager behaviour of `on_commit`.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
### Requirement 4: Shared Enqueue Path
|
|
69
|
+
|
|
70
|
+
**User Story:** As a developer, I want `on_commit` and `enqueue_from_json` to use the same underlying enqueue logic, so that eager mode, queue resolution, and SQS behaviour are always consistent between the two call sites.
|
|
71
|
+
|
|
72
|
+
#### Acceptance Criteria
|
|
73
|
+
|
|
74
|
+
1. THE `LambdaTaskWrapper` SHALL extract a private `_do_enqueue(task_name, kwargs, delay, queue)` method that contains the call to `enqueuer.enqueue()`.
|
|
75
|
+
2. BOTH `on_commit` and `enqueue_from_json` SHALL delegate to `_do_enqueue` rather than calling `enqueuer.enqueue()` directly.
|
|
76
|
+
3. ANY change to enqueue behaviour (e.g. adding a new SQS parameter) SHALL only require a change in `_do_enqueue`.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
### Requirement 5: Module-Level Standalone Enqueue Function
|
|
81
|
+
|
|
82
|
+
**User Story:** As a developer, I want a standalone function to enqueue a deferred task dict without needing a reference to the original `LambdaTaskWrapper`, so that I can enqueue from contexts where the wrapper is not easily accessible (e.g. a generic scheduler).
|
|
83
|
+
|
|
84
|
+
#### Acceptance Criteria
|
|
85
|
+
|
|
86
|
+
1. THE `Enqueuer` SHALL expose an `enqueue_deferred(deferred: dict) -> None` function that validates the dict against `SQSLambdaSQSLambdaTaskMessage` and calls `enqueuer.enqueue()`.
|
|
87
|
+
2. WHEN `enqueue_deferred` is called with an invalid dict, THE `Enqueuer` SHALL raise `pydantic.ValidationError` without calling `enqueuer.enqueue()`.
|
|
88
|
+
3. WHEN `enqueue_deferred` is called, THE `Enqueuer` SHALL use the `invocation_id`, `delay`, and `queue` values from the validated dict.
|
|
89
|
+
4. WHEN the same dict is passed to `enqueue_deferred` twice, THE second execution SHALL be deduplicated by `TaskRecord.get_or_create` because both calls use the same `invocation_id`.
|
|
90
|
+
5. WHEN `LAMBDA_TASKS_EAGER` is `True`, `enqueue_deferred` SHALL execute the task synchronously in-process, identical to the eager behaviour of `enqueue()`.
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
### Requirement 6: Round-Trip Consistency
|
|
95
|
+
|
|
96
|
+
**User Story:** As a developer, I want the serialization and enqueuing path to be consistent, so that a dict produced by `to_json` can always be enqueued without modification.
|
|
97
|
+
|
|
98
|
+
#### Acceptance Criteria
|
|
99
|
+
|
|
100
|
+
1. FOR ALL valid task invocations, calling `to_json(**kwargs)` followed by `enqueue_from_json(result)` SHALL enqueue a `SQSLambdaTaskMessage` with the same `task_name`, `invocation_id`, `kwargs`, `delay`, and `queue` as stored in the dict.
|
|
101
|
+
2. FOR ALL valid `SQSLambdaSQSLambdaTaskMessage` dicts `d`, calling `enqueue_deferred(d)` SHALL enqueue a `SQSLambdaTaskMessage` with `task_name`, `invocation_id`, `kwargs`, `delay`, and `queue` matching the fields of `d`.
|