django-lambda-tasks 0.5.0__tar.gz → 0.6.0__tar.gz

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.0}/.kiro/steering/product.md

@@ -4,37 +4,58 @@ inclusion: always
 
 # django-lambda-tasks
 
-A Django library for offloading tasks to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker.
+A Django library for offloading tasks to AWS Lambda or AWS Batch outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker or Batch container.
 
 ## Architecture
 
+### Lambda tasks
+
 ```
 View → @lambda_task.execute_on_commit() → SQS → Lambda handler → SQSLambdaTaskMessage.execute_immediately() → TaskRecord
 ```
 
+### Batch tasks
+
+```
+View → @batch_task.execute_on_commit() → SQS → Lambda handler → submit_batch_job → batch.submit_job()
+     → Container → python -m lambda_tasks.handler → handler() → execute_immediately() → TaskRecord
+```
+
+Retry path (Batch):
+```
+Container (retry_on exception) → execute_on_commit() → SQS → Lambda → submit_batch_job → new Batch job
+```
+
 Key modules:
-- `decorators.py` — `@lambda_task` decorator and `LambdaTaskWrapper`
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish or local pool submit)
+- `decorators.py` — `BaseTaskWrapper`, `LambdaTaskWrapper`, `BatchTaskWrapper`, `@lambda_task`, `@batch_task`
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish, Batch submit, or local pool submit)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution for development
-- `handler.py` — AWS Lambda entry point; cold-start init runs on first invocation (not at import time) with partial-batch failure reporting
+- `handler.py` — AWS Lambda entry point + AWS Batch container entry point (`main()`)
 - `logging.py` — `task_logger` for invocation-scoped log output
-- `settings.py` — lazy `LambdaTasksSettings` reading from Django settings
+- `settings.py` — `TaskBackend` enum, lazy `LambdaTasksSettings` reading from Django settings
 
 ## Task Definition
 
 ```python
-# Always kwargs-only — positional args raise TypeError at decoration time
+# Lambda task — max timeout 900s
 @lambda_task(retry_delay=30, soft_timeout=60, hard_timeout=120, queue="default",
              retry_on=(TransientError,),
              singleton=True, retry_singleton=True)
 def my_task(*, user_id: int, action: str) -> None:
     ...
+
+# Batch task — max timeout 3600s, for long-running or storage-heavy workloads
+@batch_task(soft_timeout=1800, hard_timeout=3500, queue="default",
+            retry_on=(TransientError,))
+def heavy_task(*, file_id: int) -> None:
+    ...
 ```
 
 - Tasks are resolved at execution time via `import_string` on the fully-qualified task name
 - Task files should be named `tasks.py` within the Django app
 - Direct call `my_task(user_id=1, action="x")` runs synchronously
 - `my_task.execute_on_commit(user_id=1, action="x")` enqueues after transaction commit
+- Both decorators share `BaseTaskWrapper` — same validation, kwargs building, retry, and singleton logic
 
 ## Enqueuing
 
@@ -132,6 +153,12 @@ class SQSLambdaTaskMessage(BaseModel):
 - A temporary `StreamHandler` is attached to the `lambda_tasks` logger for the duration of the loaders so their log output is visible before Django's `LOGGING` dictConfig has run; it is removed immediately after so that Django's configuration is the sole authority on logging from that point on
 - `resource.setrlimit(RLIMIT_DATA)` is set from `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` so that excessive allocation raises `MemoryError` instead of triggering the OOM killer
 
+`main()` in `handler.py` — AWS Batch container entry point (`python -m lambda_tasks.handler`):
+- Reads `LAMBDA_TASKS_MESSAGE` env var (the serialized task JSON, set by `submit_batch_job` via container overrides)
+- Uses `AWS_BATCH_JOB_ID` as the `message_id` (falls back to UUID4 if not set)
+- Constructs a synthetic SQS event and delegates to `handler()` — reuses the full cold-start and execution path
+- Returns exit code 0 on success, 1 on failure
+
 ## Environment Loader
 
 `resolve_environment()` in `environment_loader.py` runs once at Lambda cold start, before `resolve_secrets_into_env()` and `django.setup()`.
@@ -184,6 +211,7 @@ Statuses: `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 | Setting | Default | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | — | Dict of `{queue_name: sqs_url}`, must include `"default"` |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `{}` | Dict of `{queue_name: {"job_queue": ..., "job_definition": ...}}`, must include `"default"` when set |
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `270` | Soft timeout in seconds |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `300` | Hard timeout in seconds |
 | `LAMBDA_TASKS_EAGER` | `False` | Run tasks synchronously in-process (no SQS) |
@@ -261,6 +289,7 @@ def my_task(*, user_id: int) -> None:
 `lambda_tasks.tasks` provides maintenance tasks that ship with the library:
 
 - `cleanup_task_records(*, retention_days: int = 7) -> int` — deletes `TaskRecord` rows whose `start_time` is strictly older than `retention_days`. Returns the number of deleted rows. Decorated with `@lambda_task` so it can be enqueued via `cleanup_task_records.execute_on_commit()` or called directly. Users are responsible for scheduling (e.g. EventBridge rule, cron, management command).
+- `submit_batch_job(*, message_json: str, batch_queue: str) -> str` — submits a batch task to AWS Batch. Reads `job_queue` and `job_definition` from `LAMBDA_TASKS_BATCH_QUEUES[batch_queue]`, sanitizes the task name for use as `jobName`, and calls `batch.submit_job()` with the message passed as a `LAMBDA_TASKS_MESSAGE` environment variable override. Returns the Batch job ID. This task is enqueued automatically by `SQSLambdaTask._execute()` when `backend == TaskBackend.BATCH` — users do not call it directly.
 
 ## Conventions
 

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.0}/.kiro/steering/structure.md

@@ -12,15 +12,15 @@ django-lambda-tasks/
 │   ├── __init__.py
 │   ├── admin.py                # Django admin for TaskRecord
 │   ├── apps.py                 # Django AppConfig
-│   ├── decorators.py           # @lambda_task decorator + LambdaTaskWrapper
-│   ├── handler.py              # AWS Lambda entry point
+│   ├── decorators.py           # BaseTaskWrapper, LambdaTaskWrapper, BatchTaskWrapper, @lambda_task, @batch_task
+│   ├── handler.py              # AWS Lambda entry point + AWS Batch container entry point (main)
 │   ├── logging.py              # task_logger — invocation-scoped LoggerAdapter
 │   ├── models.py               # TaskRecord, SQSLambdaTaskMessage, SQSLambdaTask
-│   ├── settings.py             # LambdaTasksSettings (lazy Django settings reader)
+│   ├── settings.py             # LambdaTasksSettings, TaskBackend enum, constants
 │   ├── secret_loader.py        # Resolves LAMBDA_TASKS_SECRET_* env vars at cold start
 │   ├── environment_loader.py    # Loads env vars from Secrets Manager at cold start
 │   ├── local_executor.py        # ProcessPoolExecutor for async local task execution
-│   ├── tasks.py                # Built-in maintenance tasks (cleanup_task_records)
+│   ├── tasks.py                # Built-in tasks (cleanup_task_records, submit_batch_job)
 │   ├── timeouts.py             # TimeoutContext implementation
 │   └── migrations/             # Django migrations for TaskRecord
 ├── tests/                      # pytest test suite
@@ -39,23 +39,23 @@ django-lambda-tasks/
 
 ## Module Responsibilities
 
-- `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
+- `decorators.py` — defines `BaseTaskWrapper` (shared validation, kwargs building, execution logic), `LambdaTaskWrapper` and `BatchTaskWrapper` (subclasses that set `_max_timeout` and `_backend`), plus `@lambda_task` and `@batch_task` decorator factories; enforces kwargs-only at decoration time
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing + `backend` field; `_execute()` publishes to SQS, submits to Batch via `submit_batch_job`, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker (and sets `SIGINT` to `SIG_IGN` so workers ignore Ctrl+C); `_install_shutdown_handlers()` installs main-thread `SIGINT`/`SIGTERM` handlers that release the pool before chaining to the previous handler
-- `handler.py` — Lambda entry point; cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
+- `handler.py` — Lambda entry point (`handler(event, context)`); cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`. Also provides `main()` as the AWS Batch container entry point (`python -m lambda_tasks.handler`) — reads `LAMBDA_TASKS_MESSAGE` env var, constructs a synthetic SQS event, and delegates to `handler()`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel
 - `logging.py` — `task_logger` singleton; `message_id` set/cleared around each task execution
-- `settings.py` — `LambdaTasksSettings` instantiated fresh per use (reads live Django settings)
+- `settings.py` — `TaskBackend` StrEnum (`LAMBDA`, `BATCH`); constants (`MAX_DELAY`, `MAX_TIMEOUT`, `MAX_BATCH_TIMEOUT`); `LambdaTasksSettings` instantiated fresh per use (reads live Django settings including `BATCH_QUEUES`)
 - `admin.py` — Django admin registration for `TaskRecord`
 - `apps.py` — Django `AppConfig`; `ready()` installs the local-executor shutdown signal handlers when `LOCAL_WORKERS > 0`
-- `tasks.py` — built-in maintenance tasks; `cleanup_task_records` deletes old `TaskRecord` rows
+- `tasks.py` — built-in tasks; `cleanup_task_records` deletes old `TaskRecord` rows; `submit_batch_job` receives a serialized task message and submits it to AWS Batch via `batch.submit_job()`
 
 ## Conventions
 
 - Task definitions belong in `tasks.py` within each Django app
 - All task functions must use keyword-only arguments — positional args raise `TypeError` at decoration time
-- Task logic must be self-contained; no Django request context is available in Lambda
+- Task logic must be self-contained; no Django request context is available in Lambda or Batch
 - New modules go inside `lambda_tasks/`; add a corresponding `tests/test_<module>.py`
 - Tests for `models.py` (including `SQSLambdaTask._execute` and `SQSLambdaTask.execute_on_commit`) live in `tests/test_models.py`
 - Do not add application-level code to `main.py` — it is a placeholder only

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-lambda-tasks
-Version: 0.5.0
+Version: 0.6.0
 Summary: Run async tasks in a lambda function
 Requires-Python: >=3.10
 Requires-Dist: awslambdaric
@@ -110,12 +110,20 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 | Setting | Type | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `dict[str, dict]` | Map of queue name → `{"job_queue": ..., "job_definition": ...}`. Must include a `"default"` key when set. Default `{}`. |
 
 ```python
 LAMBDA_TASKS_QUEUES = {
     "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
     "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
 }
+
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
 ```
 
 `LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
@@ -623,3 +631,71 @@ result = send_welcome_email(user_id=1, template="welcome")
 ```
 
 This bypasses the queue entirely and runs the function in the current process. No `TaskRecord` is created, no `transaction.atomic()` block is used, and no timeout enforcement applies — it behaves exactly like calling the underlying function directly. Kwargs are still validated against the task's type annotations via Pydantic.
+
+---
+
+## AWS Batch tasks
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, use `@batch_task` to route execution to AWS Batch containers.
+
+### 1. Define a batch task
+
+```python
+# myapp/tasks.py
+from lambda_tasks.decorators import batch_task
+
+@batch_task(soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### 2. Configure AWS Batch queues
+
+```python
+# settings.py
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
+```
+
+### 3. Enqueue from a view
+
+```python
+process_large_file.execute_on_commit(file_id=42)
+```
+
+### How it works
+
+1. `execute_on_commit()` enqueues a message to SQS (same as Lambda tasks)
+2. The Lambda handler picks up the message and runs the built-in `submit_batch_job` task
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
+### Batch task timeouts
+
+Batch tasks support the same two-phase timeout mechanism as Lambda tasks (soft + hard), but with a maximum of **3600 seconds** (1 hour) instead of 900.
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Batch tasks respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like Lambda tasks, without requiring AWS infrastructure.

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.0}/README.md

@@ -98,12 +98,20 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 | Setting | Type | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `dict[str, dict]` | Map of queue name → `{"job_queue": ..., "job_definition": ...}`. Must include a `"default"` key when set. Default `{}`. |
 
 ```python
 LAMBDA_TASKS_QUEUES = {
     "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
     "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
 }
+
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
 ```
 
 `LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
@@ -611,3 +619,71 @@ result = send_welcome_email(user_id=1, template="welcome")
 ```
 
 This bypasses the queue entirely and runs the function in the current process. No `TaskRecord` is created, no `transaction.atomic()` block is used, and no timeout enforcement applies — it behaves exactly like calling the underlying function directly. Kwargs are still validated against the task's type annotations via Pydantic.
+
+---
+
+## AWS Batch tasks
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, use `@batch_task` to route execution to AWS Batch containers.
+
+### 1. Define a batch task
+
+```python
+# myapp/tasks.py
+from lambda_tasks.decorators import batch_task
+
+@batch_task(soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### 2. Configure AWS Batch queues
+
+```python
+# settings.py
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
+```
+
+### 3. Enqueue from a view
+
+```python
+process_large_file.execute_on_commit(file_id=42)
+```
+
+### How it works
+
+1. `execute_on_commit()` enqueues a message to SQS (same as Lambda tasks)
+2. The Lambda handler picks up the message and runs the built-in `submit_batch_job` task
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
+### Batch task timeouts
+
+Batch tasks support the same two-phase timeout mechanism as Lambda tasks (soft + hard), but with a maximum of **3600 seconds** (1 hour) instead of 900.
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Batch tasks respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like Lambda tasks, without requiring AWS infrastructure.

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.0}/lambda_tasks/admin.py

@@ -17,6 +17,8 @@ class TaskRecordAdmin(admin.ModelAdmin):
     list_display = (
         "pk",
         "task_name",
+        "backend",
+        "queue",
         "status",
         "start_time",
         "end_time",
@@ -59,10 +61,24 @@ class TaskRecordAdmin(admin.ModelAdmin):
         total_seconds = d.total_seconds()
         return f"{total_seconds:.3f}s"
 
+    @admin.display(description="Backend")
+    def backend(self, obj: TaskRecord) -> str:
+        try:
+            return import_string(obj.task_name).backend.value
+        except Exception:
+            return "?"
+
+    @admin.display(description="Queue")
+    def queue(self, obj: TaskRecord) -> str:
+        try:
+            return import_string(obj.task_name).queue
+        except Exception:
+            return "?"
+
     @admin.action(description="Replay selected tasks", permissions=("change",))
     def replay_tasks(self, request: HttpRequest, queryset: QuerySet) -> None:
         for record in queryset:
-            queue = import_string(record.task_name).queue
+            wrapper = import_string(record.task_name)
             task = SQSLambdaTask(
                 message=SQSLambdaTaskMessage(
                     task_name=record.task_name,
@@ -70,6 +86,6 @@ class TaskRecordAdmin(admin.ModelAdmin):
                     n_retries=0,
                 ),
                 delay=0,
-                queue=queue,
+                queue=wrapper.queue,
             )
             task.execute_on_commit()