django-lambda-tasks 0.4.11__tar.gz → 0.6.0__tar.gz

{django_lambda_tasks-0.4.11 → django_lambda_tasks-0.6.0}/.kiro/steering/product.md

@@ -4,37 +4,58 @@ inclusion: always
 
 # django-lambda-tasks
 
-A Django library for offloading tasks to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker.
+A Django library for offloading tasks to AWS Lambda or AWS Batch outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker or Batch container.
 
 ## Architecture
 
+### Lambda tasks
+
 ```
 View → @lambda_task.execute_on_commit() → SQS → Lambda handler → SQSLambdaTaskMessage.execute_immediately() → TaskRecord
 ```
 
+### Batch tasks
+
+```
+View → @batch_task.execute_on_commit() → SQS → Lambda handler → submit_batch_job → batch.submit_job()
+     → Container → python -m lambda_tasks.handler → handler() → execute_immediately() → TaskRecord
+```
+
+Retry path (Batch):
+```
+Container (retry_on exception) → execute_on_commit() → SQS → Lambda → submit_batch_job → new Batch job
+```
+
 Key modules:
-- `decorators.py` — `@lambda_task` decorator and `LambdaTaskWrapper`
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish or local pool submit)
+- `decorators.py` — `BaseTaskWrapper`, `LambdaTaskWrapper`, `BatchTaskWrapper`, `@lambda_task`, `@batch_task`
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish, Batch submit, or local pool submit)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution for development
-- `handler.py` — AWS Lambda entry point; cold-start init runs on first invocation (not at import time) with partial-batch failure reporting
+- `handler.py` — AWS Lambda entry point + AWS Batch container entry point (`main()`)
 - `logging.py` — `task_logger` for invocation-scoped log output
-- `settings.py` — lazy `LambdaTasksSettings` reading from Django settings
+- `settings.py` — `TaskBackend` enum, lazy `LambdaTasksSettings` reading from Django settings
 
 ## Task Definition
 
 ```python
-# Always kwargs-only — positional args raise TypeError at decoration time
+# Lambda task — max timeout 900s
 @lambda_task(retry_delay=30, soft_timeout=60, hard_timeout=120, queue="default",
              retry_on=(TransientError,),
              singleton=True, retry_singleton=True)
 def my_task(*, user_id: int, action: str) -> None:
     ...
+
+# Batch task — max timeout 3600s, for long-running or storage-heavy workloads
+@batch_task(soft_timeout=1800, hard_timeout=3500, queue="default",
+            retry_on=(TransientError,))
+def heavy_task(*, file_id: int) -> None:
+    ...
 ```
 
 - Tasks are resolved at execution time via `import_string` on the fully-qualified task name
 - Task files should be named `tasks.py` within the Django app
 - Direct call `my_task(user_id=1, action="x")` runs synchronously
 - `my_task.execute_on_commit(user_id=1, action="x")` enqueues after transaction commit
+- Both decorators share `BaseTaskWrapper` — same validation, kwargs building, retry, and singleton logic
 
 ## Enqueuing
 
@@ -132,18 +153,24 @@ class SQSLambdaTaskMessage(BaseModel):
 - A temporary `StreamHandler` is attached to the `lambda_tasks` logger for the duration of the loaders so their log output is visible before Django's `LOGGING` dictConfig has run; it is removed immediately after so that Django's configuration is the sole authority on logging from that point on
 - `resource.setrlimit(RLIMIT_DATA)` is set from `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` so that excessive allocation raises `MemoryError` instead of triggering the OOM killer
 
+`main()` in `handler.py` — AWS Batch container entry point (`python -m lambda_tasks.handler`):
+- Reads `LAMBDA_TASKS_MESSAGE` env var (the serialized task JSON, set by `submit_batch_job` via container overrides)
+- Uses `AWS_BATCH_JOB_ID` as the `message_id` (falls back to UUID4 if not set)
+- Constructs a synthetic SQS event and delegates to `handler()` — reuses the full cold-start and execution path
+- Returns exit code 0 on success, 1 on failure
+
 ## Environment Loader
 
 `resolve_environment()` in `environment_loader.py` runs once at Lambda cold start, before `resolve_secrets_into_env()` and `django.setup()`.
 
 When the environment variable `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` is set, the loader parses the reference, fetches the named Secrets Manager secret, parses its JSON content as a flat key-value mapping, and sets the resulting pairs as environment variables.
 
-Required format: `<arn>:<version-stage>:<version-id>` (9 colon-separated segments — the ARN is 7 segments, plus version-stage and version-id). Both suffix fields must be non-empty.
+Required format: `<arn>:<version-id>` (8 colon-separated segments — the ARN is 7 segments, plus the version-id). The version-id must be non-empty.
 
 Behaviour:
 - If `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` is not set, does nothing (no AWS API calls)
 - Validates the reference format before any AWS call — malformed references raise `ValueError` immediately
-- Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionStage=..., VersionId=...)`
+- Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionId=...)`
 - Validates the secret value is a flat JSON object (all values must be strings, no empty keys)
 - Sets each key-value pair in `os.environ` — existing env vars are overridden (no conflict detection)
 - Idempotent via a module-level `_loaded` sentinel — subsequent calls are free no-ops
@@ -155,12 +182,12 @@ Behaviour:
 
 Any env var prefixed `LAMBDA_TASKS_SECRET_` is treated as a Secrets Manager reference. The unprefixed name is the target env var.
 
-Required format: `<arn>:<json-key>:<version-stage>:<version-id>` (10 colon-separated segments, all fields non-empty).
+Required format: `<arn>:<json-key>:<version-stage>:<version-id>` (10 colon-separated segments). The version-stage segment must be empty. The json-key and version-id must be non-empty.
 
 Behaviour:
 - All references are validated before any AWS call — malformed references raise `ValueError` immediately
 - Setting both `LAMBDA_TASKS_SECRET_FOO` and `FOO` is a configuration error and raises `ValueError`
-- Calls are batched by `(ARN, version-stage, version-id)` — one `GetSecretValue` per unique combination
+- Calls are batched by `(ARN, version-id)` — one `GetSecretValue` per unique combination
 - Fetched secrets are cached in-process; warm invocations pay no extra cost
 
 ## Models
@@ -184,6 +211,7 @@ Statuses: `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 | Setting | Default | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | — | Dict of `{queue_name: sqs_url}`, must include `"default"` |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `{}` | Dict of `{queue_name: {"job_queue": ..., "job_definition": ...}}`, must include `"default"` when set |
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `270` | Soft timeout in seconds |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `300` | Hard timeout in seconds |
 | `LAMBDA_TASKS_EAGER` | `False` | Run tasks synchronously in-process (no SQS) |
@@ -261,6 +289,7 @@ def my_task(*, user_id: int) -> None:
 `lambda_tasks.tasks` provides maintenance tasks that ship with the library:
 
 - `cleanup_task_records(*, retention_days: int = 7) -> int` — deletes `TaskRecord` rows whose `start_time` is strictly older than `retention_days`. Returns the number of deleted rows. Decorated with `@lambda_task` so it can be enqueued via `cleanup_task_records.execute_on_commit()` or called directly. Users are responsible for scheduling (e.g. EventBridge rule, cron, management command).
+- `submit_batch_job(*, message_json: str, batch_queue: str) -> str` — submits a batch task to AWS Batch. Reads `job_queue` and `job_definition` from `LAMBDA_TASKS_BATCH_QUEUES[batch_queue]`, sanitizes the task name for use as `jobName`, and calls `batch.submit_job()` with the message passed as a `LAMBDA_TASKS_MESSAGE` environment variable override. Returns the Batch job ID. This task is enqueued automatically by `SQSLambdaTask._execute()` when `backend == TaskBackend.BATCH` — users do not call it directly.
 
 ## Conventions
 

{django_lambda_tasks-0.4.11 → django_lambda_tasks-0.6.0}/.kiro/steering/structure.md

@@ -12,15 +12,15 @@ django-lambda-tasks/
 │   ├── __init__.py
 │   ├── admin.py                # Django admin for TaskRecord
 │   ├── apps.py                 # Django AppConfig
-│   ├── decorators.py           # @lambda_task decorator + LambdaTaskWrapper
-│   ├── handler.py              # AWS Lambda entry point
+│   ├── decorators.py           # BaseTaskWrapper, LambdaTaskWrapper, BatchTaskWrapper, @lambda_task, @batch_task
+│   ├── handler.py              # AWS Lambda entry point + AWS Batch container entry point (main)
 │   ├── logging.py              # task_logger — invocation-scoped LoggerAdapter
 │   ├── models.py               # TaskRecord, SQSLambdaTaskMessage, SQSLambdaTask
-│   ├── settings.py             # LambdaTasksSettings (lazy Django settings reader)
+│   ├── settings.py             # LambdaTasksSettings, TaskBackend enum, constants
 │   ├── secret_loader.py        # Resolves LAMBDA_TASKS_SECRET_* env vars at cold start
 │   ├── environment_loader.py    # Loads env vars from Secrets Manager at cold start
 │   ├── local_executor.py        # ProcessPoolExecutor for async local task execution
-│   ├── tasks.py                # Built-in maintenance tasks (cleanup_task_records)
+│   ├── tasks.py                # Built-in tasks (cleanup_task_records, submit_batch_job)
 │   ├── timeouts.py             # TimeoutContext implementation
 │   └── migrations/             # Django migrations for TaskRecord
 ├── tests/                      # pytest test suite
@@ -39,23 +39,23 @@ django-lambda-tasks/
 
 ## Module Responsibilities
 
-- `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
+- `decorators.py` — defines `BaseTaskWrapper` (shared validation, kwargs building, execution logic), `LambdaTaskWrapper` and `BatchTaskWrapper` (subclasses that set `_max_timeout` and `_backend`), plus `@lambda_task` and `@batch_task` decorator factories; enforces kwargs-only at decoration time
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing + `backend` field; `_execute()` publishes to SQS, submits to Batch via `submit_batch_job`, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker (and sets `SIGINT` to `SIG_IGN` so workers ignore Ctrl+C); `_install_shutdown_handlers()` installs main-thread `SIGINT`/`SIGTERM` handlers that release the pool before chaining to the previous handler
-- `handler.py` — Lambda entry point; cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
+- `handler.py` — Lambda entry point (`handler(event, context)`); cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`. Also provides `main()` as the AWS Batch container entry point (`python -m lambda_tasks.handler`) — reads `LAMBDA_TASKS_MESSAGE` env var, constructs a synthetic SQS event, and delegates to `handler()`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel
 - `logging.py` — `task_logger` singleton; `message_id` set/cleared around each task execution
-- `settings.py` — `LambdaTasksSettings` instantiated fresh per use (reads live Django settings)
+- `settings.py` — `TaskBackend` StrEnum (`LAMBDA`, `BATCH`); constants (`MAX_DELAY`, `MAX_TIMEOUT`, `MAX_BATCH_TIMEOUT`); `LambdaTasksSettings` instantiated fresh per use (reads live Django settings including `BATCH_QUEUES`)
 - `admin.py` — Django admin registration for `TaskRecord`
 - `apps.py` — Django `AppConfig`; `ready()` installs the local-executor shutdown signal handlers when `LOCAL_WORKERS > 0`
-- `tasks.py` — built-in maintenance tasks; `cleanup_task_records` deletes old `TaskRecord` rows
+- `tasks.py` — built-in tasks; `cleanup_task_records` deletes old `TaskRecord` rows; `submit_batch_job` receives a serialized task message and submits it to AWS Batch via `batch.submit_job()`
 
 ## Conventions
 
 - Task definitions belong in `tasks.py` within each Django app
 - All task functions must use keyword-only arguments — positional args raise `TypeError` at decoration time
-- Task logic must be self-contained; no Django request context is available in Lambda
+- Task logic must be self-contained; no Django request context is available in Lambda or Batch
 - New modules go inside `lambda_tasks/`; add a corresponding `tests/test_<module>.py`
 - Tests for `models.py` (including `SQSLambdaTask._execute` and `SQSLambdaTask.execute_on_commit`) live in `tests/test_models.py`
 - Do not add application-level code to `main.py` — it is a placeholder only

{django_lambda_tasks-0.4.11 → django_lambda_tasks-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-lambda-tasks
-Version: 0.4.11
+Version: 0.6.0
 Summary: Run async tasks in a lambda function
 Requires-Python: >=3.10
 Requires-Dist: awslambdaric
@@ -110,12 +110,20 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 | Setting | Type | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `dict[str, dict]` | Map of queue name → `{"job_queue": ..., "job_definition": ...}`. Must include a `"default"` key when set. Default `{}`. |
 
 ```python
 LAMBDA_TASKS_QUEUES = {
     "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
     "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
 }
+
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
 ```
 
 `LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
@@ -150,7 +158,7 @@ LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT = 270
 
 | Setting | Type | Description |
 |---|---|---|
-| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | env var | Secrets Manager reference (`<arn>:<version-stage>:<version-id>`) to load as environment variables at Lambda cold start. |
+| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | env var | Secrets Manager reference (`<arn>:<version-id>`) to load as environment variables at Lambda cold start. |
 | `LAMBDA_TASKS_SECRET_*` | env var(s) | Secrets Manager references resolved into env vars at Lambda cold start. The unprefixed name becomes the target env var. |
 
 These are environment variables set on the Lambda function, not Django settings. See [Loading environment variables from Secrets Manager](#loading-environment-variables-from-secrets-manager) and [Resolving individual secrets from AWS Secrets Manager](#resolving-individual-secrets-from-aws-secrets-manager) for full details.
@@ -203,7 +211,6 @@ Key characteristics:
 
 ```python
 @lambda_task(
-    delay=0,               # seconds — SQS DelaySeconds before message becomes visible
     soft_timeout=60,       # seconds — overrides global default for this task
     hard_timeout=90,       # seconds — overrides global default for this task
     queue="default",       # named queue from LAMBDA_TASKS_QUEUES
@@ -493,20 +500,20 @@ Ensure the Lambda execution environment has `DJANGO_SETTINGS_MODULE` set and tha
 | Environment Variable | Required | Description |
 |---|---|---|
 | `DJANGO_SETTINGS_MODULE` | Yes | Django settings module path (e.g. `myapp.settings.production`). |
-| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | No | Secrets Manager reference (`<arn>:<version-stage>:<version-id>`) to load as environment variables at cold start (runs first). |
+| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | No | Secrets Manager reference (`<arn>:<version-id>`) to load as environment variables at cold start (runs first). |
 | `LAMBDA_TASKS_SECRET_*` | No | Secrets Manager references resolved into individual env vars at cold start (runs second, after environment loading). |
 
 ### Loading environment variables from Secrets Manager
 
 The Lambda handler supports loading environment variables from an AWS Secrets Manager secret at cold start. This lets you manage environment configuration centrally in Secrets Manager without baking values into the Lambda deployment package.
 
-Set the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` environment variable to a full reference including version stage and version ID:
+Set the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` environment variable to a full reference including version ID:
 
 ```
-LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod/environment:AWSCURRENT:v1
+LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod/environment:v1
 ```
 
-The format is `<arn>:<version-stage>:<version-id>` (9 colon-separated segments — the ARN is 7 segments, plus version-stage and version-id). Both suffix fields must be non-empty.
+The format is `<arn>:<version-id>` (8 colon-separated segments — the ARN is 7 segments, plus the version-id). The version-id must be non-empty.
 
 The secret value must be a flat JSON object where all keys and values are strings:
 
@@ -521,8 +528,8 @@ The secret value must be a flat JSON object where all keys and values are string
 At cold start (on the first handler invocation), before `resolve_secrets_into_env()` and `django.setup()`, the handler calls `resolve_environment()` which:
 
 1. Checks for the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` env var — if not set, does nothing
-2. Parses and validates the reference format (9 segments, non-empty version-stage and version-id)
-3. Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionStage=..., VersionId=...)`
+2. Parses and validates the reference format (8 segments, non-empty version-id)
+3. Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionId=...)`
 4. Validates the secret value is a flat JSON object (all values must be strings, no empty keys)
 5. Sets each key-value pair in `os.environ` — existing env vars are overridden
 6. Caches the result via a module-level sentinel — subsequent calls are free no-ops
@@ -533,7 +540,7 @@ Because environment loading runs first, the secret can provide `DJANGO_SETTINGS_
 
 The following raise `ValueError` at cold start, preventing the Lambda container from starting:
 
-- Reference format is invalid (wrong segment count, empty version-stage or version-id)
+- Reference format is invalid (wrong segment count, empty version-id)
 - Secret value is not valid JSON
 - JSON is not a flat object (contains non-string values) — error message lists the offending keys
 - JSON contains an empty string key
@@ -547,14 +554,14 @@ The Lambda handler supports loading individual secret values from AWS Secrets Ma
 Set any env var with the prefix `LAMBDA_TASKS_SECRET_` to a full Secrets Manager dynamic reference. The unprefixed name becomes the target env var:
 
 ```
-LAMBDA_TASKS_SECRET_DATABASE_URL=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL:AWSCURRENT:v1
+LAMBDA_TASKS_SECRET_DATABASE_URL=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL::v1
 ```
 
 At cold start (on the first handler invocation), after `resolve_environment()` and before `django.setup()`, the handler calls `resolve_secrets_into_env()` which:
 
 1. Scans all env vars for the `LAMBDA_TASKS_SECRET_` prefix
 2. Validates every reference — malformed references raise immediately so the container fails to start rather than misconfiguring Django silently
-3. Groups references by `(ARN, version-stage, version-id)` and makes one `GetSecretValue` call per unique combination
+3. Groups references by `(ARN, version-id)` and makes one `GetSecretValue` call per unique combination
 4. Extracts the named JSON key from the secret and writes it into `os.environ`
 5. Caches fetched secrets in-process — warm invocations pay no extra cost
 
@@ -566,18 +573,18 @@ Every value must follow the full dynamic reference syntax:
 <arn>:<json-key>:<version-stage>:<version-id>
 ```
 
-All four fields are required and must be non-empty. The secret value must be a JSON object; `json-key` names the field to extract.
+The version-stage segment must be empty. The json-key and version-id fields are required and must be non-empty. The secret value must be a JSON object; `json-key` names the field to extract.
 
 ```
-# arn (7 segments) : json-key : version-stage : version-id
-arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL:AWSCURRENT:v1
+# arn (7 segments) : json-key : version-stage (empty) : version-id
+arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL::v1
 ```
 
-Multiple env vars can reference different keys from the same secret — only one `GetSecretValue` call is made for that `(ARN, version-stage, version-id)` combination:
+Multiple env vars can reference different keys from the same secret — only one `GetSecretValue` call is made for that `(ARN, version-id)` combination:
 
 ```
-LAMBDA_TASKS_SECRET_DATABASE_URL=arn:...:myapp/prod:DATABASE_URL:AWSCURRENT:v1
-LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY:AWSCURRENT:v1
+LAMBDA_TASKS_SECRET_DATABASE_URL=arn:...:myapp/prod:DATABASE_URL::v1
+LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY::v1
 ```
 
 #### Validation errors
@@ -585,7 +592,8 @@ LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY:AWSCURRENT:v1
 The following all raise `ValueError` at cold start, preventing the Lambda container from starting with a misconfigured environment:
 
 - Wrong number of colon-separated segments (must be exactly 10)
-- Empty `json-key`, `version-stage`, or `version-id`
+- Empty `json-key` or `version-id`
+- Non-empty `version-stage` (version-stage is not supported)
 - Both `LAMBDA_TASKS_SECRET_FOO` and `FOO` are set — use one or the other
 - The named JSON key does not exist in the fetched secret
 - The secret value is not valid JSON
@@ -623,3 +631,71 @@ result = send_welcome_email(user_id=1, template="welcome")
 ```
 
 This bypasses the queue entirely and runs the function in the current process. No `TaskRecord` is created, no `transaction.atomic()` block is used, and no timeout enforcement applies — it behaves exactly like calling the underlying function directly. Kwargs are still validated against the task's type annotations via Pydantic.
+
+---
+
+## AWS Batch tasks
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, use `@batch_task` to route execution to AWS Batch containers.
+
+### 1. Define a batch task
+
+```python
+# myapp/tasks.py
+from lambda_tasks.decorators import batch_task
+
+@batch_task(soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### 2. Configure AWS Batch queues
+
+```python
+# settings.py
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
+```
+
+### 3. Enqueue from a view
+
+```python
+process_large_file.execute_on_commit(file_id=42)
+```
+
+### How it works
+
+1. `execute_on_commit()` enqueues a message to SQS (same as Lambda tasks)
+2. The Lambda handler picks up the message and runs the built-in `submit_batch_job` task
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
+### Batch task timeouts
+
+Batch tasks support the same two-phase timeout mechanism as Lambda tasks (soft + hard), but with a maximum of **3600 seconds** (1 hour) instead of 900.
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Batch tasks respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like Lambda tasks, without requiring AWS infrastructure.

{django_lambda_tasks-0.4.11 → django_lambda_tasks-0.6.0}/README.md

@@ -98,12 +98,20 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 | Setting | Type | Description |
 |---|---|---|
 | `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_BATCH_QUEUES` | `dict[str, dict]` | Map of queue name → `{"job_queue": ..., "job_definition": ...}`. Must include a `"default"` key when set. Default `{}`. |
 
 ```python
 LAMBDA_TASKS_QUEUES = {
     "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
     "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
 }
+
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
 ```
 
 `LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
@@ -138,7 +146,7 @@ LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT = 270
 
 | Setting | Type | Description |
 |---|---|---|
-| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | env var | Secrets Manager reference (`<arn>:<version-stage>:<version-id>`) to load as environment variables at Lambda cold start. |
+| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | env var | Secrets Manager reference (`<arn>:<version-id>`) to load as environment variables at Lambda cold start. |
 | `LAMBDA_TASKS_SECRET_*` | env var(s) | Secrets Manager references resolved into env vars at Lambda cold start. The unprefixed name becomes the target env var. |
 
 These are environment variables set on the Lambda function, not Django settings. See [Loading environment variables from Secrets Manager](#loading-environment-variables-from-secrets-manager) and [Resolving individual secrets from AWS Secrets Manager](#resolving-individual-secrets-from-aws-secrets-manager) for full details.
@@ -191,7 +199,6 @@ Key characteristics:
 
 ```python
 @lambda_task(
-    delay=0,               # seconds — SQS DelaySeconds before message becomes visible
     soft_timeout=60,       # seconds — overrides global default for this task
     hard_timeout=90,       # seconds — overrides global default for this task
     queue="default",       # named queue from LAMBDA_TASKS_QUEUES
@@ -481,20 +488,20 @@ Ensure the Lambda execution environment has `DJANGO_SETTINGS_MODULE` set and tha
 | Environment Variable | Required | Description |
 |---|---|---|
 | `DJANGO_SETTINGS_MODULE` | Yes | Django settings module path (e.g. `myapp.settings.production`). |
-| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | No | Secrets Manager reference (`<arn>:<version-stage>:<version-id>`) to load as environment variables at cold start (runs first). |
+| `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` | No | Secrets Manager reference (`<arn>:<version-id>`) to load as environment variables at cold start (runs first). |
 | `LAMBDA_TASKS_SECRET_*` | No | Secrets Manager references resolved into individual env vars at cold start (runs second, after environment loading). |
 
 ### Loading environment variables from Secrets Manager
 
 The Lambda handler supports loading environment variables from an AWS Secrets Manager secret at cold start. This lets you manage environment configuration centrally in Secrets Manager without baking values into the Lambda deployment package.
 
-Set the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` environment variable to a full reference including version stage and version ID:
+Set the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` environment variable to a full reference including version ID:
 
 ```
-LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod/environment:AWSCURRENT:v1
+LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod/environment:v1
 ```
 
-The format is `<arn>:<version-stage>:<version-id>` (9 colon-separated segments — the ARN is 7 segments, plus version-stage and version-id). Both suffix fields must be non-empty.
+The format is `<arn>:<version-id>` (8 colon-separated segments — the ARN is 7 segments, plus the version-id). The version-id must be non-empty.
 
 The secret value must be a flat JSON object where all keys and values are strings:
 
@@ -509,8 +516,8 @@ The secret value must be a flat JSON object where all keys and values are string
 At cold start (on the first handler invocation), before `resolve_secrets_into_env()` and `django.setup()`, the handler calls `resolve_environment()` which:
 
 1. Checks for the `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` env var — if not set, does nothing
-2. Parses and validates the reference format (9 segments, non-empty version-stage and version-id)
-3. Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionStage=..., VersionId=...)`
+2. Parses and validates the reference format (8 segments, non-empty version-id)
+3. Fetches the secret via `secretsmanager.get_secret_value(SecretId=..., VersionId=...)`
 4. Validates the secret value is a flat JSON object (all values must be strings, no empty keys)
 5. Sets each key-value pair in `os.environ` — existing env vars are overridden
 6. Caches the result via a module-level sentinel — subsequent calls are free no-ops
@@ -521,7 +528,7 @@ Because environment loading runs first, the secret can provide `DJANGO_SETTINGS_
 
 The following raise `ValueError` at cold start, preventing the Lambda container from starting:
 
-- Reference format is invalid (wrong segment count, empty version-stage or version-id)
+- Reference format is invalid (wrong segment count, empty version-id)
 - Secret value is not valid JSON
 - JSON is not a flat object (contains non-string values) — error message lists the offending keys
 - JSON contains an empty string key
@@ -535,14 +542,14 @@ The Lambda handler supports loading individual secret values from AWS Secrets Ma
 Set any env var with the prefix `LAMBDA_TASKS_SECRET_` to a full Secrets Manager dynamic reference. The unprefixed name becomes the target env var:
 
 ```
-LAMBDA_TASKS_SECRET_DATABASE_URL=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL:AWSCURRENT:v1
+LAMBDA_TASKS_SECRET_DATABASE_URL=arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL::v1
 ```
 
 At cold start (on the first handler invocation), after `resolve_environment()` and before `django.setup()`, the handler calls `resolve_secrets_into_env()` which:
 
 1. Scans all env vars for the `LAMBDA_TASKS_SECRET_` prefix
 2. Validates every reference — malformed references raise immediately so the container fails to start rather than misconfiguring Django silently
-3. Groups references by `(ARN, version-stage, version-id)` and makes one `GetSecretValue` call per unique combination
+3. Groups references by `(ARN, version-id)` and makes one `GetSecretValue` call per unique combination
 4. Extracts the named JSON key from the secret and writes it into `os.environ`
 5. Caches fetched secrets in-process — warm invocations pay no extra cost
 
@@ -554,18 +561,18 @@ Every value must follow the full dynamic reference syntax:
 <arn>:<json-key>:<version-stage>:<version-id>
 ```
 
-All four fields are required and must be non-empty. The secret value must be a JSON object; `json-key` names the field to extract.
+The version-stage segment must be empty. The json-key and version-id fields are required and must be non-empty. The secret value must be a JSON object; `json-key` names the field to extract.
 
 ```
-# arn (7 segments) : json-key : version-stage : version-id
-arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL:AWSCURRENT:v1
+# arn (7 segments) : json-key : version-stage (empty) : version-id
+arn:aws:secretsmanager:eu-west-1:123456789012:secret:myapp/prod:DATABASE_URL::v1
 ```
 
-Multiple env vars can reference different keys from the same secret — only one `GetSecretValue` call is made for that `(ARN, version-stage, version-id)` combination:
+Multiple env vars can reference different keys from the same secret — only one `GetSecretValue` call is made for that `(ARN, version-id)` combination:
 
 ```
-LAMBDA_TASKS_SECRET_DATABASE_URL=arn:...:myapp/prod:DATABASE_URL:AWSCURRENT:v1
-LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY:AWSCURRENT:v1
+LAMBDA_TASKS_SECRET_DATABASE_URL=arn:...:myapp/prod:DATABASE_URL::v1
+LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY::v1
 ```
 
 #### Validation errors
@@ -573,7 +580,8 @@ LAMBDA_TASKS_SECRET_SECRET_KEY=arn:...:myapp/prod:SECRET_KEY:AWSCURRENT:v1
 The following all raise `ValueError` at cold start, preventing the Lambda container from starting with a misconfigured environment:
 
 - Wrong number of colon-separated segments (must be exactly 10)
-- Empty `json-key`, `version-stage`, or `version-id`
+- Empty `json-key` or `version-id`
+- Non-empty `version-stage` (version-stage is not supported)
 - Both `LAMBDA_TASKS_SECRET_FOO` and `FOO` are set — use one or the other
 - The named JSON key does not exist in the fetched secret
 - The secret value is not valid JSON
@@ -611,3 +619,71 @@ result = send_welcome_email(user_id=1, template="welcome")
 ```
 
 This bypasses the queue entirely and runs the function in the current process. No `TaskRecord` is created, no `transaction.atomic()` block is used, and no timeout enforcement applies — it behaves exactly like calling the underlying function directly. Kwargs are still validated against the task's type annotations via Pydantic.
+
+---
+
+## AWS Batch tasks
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, use `@batch_task` to route execution to AWS Batch containers.
+
+### 1. Define a batch task
+
+```python
+# myapp/tasks.py
+from lambda_tasks.decorators import batch_task
+
+@batch_task(soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### 2. Configure AWS Batch queues
+
+```python
+# settings.py
+LAMBDA_TASKS_BATCH_QUEUES = {
+    "default": {
+        "job_queue": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
+}
+```
+
+### 3. Enqueue from a view
+
+```python
+process_large_file.execute_on_commit(file_id=42)
+```
+
+### How it works
+
+1. `execute_on_commit()` enqueues a message to SQS (same as Lambda tasks)
+2. The Lambda handler picks up the message and runs the built-in `submit_batch_job` task
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
+### Batch task timeouts
+
+Batch tasks support the same two-phase timeout mechanism as Lambda tasks (soft + hard), but with a maximum of **3600 seconds** (1 hour) instead of 900.
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Batch tasks respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like Lambda tasks, without requiring AWS infrastructure.

{django_lambda_tasks-0.4.11 → django_lambda_tasks-0.6.0}/lambda_tasks/admin.py

@@ -17,6 +17,8 @@ class TaskRecordAdmin(admin.ModelAdmin):
     list_display = (
         "pk",
         "task_name",
+        "backend",
+        "queue",
         "status",
         "start_time",
         "end_time",
@@ -59,10 +61,24 @@ class TaskRecordAdmin(admin.ModelAdmin):
         total_seconds = d.total_seconds()
         return f"{total_seconds:.3f}s"
 
+    @admin.display(description="Backend")
+    def backend(self, obj: TaskRecord) -> str:
+        try:
+            return import_string(obj.task_name).backend.value
+        except Exception:
+            return "?"
+
+    @admin.display(description="Queue")
+    def queue(self, obj: TaskRecord) -> str:
+        try:
+            return import_string(obj.task_name).queue
+        except Exception:
+            return "?"
+
     @admin.action(description="Replay selected tasks", permissions=("change",))
     def replay_tasks(self, request: HttpRequest, queryset: QuerySet) -> None:
         for record in queryset:
-            queue = import_string(record.task_name).queue
+            wrapper = import_string(record.task_name)
             task = SQSLambdaTask(
                 message=SQSLambdaTaskMessage(
                     task_name=record.task_name,
@@ -70,6 +86,6 @@ class TaskRecordAdmin(admin.ModelAdmin):
                     n_retries=0,
                 ),
                 delay=0,
-                queue=queue,
+                queue=wrapper.queue,
             )
             task.execute_on_commit()