django-lambda-tasks 0.5.0__tar.gz → 0.6.1__tar.gz

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/.kiro/steering/product.md

@@ -4,37 +4,58 @@ inclusion: always
 
 # django-lambda-tasks
 
-A Django library for offloading tasks to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker.
+A Django library for offloading tasks to AWS Lambda or AWS Batch outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda worker or Batch container.
 
 ## Architecture
 
+### Lambda tasks
+
 ```
 View → @lambda_task.execute_on_commit() → SQS → Lambda handler → SQSLambdaTaskMessage.execute_immediately() → TaskRecord
 ```
 
+### Batch tasks (via queue config)
+
+```
+View → @lambda_task(queue="heavy").execute_on_commit() → SQS → Lambda handler → submit_batch_job → batch.submit_job()
+     → Container → python -m lambda_tasks.handler → handler() → execute_immediately() → TaskRecord
+```
+
+Retry path (Batch):
+```
+Container (retry_on exception) → execute_on_commit() → SQS → Lambda → submit_batch_job → new Batch job
+```
+
 Key modules:
-- `decorators.py` — `@lambda_task` decorator and `LambdaTaskWrapper`
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish or local pool submit)
+- `decorators.py` — `LambdaTaskWrapper` and `@lambda_task` decorator factory
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish, Batch submit, or local pool submit)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution for development
-- `handler.py` — AWS Lambda entry point; cold-start init runs on first invocation (not at import time) with partial-batch failure reporting
+- `handler.py` — AWS Lambda entry point + AWS Batch container entry point (`main()`)
 - `logging.py` — `task_logger` for invocation-scoped log output
-- `settings.py` — lazy `LambdaTasksSettings` reading from Django settings
+- `settings.py` — lazy `LambdaTasksSettings` reading from Django settings, queue type helpers
 
 ## Task Definition
 
 ```python
-# Always kwargs-only — positional args raise TypeError at decoration time
+# Lambda task — max timeout determined by queue type (900s for SQS, 3600s for Batch)
 @lambda_task(retry_delay=30, soft_timeout=60, hard_timeout=120, queue="default",
              retry_on=(TransientError,),
              singleton=True, retry_singleton=True)
 def my_task(*, user_id: int, action: str) -> None:
     ...
+
+# Batch task — same decorator, just routed to a Batch queue
+@lambda_task(queue="heavy", soft_timeout=1800, hard_timeout=3500,
+            retry_on=(TransientError,))
+def heavy_task(*, file_id: int) -> None:
+    ...
 ```
 
 - Tasks are resolved at execution time via `import_string` on the fully-qualified task name
 - Task files should be named `tasks.py` within the Django app
 - Direct call `my_task(user_id=1, action="x")` runs synchronously
 - `my_task.execute_on_commit(user_id=1, action="x")` enqueues after transaction commit
+- The execution backend (Lambda vs Batch) is determined by the queue configuration, not the decorator
 
 ## Enqueuing
 
@@ -109,7 +130,7 @@ class SQSLambdaTaskMessage(BaseModel):
 `SQSLambdaTaskMessage.execute_immediately(*, message_id: str)` in `models.py`:
 1. Checks for an existing `TaskRecord` with the same `pk` (`message_id`) via `get_or_create`
 2. If a record already exists (any status), logs and returns immediately — duplicate deliveries are silently skipped
-3. Resolves timeouts: decorator default → settings defaults (soft=270s, hard=300s)
+3. Resolves timeouts: decorator default → settings defaults (soft=270s, hard=300s), validated against queue max (900 for SQS, 3600 for Batch)
 4. If `wrapper.singleton` is `True`, acquires a Redis lock via `caches[SINGLETON_CACHE].lock(lock_key)` wrapping the atomic block; if `False`, no lock is acquired
 5. Runs task inside `transaction.atomic()` + `TimeoutContext`
 6. On success: updates record to `SUCCESS` with result and `end_time`
@@ -132,6 +153,12 @@ class SQSLambdaTaskMessage(BaseModel):
 - A temporary `StreamHandler` is attached to the `lambda_tasks` logger for the duration of the loaders so their log output is visible before Django's `LOGGING` dictConfig has run; it is removed immediately after so that Django's configuration is the sole authority on logging from that point on
 - `resource.setrlimit(RLIMIT_DATA)` is set from `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` so that excessive allocation raises `MemoryError` instead of triggering the OOM killer
 
+`main()` in `handler.py` — AWS Batch container entry point (`python -m lambda_tasks.handler`):
+- Reads `LAMBDA_TASKS_MESSAGE` env var (the serialized task JSON, set by `submit_batch_job` via container overrides)
+- Uses `AWS_BATCH_JOB_ID` as the `message_id` (falls back to UUID4 if not set)
+- Constructs a synthetic SQS event and delegates to `handler()` — reuses the full cold-start and execution path
+- Returns exit code 0 on success, 1 on failure
+
 ## Environment Loader
 
 `resolve_environment()` in `environment_loader.py` runs once at Lambda cold start, before `resolve_secrets_into_env()` and `django.setup()`.
@@ -169,7 +196,7 @@ Behaviour:
 
 - `TaskRecord` — Django ORM model; persists task execution state with statuses `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 - `SQSLambdaTaskMessage` — Pydantic model; the SQS message schema; `execute_immediately()` runs the task inside `transaction.atomic()` + `TimeoutContext`
-- `SQSLambdaTask` — Pydantic model; holds a `SQSLambdaTaskMessage` plus routing fields (`delay`, `queue`); `_execute()` publishes to SQS or executes eagerly; `execute_on_commit()` registers `_execute` with `transaction.on_commit`
+- `SQSLambdaTask` — Pydantic model; holds a `SQSLambdaTaskMessage` plus routing fields (`delay`, `queue`); `_execute()` inspects queue config to publish to SQS, submit to Batch via `submit_batch_job`, or execute eagerly/locally; `execute_on_commit()` registers `_execute` with `transaction.on_commit`
 
 ## TaskRecord Model
 
@@ -183,7 +210,7 @@ Statuses: `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 
 | Setting | Default | Description |
 |---|---|---|
-| `LAMBDA_TASKS_QUEUES` | — | Dict of `{queue_name: sqs_url}`, must include `"default"` |
+| `LAMBDA_TASKS_QUEUES` | — | Dict of `{queue_name: queue_config}`. Each value is a dict with either `queue_url` (SQS) or `job_queue_arn`+`job_definition_arn` (Batch). Must include `"default"` which must be SQS. |
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `270` | Soft timeout in seconds |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `300` | Hard timeout in seconds |
 | `LAMBDA_TASKS_EAGER` | `False` | Run tasks synchronously in-process (no SQS) |
@@ -191,7 +218,7 @@ Statuses: `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 | `LAMBDA_TASKS_MAX_RETRIES` | `2880` | Maximum retry attempts before `MaxRetriesExceededError` is raised (60 × 24 × 2) |
 | `LAMBDA_TASKS_SINGLETON_CACHE` | `"default"` | Django cache backend used for singleton task locks |
 
-`LAMBDA_TASKS_QUEUES` must be set and include a `"default"` key. Both timeout values must be greater than zero and at most `900` seconds. `soft_timeout` must always be strictly less than `hard_timeout`.
+`LAMBDA_TASKS_QUEUES` must be set and include a `"default"` key. The `"default"` queue must be an SQS queue. Both timeout values must be greater than zero and `soft_timeout` must always be strictly less than `hard_timeout`. The maximum allowed timeout depends on the queue type: 900 for SQS, 3600 for Batch.
 
 ## Eager Mode
 
@@ -261,6 +288,7 @@ def my_task(*, user_id: int) -> None:
 `lambda_tasks.tasks` provides maintenance tasks that ship with the library:
 
 - `cleanup_task_records(*, retention_days: int = 7) -> int` — deletes `TaskRecord` rows whose `start_time` is strictly older than `retention_days`. Returns the number of deleted rows. Decorated with `@lambda_task` so it can be enqueued via `cleanup_task_records.execute_on_commit()` or called directly. Users are responsible for scheduling (e.g. EventBridge rule, cron, management command).
+- `submit_batch_job(*, message_json: str, batch_queue: str) -> str` — submits a batch task to AWS Batch. Reads `job_queue_arn` and `job_definition_arn` from `LAMBDA_TASKS_QUEUES[batch_queue]`, sanitizes the task name for use as `jobName`, and calls `batch.submit_job()` with the message passed as a `LAMBDA_TASKS_MESSAGE` environment variable override. Returns the Batch job ID. This task is enqueued automatically by `SQSLambdaTask._execute()` when the queue is a Batch queue — users do not call it directly.
 
 ## Conventions
 

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/.kiro/steering/structure.md

@@ -12,22 +12,22 @@ django-lambda-tasks/
 │   ├── __init__.py
 │   ├── admin.py                # Django admin for TaskRecord
 │   ├── apps.py                 # Django AppConfig
-│   ├── decorators.py           # @lambda_task decorator + LambdaTaskWrapper
-│   ├── handler.py              # AWS Lambda entry point
+│   ├── decorators.py           # LambdaTaskWrapper, @lambda_task
+│   ├── handler.py              # AWS Lambda entry point + AWS Batch container entry point (main)
 │   ├── logging.py              # task_logger — invocation-scoped LoggerAdapter
 │   ├── models.py               # TaskRecord, SQSLambdaTaskMessage, SQSLambdaTask
-│   ├── settings.py             # LambdaTasksSettings (lazy Django settings reader)
+│   ├── settings.py             # LambdaTasksSettings, constants, queue type helpers
 │   ├── secret_loader.py        # Resolves LAMBDA_TASKS_SECRET_* env vars at cold start
 │   ├── environment_loader.py    # Loads env vars from Secrets Manager at cold start
 │   ├── local_executor.py        # ProcessPoolExecutor for async local task execution
-│   ├── tasks.py                # Built-in maintenance tasks (cleanup_task_records)
+│   ├── tasks.py                # Built-in tasks (cleanup_task_records, submit_batch_job)
 │   ├── timeouts.py             # TimeoutContext implementation
 │   └── migrations/             # Django migrations for TaskRecord
 ├── tests/                      # pytest test suite
 │   ├── conftest.py
 │   ├── settings.py             # Django settings for test environment
 │   └── test_*.py               # One test file per module
-├── example/                    # Runnable Django project (EAGER mode, no AWS needed)
+├── example/                    # Runnable Django project (LOCAL_WORKERS mode, no AWS needed)
 │   ├── example_app/            # Sample app with a task definition and trigger view
 │   ├── example_project/        # Django settings, URLs, WSGI
 │   ├── manage.py
@@ -39,23 +39,23 @@ django-lambda-tasks/
 
 ## Module Responsibilities
 
-- `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
+- `decorators.py` — defines `LambdaTaskWrapper` and `@lambda_task` decorator factory; enforces kwargs-only at decoration time; validates timeouts (> 0, soft < hard) at decoration time; upper-bound timeout validation deferred to execution time based on queue type
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` inspects queue config to publish to SQS, submit to Batch via `submit_batch_job`, execute eagerly, or submit to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker (and sets `SIGINT` to `SIG_IGN` so workers ignore Ctrl+C); `_install_shutdown_handlers()` installs main-thread `SIGINT`/`SIGTERM` handlers that release the pool before chaining to the previous handler
-- `handler.py` — Lambda entry point; cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
+- `handler.py` — Lambda entry point (`handler(event, context)`); cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`. Also provides `main()` as the AWS Batch container entry point (`python -m lambda_tasks.handler`) — reads `LAMBDA_TASKS_MESSAGE` env var, constructs a synthetic SQS event, and delegates to `handler()`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel
 - `logging.py` — `task_logger` singleton; `message_id` set/cleared around each task execution
-- `settings.py` — `LambdaTasksSettings` instantiated fresh per use (reads live Django settings)
+- `settings.py` — constants (`MAX_DELAY`, `MAX_TIMEOUT`, `MAX_BATCH_TIMEOUT`); `_is_sqs_queue()` and `_is_batch_queue()` helpers; `LambdaTasksSettings` instantiated fresh per use (reads live Django settings); `queue_max_timeout()` method returns 900 or 3600 based on queue type
 - `admin.py` — Django admin registration for `TaskRecord`
 - `apps.py` — Django `AppConfig`; `ready()` installs the local-executor shutdown signal handlers when `LOCAL_WORKERS > 0`
-- `tasks.py` — built-in maintenance tasks; `cleanup_task_records` deletes old `TaskRecord` rows
+- `tasks.py` — built-in tasks; `cleanup_task_records` deletes old `TaskRecord` rows; `submit_batch_job` receives a serialized task message and submits it to AWS Batch via `batch.submit_job()`
 
 ## Conventions
 
 - Task definitions belong in `tasks.py` within each Django app
 - All task functions must use keyword-only arguments — positional args raise `TypeError` at decoration time
-- Task logic must be self-contained; no Django request context is available in Lambda
+- Task logic must be self-contained; no Django request context is available in Lambda or Batch
 - New modules go inside `lambda_tasks/`; add a corresponding `tests/test_<module>.py`
 - Tests for `models.py` (including `SQSLambdaTask._execute` and `SQSLambdaTask.execute_on_commit`) live in `tests/test_models.py`
 - Do not add application-level code to `main.py` — it is a placeholder only

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-lambda-tasks
-Version: 0.5.0
+Version: 0.6.1
 Summary: Run async tasks in a lambda function
 Requires-Python: >=3.10
 Requires-Dist: awslambdaric
@@ -12,7 +12,7 @@ Description-Content-Type: text/markdown
 
 # Django Lambda Tasks
 
-A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
+A Django library for offloading work to AWS Lambda or AWS Batch outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler or Batch container. Task results, status, and metadata are persisted in the Django database.
 
 > **Platform note:** Unix-based systems only (Linux, macOS). Timeout enforcement relies on `SIGALRM`.
 
@@ -86,7 +86,7 @@ See [AWS Lambda and SQS setup](#aws-lambda-and-sqs-setup) for queue and Lambda c
 
 ## Example app
 
-A runnable Django project is included in the [`example/`](example/) directory. It uses `LAMBDA_TASKS_EAGER = True` so no AWS infrastructure is needed.
+A runnable Django project is included in the [`example/`](example/) directory. It uses `LAMBDA_TASKS_LOCAL_WORKERS = 2` so no AWS infrastructure is needed.
 
 ```bash
 cd example
@@ -109,16 +109,25 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 
 | Setting | Type | Description |
 |---|---|---|
-| `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_QUEUES` | `dict[str, dict]` | Map of queue name → queue configuration. Must include a `"default"` key. The `"default"` queue must be an SQS queue. |
+
+Each queue value is a dict. The queue type is determined by the keys present:
+
+- **SQS queue:** `{"queue_url": "https://sqs.../queue-name"}` — tasks execute on Lambda (max timeout 900s)
+- **Batch queue:** `{"job_queue_arn": "arn:...", "job_definition_arn": "arn:..."}` — tasks execute on AWS Batch (max timeout 3600s)
 
 ```python
 LAMBDA_TASKS_QUEUES = {
-    "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
-    "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
+    "default": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/default"},
+    "high_memory": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory"},
+    "heavy": {
+        "job_queue_arn": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition_arn": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
 }
 ```
 
-`LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
+`LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use. The `"default"` queue must be an SQS queue.
 
 ### Timeout settings
 
@@ -127,7 +136,7 @@ LAMBDA_TASKS_QUEUES = {
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `int` | `270` | Seconds before `SoftTimeLimitExceeded` is raised inside the task. |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `int` | `300` | Seconds before the task is forcibly terminated. |
 
-Both values must be between `1` and `900` seconds (the AWS Lambda maximum runtime is 15 minutes). `soft_timeout` must always be strictly less than `hard_timeout`.
+Both values must be greater than zero and `soft_timeout` must always be strictly less than `hard_timeout`. The maximum allowed value depends on the queue type: 900 seconds for SQS queues (Lambda's 15-minute limit), 3600 seconds for Batch queues (1 hour). Upper bound validation happens at execution time based on the task's queue.
 
 ```python
 LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT = 240
@@ -217,9 +226,9 @@ def my_task(*, arg: str) -> None:
 
 | Parameter | Type | Default | Description |
 |---|---|---|---|
-| `soft_timeout` | `int \| None` | `None` (uses global default) | Per-task soft timeout in seconds (max 900). |
-| `hard_timeout` | `int \| None` | `None` (uses global default) | Per-task hard timeout in seconds (max 900). |
-| `queue` | `str` | `"default"` | Named queue to route this task to. |
+| `soft_timeout` | `int \| None` | `None` (uses global default) | Per-task soft timeout in seconds. Max depends on queue type (900 for SQS, 3600 for Batch). |
+| `hard_timeout` | `int \| None` | `None` (uses global default) | Per-task hard timeout in seconds. Max depends on queue type (900 for SQS, 3600 for Batch). |
+| `queue` | `str` | `"default"` | Named queue to route this task to. Determines both execution backend and timeout limits. |
 | `retry_on` | `tuple[type[BaseException], ...]` | `()` | Exception types that trigger an automatic retry (see [Automatic retries](#automatic-retries)). |
 | `retry_delay` | `int` | `0` | Base delay in seconds when enqueuing a retry. Jitter (1–5s) is always added; result capped at 900. Requires `retry_on` to be non-empty. |
 | `singleton` | `bool` | `False` | Prevent concurrent execution via a Redis lock (see [Singleton tasks](#singleton-tasks)). |
@@ -298,26 +307,46 @@ Timeout resolution order (first non-`None` value wins):
 1. Per-task decorator default (`soft_timeout` / `hard_timeout` on `@lambda_task`)
 2. Global settings (`LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` / `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT`)
 
+The maximum allowed timeout is determined by the queue type at execution time: 900 seconds for SQS queues, 3600 seconds for Batch queues.
+
 ---
 
 ## Named queues
 
-Route tasks to queues backed by Lambda functions with different hardware profiles (e.g. more memory or CPU):
+Route tasks to different execution backends by assigning them to named queues:
 
 ```python
 # settings.py
 LAMBDA_TASKS_QUEUES = {
-    "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
-    "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
+    "default": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/default"},
+    "high_memory": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory"},
+    "heavy": {
+        "job_queue_arn": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition_arn": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
 }
 ```
 
 ```python
+# Runs on Lambda with more memory
 @lambda_task(queue="high_memory")
-def process_large_file(*, file_id: int) -> None:
+def process_images(*, batch_id: int) -> None:
+    ...
+
+# Runs on AWS Batch (up to 1 hour, large disk)
+@lambda_task(queue="heavy", soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
     ...
 ```
 
+Tasks routed to a Batch queue are automatically submitted to AWS Batch via the built-in `submit_batch_job` task. The flow is:
+
+1. `execute_on_commit()` enqueues a message to the default SQS queue
+2. The Lambda handler picks up the message and runs `submit_batch_job`
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
 ---
 
 ## Task results
@@ -453,10 +482,11 @@ Each task runs inside a `django.db.transaction.atomic` block. If the task raises
 |---|---|---|
 | Task function has positional parameters | `TypeError` | At decoration time |
 | `soft_timeout >= hard_timeout` | `ValueError` | At decoration time |
-| Timeout value exceeds 900 seconds | `ValueError` | At decoration time or settings load |
+| Timeout value exceeds queue max (900 or 3600) | `ValueError` | At execution time (timeout resolution) |
 | Unknown queue name | `ImproperlyConfigured` | At `.execute_on_commit()` |
 | `LAMBDA_TASKS_QUEUES` not set | `ImproperlyConfigured` | On first settings access |
 | `LAMBDA_TASKS_QUEUES` missing `"default"` | `ImproperlyConfigured` | On first settings access |
+| `"default"` queue is not SQS | `ImproperlyConfigured` | On first settings access |
 | SQS `send_message` failure | boto3 exception (propagated) | At `.execute_on_commit()` |
 
 ---
@@ -465,7 +495,7 @@ Each task runs inside a `django.db.transaction.atomic` block. If the task raises
 
 ### SQS queue configuration
 
-Each queue in `LAMBDA_TASKS_QUEUES` should be a standard SQS queue (not FIFO) with the following recommended settings:
+Each SQS queue in `LAMBDA_TASKS_QUEUES` should be a standard SQS queue (not FIFO) with the following recommended settings:
 
 - **Visibility timeout** — set this higher than your Lambda function's `hard_timeout`. If the Lambda times out before SQS receives a deletion confirmation, the message becomes visible again and will be redelivered.
 - **Dead letter queue (DLQ)** — configure a DLQ to capture messages that cannot be processed. Without one, messages that exhaust retries are silently deleted.
@@ -592,6 +622,39 @@ The following all raise `ValueError` at cold start, preventing the Lambda contai
 
 ---
 
+## AWS Batch setup
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, route them to a Batch queue:
+
+```python
+@lambda_task(queue="heavy", soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Tasks on Batch queues respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like SQS-backed tasks, without requiring AWS infrastructure.
+
+---
+
 ## Built-in tasks
 
 ### `cleanup_task_records`

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/README.md

@@ -1,6 +1,6 @@
 # Django Lambda Tasks
 
-A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
+A Django library for offloading work to AWS Lambda or AWS Batch outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler or Batch container. Task results, status, and metadata are persisted in the Django database.
 
 > **Platform note:** Unix-based systems only (Linux, macOS). Timeout enforcement relies on `SIGALRM`.
 
@@ -74,7 +74,7 @@ See [AWS Lambda and SQS setup](#aws-lambda-and-sqs-setup) for queue and Lambda c
 
 ## Example app
 
-A runnable Django project is included in the [`example/`](example/) directory. It uses `LAMBDA_TASKS_EAGER = True` so no AWS infrastructure is needed.
+A runnable Django project is included in the [`example/`](example/) directory. It uses `LAMBDA_TASKS_LOCAL_WORKERS = 2` so no AWS infrastructure is needed.
 
 ```bash
 cd example
@@ -97,16 +97,25 @@ All settings are read from your Django settings module and prefixed with `LAMBDA
 
 | Setting | Type | Description |
 |---|---|---|
-| `LAMBDA_TASKS_QUEUES` | `dict[str, str]` | Map of queue name → SQS queue URL. Must include a `"default"` key. |
+| `LAMBDA_TASKS_QUEUES` | `dict[str, dict]` | Map of queue name → queue configuration. Must include a `"default"` key. The `"default"` queue must be an SQS queue. |
+
+Each queue value is a dict. The queue type is determined by the keys present:
+
+- **SQS queue:** `{"queue_url": "https://sqs.../queue-name"}` — tasks execute on Lambda (max timeout 900s)
+- **Batch queue:** `{"job_queue_arn": "arn:...", "job_definition_arn": "arn:..."}` — tasks execute on AWS Batch (max timeout 3600s)
 
 ```python
 LAMBDA_TASKS_QUEUES = {
-    "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
-    "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
+    "default": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/default"},
+    "high_memory": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory"},
+    "heavy": {
+        "job_queue_arn": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition_arn": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
 }
 ```
 
-`LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use.
+`LAMBDA_TASKS_QUEUES` must be set and must include a `"default"` key, otherwise `ImproperlyConfigured` is raised on first use. The `"default"` queue must be an SQS queue.
 
 ### Timeout settings
 
@@ -115,7 +124,7 @@ LAMBDA_TASKS_QUEUES = {
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `int` | `270` | Seconds before `SoftTimeLimitExceeded` is raised inside the task. |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `int` | `300` | Seconds before the task is forcibly terminated. |
 
-Both values must be between `1` and `900` seconds (the AWS Lambda maximum runtime is 15 minutes). `soft_timeout` must always be strictly less than `hard_timeout`.
+Both values must be greater than zero and `soft_timeout` must always be strictly less than `hard_timeout`. The maximum allowed value depends on the queue type: 900 seconds for SQS queues (Lambda's 15-minute limit), 3600 seconds for Batch queues (1 hour). Upper bound validation happens at execution time based on the task's queue.
 
 ```python
 LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT = 240
@@ -205,9 +214,9 @@ def my_task(*, arg: str) -> None:
 
 | Parameter | Type | Default | Description |
 |---|---|---|---|
-| `soft_timeout` | `int \| None` | `None` (uses global default) | Per-task soft timeout in seconds (max 900). |
-| `hard_timeout` | `int \| None` | `None` (uses global default) | Per-task hard timeout in seconds (max 900). |
-| `queue` | `str` | `"default"` | Named queue to route this task to. |
+| `soft_timeout` | `int \| None` | `None` (uses global default) | Per-task soft timeout in seconds. Max depends on queue type (900 for SQS, 3600 for Batch). |
+| `hard_timeout` | `int \| None` | `None` (uses global default) | Per-task hard timeout in seconds. Max depends on queue type (900 for SQS, 3600 for Batch). |
+| `queue` | `str` | `"default"` | Named queue to route this task to. Determines both execution backend and timeout limits. |
 | `retry_on` | `tuple[type[BaseException], ...]` | `()` | Exception types that trigger an automatic retry (see [Automatic retries](#automatic-retries)). |
 | `retry_delay` | `int` | `0` | Base delay in seconds when enqueuing a retry. Jitter (1–5s) is always added; result capped at 900. Requires `retry_on` to be non-empty. |
 | `singleton` | `bool` | `False` | Prevent concurrent execution via a Redis lock (see [Singleton tasks](#singleton-tasks)). |
@@ -286,26 +295,46 @@ Timeout resolution order (first non-`None` value wins):
 1. Per-task decorator default (`soft_timeout` / `hard_timeout` on `@lambda_task`)
 2. Global settings (`LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` / `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT`)
 
+The maximum allowed timeout is determined by the queue type at execution time: 900 seconds for SQS queues, 3600 seconds for Batch queues.
+
 ---
 
 ## Named queues
 
-Route tasks to queues backed by Lambda functions with different hardware profiles (e.g. more memory or CPU):
+Route tasks to different execution backends by assigning them to named queues:
 
 ```python
 # settings.py
 LAMBDA_TASKS_QUEUES = {
-    "default": "https://sqs.us-east-1.amazonaws.com/123456789/default",
-    "high_memory": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory",
+    "default": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/default"},
+    "high_memory": {"queue_url": "https://sqs.us-east-1.amazonaws.com/123456789/high-memory"},
+    "heavy": {
+        "job_queue_arn": "arn:aws:batch:eu-west-1:123456789:job-queue/my-queue",
+        "job_definition_arn": "arn:aws:batch:eu-west-1:123456789:job-definition/my-def:1",
+    },
 }
 ```
 
 ```python
+# Runs on Lambda with more memory
 @lambda_task(queue="high_memory")
-def process_large_file(*, file_id: int) -> None:
+def process_images(*, batch_id: int) -> None:
+    ...
+
+# Runs on AWS Batch (up to 1 hour, large disk)
+@lambda_task(queue="heavy", soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
     ...
 ```
 
+Tasks routed to a Batch queue are automatically submitted to AWS Batch via the built-in `submit_batch_job` task. The flow is:
+
+1. `execute_on_commit()` enqueues a message to the default SQS queue
+2. The Lambda handler picks up the message and runs `submit_batch_job`
+3. `submit_batch_job` calls `batch.submit_job()` with the task message as a `LAMBDA_TASKS_MESSAGE` environment variable override
+4. The Batch container runs `python -m lambda_tasks.handler`, which reads the env var, performs cold-start init, and calls `execute_immediately()`
+5. The task runs with full timeout enforcement (`SIGALRM`), retry support, and `TaskRecord` tracking
+
 ---
 
 ## Task results
@@ -441,10 +470,11 @@ Each task runs inside a `django.db.transaction.atomic` block. If the task raises
 |---|---|---|
 | Task function has positional parameters | `TypeError` | At decoration time |
 | `soft_timeout >= hard_timeout` | `ValueError` | At decoration time |
-| Timeout value exceeds 900 seconds | `ValueError` | At decoration time or settings load |
+| Timeout value exceeds queue max (900 or 3600) | `ValueError` | At execution time (timeout resolution) |
 | Unknown queue name | `ImproperlyConfigured` | At `.execute_on_commit()` |
 | `LAMBDA_TASKS_QUEUES` not set | `ImproperlyConfigured` | On first settings access |
 | `LAMBDA_TASKS_QUEUES` missing `"default"` | `ImproperlyConfigured` | On first settings access |
+| `"default"` queue is not SQS | `ImproperlyConfigured` | On first settings access |
 | SQS `send_message` failure | boto3 exception (propagated) | At `.execute_on_commit()` |
 
 ---
@@ -453,7 +483,7 @@ Each task runs inside a `django.db.transaction.atomic` block. If the task raises
 
 ### SQS queue configuration
 
-Each queue in `LAMBDA_TASKS_QUEUES` should be a standard SQS queue (not FIFO) with the following recommended settings:
+Each SQS queue in `LAMBDA_TASKS_QUEUES` should be a standard SQS queue (not FIFO) with the following recommended settings:
 
 - **Visibility timeout** — set this higher than your Lambda function's `hard_timeout`. If the Lambda times out before SQS receives a deletion confirmation, the message becomes visible again and will be redelivered.
 - **Dead letter queue (DLQ)** — configure a DLQ to capture messages that cannot be processed. Without one, messages that exhaust retries are silently deleted.
@@ -580,6 +610,39 @@ The following all raise `ValueError` at cold start, preventing the Lambda contai
 
 ---
 
+## AWS Batch setup
+
+For tasks that exceed Lambda's 15-minute timeout or 10GB ephemeral storage limit, route them to a Batch queue:
+
+```python
+@lambda_task(queue="heavy", soft_timeout=1800, hard_timeout=3500)
+def process_large_file(*, file_id: int) -> str:
+    # long-running work with large disk usage
+    return "done"
+```
+
+### Batch job definition setup
+
+Your Batch job definition should:
+- Use the same Docker image as your application
+- Set `DJANGO_SETTINGS_MODULE` as an environment variable
+- Configure secrets via the job definition's `secrets` field or via `LAMBDA_TASKS_ENVIRONMENT_SECRETS_MANAGER_ARN` / `LAMBDA_TASKS_SECRET_*` env vars (same mechanism as Lambda)
+- Set the default command to `["python", "-m", "lambda_tasks.handler"]`
+
+### Retries
+
+Batch tasks support `retry_on` — when a retryable exception is raised, the retry goes back through SQS → Lambda → `submit_batch_job` → new Batch job. The same `LAMBDA_TASKS_MAX_RETRIES` limit applies.
+
+### Container overrides limit
+
+The total `containerOverrides` payload (including JSON formatting) is limited to 8192 characters by AWS Batch. For typical task kwargs (IDs, short strings) this is not a concern.
+
+### Eager and local worker modes
+
+Tasks on Batch queues respect `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` — in development, they run in-process or in the local process pool just like SQS-backed tasks, without requiring AWS infrastructure.
+
+---
+
 ## Built-in tasks
 
 ### `cleanup_task_records`

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/example/example_project/settings.py

@@ -66,6 +66,9 @@ USE_TZ = True
 
 # Local mode: tasks execute in process pool (no SQS required)
 LAMBDA_TASKS_LOCAL_WORKERS = 2
+LAMBDA_TASKS_QUEUES = {
+    "default": {"queue_url": "https://sqs.localhost/default"},
+}
 
 LOGGING = {
     "version": 1,

{django_lambda_tasks-0.5.0 → django_lambda_tasks-0.6.1}/lambda_tasks/admin.py

@@ -17,6 +17,7 @@ class TaskRecordAdmin(admin.ModelAdmin):
     list_display = (
         "pk",
         "task_name",
+        "queue",
         "status",
         "start_time",
         "end_time",
@@ -59,10 +60,17 @@ class TaskRecordAdmin(admin.ModelAdmin):
         total_seconds = d.total_seconds()
         return f"{total_seconds:.3f}s"
 
+    @admin.display(description="Queue")
+    def queue(self, obj: TaskRecord) -> str:
+        try:
+            return import_string(obj.task_name).queue
+        except Exception:
+            return "?"
+
     @admin.action(description="Replay selected tasks", permissions=("change",))
     def replay_tasks(self, request: HttpRequest, queryset: QuerySet) -> None:
         for record in queryset:
-            queue = import_string(record.task_name).queue
+            wrapper = import_string(record.task_name)
             task = SQSLambdaTask(
                 message=SQSLambdaTaskMessage(
                     task_name=record.task_name,
@@ -70,6 +78,6 @@ class TaskRecordAdmin(admin.ModelAdmin):
                     n_retries=0,
                 ),
                 delay=0,
-                queue=queue,
+                queue=wrapper.queue,
             )
             task.execute_on_commit()