django-lambda-tasks 0.3.0__tar.gz → 0.4.0__tar.gz

django_lambda_tasks-0.4.0/.kiro/specs/async-local-execution/.config.kiro

@@ -0,0 +1 @@
+{"specId": "9b6bcf96-d290-4107-8540-6dfeb1b6b6cb", "workflowType": "requirements-first", "specType": "feature"}

django_lambda_tasks-0.4.0/.kiro/specs/async-local-execution/design.md

@@ -0,0 +1,281 @@
+# Design Document: Async Local Execution
+
+## Overview
+
+This feature adds a third execution mode to django-lambda-tasks: **async local mode**. When `LAMBDA_TASKS_LOCAL_WORKERS` is set to a positive integer, tasks are submitted to a `concurrent.futures.ProcessPoolExecutor` instead of SQS. Tasks execute in background worker processes with full timeout enforcement via `SIGALRM`, providing true parallelism without AWS infrastructure.
+
+The execution mode hierarchy becomes:
+1. **Eager mode** (`LAMBDA_TASKS_EAGER=True`) — synchronous, in-process, no timeouts
+2. **Async local mode** (`LOCAL_WORKERS > 0`) — async, separate processes, timeouts enforced
+3. **SQS mode** (default) — async, Lambda workers, timeouts enforced
+
+This is a development-only feature that bridges the gap between eager mode (no parallelism, no timeouts) and full SQS/Lambda deployment.
+
+## Architecture
+
+```mermaid
+graph TD
+    A[View calls execute_on_commit] --> B[transaction.on_commit fires]
+    B --> C{SQSLambdaTask._execute}
+    C -->|EAGER=True| D[execute_immediately in-process]
+    C -->|LOCAL_WORKERS > 0| E[Submit to ProcessPoolExecutor]
+    C -->|else| F[Send to SQS]
+    E --> G[Worker Process]
+    G --> H[model_validate_json]
+    H --> I[execute_immediately with SIGALRM]
+```
+
+### Key Design Decisions
+
+1. **ProcessPoolExecutor over multiprocessing.Pool**: `concurrent.futures` provides a cleaner API, automatic worker replacement on crash, and a simpler lifecycle model.
+
+2. **JSON string serialization for IPC**: Rather than relying on pickle for the task message (which would work but couples to internal object layout), we serialize via `model_dump_json()` and deserialize via `model_validate_json()`. This mirrors the SQS path and guarantees picklability since strings are always picklable.
+
+3. **Module-level pool storage**: The pool is stored in a module-level variable so it persists across Django requests for the lifetime of the server process. This avoids repeated process spawning.
+
+4. **Fire-and-forget submission**: The dispatcher discards the `Future` returned by `submit()`. No callbacks, no `result()` calls. Worker failures are isolated and logged within the worker process via the existing `execute_immediately()` error handling.
+
+5. **Pool initializer calls `django.setup()`**: Each worker process needs Django configured. The initializer runs once per worker and sets up Django using the `DJANGO_SETTINGS_MODULE` inherited from the parent process environment.
+
+6. **Timeouts ARE enforced**: Unlike eager mode, async local workers are separate processes where `SIGALRM` works safely. The `TimeoutContext` condition changes from `if not conf.EAGER` to `if not conf.EAGER or conf.LOCAL_WORKERS > 0` — actually, since workers are separate processes that don't have `EAGER=True`, the existing `TimeoutContext` logic works unchanged in worker processes.
+
+## Components and Interfaces
+
+### New Module: `lambda_tasks/local_executor.py`
+
+This module owns the process pool lifecycle and the worker entry point.
+
+```python
+"""Process pool executor for async local task execution."""
+
+import uuid
+from concurrent.futures import ProcessPoolExecutor
+
+from lambda_tasks.settings import LambdaTasksSettings
+
+# Module-level pool — lazily created, reused for server lifetime
+_pool: ProcessPoolExecutor | None = None
+
+
+def _pool_initializer() -> None:
+    """Run once per worker process. Sets up Django."""
+    import django
+    django.setup()
+
+
+def get_pool() -> ProcessPoolExecutor:
+    """Return the shared ProcessPoolExecutor, creating it on first call."""
+    global _pool
+    if _pool is None:
+        conf = LambdaTasksSettings()
+        _pool = ProcessPoolExecutor(
+            max_workers=conf.LOCAL_WORKERS,
+            initializer=_pool_initializer,
+        )
+    return _pool
+
+
+def _execute_in_worker(*, message_json: str, message_id: str) -> None:
+    """Worker entry point. Deserializes and executes the task.
+
+    Runs in a child process. Django is already set up via the pool initializer.
+    """
+    from lambda_tasks.models import SQSLambdaTaskMessage
+
+    message = SQSLambdaTaskMessage.model_validate_json(message_json)
+    message.execute_immediately(message_id=message_id)
+
+
+def submit_task(*, message_json: str) -> None:
+    """Submit a task to the process pool. Fire-and-forget."""
+    pool = get_pool()
+    message_id = str(uuid.uuid4())
+    pool.submit(_execute_in_worker, message_json=message_json, message_id=message_id)
+```
+
+### Modified: `lambda_tasks/settings.py`
+
+Add the `LOCAL_WORKERS` property with validation:
+
+```python
+@property
+def LOCAL_WORKERS(self) -> int:
+    value = int(getattr(django_settings, "LAMBDA_TASKS_LOCAL_WORKERS", 0))
+    if value < 0:
+        raise ImproperlyConfigured(
+            "LAMBDA_TASKS_LOCAL_WORKERS must be a non-negative integer."
+        )
+    if value > 0 and self.EAGER:
+        raise ImproperlyConfigured(
+            "LAMBDA_TASKS_LOCAL_WORKERS and LAMBDA_TASKS_EAGER are mutually exclusive. "
+            "Set one or the other, not both."
+        )
+    return value
+```
+
+### Modified: `lambda_tasks/models.py` — `SQSLambdaTask._execute()`
+
+The dispatch logic gains a third branch:
+
+```python
+def _execute(self) -> None:
+    conf = LambdaTasksSettings()
+
+    if conf.EAGER:
+        self.message.execute_immediately(message_id=str(uuid.uuid4()))
+    elif conf.LOCAL_WORKERS > 0:
+        from lambda_tasks.local_executor import submit_task
+        submit_task(message_json=self.message.model_dump_json())
+    else:
+        # existing SQS path
+        ...
+```
+
+### Modified: `lambda_tasks/timeouts.py`
+
+No changes needed. The `TimeoutContext` checks `conf.EAGER` to decide whether to arm `SIGALRM`. In worker processes, `EAGER` is `False` (the setting is read from Django settings which are inherited), so timeouts are enforced automatically. The worker processes are separate OS processes where `SIGALRM` is safe to use.
+
+## Data Models
+
+### Settings Model (conceptual)
+
+| Setting | Type | Default | Validation |
+|---|---|---|---|
+| `LAMBDA_TASKS_LOCAL_WORKERS` | `int` | `0` | Must be ≥ 0; mutually exclusive with `EAGER=True` |
+
+### IPC Data Flow
+
+```
+Dispatcher (main process)          Worker Process
+─────────────────────────────      ──────────────────────────────
+SQSLambdaTaskMessage
+  → model_dump_json()
+  → JSON string (picklable)
+  → submit() via ProcessPool
+                                   → _execute_in_worker(message_json=..., message_id=...)
+                                   → SQSLambdaTaskMessage.model_validate_json(message_json)
+                                   → message.execute_immediately(message_id=message_id)
+                                   → TaskRecord created, task runs with SIGALRM
+```
+
+### Module-Level State
+
+| Variable | Module | Type | Lifecycle |
+|---|---|---|---|
+| `_pool` | `local_executor.py` | `ProcessPoolExecutor \| None` | Created on first `submit_task()` call, lives until process exit |
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Property 1: Positive LOCAL_WORKERS is preserved
+
+*For any* positive integer `n`, when `LAMBDA_TASKS_LOCAL_WORKERS` is set to `n` and `LAMBDA_TASKS_EAGER` is `False`, the `LambdaTasksSettings().LOCAL_WORKERS` property SHALL return exactly `n`.
+
+**Validates: Requirements 1.1**
+
+### Property 2: Negative LOCAL_WORKERS is rejected
+
+*For any* negative integer `n`, when `LAMBDA_TASKS_LOCAL_WORKERS` is set to `n`, accessing `LambdaTasksSettings().LOCAL_WORKERS` SHALL raise `ImproperlyConfigured`.
+
+**Validates: Requirements 1.3**
+
+### Property 3: Mutual exclusion of EAGER and LOCAL_WORKERS
+
+*For any* positive integer `n`, when both `LAMBDA_TASKS_EAGER` is `True` and `LAMBDA_TASKS_LOCAL_WORKERS` is set to `n`, accessing `LambdaTasksSettings().LOCAL_WORKERS` SHALL raise `ImproperlyConfigured`.
+
+**Validates: Requirements 1.4**
+
+### Property 4: Pool created with correct worker count
+
+*For any* positive integer `n` (within reasonable bounds, e.g. 1–32), when `LOCAL_WORKERS` is `n`, the `ProcessPoolExecutor` created by `get_pool()` SHALL have `_max_workers` equal to `n`.
+
+**Validates: Requirements 2.1**
+
+### Property 5: Async local dispatch routes to pool
+
+*For any* valid `SQSLambdaTaskMessage` and any positive `LOCAL_WORKERS` value, when `EAGER` is `False`, calling `SQSLambdaTask._execute()` SHALL call `ProcessPoolExecutor.submit()` and SHALL NOT call `boto3.client('sqs').send_message()`.
+
+**Validates: Requirements 3.1, 7.2**
+
+### Property 6: Task message serialization round-trip
+
+*For any* valid `SQSLambdaTaskMessage` (with arbitrary `task_name`, `kwargs` containing JSON-serializable values, and non-negative `n_retries`), serializing via `model_dump_json()` and deserializing via `model_validate_json()` SHALL produce an equivalent message object.
+
+**Validates: Requirements 6.1, 6.2, 3.2, 3.3**
+
+## Error Handling
+
+### Configuration Errors (startup time)
+
+| Error Condition | Raised Exception | When |
+|---|---|---|
+| `LOCAL_WORKERS < 0` | `ImproperlyConfigured` | On first access to `LambdaTasksSettings().LOCAL_WORKERS` |
+| `EAGER=True` and `LOCAL_WORKERS > 0` | `ImproperlyConfigured` | On first access to `LambdaTasksSettings().LOCAL_WORKERS` |
+
+### Runtime Errors (task execution)
+
+| Error Condition | Behavior | Impact on Main Process |
+|---|---|---|
+| Task raises exception | `execute_immediately()` catches it, writes `FAILED` TaskRecord | None — worker is isolated |
+| Worker process crashes | `ProcessPoolExecutor` replaces worker automatically | None — pool self-heals |
+| Soft timeout exceeded | `SoftTimeLimitExceeded` raised in worker | None — signal is per-process |
+| Hard timeout exceeded | `HardTimeLimitExceeded` raised in worker | None — signal is per-process |
+| `model_validate_json()` fails | Exception in worker, no TaskRecord written | None — worker is isolated |
+
+### Fire-and-Forget Implications
+
+Since the dispatcher discards the `Future`:
+- There is no mechanism to propagate worker exceptions back to the caller
+- Failed tasks are only observable via `TaskRecord` entries in the database
+- This matches the SQS behavior where the caller never sees task outcomes synchronously
+
+## Testing Strategy
+
+### Property-Based Tests (Hypothesis)
+
+Property-based testing is appropriate for this feature because:
+- Settings validation has clear input/output behavior across a range of integers
+- Serialization round-trip is a classic PBT pattern
+- Dispatch routing is a pure decision based on configuration values
+
+**Library**: `hypothesis` (already in dev dependencies)
+**Minimum iterations**: 100 per property test
+**Tag format**: `Feature: async-local-execution, Property {number}: {property_text}`
+
+Each correctness property maps to a single `@given`-decorated test function.
+
+### Unit Tests (Example-Based)
+
+| Test | Validates |
+|---|---|
+| `test_local_workers_default_is_zero` | Req 1.2 |
+| `test_pool_reused_across_calls` | Req 2.2 |
+| `test_pool_stored_at_module_level` | Req 2.4 |
+| `test_dispatcher_does_not_wait_on_future` | Req 3.4, 5.3 |
+| `test_eager_mode_with_zero_local_workers` | Req 7.1 |
+| `test_sqs_mode_when_local_workers_zero` | Req 7.3 |
+
+### Integration Tests
+
+| Test | Validates |
+|---|---|
+| `test_pool_initializer_calls_django_setup` | Req 2.3 |
+| `test_on_commit_submits_after_transaction` | Req 4.1 |
+| `test_rollback_prevents_pool_submission` | Req 4.2 |
+| `test_pool_survives_worker_exception` | Req 5.1 |
+| `test_pool_replaces_crashed_worker` | Req 5.2 |
+| `test_timeout_enforced_in_worker` | Req 8.1, 8.4 |
+| `test_soft_timeout_isolated_to_worker` | Req 8.2 |
+| `test_hard_timeout_isolated_to_worker` | Req 8.3 |
+
+### Test File Location
+
+All tests go in `tests/test_local_executor.py` following the project convention of one test file per source module.
+
+### Mocking Strategy
+
+- **`ProcessPoolExecutor`**: Mocked in dispatch routing tests to verify `submit()` is called with correct args without spawning real processes
+- **`django.setup()`**: Mocked in initializer tests
+- **`boto3.client`**: Mocked to verify SQS is NOT called in async local mode
+- **Real pool**: Used in integration tests that verify crash isolation and timeout behavior (these tests spawn actual worker processes)

django_lambda_tasks-0.4.0/.kiro/specs/async-local-execution/requirements.md

@@ -0,0 +1,97 @@
+# Requirements Document
+
+## Introduction
+
+This feature adds an async local execution mode to django-lambda-tasks using `concurrent.futures.ProcessPoolExecutor`. When enabled, tasks are submitted to a process pool and execute in the background without blocking the calling request. This is a development-only feature (like `LAMBDA_TASKS_EAGER`) that provides true parallelism for CPU-bound tasks without requiring AWS infrastructure or a separate service.
+
+## Glossary
+
+- **Process_Pool**: A `concurrent.futures.ProcessPoolExecutor` instance that manages a fixed number of worker processes for executing tasks in the background.
+- **Worker_Process**: A child process in the Process_Pool that calls `django.setup()` once via the pool initializer and then executes submitted tasks.
+- **Dispatcher**: The component within `SQSLambdaTask._execute()` that decides whether to send a task to SQS, execute eagerly, or submit to the Process_Pool.
+- **Pool_Initializer**: A function passed to `ProcessPoolExecutor(initializer=...)` that runs once per Worker_Process to configure the Django environment.
+- **Async_Local_Mode**: The execution mode activated by `LAMBDA_TASKS_LOCAL_WORKERS` being set to a positive integer, causing tasks to be submitted to the Process_Pool instead of SQS.
+
+## Requirements
+
+### Requirement 1: Pool Configuration Setting
+
+**User Story:** As a developer, I want to configure the local process pool size via a Django setting, so that I can control resource usage during development.
+
+#### Acceptance Criteria
+
+1. WHEN `LAMBDA_TASKS_LOCAL_WORKERS` is set to a positive integer, THE LambdaTasksSettings SHALL expose that value as the `LOCAL_WORKERS` property.
+2. WHEN `LAMBDA_TASKS_LOCAL_WORKERS` is not set, THE LambdaTasksSettings SHALL return `0` as the `LOCAL_WORKERS` property.
+3. IF `LAMBDA_TASKS_LOCAL_WORKERS` is set to a value less than zero, THEN THE LambdaTasksSettings SHALL raise `ImproperlyConfigured`.
+4. IF both `LAMBDA_TASKS_EAGER` and `LAMBDA_TASKS_LOCAL_WORKERS` (greater than zero) are set, THEN THE LambdaTasksSettings SHALL raise `ImproperlyConfigured` indicating the two modes are mutually exclusive.
+
+### Requirement 2: Process Pool Lifecycle
+
+**User Story:** As a developer, I want the process pool to be created lazily and reused across requests, so that worker startup cost is paid only once.
+
+#### Acceptance Criteria
+
+1. WHEN the first task is submitted in Async_Local_Mode, THE Dispatcher SHALL create a Process_Pool with `max_workers` equal to the `LOCAL_WORKERS` setting.
+2. WHILE the Process_Pool has been created, THE Dispatcher SHALL reuse the same Process_Pool instance for all subsequent task submissions.
+3. THE Pool_Initializer SHALL call `django.setup()` once per Worker_Process using the `DJANGO_SETTINGS_MODULE` environment variable from the parent process.
+4. THE Process_Pool SHALL be stored at module level so that it persists for the lifetime of the Django server process.
+
+### Requirement 3: Task Submission
+
+**User Story:** As a developer, I want tasks to be submitted to the process pool after transaction commit, so that the request returns immediately and the task runs in the background.
+
+#### Acceptance Criteria
+
+1. WHEN `LOCAL_WORKERS` is greater than zero and `EAGER` is `False`, THE Dispatcher SHALL submit the task to the Process_Pool via `ProcessPoolExecutor.submit()` instead of sending to SQS.
+2. WHEN a task is submitted to the Process_Pool, THE Dispatcher SHALL pass the serialized `SQSLambdaTaskMessage` data and a new UUID4 `message_id` to the Worker_Process.
+3. THE Worker_Process SHALL call `SQSLambdaTaskMessage.execute_immediately()` with the provided `message_id` to execute the task and write the TaskRecord.
+4. WHEN a task is submitted to the Process_Pool, THE Dispatcher SHALL return immediately without waiting for the task to complete.
+
+### Requirement 4: Transaction Commit Integration
+
+**User Story:** As a developer, I want async local tasks to respect `transaction.on_commit` just like SQS tasks, so that tasks only run after the triggering transaction succeeds.
+
+#### Acceptance Criteria
+
+1. WHEN `execute_on_commit()` is called in Async_Local_Mode, THE Dispatcher SHALL register the pool submission with `transaction.on_commit` so the task is only submitted after the current transaction commits.
+2. IF the transaction is rolled back, THEN THE Dispatcher SHALL not submit the task to the Process_Pool.
+
+### Requirement 5: Worker Process Error Isolation
+
+**User Story:** As a developer, I want worker process failures to be isolated from the Django server process, so that a crashing task does not bring down the dev server.
+
+#### Acceptance Criteria
+
+1. IF a task raises an unhandled exception in the Worker_Process, THEN THE Process_Pool SHALL continue operating and accept new task submissions.
+2. IF a Worker_Process crashes, THEN THE Process_Pool SHALL replace the crashed Worker_Process with a new one that runs the Pool_Initializer.
+3. WHEN a task is submitted to the Process_Pool, THE Dispatcher SHALL not attach any callbacks or wait on the returned Future object.
+
+### Requirement 6: Picklability of Task Arguments
+
+**User Story:** As a developer, I want clear feedback when task arguments cannot be sent to the worker process, so that I can fix serialization issues quickly.
+
+#### Acceptance Criteria
+
+1. THE Dispatcher SHALL submit the task message as a JSON string (the output of `model_dump_json()`) to the Worker_Process, ensuring picklability via standard string serialization.
+2. WHEN the Worker_Process receives the JSON string, THE Worker_Process SHALL reconstruct the `SQSLambdaTaskMessage` via `model_validate_json()` before calling `execute_immediately()`.
+
+### Requirement 7: Mode Precedence
+
+**User Story:** As a developer, I want a clear precedence between execution modes, so that configuration is predictable.
+
+#### Acceptance Criteria
+
+1. WHEN `LAMBDA_TASKS_EAGER` is `True`, THE Dispatcher SHALL execute tasks synchronously regardless of the `LOCAL_WORKERS` setting (enforced by the mutual exclusion constraint in Requirement 1).
+2. WHEN `LAMBDA_TASKS_EAGER` is `False` and `LOCAL_WORKERS` is greater than zero, THE Dispatcher SHALL submit tasks to the Process_Pool.
+3. WHEN `LAMBDA_TASKS_EAGER` is `False` and `LOCAL_WORKERS` is zero, THE Dispatcher SHALL send tasks to SQS.
+
+### Requirement 8: Timeout Behaviour in Async Local Mode
+
+**User Story:** As a developer, I want timeouts to be enforced in worker processes, so that runaway tasks are terminated just like they would be in Lambda.
+
+#### Acceptance Criteria
+
+1. WHILE a task executes in a Worker_Process in Async_Local_Mode, THE TimeoutContext SHALL set up `SIGALRM`-based timeouts (same behaviour as Lambda execution).
+2. IF a soft timeout fires in a Worker_Process, THEN THE TimeoutContext SHALL raise `SoftTimeoutError` in that Worker_Process without affecting the Django server process.
+3. IF a hard timeout fires in a Worker_Process, THEN THE TimeoutContext SHALL raise `HardTimeoutError` in that Worker_Process without affecting the Django server process.
+4. THE TimeoutContext SHALL NOT treat Async_Local_Mode the same as eager mode — timeouts are fully enforced because each Worker_Process is isolated from the dev server.

django_lambda_tasks-0.4.0/.kiro/specs/async-local-execution/tasks.md

@@ -0,0 +1,103 @@
+# Implementation Plan: Async Local Execution
+
+## Overview
+
+Adds a third execution mode to django-lambda-tasks using `concurrent.futures.ProcessPoolExecutor`. Tasks are submitted to a process pool when `LAMBDA_TASKS_LOCAL_WORKERS` is set to a positive integer, providing true async parallelism with timeout enforcement for local development. Implementation follows TDD: write failing tests first, then implement.
+
+## Tasks
+
+- [x] 1. Add `LOCAL_WORKERS` setting with validation
+  - [x] 1.1 Write tests for `LOCAL_WORKERS` property in `tests/test_local_executor.py`
+    - Test that `LOCAL_WORKERS` defaults to `0` when setting is absent
+    - Test that a positive integer is returned correctly
+    - Test that a negative integer raises `ImproperlyConfigured`
+    - Test that setting both `EAGER=True` and `LOCAL_WORKERS > 0` raises `ImproperlyConfigured`
+    - _Requirements: 1.1, 1.2, 1.3, 1.4_
+
+  - [x] 1.2 Write property tests for `LOCAL_WORKERS` validation
+    - **Property 1: Positive LOCAL_WORKERS is preserved**
+    - **Validates: Requirements 1.1**
+    - **Property 2: Negative LOCAL_WORKERS is rejected**
+    - **Validates: Requirements 1.3**
+    - **Property 3: Mutual exclusion of EAGER and LOCAL_WORKERS**
+    - **Validates: Requirements 1.4**
+
+  - [x] 1.3 Implement `LOCAL_WORKERS` property in `lambda_tasks/settings.py`
+    - Add `LOCAL_WORKERS` property to `LambdaTasksSettings`
+    - Read from `LAMBDA_TASKS_LOCAL_WORKERS` Django setting with default `0`
+    - Raise `ImproperlyConfigured` if value is negative
+    - Raise `ImproperlyConfigured` if both `EAGER` and `LOCAL_WORKERS > 0`
+    - _Requirements: 1.1, 1.2, 1.3, 1.4_
+
+- [x] 2. Create `lambda_tasks/local_executor.py` module
+  - [x] 2.1 Write tests for `get_pool()` in `tests/test_local_executor.py`
+    - Test that `get_pool()` returns a `ProcessPoolExecutor` with `_max_workers` equal to `LOCAL_WORKERS`
+    - Test that `get_pool()` returns the same instance on repeated calls (pool reuse)
+    - Test that the pool is stored at module level (`local_executor._pool`)
+    - _Requirements: 2.1, 2.2, 2.4_
+
+  - [x] 2.2 Write property test for pool worker count
+    - **Property 4: Pool created with correct worker count**
+    - **Validates: Requirements 2.1**
+
+  - [x] 2.3 Implement `get_pool()` and `_pool_initializer()` in `lambda_tasks/local_executor.py`
+    - Create module-level `_pool: ProcessPoolExecutor | None = None`
+    - Implement `_pool_initializer()` that calls `django.setup()`
+    - Implement `get_pool()` that lazily creates the pool with `max_workers=conf.LOCAL_WORKERS`
+    - _Requirements: 2.1, 2.2, 2.3, 2.4_
+
+  - [x] 2.4 Write tests for `submit_task()` in `tests/test_local_executor.py`
+    - Test that `submit_task()` calls `pool.submit()` with `_execute_in_worker`, the JSON string, and a UUID message_id
+    - Test that `submit_task()` does not wait on the returned Future
+    - _Requirements: 3.1, 3.2, 3.4, 5.3_
+
+  - [x] 2.5 Implement `submit_task()` and `_execute_in_worker()` in `lambda_tasks/local_executor.py`
+    - Implement `_execute_in_worker(*, message_json: str, message_id: str)` that deserializes and executes
+    - Implement `submit_task(*, message_json: str)` that generates a UUID and submits to the pool
+    - _Requirements: 3.1, 3.2, 3.3, 3.4, 6.1, 6.2_
+
+  - [x] 2.6 Write property test for message serialization round-trip
+    - **Property 6: Task message serialization round-trip**
+    - **Validates: Requirements 6.1, 6.2, 3.2, 3.3**
+
+- [x] 3. Checkpoint
+  - Ensure all tests pass, ask the user if questions arise.
+
+- [x] 4. Add async local dispatch branch to `SQSLambdaTask._execute()`
+  - [x] 4.1 Write tests for the dispatch routing in `tests/test_local_executor.py`
+    - Test that when `LOCAL_WORKERS > 0` and `EAGER=False`, `_execute()` calls `submit_task()` with the JSON-serialized message
+    - Test that when `LOCAL_WORKERS > 0` and `EAGER=False`, `_execute()` does NOT call `boto3.client('sqs').send_message()`
+    - Test that when `LOCAL_WORKERS=0` and `EAGER=False`, `_execute()` sends to SQS (existing behaviour preserved)
+    - Test that when `EAGER=True`, `_execute()` calls `execute_immediately()` (existing behaviour preserved)
+    - _Requirements: 3.1, 7.1, 7.2, 7.3_
+
+  - [x] 4.2 Write property test for async local dispatch routing
+    - **Property 5: Async local dispatch routes to pool**
+    - **Validates: Requirements 3.1, 7.2**
+
+  - [x] 4.3 Implement the third dispatch branch in `lambda_tasks/models.py`
+    - Add `elif conf.LOCAL_WORKERS > 0` branch in `SQSLambdaTask._execute()`
+    - Import `submit_task` from `lambda_tasks.local_executor`
+    - Call `submit_task(message_json=self.message.model_dump_json())`
+    - _Requirements: 3.1, 7.2_
+
+- [x] 5. Write integration tests for transaction commit and error isolation
+  - [x] 5.1 Write integration tests in `tests/test_local_executor.py`
+    - Test that `execute_on_commit()` in async local mode submits after transaction commit
+    - Test that a rolled-back transaction does not submit to the pool
+    - Test that a worker exception does not crash the pool (pool continues accepting tasks)
+    - Test that `_pool_initializer` calls `django.setup()`
+    - _Requirements: 2.3, 4.1, 4.2, 5.1, 5.2_
+
+- [x] 6. Final checkpoint
+  - Ensure all tests pass, ask the user if questions arise.
+
+## Notes
+
+- Tasks marked with `*` are optional and can be skipped for faster MVP
+- Each task references specific requirements for traceability
+- Checkpoints ensure incremental validation
+- Property tests validate universal correctness properties from the design document
+- Unit tests validate specific examples and edge cases
+- TDD order: tests are written before implementation in each group
+- No changes to `lambda_tasks/timeouts.py` — workers inherit `EAGER=False` so `SIGALRM` works automatically

{django_lambda_tasks-0.3.0 → django_lambda_tasks-0.4.0}/.kiro/steering/product.md

@@ -14,7 +14,8 @@ View → @lambda_task.execute_on_commit() → SQS → Lambda handler → SQSLamb
 
 Key modules:
 - `decorators.py` — `@lambda_task` decorator and `LambdaTaskWrapper`
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish)
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (SQS schema + execution), `SQSLambdaTask` (routing + SQS publish or local pool submit)
+- `local_executor.py` — `ProcessPoolExecutor`-based async local execution for development
 - `handler.py` — AWS Lambda entry point; cold-start init runs on first invocation (not at import time) with partial-batch failure reporting
 - `logging.py` — `task_logger` for invocation-scoped log output
 - `settings.py` — lazy `LambdaTasksSettings` reading from Django settings
@@ -195,6 +196,7 @@ Statuses: `RUNNING`, `SUCCESS`, `FAILED`, `RETRYING`
 | `LAMBDA_TASKS_DEFAULT_SOFT_TIMEOUT` | `270` | Soft timeout in seconds |
 | `LAMBDA_TASKS_DEFAULT_HARD_TIMEOUT` | `300` | Hard timeout in seconds |
 | `LAMBDA_TASKS_EAGER` | `False` | Run tasks synchronously in-process (no SQS) |
+| `LAMBDA_TASKS_LOCAL_WORKERS` | `0` | Number of worker processes for async local execution (development only; mutually exclusive with `EAGER`) |
 | `LAMBDA_TASKS_MAX_RETRIES` | `2880` | Maximum retry attempts before `MaxRetriesExceededError` is raised (60 × 24 × 2) |
 | `LAMBDA_TASKS_SINGLETON_CACHE` | `"default"` | Django cache backend used for singleton task locks |
 
@@ -208,6 +210,36 @@ In eager mode a random UUID4 is generated as the `message_id` passed to `execute
 
 **Timeouts are not enforced in eager mode.** `TimeoutContext` is still entered but becomes a no-op — it checks `LAMBDA_TASKS_EAGER` internally and skips `SIGALRM` setup. `SIGALRM`-based timeouts require a Lambda worker process, not a Django dev server thread. Timeout values are still validated at decoration time.
 
+## Async Local Mode
+
+Set `LAMBDA_TASKS_LOCAL_WORKERS` to a positive integer to run tasks in a background `ProcessPoolExecutor`. Tasks are submitted after transaction commit (same as SQS mode) but execute in local worker processes instead of Lambda. This provides true parallelism with timeout enforcement for local development.
+
+```python
+# settings/local.py
+LAMBDA_TASKS_LOCAL_WORKERS = 4
+```
+
+The execution mode hierarchy is:
+1. **Eager mode** (`LAMBDA_TASKS_EAGER=True`) — synchronous, in-process, no timeouts
+2. **Async local mode** (`LOCAL_WORKERS > 0`) — async, separate processes, timeouts enforced
+3. **SQS mode** (default) — async, Lambda workers, timeouts enforced
+
+`LAMBDA_TASKS_LOCAL_WORKERS` and `LAMBDA_TASKS_EAGER` are mutually exclusive — setting both raises `ImproperlyConfigured`. A negative value also raises `ImproperlyConfigured`.
+
+Key behaviours:
+- The process pool is created lazily on first task submission and reused for the server lifetime
+- Each worker process calls `django.setup()` once via the pool initializer
+- Tasks are serialized as JSON strings (via `model_dump_json()`) for IPC — same path as SQS
+- The dispatcher discards the `Future` (fire-and-forget) — worker failures are isolated
+- `SIGALRM`-based timeouts work in worker processes because they are separate OS processes
+- `transaction.on_commit` is respected — tasks only submit after the transaction commits
+
+Implementation lives in `lambda_tasks/local_executor.py`:
+- `get_pool()` — lazily creates and returns the shared `ProcessPoolExecutor`
+- `submit_task(*, message_json: str)` — generates a UUID4 message_id and submits to the pool
+- `_execute_in_worker(*, message_json: str, message_id: str)` — worker entry point; deserializes and calls `execute_immediately()`
+- `_pool_initializer()` — calls `django.setup()` once per worker
+
 ## Logging
 
 Import `task_logger` to emit log records that are automatically prefixed with the active `message_id`:

{django_lambda_tasks-0.3.0 → django_lambda_tasks-0.4.0}/.kiro/steering/structure.md

@@ -19,6 +19,7 @@ django-lambda-tasks/
 │   ├── settings.py             # LambdaTasksSettings (lazy Django settings reader)
 │   ├── secret_loader.py        # Resolves LAMBDA_TASKS_SECRET_* env vars at cold start
 │   ├── environment_loader.py    # Loads env vars from Secrets Manager at cold start
+│   ├── local_executor.py        # ProcessPoolExecutor for async local task execution
 │   ├── tasks.py                # Built-in maintenance tasks (cleanup_task_records)
 │   ├── timeouts.py             # TimeoutContext implementation
 │   └── migrations/             # Django migrations for TaskRecord
@@ -39,7 +40,8 @@ django-lambda-tasks/
 ## Module Responsibilities
 
 - `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
-- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS or executes eagerly; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
+- `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
+- `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker
 - `handler.py` — Lambda entry point; cold-start init (temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel

django_lambda_tasks-0.3.0/README.md → django_lambda_tasks-0.4.0/PKG-INFO

@@ -1,3 +1,15 @@
+Metadata-Version: 2.4
+Name: django-lambda-tasks
+Version: 0.4.0
+Summary: Run async tasks in a lambda function
+Requires-Python: >=3.10
+Requires-Dist: awslambdaric
+Requires-Dist: boto3
+Requires-Dist: django
+Requires-Dist: pydantic
+Requires-Dist: redis
+Description-Content-Type: text/markdown
+
 # Django Lambda Tasks
 
 A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
@@ -151,6 +163,33 @@ With eager mode enabled, `.execute_on_commit()` executes the task immediately wi
 
 > **Note:** Timeouts are not enforced in eager mode. `soft_timeout` and `hard_timeout` values are accepted and stored but `TimeoutContext` becomes a no-op — it checks `LAMBDA_TASKS_EAGER` internally and skips `SIGALRM` setup. This is intentional: `SIGALRM`-based timeouts require a Lambda/Unix worker process, not a Django dev server thread.
 
+### Async local execution (development)
+
+| Setting | Type | Default | Description |
+|---|---|---|---|
+| `LAMBDA_TASKS_LOCAL_WORKERS` | `int` | `0` | Number of worker processes for async local execution. When set to a positive integer, tasks run in a background process pool instead of SQS. |
+
+```python
+# settings/local.py
+LAMBDA_TASKS_LOCAL_WORKERS = 4
+```
+
+Async local mode bridges the gap between eager mode and full SQS/Lambda deployment. Tasks execute in background worker processes with true parallelism and full timeout enforcement via `SIGALRM`, without requiring AWS infrastructure.
+
+The execution mode hierarchy is:
+1. **Eager** (`LAMBDA_TASKS_EAGER=True`) — synchronous, in-process, no timeouts
+2. **Async local** (`LAMBDA_TASKS_LOCAL_WORKERS > 0`) — async, separate processes, timeouts enforced
+3. **SQS** (default) — async, Lambda workers, timeouts enforced
+
+Key characteristics:
+- `LAMBDA_TASKS_LOCAL_WORKERS` and `LAMBDA_TASKS_EAGER` are mutually exclusive — setting both raises `ImproperlyConfigured`
+- The process pool is created lazily on first task submission and reused for the server lifetime
+- Each worker process calls `django.setup()` once at startup
+- Tasks are serialized as JSON for IPC (same code path as SQS)
+- Worker failures are isolated from the Django server process — a crashing task does not bring down the dev server
+- `transaction.on_commit` is respected — tasks only submit after the transaction commits
+- Timeouts (`SIGALRM`) are fully enforced because workers are separate OS processes
+
 ---
 
 ## Decorator options

django_lambda_tasks-0.3.0/PKG-INFO → django_lambda_tasks-0.4.0/README.md

@@ -1,15 +1,3 @@
-Metadata-Version: 2.4
-Name: django-lambda-tasks
-Version: 0.3.0
-Summary: Run async tasks in a lambda function
-Requires-Python: >=3.10
-Requires-Dist: awslambdaric
-Requires-Dist: boto3
-Requires-Dist: django
-Requires-Dist: pydantic
-Requires-Dist: redis
-Description-Content-Type: text/markdown
-
 # Django Lambda Tasks
 
 A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
@@ -163,6 +151,33 @@ With eager mode enabled, `.execute_on_commit()` executes the task immediately wi
 
 > **Note:** Timeouts are not enforced in eager mode. `soft_timeout` and `hard_timeout` values are accepted and stored but `TimeoutContext` becomes a no-op — it checks `LAMBDA_TASKS_EAGER` internally and skips `SIGALRM` setup. This is intentional: `SIGALRM`-based timeouts require a Lambda/Unix worker process, not a Django dev server thread.
 
+### Async local execution (development)
+
+| Setting | Type | Default | Description |
+|---|---|---|---|
+| `LAMBDA_TASKS_LOCAL_WORKERS` | `int` | `0` | Number of worker processes for async local execution. When set to a positive integer, tasks run in a background process pool instead of SQS. |
+
+```python
+# settings/local.py
+LAMBDA_TASKS_LOCAL_WORKERS = 4
+```
+
+Async local mode bridges the gap between eager mode and full SQS/Lambda deployment. Tasks execute in background worker processes with true parallelism and full timeout enforcement via `SIGALRM`, without requiring AWS infrastructure.
+
+The execution mode hierarchy is:
+1. **Eager** (`LAMBDA_TASKS_EAGER=True`) — synchronous, in-process, no timeouts
+2. **Async local** (`LAMBDA_TASKS_LOCAL_WORKERS > 0`) — async, separate processes, timeouts enforced
+3. **SQS** (default) — async, Lambda workers, timeouts enforced
+
+Key characteristics:
+- `LAMBDA_TASKS_LOCAL_WORKERS` and `LAMBDA_TASKS_EAGER` are mutually exclusive — setting both raises `ImproperlyConfigured`
+- The process pool is created lazily on first task submission and reused for the server lifetime
+- Each worker process calls `django.setup()` once at startup
+- Tasks are serialized as JSON for IPC (same code path as SQS)
+- Worker failures are isolated from the Django server process — a crashing task does not bring down the dev server
+- `transaction.on_commit` is respected — tasks only submit after the transaction commits
+- Timeouts (`SIGALRM`) are fully enforced because workers are separate OS processes
+
 ---
 
 ## Decorator options